@hue-run/sdk 0.12.0 → 0.12.1

package/CLI.md

@@ -42,7 +42,7 @@ project manifest are refused because managers can update ancestor locks; Python
 The generated `hue.setup.mjs` or `hue_setup.py` always selects `captureContent: false` /
 `capture_content=False`. For a supported application, setup installs the dependency and adds the
 managed import and middleware registration to the existing entrypoint; an unreferenced helper is
-not a completed integration. TypeScript uses `@hue-run/sdk@0.12.0`, `@opentelemetry/api@1.9.1` and
+not a completed integration. TypeScript uses `@hue-run/sdk@0.12.1`, `@opentelemetry/api@1.9.1` and
 `@opentelemetry/context-async-hooks@2.11.0`; Python setup uses its separately tested package pin.
 Content capture requires an ordinary account-managed key and a later explicit application decision.
 
@@ -469,7 +469,7 @@ node packages/sdk-typescript/scripts/verify-package.mjs --artifacts-dir .artifac
 # Set project to an existing supported fixture; use the same directory on resume.
 project=/absolute/path/to/supported-fixture
 node packages/sdk-typescript/scripts/verify-setup-live.mjs \
-  --archive .artifacts/typescript/hue-run-sdk-0.12.0.tgz \
+  --archive .artifacts/typescript/hue-run-sdk-0.12.1.tgz \
   --origin https://STAGING_ORIGIN \
   --project "$project" --command setup \
   --evidence .context/setup-staging-before-claim.json
@@ -480,7 +480,7 @@ the private local handoff and finish the real browser claim, then reconcile the
 
 ```sh
 node packages/sdk-typescript/scripts/verify-setup-live.mjs \
-  --archive .artifacts/typescript/hue-run-sdk-0.12.0.tgz \
+  --archive .artifacts/typescript/hue-run-sdk-0.12.1.tgz \
   --origin https://STAGING_ORIGIN \
   --project "$project" --command claim \
   --evidence .context/setup-staging-after-claim.json
@@ -618,6 +618,11 @@ for example, needs a reviewed `attemptBaselineV2` (see
 registers none, so put `"attemptBaselineV2"` in the run's configuration JSON when you launch the
 run from Hue.
 
+`--trace-not-accepted pending` completes a case whose telemetry Hue did not accept in time in its
+true state with the trace still to arrive, carrying the count of spans the case ended; Hue scores
+the sealed world at once and attaches the trace once its export is complete. The default,
+`fail_case`, completes such a case as failed with its evidence omitted.
+
 A Scenario or eval set whose dataset version is not saved cannot back an experiment: the command
 exits 1 and asks for **Save eval-set version** in Hue or `--save-version`, which freezes that
 version at its current revision. Connection settings are `HUE_API_KEY` and `HUE_BASE_URL`
@@ -663,9 +668,9 @@ an interrupted run without invoking the agent again; a different selection is re
 unfinished one is resumed or its directory is removed.
 
 Exit codes: `0` every case passed the evaluators that apply to it (advisory ones never count),
-`1` a case failed, errored (including one no pinned evaluator applies to or decides), was skipped
-or Hue's checks were
-still pending at `--wait`, `2` usage or configuration error (including a missing key), `130`
+`1` a case failed, errored (including one no pinned evaluator applies to or decides), was skipped,
+was inconclusive (a required judge could not run or answer; the table says `INCONCLUSIVE` and the
+summary counts it apart from skipped cases) or Hue's checks were still pending at `--wait`, `2` usage or configuration error (including a missing key), `130`
 interrupted. On Node.js 22, load TypeScript adapters with `NODE_OPTIONS=--experimental-strip-types`;
 non-erasable syntax (enums, parameter properties, namespaces) needs a loader such as `--import tsx`
 on any Node.js version.

package/EVALUATIONS.md

@@ -116,7 +116,7 @@ Both choices are required and independent:
 
 Frozen dataset inputs/references already exist on Hue. Their presence is independent of these switches. No output is inferred from telemetry. JavaScript `undefined` means unavailable output; JSON `null`, false, zero and empty string remain present values. Historical subjects with unavailable output produce skipped results without executing a scorer callback. A failed quality metric remains `state:"scored"` with `passed:false`. Target errors and scorer errors remain separate. Target error types are generalized to `TargetError`; when content is enabled, bounded messages may be stored. Never put credentials in error messages, output, custom metric values, names or metadata.
 
-`traceEvidence:{mode:"required"}` waits for trace and log export acknowledgement after the root span ends. The transport is shared by every case in flight, so each export issue names the traces of the records it concerned (`ExportIssue.traceIds`): a refused or rejected batch, a dropped record, a capture failure inside a case. A failure that names the case's trace is that case's own and final (its checkpoint reads `failed`, and resume refuses it); one that names only other traces leaves the case accepted; one that names no trace (a processor flush that failed) or a rejection in a batch of several traces is decided by the case's trace receipt, at once or on resume for a checkpoint still `pending`: the receipt must hold the case's root span and at least as many spans as the case ended (the checkpoint records the count at the flush; without it the receipt does not accept). The explicit alternative `{mode:"omit",reason:"..."}` stores the omission reason and declared trace ID without a fake snapshot. It can complete despite an export failure; export diagnostics remain available on the Hue client. There is no automatic fallback to omission. When required telemetry is not accepted, the runner by default keeps the saved outcome, leaves the execution started and rejects (`traceNotAccepted: "stop"`). With `traceNotAccepted: "fail_case"` it instead completes that case as failed (a `TelemetryNotAccepted` error, the evidence omitted with a reason starting `telemetry_not_accepted`, and no output or generated files attached, so no scorer can pass it), calls `onTelemetryNotAccepted(entry)` once, as soon as it completes the case (a resumed call only lists it), lists it in `report.telemetryNotAccepted` with sanitized issue counts, and goes on; `hue eval` uses this. A scorer that grades only the world can still score such a case, so treat a listed case as failed whatever its results say, as `hue eval` does.
+`traceEvidence:{mode:"required"}` waits for trace and log export acknowledgement after the root span ends. The transport is shared by every case in flight, so each export issue names the traces of the records it concerned (`ExportIssue.traceIds`): a refused or rejected batch, a dropped record, a capture failure inside a case. A failure that names the case's trace is that case's own and final (its checkpoint reads `failed`, and resume refuses it); one that names only other traces leaves the case accepted; one that names no trace (a processor flush that failed) or a rejection in a batch of several traces is decided by the case's trace receipt, at once or on resume for a checkpoint still `pending`: the receipt must hold the case's root span and at least as many spans as the case ended (the checkpoint records the count at the flush; without it the receipt does not accept). The explicit alternative `{mode:"omit",reason:"..."}` stores the omission reason and declared trace ID without a fake snapshot. It can complete despite an export failure; export diagnostics remain available on the Hue client. There is no automatic fallback to omission. When required telemetry is not accepted, the runner by default keeps the saved outcome, leaves the execution started and rejects (`traceNotAccepted: "stop"`). With `traceNotAccepted: "fail_case"` it instead completes that case as failed (a `TelemetryNotAccepted` error, the evidence omitted with a reason starting `telemetry_not_accepted`, and no output or generated files attached, so no scorer can pass it), calls `onTelemetryNotAccepted(entry)` once, as soon as it completes the case (a resumed call only lists it), lists it in `report.telemetryNotAccepted` with sanitized issue counts, and goes on; `hue eval` uses this. A scorer that grades only the world can still score such a case, so treat a listed case as failed whatever its results say, as `hue eval` does. `traceNotAccepted:"pending"` completes a case whose telemetry Hue did not accept in time in its true state with `traceEvidence:"pending"` and the count of spans the case ended (`traceSpanCount`): Hue scores the sealed world at once, attaches the trace once it holds every span (a root span alone is not the trace), or records the evidence omitted a day later; such cases are listed in `RunnerReport.tracePending`. A saved outcome whose export the receipt does not accept on resume completes the same way. `hue eval --trace-not-accepted pending` chooses it for the CLI.
 
 ## Local scorers
 
@@ -371,6 +371,8 @@ files directory must be owned by the current user and closed to everyone else (m
 target again; if result content is not persisted, the JSON output cannot be reconstructed and the
 case is reported as uncertain.
 
+A target that throws completes its execution as `error` with a type that says what stopped the case. `TargetError` is the agent's own failure. A service the agent called that refused with a retryable status (408, 429 or 5xx, such as a model provider's rate limit or outage, when the error carries the exchange: headers, a response or a URL) is `ServiceRefused`; a connection that failed (`ECONNRESET`, `ECONNREFUSED`, `fetch failed`, an `APIConnectionError`) is `ConnectionFailed`; a call that timed out (`TimeoutError`, `APITimeoutError`) is `TimedOut`. The runner reads the error's `cause`, `lastError` and `errors` chains for these signals. Hue counts the three as infrastructure and leaves them out of the agent's pass rate; a refusal the agent caused (a 400 from a model provider or another service) stays `TargetError`. Any other status Hue's own clients answered with (a 4xx: a key, a project, an environment version or a world Hue refused) is `ConfigurationRejected`, which Hue counts as configuration and also leaves out of the agent's pass rate: an error from Hue's own clients is never the agent's. The clients fetch with `redirect: "error"`, so a redirect from the configured endpoint surfaces without a status, as `ConnectionFailed`. A runner of your own that completes an execution with a `type` of its own can say which class Hue files it under with the error's `class`.
+
 `rescore` downloads a subject's frozen `files` — the pinned inputs and the generated outputs, with
 `role: "output"` for the documents a run produced — so a code evaluator can grade saved documents
 without invoking an agent. Local scorers receive them as `context.files` exactly as they do during

package/dist/cli/eval.js

@@ -83,16 +83,28 @@ Output and limits:
   --save-version                  Freeze an unsaved eval-set version before running
   --checkpoint-dir <path>         Private checkpoint directory (default: .hue/eval/<agent-key>)
   --concurrency <n>               Cases in flight, 1-64 (default: 1)
+  --trace-not-accepted <policy>   A case whose telemetry Hue did not accept in time: fail_case
+                                  (default) completes it as failed with the evidence omitted;
+                                  pending completes it in its true state with the trace still to
+                                  arrive, which Hue attaches once its export is complete
   --timeout <seconds>             Per-case --command timeout (default: 600)
   --wait <seconds>                Verdict wait after the run finishes (default: 300)
   -h, --help                      Show this help
 
 HUE_API_KEY must be a "Read and write" project key; it is never printed.
 Code evaluators pinned to a direct run are graded by Hue's executor after the upload; the wait
-covers them. Exit codes: 0 every case passed, 1 a case failed, errored or is incomplete,
-2 usage error, 130 interrupted.
+covers them. Exit codes: 0 every case passed, 1 a case failed, errored, is inconclusive or is
+incomplete, 2 usage error, 130 interrupted.
 `;
 /** Thrown for invalid arguments or configuration; exits with status 2. */
+/** The `--trace-not-accepted` policy: the failed-case default, or pending evidence. */
+function traceNotAcceptedPolicy(value) {
+    if (value === undefined || value === "fail_case")
+        return "fail_case";
+    if (value === "pending")
+        return "pending";
+    throw new UsageError("--trace-not-accepted must be fail_case or pending");
+}
 class UsageError extends Error {
     constructor(message) {
         super(message);
@@ -129,6 +141,7 @@ function parse(argv) {
                 json: { type: "boolean", default: false },
                 content: { type: "boolean", default: false },
                 "no-output": { type: "boolean", default: false },
+                "trace-not-accepted": { type: "string" },
                 "save-version": { type: "boolean", default: false },
                 "checkpoint-dir": { type: "string" },
                 concurrency: { type: "string" },
@@ -702,6 +715,11 @@ const STATE_LABEL = {
     skipped: "SKIPPED",
     pending: "PENDING",
 };
+/** A skipped case whose deciding evaluator could not decide it, as an answer-graded case whose
+ * required judge could not run or answer: Hue's explanation begins "Inconclusive:". It is named
+ * apart from a case its evaluators did not grade, since it needs a judge, not another pin. */
+const inconclusive = (item) => item.state === "skipped" && item.explanations.some((text) => /^inconclusive:/i.test(text));
+const stateLabel = (item) => inconclusive(item) ? "INCONCLUSIVE" : STATE_LABEL[item.state];
 export function renderTable(verdicts, output) {
     const cases = verdicts.summary.cases;
     // A column per metric name; a name several evaluators report, such as two judges' `verdict`,
@@ -736,7 +754,7 @@ export function renderTable(verdicts, output) {
             // reporting the metric shows "-".
             return item.notApplicable?.includes(column.version) ? "n/a" : "-";
         }),
-        STATE_LABEL[item.state],
+        stateLabel(item),
     ]);
     const widths = header.map((cell, index) => Math.max(cell.length, ...rows.map((row) => row[index].length)));
     const line = (cells) => cells
@@ -751,15 +769,18 @@ export function renderTable(verdicts, output) {
         if (item.state === "passed")
             continue;
         const details = [...item.errors.map((type) => `scorer error ${type}`), ...item.explanations];
-        for (const detail of details.length ? details : [STATE_LABEL[item.state].toLowerCase()])
+        for (const detail of details.length ? details : [stateLabel(item).toLowerCase()])
             output.log(`  ${item.externalKey}: ${detail}`);
     }
     const totals = verdicts.summary.totals;
+    const inconclusiveCount = cases.filter(inconclusive).length;
+    const skipped = totals.skipped - inconclusiveCount;
     const advisoryFailures = cases.reduce((count, item) => count +
         item.metrics.filter((metric) => item.advisory?.includes(metric.scorerVersionId) && !metricPassed(metric)).length, 0);
     const extra = [
         totals.error ? `${totals.error} error` : "",
-        totals.skipped ? `${totals.skipped} skipped` : "",
+        inconclusiveCount ? `${inconclusiveCount} inconclusive` : "",
+        skipped > 0 ? `${skipped} skipped` : "",
         totals.pending ? `${totals.pending} pending` : "",
         totals.notApplicable
             ? `${totals.notApplicable} evaluator result${totals.notApplicable === 1 ? "" : "s"} not applicable`
@@ -1116,7 +1137,7 @@ async function runOnce(values, connection, agents, agent, hue, output, signal) {
         persistResultContent: !values["no-output"],
         traceEvidence: { mode: "required" },
         // A case whose telemetry Hue did not accept fails instead of staying started.
-        traceNotAccepted: "fail_case",
+        traceNotAccepted: traceNotAcceptedPolicy(values["trace-not-accepted"]),
         onTelemetryNotAccepted: telemetry.report,
         concurrency,
         agentRevision: agent.revision,
@@ -1221,7 +1242,7 @@ async function runDirect(run, connection) {
             // only the telemetry.
             persistResultContent: !values["no-output"],
             traceEvidence: { mode: "required" },
-            traceNotAccepted: "fail_case",
+            traceNotAccepted: traceNotAcceptedPolicy(values["trace-not-accepted"]),
             onTelemetryNotAccepted: telemetry.report,
             concurrency: run.concurrency,
             scorers: [],
@@ -1297,7 +1318,7 @@ async function runWorker(values, connection, adapter, agent, hue, output, signal
         },
         scorers: [],
         concurrency,
-        traceNotAccepted: "fail_case",
+        traceNotAccepted: traceNotAcceptedPolicy(values["trace-not-accepted"]),
         onTelemetryNotAccepted: telemetry.report,
         deprecationWarnings: false,
         signal,
@@ -1402,6 +1423,8 @@ export async function runEvalCommand(argv) {
     let json = false;
     try {
         const { values, positionals } = parse(argv);
+        // A usage error before anything is prepared or created: no run is left to recover from it.
+        traceNotAcceptedPolicy(values["trace-not-accepted"]);
         json = values.json;
         const output = {
             log: (line) => (json ? process.stderr : process.stdout).write(`${line}\n`),

package/dist/evals/failure.d.ts

@@ -0,0 +1,18 @@
+/**
+ * What a target's throw shows about a service the agent called, when a service rather than the
+ * agent's own code stopped it: `ServiceRefused` for a retryable HTTP status (408, 429 or 5xx) a
+ * model provider, Hue or another service answered with, such as a rate limit or an outage;
+ * `ConnectionFailed` for a connection that failed; `TimedOut` for a call that timed out; and
+ * `ConfigurationRejected` for any other status Hue's own clients answered with (a 4xx: a key, a
+ * project, an environment version or a world Hue refused), which is the caller's configuration;
+ * the clients fetch with `redirect: "error"`, so a redirect surfaces without a status, as a
+ * failed connection.
+ * The runner records the type on the execution in place of `TargetError`; Hue classes the first
+ * three as infrastructure and the last as configuration, so an error from Hue's own clients is
+ * never filed as the agent's. Anything else is the agent's own error.
+ */
+export type ServiceFailureType = "ServiceRefused" | "ConnectionFailed" | "TimedOut" | "ConfigurationRejected";
+/** The service failure a target's throw shows, by the errors linked from it, in precedence
+ * order: a timeout, a failed connection, a retryable refusal, a status Hue's own client was
+ * refused with; null for the agent's own error. */
+export declare function serviceFailureType(error: unknown): ServiceFailureType | null;

package/dist/evals/failure.js

@@ -0,0 +1,150 @@
+import { HueConnectionError } from "../client.js";
+import { HueEnvironmentError } from "../environment/client.js";
+import { HueApiError } from "./client.js";
+const TIMEOUT_NAMES = new Set(["TimeoutError", "APIConnectionTimeoutError", "APITimeoutError"]);
+/** Node's and Undici's codes for a call that timed out, connecting or waiting on the answer. */
+const TIMEOUT_CODES = new Set([
+    "ETIMEDOUT",
+    "UND_ERR_CONNECT_TIMEOUT",
+    "UND_ERR_HEADERS_TIMEOUT",
+    "UND_ERR_BODY_TIMEOUT",
+]);
+const NETWORK_CODES = new Set([
+    "ECONNRESET",
+    "ECONNREFUSED",
+    "ENOTFOUND",
+    "EAI_AGAIN",
+    "EPIPE",
+    "ENETUNREACH",
+    "EHOSTUNREACH",
+    "UND_ERR_SOCKET",
+]);
+/** Links followed from the thrown error, and errors read at most. */
+const MAX_DEPTH = 6;
+const MAX_ERRORS = 64;
+/** The errors an error links to: its `cause`, and the `errors` and `lastError` of an
+ * `AggregateError` or a retry error. A property that throws when read links nothing. */
+function links(error) {
+    try {
+        const { cause, lastError, errors } = error;
+        return [cause, lastError, ...(Array.isArray(errors) ? errors.slice(0, MAX_ERRORS) : [])];
+    }
+    catch {
+        return [];
+    }
+}
+/** The thrown error and every error within `MAX_DEPTH` links of it, each once, so a cycle ends
+ * the walk. */
+function linked(error) {
+    const found = [];
+    const seen = new Set();
+    let level = [error];
+    for (let depth = 0; depth <= MAX_DEPTH && level.length; depth++) {
+        const next = [];
+        for (const item of level) {
+            if (typeof item !== "object" || item === null || seen.has(item))
+                continue;
+            if (found.length === MAX_ERRORS)
+                return found;
+            seen.add(item);
+            found.push(item);
+            next.push(...links(item));
+        }
+        level = next;
+    }
+    return found;
+}
+/** The error's `name` and its class names, since some SDKs never set `name`. */
+function names(error) {
+    const found = [];
+    const name = error.name;
+    if (typeof name === "string")
+        found.push(name);
+    let prototype = Object.getPrototypeOf(error);
+    for (let depth = 0; depth < MAX_DEPTH && prototype; depth++) {
+        const constructor = prototype.constructor;
+        if (typeof constructor?.name === "string")
+            found.push(constructor.name);
+        prototype = Object.getPrototypeOf(prototype);
+    }
+    return found;
+}
+const retryable = (status) => typeof status === "number" &&
+    (status === 408 || status === 429 || (status >= 500 && status < 600));
+function timedOut(error) {
+    const { code } = error;
+    return ((typeof code === "string" && TIMEOUT_CODES.has(code)) ||
+        names(error).some((name) => TIMEOUT_NAMES.has(name)));
+}
+/** An error one of Hue's own clients threw, which carries the status Hue answered or none. */
+function hueClientError(error) {
+    return (error instanceof HueConnectionError ||
+        error instanceof HueEnvironmentError ||
+        error instanceof HueApiError);
+}
+function disconnected(error) {
+    // Hue's clients report a request that got no usable response without a status.
+    if (hueClientError(error) && error.status === undefined)
+        return true;
+    const { code, message } = error;
+    const classes = names(error);
+    return ((typeof code === "string" && NETWORK_CODES.has(code)) ||
+        classes.includes("APIConnectionError") ||
+        (classes.includes("TypeError") && message === "fetch failed"));
+}
+/** A retryable status a service answered with: on Hue's own clients, or on an error that also
+ * carries the exchange it came from (the OpenAI and Anthropic SDKs' `headers`, the AI SDK's
+ * `url` or `responseHeaders`, a fetch `response`), so an agent's own error that only names a
+ * status stays the agent's. */
+function refused(error) {
+    if (hueClientError(error))
+        return retryable(error.status);
+    const { status, statusCode, headers, response, url, responseHeaders } = error;
+    const exchanged = [headers, response, url, responseHeaders].some((item) => item !== undefined && item !== null);
+    // A fetch `Response` carries the status itself.
+    const answered = response !== null && typeof response === "object"
+        ? response
+        : {};
+    return (exchanged &&
+        (retryable(status) ||
+            retryable(statusCode) ||
+            retryable(answered.status) ||
+            retryable(answered.statusCode)));
+}
+/** A status Hue's own client answered with that is neither a success nor retryable, a 3xx or a
+ * 4xx: Hue refused the caller's key, project, environment version or world, so the case could
+ * not run as configured. The clients fetch with `redirect: "error"`, so a redirect reaches here
+ * only as a status some caller constructed; a real one surfaces without a status, as a failed
+ * connection. Only Hue's clients show it; another service's 4xx stays the agent's. */
+function rejected(error) {
+    const { status } = error;
+    return (hueClientError(error) &&
+        typeof status === "number" &&
+        status >= 300 &&
+        status < 500 &&
+        !retryable(status));
+}
+/** Whether any error shows the signal; one whose properties throw when read shows none. */
+const any = (errors, signal) => errors.some((error) => {
+    try {
+        return signal(error);
+    }
+    catch {
+        return false;
+    }
+});
+/** The service failure a target's throw shows, by the errors linked from it, in precedence
+ * order: a timeout, a failed connection, a retryable refusal, a status Hue's own client was
+ * refused with; null for the agent's own error. */
+export function serviceFailureType(error) {
+    const errors = linked(error);
+    if (any(errors, timedOut))
+        return "TimedOut";
+    if (any(errors, disconnected))
+        return "ConnectionFailed";
+    if (any(errors, refused))
+        return "ServiceRefused";
+    if (any(errors, rejected))
+        return "ConfigurationRejected";
+    return null;
+}

package/dist/evals/local-worker.d.ts

@@ -104,7 +104,7 @@ export interface RunLocalAgentOptions {
     concurrency?: number;
     /** What a case does when its telemetry is not accepted; see
      * `traceNotAccepted` of {@link runExperiment}. Defaults to `"stop"`. */
-    traceNotAccepted?: "stop" | "fail_case";
+    traceNotAccepted?: "stop" | "fail_case" | "pending";
     /** Called as each case is failed for its telemetry; see `onTelemetryNotAccepted` of
      * {@link runExperiment}. */
     onTelemetryNotAccepted?(entry: TelemetryNotAccepted): void | Promise<void>;

package/dist/evals/runner.d.ts

@@ -123,9 +123,13 @@ export interface RunExperimentOptions extends RunnerOptions {
      * rejects. `"fail_case"` completes the execution as failed instead, with the evidence omitted
      * under the reason `telemetry_not_accepted` and without its output or generated files, so no
      * scorer can pass it; reports it in {@link RunnerReport.telemetryNotAccepted} and goes on with
-     * the other cases.
+     * the other cases. `"pending"` completes the execution in its true state with
+     * `traceEvidence: "pending"` and the count of spans the case ended: Hue scores the sealed world
+     * at once and attaches the trace once it holds every span, or records the evidence omitted a
+     * day later; reports it in {@link RunnerReport.tracePending} and goes on. A saved outcome whose
+     * export the receipt does not accept on resume completes the same way.
      */
-    traceNotAccepted?: "stop" | "fail_case";
+    traceNotAccepted?: "stop" | "fail_case" | "pending";
     /** Called when this call completes a case as failed under `traceNotAccepted: "fail_case"`,
      * before the run goes on, so the failure is known even if a later case stops the run. A call
      * that resumes an already completed case lists it in the report without calling again. */
@@ -152,6 +156,21 @@ export interface RunnerReport {
     /** Cases completed as failed because their telemetry was not accepted, under
      * `traceNotAccepted: "fail_case"`; absent when there were none. */
     telemetryNotAccepted?: TelemetryNotAccepted[];
+    /** Cases completed with their trace evidence pending, under `traceNotAccepted: "pending"`;
+     * absent when there were none. */
+    tracePending?: TracePending[];
+}
+/** A case completed with its trace still to arrive. */
+export interface TracePending {
+    /** Frozen case ID. */
+    caseId: string;
+    /** The case's external key. */
+    caseKey: string;
+    /** Execution completed pending. */
+    executionId: string;
+    /** How many spans the case ended, which Hue holds the arriving trace to; absent when the
+     * checkpoint recorded no count. */
+    spanCount?: number;
 }
 /** Export issue counts, never content: which signal, what happened, the HTTP status when Hue
  * answered, and how many records. */

package/dist/evals/runner.js

@@ -6,6 +6,7 @@ import { ROOT_CONTEXT } from "@opentelemetry/api";
 import { HueExportError } from "../transport.js";
 import { HueApiError } from "./client.js";
 import { loadEnvironmentEvidence } from "./environment-evidence.js";
+import { serviceFailureType } from "./failure.js";
 import { CheckpointStore } from "./checkpoint.js";
 import { onForcedExit } from "./exit-cleanup.js";
 import { assertSafeFileNames, downloadCaseFiles, downloadCaseInputs, localOutputFiles, verifyCaseFiles, OutputFileError, stageOutputFiles, targetFileRoles, uploadOutputFiles, } from "./files.js";
@@ -252,6 +253,20 @@ async function pool(items, concurrency, execute) {
 /** Whether Hue holds the case's telemetry, read from the trace receipt: its root span and at
  * least as many spans as the case ended; false when they are not there by the wait's end or the
  * receipt cannot be read, and for a checkpoint an older runner saved without the span. */
+/** Turn a prepared completion into a pending one: the trace is declared but still to arrive,
+ * and the span count the checkpoint recorded at the flush goes with it, so Hue freezes the trace
+ * only once it holds every span the case ended, never a root span alone. */
+function pendingCompletion(prepared) {
+    prepared.complete.traceEvidence = "pending";
+    delete prepared.complete.omissionReason;
+    if (prepared.trace?.spans !== undefined)
+        prepared.complete.traceSpanCount = prepared.trace.spans;
+}
+/** The reverse, once the receipt accepted the trace after all: required evidence, no count. */
+function acceptedCompletion(prepared) {
+    prepared.complete.traceEvidence = "required";
+    delete prepared.complete.traceSpanCount;
+}
 async function traceLanded(hue, trace) {
     // Without the count of spans the case ended, the receipt cannot tell a lost child span from a
     // landed trace, and does not accept.
@@ -417,13 +432,15 @@ export async function runExperiment(options) {
         const filesRoot = resolve(options.filesDirectory ?? join(options.checkpointDirectory, "files"));
         const sanitize = (message) => message.slice(0, 4000).toWellFormed().replaceAll("\u0000", "");
         // A target stopped at its world's deadline is a `TargetTimeout`, told apart from an error the
-        // target raised itself: the agent did not finish, rather than finishing wrongly.
+        // target raised itself: the agent did not finish, rather than finishing wrongly. A service the
+        // agent called that refused, dropped the connection or timed out is named apart from the
+        // agent's own error (`failure.ts`), so Hue counts it as infrastructure, not as the agent.
         const errorPayload = (error) => ({
             type: error instanceof OutputTooLargeError
                 ? "OutputTooLarge"
                 : error instanceof TargetTimeoutError
                     ? "TargetTimeout"
-                    : "TargetError",
+                    : (serviceFailureType(error) ?? "TargetError"),
             ...(options.persistResultContent && error instanceof Error
                 ? { message: sanitize(error.message) }
                 : {}),
@@ -707,7 +724,15 @@ export async function runExperiment(options) {
                     if (options.traceEvidence.mode !== "omit")
                         throw error;
                 }
-                if (exportError !== undefined && options.traceEvidence.mode !== "omit") {
+                if (exportError !== undefined &&
+                    options.traceEvidence.mode !== "omit" &&
+                    options.traceNotAccepted === "pending") {
+                    // The outcome stands as the target gave it; Hue attaches the trace once it holds
+                    // every span the case ended, which the checkpoint's count tells it.
+                    pendingCompletion(checkpoint);
+                    await store.write(file, checkpoint);
+                }
+                else if (exportError !== undefined && options.traceEvidence.mode !== "omit") {
                     if (options.traceNotAccepted !== "fail_case")
                         throw exportError;
                     // Required evidence that Hue did not accept fails the case rather than leaving its
@@ -746,9 +771,26 @@ export async function runExperiment(options) {
                 prepared.complete.traceEvidence !== "omit" &&
                 (await traceLanded(options.hue, prepared.trace))) {
                 prepared.exportState = "accepted";
+                // A completion made pending by an earlier receipt read is required again: the trace
+                // landed, so Hue freezes it at completion as for any accepted case.
+                if (prepared.complete.traceEvidence === "pending")
+                    acceptedCompletion(prepared);
+                await store.write(file, prepared);
+            }
+            // Under the pending policy a saved outcome the receipt does not accept, whether no flush
+            // decided it or an export failure named its trace, completes with the trace pending rather
+            // than being refused: Hue attaches the trace when it arrives in full.
+            if (options.traceNotAccepted === "pending" &&
+                prepared.exportState !== "accepted" &&
+                prepared.exportState !== "not_accepted" &&
+                prepared.complete.traceEvidence !== "omit" &&
+                prepared.complete.traceEvidence !== "pending") {
+                pendingCompletion(prepared);
                 await store.write(file, prepared);
             }
-            if (prepared.exportState !== "accepted" && prepared.complete.traceEvidence !== "omit")
+            if (prepared.exportState !== "accepted" &&
+                prepared.complete.traceEvidence !== "omit" &&
+                prepared.complete.traceEvidence !== "pending")
                 throw new TraceExportUnacknowledgedError(prepared.executionId);
             const save = () => store.write(file, prepared);
             const notAccepted = prepared.exportState === "not_accepted"
@@ -770,6 +812,15 @@ export async function runExperiment(options) {
             }
             if (notAccepted)
                 (report.telemetryNotAccepted ??= []).push(notAccepted);
+            if (prepared.complete.traceEvidence === "pending")
+                (report.tracePending ??= []).push({
+                    caseId: item.id,
+                    caseKey: item.externalKey,
+                    executionId: prepared.executionId,
+                    ...(prepared.complete.traceSpanCount !== undefined
+                        ? { spanCount: prepared.complete.traceSpanCount }
+                        : {}),
+                });
             const results = await uploadScores(options, experiment.evaluation.id, prepared.scores, save, versions);
             report.subjectIds.push(prepared.completion.subjectId);
             report.resultIds.push(...results);

package/dist/evals/simulation.d.ts

@@ -160,7 +160,7 @@ export interface RunSimulationOptions {
     };
     /** What a case does when required telemetry is not accepted; see
      * `traceNotAccepted` of {@link runExperiment}. Defaults to `"stop"`. */
-    traceNotAccepted?: "stop" | "fail_case";
+    traceNotAccepted?: "stop" | "fail_case" | "pending";
     /** Called as each case is failed for its telemetry; see `onTelemetryNotAccepted` of
      * {@link runExperiment}. */
     onTelemetryNotAccepted?(entry: TelemetryNotAccepted): void | Promise<void>;

package/dist/evals/types.d.ts

@@ -542,6 +542,10 @@ export interface TypedError {
     type: string;
     /** Optional message; stored only when result content is persisted. */
     message?: string;
+    /** For a runner of your own that writes a `type` of its own: the class Hue files it under
+     * (`agent`, `infrastructure`, `configuration`, `telemetry` or `unsupported`), which Hue reads
+     * only for a name it does not know. The SDK's own types need none. */
+    class?: "agent" | "infrastructure" | "configuration" | "telemetry" | "unsupported";
 }
 /** Input for {@link EvaluationClient.startExecution}. */
 export interface StartExecution {
@@ -570,10 +574,13 @@ export interface CompleteExecution {
     error?: TypedError;
     /** Trace revision the stored snapshot must have reached. */
     expectedTraceRevision?: number;
-    /** Whether stored trace evidence is required or explicitly omitted. */
-    traceEvidence?: "required" | "omit";
+    /** Whether stored trace evidence is required, explicitly omitted, or pending: declared and
+     * still to arrive, which Hue attaches once the export is complete. */
+    traceEvidence?: "required" | "omit" | "pending";
     /** Why trace evidence was omitted. */
     omissionReason?: string;
+    /** With `pending`: how many spans the case ended, which Hue holds the arriving trace to. */
+    traceSpanCount?: number;
 }
 /** Result of {@link EvaluationClient.completeExecution}. */
 export interface Completion {
@@ -633,8 +640,9 @@ export interface Subject {
     experimentId: string;
     /** Attempt number of the execution. */
     attempt: number;
-    /** Whether trace evidence was captured, omitted with a reason or not requested. */
-    traceEvidence: "captured" | "omitted" | "not_requested";
+    /** Whether trace evidence was captured, omitted with a reason, not requested, or is pending
+     * (declared and still to arrive; Hue attaches it once the export is complete). */
+    traceEvidence: "captured" | "omitted" | "not_requested" | "pending";
     /** Declared OpenTelemetry trace ID, or `null`. */
     traceExternalId: string | null;
     /** Reason trace evidence was omitted, or `null`. */

package/dist/evals.d.ts

@@ -1,7 +1,7 @@
 export { ArtifactSizeError, createEvaluationClient, EvaluationClient, HueApiError, isTransientApiError, } from "./evals/client.js";
 export type { EvaluationClientOptions } from "./evals/client.js";
 export { runExperiment, rescore, UncertainExecutionError, OutcomeSerializationError, TargetCancelledError, TargetOutcomeUncertainError, TargetTimeoutError, TraceExportUnacknowledgedError, } from "./evals/runner.js";
-export type { RunExperimentOptions, RunExperimentTargetContext, RescoreOptions, RunnerReport, TelemetryIssueCount, TelemetryNotAccepted, } from "./evals/runner.js";
+export type { RunExperimentOptions, RunExperimentTargetContext, RescoreOptions, RunnerReport, TelemetryIssueCount, TelemetryNotAccepted, TracePending, } from "./evals/runner.js";
 export { TargetResult, withFiles } from "./evals/types.js";
 export { CaseFileError, outputContentTypes, outputFileLimits, OutputFileError, safeFilename, targetFileRoles, } from "./evals/files.js";
 export type { CaseFileErrorCode } from "./evals/files.js";

package/dist/version.d.ts

@@ -1,2 +1,2 @@
 /** Package version shared by the instrumentation scope and the export User-Agent. */
-export declare const sdkVersion = "0.12.0";
+export declare const sdkVersion = "0.12.1";

package/dist/version.js

@@ -1,3 +1,3 @@
 // Generated by scripts/write-version.mjs from package.json; do not edit by hand.
 /** Package version shared by the instrumentation scope and the export User-Agent. */
-export const sdkVersion = "0.12.0";
+export const sdkVersion = "0.12.1";

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hue-run/sdk",
-  "version": "0.12.0",
+  "version": "0.12.1",
   "private": false,
   "license": "MIT",
   "publishConfig": {