@hue-run/sdk 0.11.4 → 0.12.1

package/CLI.md

@@ -42,7 +42,7 @@ project manifest are refused because managers can update ancestor locks; Python
 The generated `hue.setup.mjs` or `hue_setup.py` always selects `captureContent: false` /
 `capture_content=False`. For a supported application, setup installs the dependency and adds the
 managed import and middleware registration to the existing entrypoint; an unreferenced helper is
-not a completed integration. TypeScript uses `@hue-run/sdk@0.11.4`, `@opentelemetry/api@1.9.1` and
+not a completed integration. TypeScript uses `@hue-run/sdk@0.12.1`, `@opentelemetry/api@1.9.1` and
 `@opentelemetry/context-async-hooks@2.11.0`; Python setup uses its separately tested package pin.
 Content capture requires an ordinary account-managed key and a later explicit application decision.
 
@@ -469,7 +469,7 @@ node packages/sdk-typescript/scripts/verify-package.mjs --artifacts-dir .artifac
 # Set project to an existing supported fixture; use the same directory on resume.
 project=/absolute/path/to/supported-fixture
 node packages/sdk-typescript/scripts/verify-setup-live.mjs \
-  --archive .artifacts/typescript/hue-run-sdk-0.11.4.tgz \
+  --archive .artifacts/typescript/hue-run-sdk-0.12.1.tgz \
   --origin https://STAGING_ORIGIN \
   --project "$project" --command setup \
   --evidence .context/setup-staging-before-claim.json
@@ -480,7 +480,7 @@ the private local handoff and finish the real browser claim, then reconcile the
 
 ```sh
 node packages/sdk-typescript/scripts/verify-setup-live.mjs \
-  --archive .artifacts/typescript/hue-run-sdk-0.11.4.tgz \
+  --archive .artifacts/typescript/hue-run-sdk-0.12.1.tgz \
   --origin https://STAGING_ORIGIN \
   --project "$project" --command claim \
   --evidence .context/setup-staging-after-claim.json
@@ -618,6 +618,11 @@ for example, needs a reviewed `attemptBaselineV2` (see
 registers none, so put `"attemptBaselineV2"` in the run's configuration JSON when you launch the
 run from Hue.
 
+`--trace-not-accepted pending` completes a case whose telemetry Hue did not accept in time in its
+true state with the trace still to arrive, carrying the count of spans the case ended; Hue scores
+the sealed world at once and attaches the trace once its export is complete. The default,
+`fail_case`, completes such a case as failed with its evidence omitted.
+
 A Scenario or eval set whose dataset version is not saved cannot back an experiment: the command
 exits 1 and asks for **Save eval-set version** in Hue or `--save-version`, which freezes that
 version at its current revision. Connection settings are `HUE_API_KEY` and `HUE_BASE_URL`
@@ -663,9 +668,9 @@ an interrupted run without invoking the agent again; a different selection is re
 unfinished one is resumed or its directory is removed.
 
 Exit codes: `0` every case passed the evaluators that apply to it (advisory ones never count),
-`1` a case failed, errored (including one no pinned evaluator applies to or decides), was skipped
-or Hue's checks were
-still pending at `--wait`, `2` usage or configuration error (including a missing key), `130`
+`1` a case failed, errored (including one no pinned evaluator applies to or decides), was skipped,
+was inconclusive (a required judge could not run or answer; the table says `INCONCLUSIVE` and the
+summary counts it apart from skipped cases) or Hue's checks were still pending at `--wait`, `2` usage or configuration error (including a missing key), `130`
 interrupted. On Node.js 22, load TypeScript adapters with `NODE_OPTIONS=--experimental-strip-types`;
 non-erasable syntax (enums, parameter properties, namespaces) needs a loader such as `--import tsx`
 on any Node.js version.

package/EVALUATIONS.md

@@ -116,7 +116,7 @@ Both choices are required and independent:
 
 Frozen dataset inputs/references already exist on Hue. Their presence is independent of these switches. No output is inferred from telemetry. JavaScript `undefined` means unavailable output; JSON `null`, false, zero and empty string remain present values. Historical subjects with unavailable output produce skipped results without executing a scorer callback. A failed quality metric remains `state:"scored"` with `passed:false`. Target errors and scorer errors remain separate. Target error types are generalized to `TargetError`; when content is enabled, bounded messages may be stored. Never put credentials in error messages, output, custom metric values, names or metadata.
 
-`traceEvidence:{mode:"required"}` waits for trace and log export acknowledgement after the root span ends. The explicit alternative `{mode:"omit",reason:"..."}` stores the omission reason and declared trace ID without a fake snapshot. It can complete despite an export failure; export diagnostics remain available on the Hue client. There is no automatic fallback to omission. When required telemetry is not accepted, the runner by default keeps the saved outcome, leaves the execution started and rejects (`traceNotAccepted: "stop"`). With `traceNotAccepted: "fail_case"` it instead completes that case as failed (a `TelemetryNotAccepted` error, the evidence omitted with a reason starting `telemetry_not_accepted`, and no output or generated files attached, so no scorer can pass it), calls `onTelemetryNotAccepted(entry)` once, as soon as it completes the case (a resumed call only lists it), lists it in `report.telemetryNotAccepted` with sanitized issue counts, and goes on; `hue eval` uses this. A scorer that grades only the world can still score such a case, so treat a listed case as failed whatever its results say, as `hue eval` does.
+`traceEvidence:{mode:"required"}` waits for trace and log export acknowledgement after the root span ends. The transport is shared by every case in flight, so each export issue names the traces of the records it concerned (`ExportIssue.traceIds`): a refused or rejected batch, a dropped record, a capture failure inside a case. A failure that names the case's trace is that case's own and final (its checkpoint reads `failed`, and resume refuses it); one that names only other traces leaves the case accepted; one that names no trace (a processor flush that failed) or a rejection in a batch of several traces is decided by the case's trace receipt, at once or on resume for a checkpoint still `pending`: the receipt must hold the case's root span and at least as many spans as the case ended (the checkpoint records the count at the flush; without it the receipt does not accept). The explicit alternative `{mode:"omit",reason:"..."}` stores the omission reason and declared trace ID without a fake snapshot. It can complete despite an export failure; export diagnostics remain available on the Hue client. There is no automatic fallback to omission. When required telemetry is not accepted, the runner by default keeps the saved outcome, leaves the execution started and rejects (`traceNotAccepted: "stop"`). With `traceNotAccepted: "fail_case"` it instead completes that case as failed (a `TelemetryNotAccepted` error, the evidence omitted with a reason starting `telemetry_not_accepted`, and no output or generated files attached, so no scorer can pass it), calls `onTelemetryNotAccepted(entry)` once, as soon as it completes the case (a resumed call only lists it), lists it in `report.telemetryNotAccepted` with sanitized issue counts, and goes on; `hue eval` uses this. A scorer that grades only the world can still score such a case, so treat a listed case as failed whatever its results say, as `hue eval` does. `traceNotAccepted:"pending"` completes a case whose telemetry Hue did not accept in time in its true state with `traceEvidence:"pending"` and the count of spans the case ended (`traceSpanCount`): Hue scores the sealed world at once, attaches the trace once it holds every span (a root span alone is not the trace), or records the evidence omitted a day later; such cases are listed in `RunnerReport.tracePending`. A saved outcome whose export the receipt does not accept on resume completes the same way. `hue eval --trace-not-accepted pending` chooses it for the CLI.
 
 ## Local scorers
 
@@ -252,14 +252,30 @@ without invoking the target or scorers. A lost preparation acknowledgement remai
 and is never recovered through binding reads, credential refresh or target replay. Synthetic
 acceptance does not contact official Gmail or claim universal provider parity.
 
-Completion or result-upload failures keep the run claimed by the durable worker identity.
-Restart with the same checkpoint directory to resume saved uploads without invoking the
-candidate again. A lost world-seal acknowledgement is recovered by reading authoritative world
-state. If the seal or candidate outcome cannot be confirmed, or an outcome cannot be serialized,
-the worker reports `attention` and stops; operator investigation is required. Such runs are not
-automatically reclaimed, and presenting the same uncertain checkpoint again cannot replay the
-candidate. Public package acceptance proves this lifecycle against local fixtures; exact
-installed-registry-package to hosted-facade acceptance remains a post-publication Fern gate.
+The worker does not die with one failure. A read, or a mutation the server deduplicates by its
+idempotency key (the worker's own register, claim, heartbeat and completion included), is sent
+again after a connection failure, a timeout or a 408 or 5xx that carried no `Retry-After`, up to
+`maxAttempts` (default 4) times; a mutation without a key is sent once. A registration or claim
+that still fails transiently is reported through `onPollError` and polled again with a growing
+wait; a refusal Hue decided on (a revoked key, a disabled agent) is thrown. A transient failure
+of a completion or result upload keeps the run claimed by the durable worker identity: the next
+claim returns it to the same process, which resumes saved uploads without invoking the candidate
+again (`onRunFailed` says `retry`), and after `maxRunAttempts` (default 5) such failures the run
+is given up as `attention` for a project member to requeue or cancel from its run page; a failure
+that would recur (a refusal Hue decided on, a case file whose bytes differ from its manifest) is
+given up at once. A lost
+world-seal acknowledgement is recovered by reading authoritative world state. If the seal or
+candidate outcome cannot be confirmed, or an outcome cannot be serialized, the run is given up
+as `attention` at once and the worker goes on to other runs; when Hue cannot record that either,
+the worker stops with the error. Presenting the same uncertain checkpoint again cannot replay the
+candidate. `maxRuns` counts runs that settled either way. Inside a run, one case's own failure
+does not stop the cases beside it; a failure every case would share (Hue not answering or
+refusing, telemetry not accepted, a world that could not be sealed) stops new cases from
+starting until the run resumes. Each world's lifetime is `worldTtlSeconds` (the server's default
+is an hour): a target still running past it is told through `context.signal`, its case ends as
+an `error` of type `TargetTimeout` and the world is sealed abandoned. Public package acceptance
+proves this lifecycle against local fixtures; exact installed-registry-package to hosted-facade
+acceptance remains a post-publication Fern gate.
 
 ## Direct cases and files
 
@@ -355,6 +371,8 @@ files directory must be owned by the current user and closed to everyone else (m
 target again; if result content is not persisted, the JSON output cannot be reconstructed and the
 case is reported as uncertain.
 
+A target that throws completes its execution as `error` with a type that says what stopped the case. `TargetError` is the agent's own failure. A service the agent called that refused with a retryable status (408, 429 or 5xx, such as a model provider's rate limit or outage, when the error carries the exchange: headers, a response or a URL) is `ServiceRefused`; a connection that failed (`ECONNRESET`, `ECONNREFUSED`, `fetch failed`, an `APIConnectionError`) is `ConnectionFailed`; a call that timed out (`TimeoutError`, `APITimeoutError`) is `TimedOut`. The runner reads the error's `cause`, `lastError` and `errors` chains for these signals. Hue counts the three as infrastructure and leaves them out of the agent's pass rate; a refusal the agent caused (a 400 from a model provider or another service) stays `TargetError`. Any other status Hue's own clients answered with (a 4xx: a key, a project, an environment version or a world Hue refused) is `ConfigurationRejected`, which Hue counts as configuration and also leaves out of the agent's pass rate: an error from Hue's own clients is never the agent's. The clients fetch with `redirect: "error"`, so a redirect from the configured endpoint surfaces without a status, as `ConnectionFailed`. A runner of your own that completes an execution with a `type` of its own can say which class Hue files it under with the error's `class`.
+
 `rescore` downloads a subject's frozen `files` — the pinned inputs and the generated outputs, with
 `role: "output"` for the documents a run produced — so a code evaluator can grade saved documents
 without invoking an agent. Local scorers receive them as `context.files` exactly as they do during

package/dist/cli/eval.js

@@ -83,16 +83,28 @@ Output and limits:
   --save-version                  Freeze an unsaved eval-set version before running
   --checkpoint-dir <path>         Private checkpoint directory (default: .hue/eval/<agent-key>)
   --concurrency <n>               Cases in flight, 1-64 (default: 1)
+  --trace-not-accepted <policy>   A case whose telemetry Hue did not accept in time: fail_case
+                                  (default) completes it as failed with the evidence omitted;
+                                  pending completes it in its true state with the trace still to
+                                  arrive, which Hue attaches once its export is complete
   --timeout <seconds>             Per-case --command timeout (default: 600)
   --wait <seconds>                Verdict wait after the run finishes (default: 300)
   -h, --help                      Show this help
 
 HUE_API_KEY must be a "Read and write" project key; it is never printed.
 Code evaluators pinned to a direct run are graded by Hue's executor after the upload; the wait
-covers them. Exit codes: 0 every case passed, 1 a case failed, errored or is incomplete,
-2 usage error, 130 interrupted.
+covers them. Exit codes: 0 every case passed, 1 a case failed, errored, is inconclusive or is
+incomplete, 2 usage error, 130 interrupted.
 `;
 /** Thrown for invalid arguments or configuration; exits with status 2. */
+/** The `--trace-not-accepted` policy: the failed-case default, or pending evidence. */
+function traceNotAcceptedPolicy(value) {
+    if (value === undefined || value === "fail_case")
+        return "fail_case";
+    if (value === "pending")
+        return "pending";
+    throw new UsageError("--trace-not-accepted must be fail_case or pending");
+}
 class UsageError extends Error {
     constructor(message) {
         super(message);
@@ -129,6 +141,7 @@ function parse(argv) {
                 json: { type: "boolean", default: false },
                 content: { type: "boolean", default: false },
                 "no-output": { type: "boolean", default: false },
+                "trace-not-accepted": { type: "string" },
                 "save-version": { type: "boolean", default: false },
                 "checkpoint-dir": { type: "string" },
                 concurrency: { type: "string" },
@@ -702,6 +715,11 @@ const STATE_LABEL = {
     skipped: "SKIPPED",
     pending: "PENDING",
 };
+/** A skipped case whose deciding evaluator could not decide it, as an answer-graded case whose
+ * required judge could not run or answer: Hue's explanation begins "Inconclusive:". It is named
+ * apart from a case its evaluators did not grade, since it needs a judge, not another pin. */
+const inconclusive = (item) => item.state === "skipped" && item.explanations.some((text) => /^inconclusive:/i.test(text));
+const stateLabel = (item) => inconclusive(item) ? "INCONCLUSIVE" : STATE_LABEL[item.state];
 export function renderTable(verdicts, output) {
     const cases = verdicts.summary.cases;
     // A column per metric name; a name several evaluators report, such as two judges' `verdict`,
@@ -736,7 +754,7 @@ export function renderTable(verdicts, output) {
             // reporting the metric shows "-".
             return item.notApplicable?.includes(column.version) ? "n/a" : "-";
         }),
-        STATE_LABEL[item.state],
+        stateLabel(item),
     ]);
     const widths = header.map((cell, index) => Math.max(cell.length, ...rows.map((row) => row[index].length)));
     const line = (cells) => cells
@@ -751,15 +769,18 @@ export function renderTable(verdicts, output) {
         if (item.state === "passed")
             continue;
         const details = [...item.errors.map((type) => `scorer error ${type}`), ...item.explanations];
-        for (const detail of details.length ? details : [STATE_LABEL[item.state].toLowerCase()])
+        for (const detail of details.length ? details : [stateLabel(item).toLowerCase()])
             output.log(`  ${item.externalKey}: ${detail}`);
     }
     const totals = verdicts.summary.totals;
+    const inconclusiveCount = cases.filter(inconclusive).length;
+    const skipped = totals.skipped - inconclusiveCount;
     const advisoryFailures = cases.reduce((count, item) => count +
         item.metrics.filter((metric) => item.advisory?.includes(metric.scorerVersionId) && !metricPassed(metric)).length, 0);
     const extra = [
         totals.error ? `${totals.error} error` : "",
-        totals.skipped ? `${totals.skipped} skipped` : "",
+        inconclusiveCount ? `${inconclusiveCount} inconclusive` : "",
+        skipped > 0 ? `${skipped} skipped` : "",
         totals.pending ? `${totals.pending} pending` : "",
         totals.notApplicable
             ? `${totals.notApplicable} evaluator result${totals.notApplicable === 1 ? "" : "s"} not applicable`
@@ -1116,7 +1137,7 @@ async function runOnce(values, connection, agents, agent, hue, output, signal) {
         persistResultContent: !values["no-output"],
         traceEvidence: { mode: "required" },
         // A case whose telemetry Hue did not accept fails instead of staying started.
-        traceNotAccepted: "fail_case",
+        traceNotAccepted: traceNotAcceptedPolicy(values["trace-not-accepted"]),
         onTelemetryNotAccepted: telemetry.report,
         concurrency,
         agentRevision: agent.revision,
@@ -1221,7 +1242,7 @@ async function runDirect(run, connection) {
             // only the telemetry.
             persistResultContent: !values["no-output"],
             traceEvidence: { mode: "required" },
-            traceNotAccepted: "fail_case",
+            traceNotAccepted: traceNotAcceptedPolicy(values["trace-not-accepted"]),
             onTelemetryNotAccepted: telemetry.report,
             concurrency: run.concurrency,
             scorers: [],
@@ -1297,11 +1318,25 @@ async function runWorker(values, connection, adapter, agent, hue, output, signal
         },
         scorers: [],
         concurrency,
-        traceNotAccepted: "fail_case",
+        traceNotAccepted: traceNotAcceptedPolicy(values["trace-not-accepted"]),
         onTelemetryNotAccepted: telemetry.report,
         deprecationWarnings: false,
         signal,
         ...(maxRuns === undefined ? {} : { maxRuns }),
+        onPollError(error) {
+            output.error(`Hue did not answer the poll (${explain(error)}); polling again`);
+        },
+        onRunFailed(failure) {
+            const run = `Run ${failure.runId}`;
+            if (failure.outcome === "retry") {
+                const waitMillis = failure.waitMillis ?? 0;
+                output.error(`${run} attempt ${failure.attempt} failed (${explain(failure.error)}); resuming in ${Math.round(waitMillis / 1000)} s`);
+            }
+            else if (failure.outcome === "attention")
+                output.error(`${run} needs attention (${explain(failure.error)}); requeue or cancel it from its run page`);
+            else
+                output.error(`${run} was released or cancelled in Hue; its outcome here is not recorded`);
+        },
         target(inputs, tools, context) {
             output.log(`[${context.item.externalKey}] agent started`);
             return adapter(inputs, {
@@ -1315,7 +1350,9 @@ async function runWorker(values, connection, adapter, agent, hue, output, signal
                 ...(context.connectionBundle ? { connectionBundle: context.connectionBundle } : {}),
                 files: context.files,
                 outputDirectory: context.outputDirectory,
-                signal,
+                // The case's own signal: the worker stopping, or the world's deadline passing, both
+                // end a command that would otherwise keep running past its timed-out case.
+                signal: context.signal ?? signal,
             });
         },
         async onCompleted(report) {
@@ -1386,6 +1423,8 @@ export async function runEvalCommand(argv) {
     let json = false;
     try {
         const { values, positionals } = parse(argv);
+        // A usage error before anything is prepared or created: no run is left to recover from it.
+        traceNotAcceptedPolicy(values["trace-not-accepted"]);
         json = values.json;
         const output = {
             log: (line) => (json ? process.stderr : process.stdout).write(`${line}\n`),

package/dist/client.d.ts

@@ -43,6 +43,9 @@ export declare class HueClient {
     readonly enabled: boolean;
     private logger;
     private storage;
+    /** Reports a contained capture or instrumentation failure, named to the trace active where
+     * it happened when there is one. */
+    private instrumentationFailed;
     private tracerProvider;
     private loggerProvider;
     private ownedProviders?;

package/dist/client.js

@@ -169,6 +169,20 @@ export class HueClient {
     enabled;
     logger;
     storage = new AsyncLocalStorage();
+    /** Reports a contained capture or instrumentation failure, named to the trace active where
+     * it happened when there is one. */
+    instrumentationFailed(signal = "traces", count = 1) {
+        let traceId;
+        try {
+            traceId = trace
+                .getSpan(this.storage.getStore()?.context ?? context.active())
+                ?.spanContext().traceId;
+        }
+        catch {
+            // The failure is recorded without a trace.
+        }
+        this.transport.instrumentationFailure(signal, undefined, count, traceId);
+    }
     tracerProvider;
     loggerProvider;
     ownedProviders;
@@ -210,7 +224,7 @@ export class HueClient {
         }
         this.captureContent = this.transport.options.captureContent;
         this.enabled = this.transport.options.enabled !== false;
-        this.tracer = new ContextualTracer(this.tracerProvider.getTracer(HUE_SCOPE, sdkVersion), this.storage, () => this.enabled && !this.closed, () => this.transport.instrumentationFailure());
+        this.tracer = new ContextualTracer(this.tracerProvider.getTracer(HUE_SCOPE, sdkVersion), this.storage, () => this.enabled && !this.closed, () => this.instrumentationFailed());
         this.logger = this.loggerProvider.getLogger(HUE_SCOPE, sdkVersion);
     }
     /**
@@ -232,7 +246,7 @@ export class HueClient {
             return this.storage.getStore()?.context ?? context.active();
         }
         catch {
-            this.transport.instrumentationFailure();
+            this.instrumentationFailed();
             return ROOT_CONTEXT;
         }
     }
@@ -256,7 +270,7 @@ export class HueClient {
                 span = this.storage.run(active, () => this.tracer.startSpan(name, { kind: options.kind ?? SpanKind.INTERNAL, attributes: options.attributes }, active.context));
             }
             catch {
-                this.transport.instrumentationFailure();
+                this.instrumentationFailed();
             }
         }
         // A disabled or failed client creates no span; the application's own active span then stays
@@ -268,7 +282,7 @@ export class HueClient {
                 spanContext = trace.setSpan(active.context, span);
             }
             catch {
-                this.transport.instrumentationFailure();
+                this.instrumentationFailed();
                 spanContext = trace.setSpan(ROOT_CONTEXT, span);
             }
         const handle = {
@@ -287,7 +301,7 @@ export class HueClient {
                     handle.setInput(options.input);
             }
             catch {
-                this.transport.instrumentationFailure();
+                this.instrumentationFailed();
             }
             try {
                 return await callback(handle);
@@ -303,9 +317,7 @@ export class HueClient {
         // The span is also OpenTelemetry's active span while the callback runs, so spans from other
         // instrumentations (HTTP clients, provider SDKs) join this trace when the application has
         // registered a context manager. Hue still registers none itself.
-        return this.storage.run({ ...active, context: spanContext }, () => created
-            ? runInContext(spanContext, execute, () => this.transport.instrumentationFailure())
-            : execute());
+        return this.storage.run({ ...active, context: spanContext }, () => created ? runInContext(spanContext, execute, () => this.instrumentationFailed()) : execute());
     }
     /**
      * Runs `execute` inside an `execute_tool` span named after the tool. `input` and a defined result
@@ -329,7 +341,7 @@ export class HueClient {
             if (isLabel(value))
                 attributes[key] = value;
             else if (this.enabled && !this.closed)
-                this.transport.instrumentationFailure();
+                this.instrumentationFailed();
         };
         stamp("gen_ai.tool.call.id", options.callId);
         stamp("mcp.server.name", options.mcp?.name);
@@ -364,7 +376,7 @@ export class HueClient {
             if (isLabel(value))
                 return value;
             if (active)
-                this.transport.instrumentationFailure();
+                this.instrumentationFailed();
             return fallback;
         };
         const requestModel = label(model, "unknown");
@@ -413,14 +425,14 @@ export class HueClient {
             if (value === undefined)
                 continue;
             if (!Number.isInteger(value) || value < 0) {
-                this.transport.instrumentationFailure();
+                this.instrumentationFailed();
                 continue;
             }
             try {
                 span.setAttribute(key, value);
             }
             catch {
-                this.transport.instrumentationFailure();
+                this.instrumentationFailed();
             }
         }
     }
@@ -433,7 +445,7 @@ export class HueClient {
             propagator.inject(activeContext, carrier, defaultTextMapSetter);
         }
         catch {
-            this.transport.instrumentationFailure();
+            this.instrumentationFailed();
         }
     }
     /** Reads W3C trace context from a carrier for use as `parentContext`. */
@@ -442,7 +454,7 @@ export class HueClient {
             return propagator.extract(ROOT_CONTEXT, carrier, defaultTextMapGetter);
         }
         catch {
-            this.transport.instrumentationFailure();
+            this.instrumentationFailed();
             return ROOT_CONTEXT;
         }
     }
@@ -461,7 +473,7 @@ export class HueClient {
             span.addEvent("exception", { "exception.type": type });
         }
         catch {
-            this.transport.instrumentationFailure();
+            this.instrumentationFailed();
         }
     }
     /**
@@ -498,7 +510,7 @@ export class HueClient {
                 else if (typeof explicit === "string" && explicit.trim() && explicit.length <= 256)
                     attributes[key] = explicit;
                 else
-                    this.transport.instrumentationFailure("logs");
+                    this.instrumentationFailed("logs");
             };
             stamp("gen_ai.operation.name", messages.operation, enclosing?.model?.operation);
             stamp("gen_ai.provider.name", messages.provider, enclosing?.model?.provider);
@@ -513,7 +525,7 @@ export class HueClient {
             });
         }
         catch {
-            this.transport.instrumentationFailure("logs");
+            this.instrumentationFailed("logs");
         }
     }
     /**
@@ -545,7 +557,7 @@ export class HueClient {
             const addresses = hostedServerAddresses(provider, options.request);
             const activity = hostedToolActivity(provider, response);
             if (activity.skipped > 0)
-                this.transport.instrumentationFailure("traces", undefined, activity.skipped);
+                this.instrumentationFailed("traces", activity.skipped);
             const server = (label) => {
                 const attributes = {};
                 if (label === undefined)
@@ -564,7 +576,7 @@ export class HueClient {
                     if (isLabel(value))
                         attributes[key] = value;
                     else
-                        this.transport.instrumentationFailure();
+                        this.instrumentationFailed();
                 }
                 const address = addresses.get(label);
                 if (address !== undefined)
@@ -617,7 +629,7 @@ export class HueClient {
                         span.setAttributes(toolCatalogSummary(encodeContent(listing.definitions, catalogLimits)));
                     }
                     catch {
-                        this.transport.instrumentationFailure();
+                        this.instrumentationFailed();
                     }
                 if (listing.errorType !== undefined)
                     fail(span, listing.errorType);
@@ -625,7 +637,7 @@ export class HueClient {
             }
         }
         catch {
-            this.transport.instrumentationFailure();
+            this.instrumentationFailed();
         }
     }
     /**
@@ -692,12 +704,12 @@ export class HueClient {
                 if (isLabel(name))
                     attributes["hue.file.name"] = name;
                 else
-                    this.transport.instrumentationFailure();
+                    this.instrumentationFailed();
             }
             span.addEvent("hue.file", attributes);
         }
         catch {
-            this.transport.instrumentationFailure();
+            this.instrumentationFailed();
         }
     }
     setContent(span, key, value) {
@@ -707,7 +719,7 @@ export class HueClient {
             span.setAttribute(key, encodeContent(value));
         }
         catch {
-            this.transport.instrumentationFailure();
+            this.instrumentationFailed();
         }
     }
     /**

package/dist/evals/client.d.ts

@@ -9,6 +9,10 @@ export interface EvaluationClientOptions {
     baseUrl?: string;
     /** Per-request budget in milliseconds, 100–60000. Default 10000. */
     timeoutMillis?: number;
+    /** Bounded attempts for a read or a mutation the server deduplicates by its key, after a
+     * connection failure, a timeout or a 408 or 5xx Hue did not time with `Retry-After`. Default 4,
+     * maximum 10; 1 sends every such request once. */
+    maxAttempts?: number;
 }
 /** Thrown for a failed evaluation API request; the message is fixed and never includes response text. */
 export declare class HueApiError extends Error {
@@ -16,12 +20,22 @@ export declare class HueApiError extends Error {
     readonly status?: number | undefined;
     /** Seconds Hue asked the caller to wait (`Retry-After` on a 429 or 503), when it said. */
     readonly retryAfterSeconds?: number | undefined;
+    /** Hue's `X-Hue-Diagnostic` code, which tells refusals of one status apart, when it sent one. */
+    readonly diagnostic?: string | undefined;
     constructor(
     /** HTTP status when Hue answered; absent for network, timeout and parsing failures. */
     status?: number | undefined, 
     /** Seconds Hue asked the caller to wait (`Retry-After` on a 429 or 503), when it said. */
-    retryAfterSeconds?: number | undefined);
+    retryAfterSeconds?: number | undefined, 
+    /** Hue's `X-Hue-Diagnostic` code, which tells refusals of one status apart, when it sent one. */
+    diagnostic?: string | undefined);
 }
+/**
+ * Whether the same request, sent again, can succeed: a connection failure, a timeout, or a status
+ * Hue answers while it cannot act yet (408, 429, 500, 502, 503, 504). A refusal Hue decided on,
+ * any other 4xx, is not, and no number of attempts changes it.
+ */
+export declare function isTransientApiError(error: unknown): boolean;
 /** An artifact download that ran past the size its caller expected; the rest is not read. */
 export declare class ArtifactSizeError extends Error {
     /** The size the caller expected, in bytes. */
@@ -32,22 +46,28 @@ export declare class ArtifactSizeError extends Error {
 }
 /**
  * Typed client for Hue's evaluation REST API: datasets, scorers, experiments, executions, runs,
- * results and hosted judge jobs. No implicit mutation retry: callers retain stable idempotency keys
- * for experiments and results. The one exception is a request Hue refused before acting on it with
- * a short `Retry-After` (HTTP 429 or 503, at most 5 seconds): it is sent again after that wait, up
- * to four times. Responses are bounded to 4 MiB.
+ * results and hosted judge jobs. A request Hue refused before acting on it with a short
+ * `Retry-After` (HTTP 429 or 503, at most 5 seconds) is sent again after that wait whatever its
+ * method, up to four times; a longer wait is the caller's error, carried as `retryAfterSeconds`.
+ * A read, or a mutation the server deduplicates by its idempotency key, is also sent again after
+ * a failure Hue did not time, a connection failure, a timeout or a 408 or 5xx without the header,
+ * up to `maxAttempts` times with a jittered backoff. A mutation without a key is never retried
+ * implicitly: callers retain stable idempotency keys for experiments and results. Responses are
+ * bounded to 4 MiB.
  */
 export declare class EvaluationClient {
     /** Validated Hue origin. */
     readonly baseUrl: string;
     private readonly apiKey;
     private readonly timeoutMillis;
+    private readonly maxAttempts;
     constructor(options: EvaluationClientOptions);
     /** Sends one request to Hue, again after the wait while Hue refuses it with a short
      * `Retry-After`. `init` runs for every attempt, so each has its own timeout; `signal` also ends
      * each attempt, and a wait to send one again, whose refusal is then the caller's error. */
     private send;
     private request;
+    private requestOnce;
     /** Verified bytes of one ready artifact in this project, bounded to the 25 MiB pilot file size.
      * With `maxBytes`, the download stops one byte past it and throws `ArtifactSizeError`. */
     downloadArtifact(id: string, options?: {