@hue-run/sdk 0.11.4 → 0.12.0

package/CLI.md

@@ -42,7 +42,7 @@ project manifest are refused because managers can update ancestor locks; Python
 The generated `hue.setup.mjs` or `hue_setup.py` always selects `captureContent: false` /
 `capture_content=False`. For a supported application, setup installs the dependency and adds the
 managed import and middleware registration to the existing entrypoint; an unreferenced helper is
-not a completed integration. TypeScript uses `@hue-run/sdk@0.11.4`, `@opentelemetry/api@1.9.1` and
+not a completed integration. TypeScript uses `@hue-run/sdk@0.12.0`, `@opentelemetry/api@1.9.1` and
 `@opentelemetry/context-async-hooks@2.11.0`; Python setup uses its separately tested package pin.
 Content capture requires an ordinary account-managed key and a later explicit application decision.
 
@@ -469,7 +469,7 @@ node packages/sdk-typescript/scripts/verify-package.mjs --artifacts-dir .artifac
 # Set project to an existing supported fixture; use the same directory on resume.
 project=/absolute/path/to/supported-fixture
 node packages/sdk-typescript/scripts/verify-setup-live.mjs \
-  --archive .artifacts/typescript/hue-run-sdk-0.11.4.tgz \
+  --archive .artifacts/typescript/hue-run-sdk-0.12.0.tgz \
   --origin https://STAGING_ORIGIN \
   --project "$project" --command setup \
   --evidence .context/setup-staging-before-claim.json
@@ -480,7 +480,7 @@ the private local handoff and finish the real browser claim, then reconcile the
 
 ```sh
 node packages/sdk-typescript/scripts/verify-setup-live.mjs \
-  --archive .artifacts/typescript/hue-run-sdk-0.11.4.tgz \
+  --archive .artifacts/typescript/hue-run-sdk-0.12.0.tgz \
   --origin https://STAGING_ORIGIN \
   --project "$project" --command claim \
   --evidence .context/setup-staging-after-claim.json

package/EVALUATIONS.md

@@ -116,7 +116,7 @@ Both choices are required and independent:
 
 Frozen dataset inputs/references already exist on Hue. Their presence is independent of these switches. No output is inferred from telemetry. JavaScript `undefined` means unavailable output; JSON `null`, false, zero and empty string remain present values. Historical subjects with unavailable output produce skipped results without executing a scorer callback. A failed quality metric remains `state:"scored"` with `passed:false`. Target errors and scorer errors remain separate. Target error types are generalized to `TargetError`; when content is enabled, bounded messages may be stored. Never put credentials in error messages, output, custom metric values, names or metadata.
 
-`traceEvidence:{mode:"required"}` waits for trace and log export acknowledgement after the root span ends. The explicit alternative `{mode:"omit",reason:"..."}` stores the omission reason and declared trace ID without a fake snapshot. It can complete despite an export failure; export diagnostics remain available on the Hue client. There is no automatic fallback to omission. When required telemetry is not accepted, the runner by default keeps the saved outcome, leaves the execution started and rejects (`traceNotAccepted: "stop"`). With `traceNotAccepted: "fail_case"` it instead completes that case as failed (a `TelemetryNotAccepted` error, the evidence omitted with a reason starting `telemetry_not_accepted`, and no output or generated files attached, so no scorer can pass it), calls `onTelemetryNotAccepted(entry)` once, as soon as it completes the case (a resumed call only lists it), lists it in `report.telemetryNotAccepted` with sanitized issue counts, and goes on; `hue eval` uses this. A scorer that grades only the world can still score such a case, so treat a listed case as failed whatever its results say, as `hue eval` does.
+`traceEvidence:{mode:"required"}` waits for trace and log export acknowledgement after the root span ends. The transport is shared by every case in flight, so each export issue names the traces of the records it concerned (`ExportIssue.traceIds`): a refused or rejected batch, a dropped record, a capture failure inside a case. A failure that names the case's trace is that case's own and final (its checkpoint reads `failed`, and resume refuses it); one that names only other traces leaves the case accepted; one that names no trace (a processor flush that failed) or a rejection in a batch of several traces is decided by the case's trace receipt, at once or on resume for a checkpoint still `pending`: the receipt must hold the case's root span and at least as many spans as the case ended (the checkpoint records the count at the flush; without it the receipt does not accept). The explicit alternative `{mode:"omit",reason:"..."}` stores the omission reason and declared trace ID without a fake snapshot. It can complete despite an export failure; export diagnostics remain available on the Hue client. There is no automatic fallback to omission. When required telemetry is not accepted, the runner by default keeps the saved outcome, leaves the execution started and rejects (`traceNotAccepted: "stop"`). With `traceNotAccepted: "fail_case"` it instead completes that case as failed (a `TelemetryNotAccepted` error, the evidence omitted with a reason starting `telemetry_not_accepted`, and no output or generated files attached, so no scorer can pass it), calls `onTelemetryNotAccepted(entry)` once, as soon as it completes the case (a resumed call only lists it), lists it in `report.telemetryNotAccepted` with sanitized issue counts, and goes on; `hue eval` uses this. A scorer that grades only the world can still score such a case, so treat a listed case as failed whatever its results say, as `hue eval` does.
 
 ## Local scorers
 
@@ -252,14 +252,30 @@ without invoking the target or scorers. A lost preparation acknowledgement remai
 and is never recovered through binding reads, credential refresh or target replay. Synthetic
 acceptance does not contact official Gmail or claim universal provider parity.
 
-Completion or result-upload failures keep the run claimed by the durable worker identity.
-Restart with the same checkpoint directory to resume saved uploads without invoking the
-candidate again. A lost world-seal acknowledgement is recovered by reading authoritative world
-state. If the seal or candidate outcome cannot be confirmed, or an outcome cannot be serialized,
-the worker reports `attention` and stops; operator investigation is required. Such runs are not
-automatically reclaimed, and presenting the same uncertain checkpoint again cannot replay the
-candidate. Public package acceptance proves this lifecycle against local fixtures; exact
-installed-registry-package to hosted-facade acceptance remains a post-publication Fern gate.
+The worker does not die with one failure. A read, or a mutation the server deduplicates by its
+idempotency key (the worker's own register, claim, heartbeat and completion included), is sent
+again after a connection failure, a timeout or a 408 or 5xx that carried no `Retry-After`, up to
+`maxAttempts` (default 4) times; a mutation without a key is sent once. A registration or claim
+that still fails transiently is reported through `onPollError` and polled again with a growing
+wait; a refusal Hue decided on (a revoked key, a disabled agent) is thrown. A transient failure
+of a completion or result upload keeps the run claimed by the durable worker identity: the next
+claim returns it to the same process, which resumes saved uploads without invoking the candidate
+again (`onRunFailed` says `retry`), and after `maxRunAttempts` (default 5) such failures the run
+is given up as `attention` for a project member to requeue or cancel from its run page; a failure
+that would recur (a refusal Hue decided on, a case file whose bytes differ from its manifest) is
+given up at once. A lost
+world-seal acknowledgement is recovered by reading authoritative world state. If the seal or
+candidate outcome cannot be confirmed, or an outcome cannot be serialized, the run is given up
+as `attention` at once and the worker goes on to other runs; when Hue cannot record that either,
+the worker stops with the error. Presenting the same uncertain checkpoint again cannot replay the
+candidate. `maxRuns` counts runs that settled either way. Inside a run, one case's own failure
+does not stop the cases beside it; a failure every case would share (Hue not answering or
+refusing, telemetry not accepted, a world that could not be sealed) stops new cases from
+starting until the run resumes. Each world's lifetime is `worldTtlSeconds` (the server's default
+is an hour): a target still running past it is told through `context.signal`, its case ends as
+an `error` of type `TargetTimeout` and the world is sealed abandoned. Public package acceptance
+proves this lifecycle against local fixtures; exact installed-registry-package to hosted-facade
+acceptance remains a post-publication Fern gate.
 
 ## Direct cases and files
 

package/dist/cli/eval.js

@@ -1302,6 +1302,20 @@ async function runWorker(values, connection, adapter, agent, hue, output, signal
         deprecationWarnings: false,
         signal,
         ...(maxRuns === undefined ? {} : { maxRuns }),
+        onPollError(error) {
+            output.error(`Hue did not answer the poll (${explain(error)}); polling again`);
+        },
+        onRunFailed(failure) {
+            const run = `Run ${failure.runId}`;
+            if (failure.outcome === "retry") {
+                const waitMillis = failure.waitMillis ?? 0;
+                output.error(`${run} attempt ${failure.attempt} failed (${explain(failure.error)}); resuming in ${Math.round(waitMillis / 1000)} s`);
+            }
+            else if (failure.outcome === "attention")
+                output.error(`${run} needs attention (${explain(failure.error)}); requeue or cancel it from its run page`);
+            else
+                output.error(`${run} was released or cancelled in Hue; its outcome here is not recorded`);
+        },
         target(inputs, tools, context) {
             output.log(`[${context.item.externalKey}] agent started`);
             return adapter(inputs, {
@@ -1315,7 +1329,9 @@ async function runWorker(values, connection, adapter, agent, hue, output, signal
                 ...(context.connectionBundle ? { connectionBundle: context.connectionBundle } : {}),
                 files: context.files,
                 outputDirectory: context.outputDirectory,
-                signal,
+                // The case's own signal: the worker stopping, or the world's deadline passing, both
+                // end a command that would otherwise keep running past its timed-out case.
+                signal: context.signal ?? signal,
             });
         },
         async onCompleted(report) {

package/dist/client.d.ts

@@ -43,6 +43,9 @@ export declare class HueClient {
     readonly enabled: boolean;
     private logger;
     private storage;
+    /** Reports a contained capture or instrumentation failure, named to the trace active where
+     * it happened when there is one. */
+    private instrumentationFailed;
     private tracerProvider;
     private loggerProvider;
     private ownedProviders?;

package/dist/client.js

@@ -169,6 +169,20 @@ export class HueClient {
     enabled;
     logger;
     storage = new AsyncLocalStorage();
+    /** Reports a contained capture or instrumentation failure, named to the trace active where
+     * it happened when there is one. */
+    instrumentationFailed(signal = "traces", count = 1) {
+        let traceId;
+        try {
+            traceId = trace
+                .getSpan(this.storage.getStore()?.context ?? context.active())
+                ?.spanContext().traceId;
+        }
+        catch {
+            // The failure is recorded without a trace.
+        }
+        this.transport.instrumentationFailure(signal, undefined, count, traceId);
+    }
     tracerProvider;
     loggerProvider;
     ownedProviders;
@@ -210,7 +224,7 @@ export class HueClient {
         }
         this.captureContent = this.transport.options.captureContent;
         this.enabled = this.transport.options.enabled !== false;
-        this.tracer = new ContextualTracer(this.tracerProvider.getTracer(HUE_SCOPE, sdkVersion), this.storage, () => this.enabled && !this.closed, () => this.transport.instrumentationFailure());
+        this.tracer = new ContextualTracer(this.tracerProvider.getTracer(HUE_SCOPE, sdkVersion), this.storage, () => this.enabled && !this.closed, () => this.instrumentationFailed());
         this.logger = this.loggerProvider.getLogger(HUE_SCOPE, sdkVersion);
     }
     /**
@@ -232,7 +246,7 @@ export class HueClient {
             return this.storage.getStore()?.context ?? context.active();
         }
         catch {
-            this.transport.instrumentationFailure();
+            this.instrumentationFailed();
             return ROOT_CONTEXT;
         }
     }
@@ -256,7 +270,7 @@ export class HueClient {
                 span = this.storage.run(active, () => this.tracer.startSpan(name, { kind: options.kind ?? SpanKind.INTERNAL, attributes: options.attributes }, active.context));
             }
             catch {
-                this.transport.instrumentationFailure();
+                this.instrumentationFailed();
             }
         }
         // A disabled or failed client creates no span; the application's own active span then stays
@@ -268,7 +282,7 @@ export class HueClient {
                 spanContext = trace.setSpan(active.context, span);
             }
             catch {
-                this.transport.instrumentationFailure();
+                this.instrumentationFailed();
                 spanContext = trace.setSpan(ROOT_CONTEXT, span);
             }
         const handle = {
@@ -287,7 +301,7 @@ export class HueClient {
                     handle.setInput(options.input);
             }
             catch {
-                this.transport.instrumentationFailure();
+                this.instrumentationFailed();
             }
             try {
                 return await callback(handle);
@@ -303,9 +317,7 @@ export class HueClient {
         // The span is also OpenTelemetry's active span while the callback runs, so spans from other
         // instrumentations (HTTP clients, provider SDKs) join this trace when the application has
         // registered a context manager. Hue still registers none itself.
-        return this.storage.run({ ...active, context: spanContext }, () => created
-            ? runInContext(spanContext, execute, () => this.transport.instrumentationFailure())
-            : execute());
+        return this.storage.run({ ...active, context: spanContext }, () => created ? runInContext(spanContext, execute, () => this.instrumentationFailed()) : execute());
     }
     /**
      * Runs `execute` inside an `execute_tool` span named after the tool. `input` and a defined result
@@ -329,7 +341,7 @@ export class HueClient {
             if (isLabel(value))
                 attributes[key] = value;
             else if (this.enabled && !this.closed)
-                this.transport.instrumentationFailure();
+                this.instrumentationFailed();
         };
         stamp("gen_ai.tool.call.id", options.callId);
         stamp("mcp.server.name", options.mcp?.name);
@@ -364,7 +376,7 @@ export class HueClient {
             if (isLabel(value))
                 return value;
             if (active)
-                this.transport.instrumentationFailure();
+                this.instrumentationFailed();
             return fallback;
         };
         const requestModel = label(model, "unknown");
@@ -413,14 +425,14 @@ export class HueClient {
             if (value === undefined)
                 continue;
             if (!Number.isInteger(value) || value < 0) {
-                this.transport.instrumentationFailure();
+                this.instrumentationFailed();
                 continue;
             }
             try {
                 span.setAttribute(key, value);
             }
             catch {
-                this.transport.instrumentationFailure();
+                this.instrumentationFailed();
             }
         }
     }
@@ -433,7 +445,7 @@ export class HueClient {
             propagator.inject(activeContext, carrier, defaultTextMapSetter);
         }
         catch {
-            this.transport.instrumentationFailure();
+            this.instrumentationFailed();
         }
     }
     /** Reads W3C trace context from a carrier for use as `parentContext`. */
@@ -442,7 +454,7 @@ export class HueClient {
             return propagator.extract(ROOT_CONTEXT, carrier, defaultTextMapGetter);
         }
         catch {
-            this.transport.instrumentationFailure();
+            this.instrumentationFailed();
             return ROOT_CONTEXT;
         }
     }
@@ -461,7 +473,7 @@ export class HueClient {
             span.addEvent("exception", { "exception.type": type });
         }
         catch {
-            this.transport.instrumentationFailure();
+            this.instrumentationFailed();
         }
     }
     /**
@@ -498,7 +510,7 @@ export class HueClient {
                 else if (typeof explicit === "string" && explicit.trim() && explicit.length <= 256)
                     attributes[key] = explicit;
                 else
-                    this.transport.instrumentationFailure("logs");
+                    this.instrumentationFailed("logs");
             };
             stamp("gen_ai.operation.name", messages.operation, enclosing?.model?.operation);
             stamp("gen_ai.provider.name", messages.provider, enclosing?.model?.provider);
@@ -513,7 +525,7 @@ export class HueClient {
             });
         }
         catch {
-            this.transport.instrumentationFailure("logs");
+            this.instrumentationFailed("logs");
         }
     }
     /**
@@ -545,7 +557,7 @@ export class HueClient {
             const addresses = hostedServerAddresses(provider, options.request);
             const activity = hostedToolActivity(provider, response);
             if (activity.skipped > 0)
-                this.transport.instrumentationFailure("traces", undefined, activity.skipped);
+                this.instrumentationFailed("traces", activity.skipped);
             const server = (label) => {
                 const attributes = {};
                 if (label === undefined)
@@ -564,7 +576,7 @@ export class HueClient {
                     if (isLabel(value))
                         attributes[key] = value;
                     else
-                        this.transport.instrumentationFailure();
+                        this.instrumentationFailed();
                 }
                 const address = addresses.get(label);
                 if (address !== undefined)
@@ -617,7 +629,7 @@ export class HueClient {
                         span.setAttributes(toolCatalogSummary(encodeContent(listing.definitions, catalogLimits)));
                     }
                     catch {
-                        this.transport.instrumentationFailure();
+                        this.instrumentationFailed();
                     }
                 if (listing.errorType !== undefined)
                     fail(span, listing.errorType);
@@ -625,7 +637,7 @@ export class HueClient {
             }
         }
         catch {
-            this.transport.instrumentationFailure();
+            this.instrumentationFailed();
         }
     }
     /**
@@ -692,12 +704,12 @@ export class HueClient {
                 if (isLabel(name))
                     attributes["hue.file.name"] = name;
                 else
-                    this.transport.instrumentationFailure();
+                    this.instrumentationFailed();
             }
             span.addEvent("hue.file", attributes);
         }
         catch {
-            this.transport.instrumentationFailure();
+            this.instrumentationFailed();
         }
     }
     setContent(span, key, value) {
@@ -707,7 +719,7 @@ export class HueClient {
             span.setAttribute(key, encodeContent(value));
         }
         catch {
-            this.transport.instrumentationFailure();
+            this.instrumentationFailed();
         }
     }
     /**

package/dist/evals/client.d.ts

@@ -9,6 +9,10 @@ export interface EvaluationClientOptions {
     baseUrl?: string;
     /** Per-request budget in milliseconds, 100–60000. Default 10000. */
     timeoutMillis?: number;
+    /** Bounded attempts for a read or a mutation the server deduplicates by its key, after a
+     * connection failure, a timeout or a 408 or 5xx Hue did not time with `Retry-After`. Default 4,
+     * maximum 10; 1 sends every such request once. */
+    maxAttempts?: number;
 }
 /** Thrown for a failed evaluation API request; the message is fixed and never includes response text. */
 export declare class HueApiError extends Error {
@@ -16,12 +20,22 @@ export declare class HueApiError extends Error {
     readonly status?: number | undefined;
     /** Seconds Hue asked the caller to wait (`Retry-After` on a 429 or 503), when it said. */
     readonly retryAfterSeconds?: number | undefined;
+    /** Hue's `X-Hue-Diagnostic` code, which tells refusals of one status apart, when it sent one. */
+    readonly diagnostic?: string | undefined;
     constructor(
     /** HTTP status when Hue answered; absent for network, timeout and parsing failures. */
     status?: number | undefined, 
     /** Seconds Hue asked the caller to wait (`Retry-After` on a 429 or 503), when it said. */
-    retryAfterSeconds?: number | undefined);
+    retryAfterSeconds?: number | undefined, 
+    /** Hue's `X-Hue-Diagnostic` code, which tells refusals of one status apart, when it sent one. */
+    diagnostic?: string | undefined);
 }
+/**
+ * Whether the same request, sent again, can succeed: a connection failure, a timeout, or a status
+ * Hue answers while it cannot act yet (408, 429, 500, 502, 503, 504). A refusal Hue decided on,
+ * any other 4xx, is not, and no number of attempts changes it.
+ */
+export declare function isTransientApiError(error: unknown): boolean;
 /** An artifact download that ran past the size its caller expected; the rest is not read. */
 export declare class ArtifactSizeError extends Error {
     /** The size the caller expected, in bytes. */
@@ -32,22 +46,28 @@ export declare class ArtifactSizeError extends Error {
 }
 /**
  * Typed client for Hue's evaluation REST API: datasets, scorers, experiments, executions, runs,
- * results and hosted judge jobs. No implicit mutation retry: callers retain stable idempotency keys
- * for experiments and results. The one exception is a request Hue refused before acting on it with
- * a short `Retry-After` (HTTP 429 or 503, at most 5 seconds): it is sent again after that wait, up
- * to four times. Responses are bounded to 4 MiB.
+ * results and hosted judge jobs. A request Hue refused before acting on it with a short
+ * `Retry-After` (HTTP 429 or 503, at most 5 seconds) is sent again after that wait whatever its
+ * method, up to four times; a longer wait is the caller's error, carried as `retryAfterSeconds`.
+ * A read, or a mutation the server deduplicates by its idempotency key, is also sent again after
+ * a failure Hue did not time, a connection failure, a timeout or a 408 or 5xx without the header,
+ * up to `maxAttempts` times with a jittered backoff. A mutation without a key is never retried
+ * implicitly: callers retain stable idempotency keys for experiments and results. Responses are
+ * bounded to 4 MiB.
  */
 export declare class EvaluationClient {
     /** Validated Hue origin. */
     readonly baseUrl: string;
     private readonly apiKey;
     private readonly timeoutMillis;
+    private readonly maxAttempts;
     constructor(options: EvaluationClientOptions);
     /** Sends one request to Hue, again after the wait while Hue refuses it with a short
      * `Retry-After`. `init` runs for every attempt, so each has its own timeout; `signal` also ends
      * each attempt, and a wait to send one again, whose refusal is then the caller's error. */
     private send;
     private request;
+    private requestOnce;
     /** Verified bytes of one ready artifact in this project, bounded to the 25 MiB pilot file size.
      * With `maxBytes`, the download stops one byte past it and throws `ArtifactSizeError`. */
     downloadArtifact(id: string, options?: {

package/dist/evals/client.js

@@ -5,17 +5,51 @@ import { attemptBindingRead, parsePrepareAttemptResultV2, parseRefreshedAttemptR
 export class HueApiError extends Error {
     status;
     retryAfterSeconds;
+    diagnostic;
     constructor(
     /** HTTP status when Hue answered; absent for network, timeout and parsing failures. */
     status, 
     /** Seconds Hue asked the caller to wait (`Retry-After` on a 429 or 503), when it said. */
-    retryAfterSeconds) {
-        super(status ? `Hue API request failed (HTTP ${status})` : "Hue API connection or response failed");
+    retryAfterSeconds, 
+    /** Hue's `X-Hue-Diagnostic` code, which tells refusals of one status apart, when it sent one. */
+    diagnostic) {
+        super(status
+            ? `Hue API request failed (HTTP ${status}${diagnostic ? `, ${diagnostic}` : ""})`
+            : "Hue API connection or response failed");
         this.status = status;
         this.retryAfterSeconds = retryAfterSeconds;
+        this.diagnostic = diagnostic;
         this.name = "HueApiError";
     }
 }
+const TRANSIENT = new Set([408, 429, 500, 502, 503, 504]);
+/**
+ * Whether the same request, sent again, can succeed: a connection failure, a timeout, or a status
+ * Hue answers while it cannot act yet (408, 429, 500, 502, 503, 504). A refusal Hue decided on,
+ * any other 4xx, is not, and no number of attempts changes it.
+ */
+export function isTransientApiError(error) {
+    return (error instanceof HueApiError && (error.status === undefined || TRANSIENT.has(error.status)));
+}
+const DIAGNOSTIC = /^[a-z_]{1,64}$/;
+function diagnosticOf(response) {
+    const value = response.headers.get("x-hue-diagnostic");
+    return value !== null && DIAGNOSTIC.test(value) ? value : undefined;
+}
+/** Whether a request may be sent again after a transient failure: every read, and a mutation the
+ * server deduplicates, either by the `idempotencyKey` in its body or because its route is
+ * idempotent by design (the local worker's register, claim, heartbeat, completion and capability
+ * routes). A mutation without a key is sent once; its caller resolves the outcome before asking
+ * again. */
+function retryable(method, body, idempotent) {
+    if (idempotent !== undefined)
+        return idempotent;
+    if (method === "GET")
+        return true;
+    return (body !== null &&
+        typeof body === "object" &&
+        typeof body.idempotencyKey === "string");
+}
 /** An artifact download that ran past the size its caller expected; the rest is not read. */
 export class ArtifactSizeError extends Error {
     maxBytes;
@@ -128,16 +162,21 @@ function refusalRetryAfter(response) {
 }
 /**
  * Typed client for Hue's evaluation REST API: datasets, scorers, experiments, executions, runs,
- * results and hosted judge jobs. No implicit mutation retry: callers retain stable idempotency keys
- * for experiments and results. The one exception is a request Hue refused before acting on it with
- * a short `Retry-After` (HTTP 429 or 503, at most 5 seconds): it is sent again after that wait, up
- * to four times. Responses are bounded to 4 MiB.
+ * results and hosted judge jobs. A request Hue refused before acting on it with a short
+ * `Retry-After` (HTTP 429 or 503, at most 5 seconds) is sent again after that wait whatever its
+ * method, up to four times; a longer wait is the caller's error, carried as `retryAfterSeconds`.
+ * A read, or a mutation the server deduplicates by its idempotency key, is also sent again after
+ * a failure Hue did not time, a connection failure, a timeout or a 408 or 5xx without the header,
+ * up to `maxAttempts` times with a jittered backoff. A mutation without a key is never retried
+ * implicitly: callers retain stable idempotency keys for experiments and results. Responses are
+ * bounded to 4 MiB.
  */
 export class EvaluationClient {
     /** Validated Hue origin. */
     baseUrl;
     apiKey;
     timeoutMillis;
+    maxAttempts;
     constructor(options) {
         const validated = validateOptions({
             ...options,
@@ -147,6 +186,10 @@ export class EvaluationClient {
         this.baseUrl = validated.baseUrl;
         this.apiKey = validated.apiKey;
         this.timeoutMillis = validated.timeoutMillis;
+        const attempts = options.maxAttempts ?? 4;
+        if (!Number.isInteger(attempts) || attempts < 1 || attempts > 10)
+            throw new RangeError("maxAttempts must be 1–10");
+        this.maxAttempts = attempts;
     }
     /** Sends one request to Hue, again after the wait while Hue refuses it with a short
      * `Retry-After`. `init` runs for every attempt, so each has its own timeout; `signal` also ends
@@ -183,11 +226,44 @@ export class EvaluationClient {
                 throw new HueApiError(response.status, askedRetryAfter(response));
         }
     }
-    async request(method, path, body, bounds = { ...valueBounds, bytes: 1024 * 1024 }, signal) {
-        // Optional top-level fields are omitted intentionally; nested undefined remains invalid.
+    async request(method, path, body, bounds = { ...valueBounds, bytes: 1024 * 1024 }, signal, 
+    /** Overrides the retry decision `retryable` takes from the method and body. */
+    idempotent) {
+        // Serialized once, outside the attempts: a body this client cannot encode is a caller error
+        // no attempt fixes. Optional top-level fields are omitted intentionally; nested undefined
+        // remains invalid.
         const payload = body === undefined
             ? undefined
             : JSON.stringify(json(Object.fromEntries(Object.entries(body).filter(([, value]) => value !== undefined)), bounds));
+        const attempts = retryable(method, body, idempotent) ? this.maxAttempts : 1;
+        for (let attempt = 1;; attempt++) {
+            try {
+                return await this.requestOnce(method, path, payload, signal);
+            }
+            catch (error) {
+                // A refusal that carried `Retry-After` was already decided by `send`: sent again while
+                // the wait was short, the caller's error when it was long. This loop covers the failures
+                // Hue did not time: a lost connection, a timeout, a 5xx or 408 without the header.
+                if (!isTransientApiError(error) ||
+                    error.retryAfterSeconds !== undefined ||
+                    attempt >= attempts ||
+                    signal?.aborted)
+                    throw error;
+                const backoff = Math.min(100 * 2 ** (attempt - 1), 2000);
+                const wait = backoff + Math.random() * backoff;
+                await new Promise((resolve) => {
+                    const done = () => {
+                        clearTimeout(timer);
+                        signal?.removeEventListener("abort", done);
+                        resolve();
+                    };
+                    const timer = setTimeout(done, wait);
+                    signal?.addEventListener("abort", done, { once: true });
+                });
+            }
+        }
+    }
+    async requestOnce(method, path, payload, signal) {
         const response = await this.send(`${this.baseUrl}/api/v1${path}`, () => ({
             method,
             headers: {
@@ -200,7 +276,7 @@ export class EvaluationClient {
         }), signal);
         if (!response.ok) {
             await response.body?.cancel();
-            throw new HueApiError(response.status, askedRetryAfter(response));
+            throw new HueApiError(response.status, askedRetryAfter(response), diagnosticOf(response));
         }
         try {
             const reader = response.body?.getReader();
@@ -474,7 +550,9 @@ export class EvaluationClient {
     async prepareAttempt(input) {
         const request = prepareAttemptInputV2.parse(input);
         const { executionId, ...body } = request;
-        const response = await this.request("POST", `/experiment-executions/${executionId}/prepare-attempt`, body, { ...valueBounds, bytes: 128_000 });
+        // Sent once although keyed: a lost acknowledgement of a credential-bearing decision stays
+        // uncertain, and the worker never reacquires credentials.
+        const response = await this.request("POST", `/experiment-executions/${executionId}/prepare-attempt`, body, { ...valueBounds, bytes: 128_000 }, undefined, false);
         try {
             return parsePrepareAttemptResultV2(response, request);
         }
@@ -662,21 +740,21 @@ export class EvaluationClient {
     }
     /** Register or refresh the fixed local agent key and revision. */
     async registerLocalAgent(input) {
-        const agent = await this.request("POST", "/local-agent-worker/register", input);
+        const agent = await this.request("POST", "/local-agent-worker/register", input, undefined, undefined, true);
         // Hue's response names the key `agentKey`; read either name.
         return { ...agent, key: agent.key ?? agent.agentKey ?? input.key };
     }
     /** Claim a queued run for this agent and durable worker identity. */
     claimLocalAgentRun(input) {
-        return this.request("POST", "/local-agent-worker/claim", input);
+        return this.request("POST", "/local-agent-worker/claim", input, undefined, undefined, true);
     }
     /** Refresh the lease of a claimed local run. */
     heartbeatLocalAgentRun(input) {
-        return this.request("POST", "/local-agent-worker/runs/heartbeat", input);
+        return this.request("POST", "/local-agent-worker/runs/heartbeat", input, undefined, undefined, true);
     }
     /** Report acknowledged completion or an execution requiring attention. */
     completeLocalAgentRun(input) {
-        return this.request("POST", "/local-agent-worker/runs/complete", input);
+        return this.request("POST", "/local-agent-worker/runs/complete", input, undefined, undefined, true);
     }
     /** Lists Scenarios (draft and published) of the project; requires a Read and write key. */
     listCaseConversions(page) {
@@ -688,7 +766,7 @@ export class EvaluationClient {
     }
     /** Creates the legacy execution-scoped generic MCP capability for one world. */
     createSimulationMcpCapability(input) {
-        return this.request("POST", "/local-agent-worker/mcp-capability", input);
+        return this.request("POST", "/local-agent-worker/mcp-capability", input, undefined, undefined, true);
     }
 }
 /**

package/dist/evals/environment-target.d.ts

@@ -105,6 +105,9 @@ export interface RunEnvironmentTargetOptions {
 export declare const MAX_GRACE_WAIT_MS = 10000;
 export declare const SEAL_POLL_MS = 250;
 export declare const SEAL_WAIT_MS = 30000;
+/** How long past the world's `expiresAt` the target is given before it is stopped: room for a
+ * last call in flight at the deadline and for clock skew between the worker and Hue. */
+export declare const DEADLINE_MARGIN_MS = 5000;
 /** The seal wait's bounds. Only tests shorten them; the helpers always use the defaults. */
 export interface SealTiming {
     maxGraceWaitMs: number;
@@ -123,16 +126,20 @@ export declare function caseTraceparent(span: {
         };
     };
 }): string;
-/** What a deployment's credential-free gateway health said: `on` (200 with
- * `gateway: "simulation"`), `off` (the empty 404 the disabled handler answers, with no
- * `x-hue-diagnostic`), or `unknown` for anything else. */
+/** What a deployment's credential-free gateway probe said: `on` (the gateway answered, with its
+ * health or with a refusal carrying `x-hue-diagnostic`), `off` (the empty 404 the disabled
+ * handler answers, with no `x-hue-diagnostic`), or `unknown` for anything else. */
 export type GatewayState = "on" | "off" | "unknown";
 /**
- * Whether the deployment serves the simulation gateway, from its credential-free health
- * endpoint: 200 with `gateway: "simulation"` is on, the disabled handler's empty 404 (no
- * `x-hue-diagnostic`) is off, and anything else (a network failure, a timeout, a redirect, a
- * refusal carrying a diagnostic, another status or body) is unknown. Probed only after a create
- * was refused; on and off are remembered per origin, unknown is probed again next time.
+ * Whether the deployment serves the simulation gateway, from a credential-free probe of the
+ * gateway's reserved health path with no app named (`/api/sim/_hue/health`), so the answer does
+ * not depend on which apps the deployment mirrors. The gateway answers it on its own host with
+ * its health, 200 with `gateway: "simulation"`, and on the application host with a refusal that
+ * carries `x-hue-diagnostic` (`unmirrored_provider`: the reserved segment is no app); either is
+ * on, since only the enabled gateway writes that header. The disabled handler's empty 404 (no
+ * `x-hue-diagnostic`) is off, and anything else (a network failure, a timeout, a redirect, another
+ * status or body) is unknown. Probed only after a create was refused; on and off are remembered
+ * per origin, unknown is probed again next time.
  */
 export declare function gatewayState(baseUrl: string, fetchImpl?: (url: string, init?: RequestInit) => Promise<Response>): Promise<GatewayState>;
 /**