@hue-run/sdk 0.2.1 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/ENVIRONMENTS.md CHANGED
@@ -1,9 +1,10 @@
1
1
  # Simulated environments
2
2
 
3
- Install the optional evaluation runtime-contract peer with the SDK:
3
+ This guide documents the TypeScript `0.3.0` release candidate. Until registry acceptance, use
4
+ the exact reviewed archive rather than requesting `0.3.0` from npm:
4
5
 
5
6
  ```bash
6
- npm install @hue-run/sdk zod
7
+ npm install /path/to/reviewed/hue-run-sdk-0.3.0.tgz zod
7
8
  ```
8
9
 
9
10
  Your agent runs in your process while a disposable simulated world runs in Hue. The world is
@@ -77,11 +78,12 @@ them to checkpoints or progress events, and it never mutates global `process.env
77
78
  world seal cannot be confirmed, the checkpoint remains uncertain and resume neither reacquires
78
79
  credentials nor invokes the callback again.
79
80
 
80
- This is currently a control-plane contract. Hue can issue provider endpoints under
81
- `/api/v1/provider-facades/{bindingId}/{grantId}`, but a provider data-plane facade call has not
82
- yet been proven by the released integration. The existing generic Hue MCP capability remains the
83
- runnable hosted-tool path; do not interpret preparation or local-tool tests as evidence of a
84
- hosted Gmail or Slack MCP call.
81
+ The SDK consumes the versioned connection contract but does not itself establish provider
82
+ fidelity. Public package acceptance exercises local control-plane responses and connection-bundle
83
+ handling; it does not call the issued facade endpoint. Exact installed-registry-package to hosted-
84
+ facade acceptance remains a post-publication Fern integration gate. No public SDK test calls the
85
+ official Gmail service, and passing these tests is not evidence of universal Gmail or Slack
86
+ parity. A matching Hue deployment and verified provider profile remain required.
85
87
 
86
88
  ## Repository-authored scenarios
87
89
 
@@ -97,6 +99,13 @@ identity-affecting defaults as Hue before resolving versions and rejects unknown
97
99
  kinds. In particular, `document_verifier` is not part of this SDK contract and is rejected rather
98
100
  than published with a guessed digest.
99
101
 
102
+ The extendable `EnvironmentDefinition` name remains the V1 contract and is also exported as
103
+ `EnvironmentDefinitionV1`. Use `EnvironmentDefinitionV2` to add immutable Gmail
104
+ `providerInstances`; `PublishableEnvironmentDefinition` is the publication/repository union.
105
+ Hue canonicalizes valid synthetic-principal UUIDs to lowercase, and repository resolution does
106
+ the same before comparing immutable digests, so casing-only UUID changes reuse the stored
107
+ version without dropping provider bindings.
108
+
100
109
  ```ts
101
110
  const scenario = {
102
111
  kind: "repository" as const,
@@ -180,3 +189,13 @@ that cannot be confirmed stays uncertain and never causes the agent to be replay
180
189
  The hosted MCP connection exposes Hue's bounded native actions; it is not general Gmail or
181
190
  Slack HTTP parity and does not proxy arbitrary provider traffic. Forking, in-place reset and
182
191
  arbitrary-step diffs are outside this interface.
192
+
193
+ ## Candidate context migration
194
+
195
+ This candidate-context restriction is part of the `@hue-run/sdk@0.3.0` release candidate.
196
+
197
+ `runSimulation` now supplies `context.item` as `{ id, externalKey }`. Read candidate inputs
198
+ from the callback's first argument. Expected outcomes, case metadata and environment-version
199
+ pins are available to evaluation and scoring code, and are omitted from the candidate callback.
200
+ Inputs and configuration are cloned before invocation so candidate mutations cannot change
201
+ pinned grading data. The generic `runExperiment` evaluator interface is unchanged.
package/EVALUATIONS.md CHANGED
@@ -9,6 +9,8 @@ npm install @hue-run/sdk zod
9
9
 
10
10
  The SDK executes targets and scorers on your machine. Hue stores pinned definitions, experiment progress and results. It does not execute uploaded source code. Follow the [installation guide](https://docs.hue.run/installation) to add `@hue-run/sdk` to your application.
11
11
 
12
+ Create a **Tracing and evaluations** project service key under **Settings → Integrations & API keys** and expose it to this server-side process as `HUE_API_KEY`. A **Tracing only** key cannot author datasets or evaluation runs.
13
+
12
14
  ```ts
13
15
  import { randomUUID } from "node:crypto";
14
16
  import { createHue } from "@hue-run/sdk";
@@ -150,4 +152,66 @@ The runner stops scheduling more cases after an operational failure and waits fo
150
152
 
151
153
  ## Verification boundaries
152
154
 
153
- `scripts/verify-package.mjs` installs a real packed tarball outside the monorepo and runs HTTP contract tests against a synthetic service plus actual OpenTelemetry exporters. It checks two configurations, rescoring without target invocation, absent/null output, upload resume, uncertain execution, exclusive checkpoints, source/metric contracts, content policy and terminating schema workers. `scripts/verify-evaluation-api.mjs` is a separate opt-in acceptance against a real Hue receiver/API; it creates synthetic datasets/scorers/experiments in the project associated with the supplied development key.
155
+ `scripts/verify-package.mjs` installs a real packed tarball outside the monorepo and runs HTTP contract tests against a synthetic service plus actual OpenTelemetry exporters. It checks two configurations, rescoring without target invocation, absent/null output, upload resume, uncertain execution, exclusive checkpoints, source/metric contracts, content policy and terminating schema workers. It also exercises the local worker's ready, incomplete and uncertain provider-attempt control-plane paths, but does not call an issued provider facade. `scripts/verify-evaluation-api.mjs` is a separate opt-in acceptance against a real Hue receiver/API; it creates synthetic datasets/scorers/experiments in the project associated with the supplied development key.
156
+
157
+ ## Outbound local agent worker
158
+
159
+ `runLocalAgent` is included in the `@hue-run/sdk@0.3.0` release candidate. Queue registration,
160
+ claims, scoped MCP capabilities and sealed evidence require a supporting Hue server and project
161
+ access; the package version alone does not establish hosted availability. Until registry
162
+ acceptance, install the exact reviewed `hue-run-sdk-0.3.0.tgz` archive with the optional `zod`
163
+ peer. `node packages/sdk-typescript/scripts/verify-package.mjs` creates and verifies that archive;
164
+ do not request `0.3.0` from npm yet.
165
+
166
+ `runLocalAgent` registers one fixed application callback and polls for queued runs. Hue selects
167
+ the registered key/revision; it does not send executable code or shell commands. Keep the
168
+ checkpoint directory private and durable. The worker persists result content and requires
169
+ acknowledged trace and sealed environment evidence.
170
+
171
+ ```ts
172
+ import { createHue } from "@hue-run/sdk";
173
+ import { createEnvironmentClient } from "@hue-run/sdk/environment";
174
+ import { createEvaluationClient, runLocalAgent } from "@hue-run/sdk/evals";
175
+ import { runMyAgent } from "./agent.js"; // Your existing application entry point.
176
+
177
+ const connection = { apiKey: process.env.HUE_API_KEY! };
178
+ const hue = createHue({ ...connection, serviceName: "local-worker", captureContent: false });
179
+ try {
180
+ await runLocalAgent({
181
+ client: createEvaluationClient(connection),
182
+ environmentClient: createEnvironmentClient(connection),
183
+ hue,
184
+ agent: { key: "support-agent", name: "Support agent", revision: "1" },
185
+ checkpointDirectory: ".hue-checkpoints/support-agent",
186
+ target: (inputs, tools, context) => runMyAgent({ inputs, tools, config: context.config }),
187
+ });
188
+ } finally {
189
+ await hue.shutdownSafe();
190
+ }
191
+ ```
192
+
193
+ The callback receives cloned inputs, local tools, and an allowlisted context containing
194
+ `config`, `item: {id, externalKey}`, `executionId`, `environmentRunId`,
195
+ `trace: {traceId,spanId}` and a short-lived `mcp` capability. Expected outcomes, case metadata
196
+ and original source pins remain private to grading. Pass the tools or scoped MCP capability into
197
+ the agent's actual tool boundary; their presence does not redirect provider calls. Capabilities
198
+ are not written to checkpoints. `maxRuns` limits completed runs for one-shot workers, while
199
+ `signal` stops polling. A stop signal does not forcibly cancel an already executing callback.
200
+
201
+ For an experiment with an immutable V2 attempt baseline, also supply `actualAgentManifest`, the
202
+ exact ordered `requestedProviders`, and an `mcpSurface` selected from that request. The worker
203
+ creates the world and prepares once before target code. A ready response exposes the memory-only
204
+ `connectionBundle` and keeps `context.mcp` as its selected MCP projection; it never mints the
205
+ legacy generic capability for that attempt. An incomplete response seals the world as completed
206
+ without invoking the target or scorers. A lost preparation acknowledgement remains uncertain
207
+ and is never recovered through binding reads, credential refresh or target replay. Synthetic
208
+ acceptance does not contact official Gmail or claim universal provider parity.
209
+
210
+ Completion or result-upload failures keep the run claimed by the durable worker identity.
211
+ Restart with the same checkpoint directory to resume saved uploads without invoking the
212
+ candidate again. A lost world-seal acknowledgement is recovered by reading authoritative world
213
+ state. If the seal or candidate outcome cannot be confirmed, or an outcome cannot be serialized,
214
+ the worker reports `attention` and stops; operator investigation is required. Such runs are not
215
+ automatically reclaimed, and presenting the same uncertain checkpoint again cannot replay the
216
+ candidate. Public package acceptance proves this lifecycle against local fixtures; exact
217
+ installed-registry-package to hosted-facade acceptance remains a post-publication Fern gate.
package/README.md CHANGED
@@ -395,6 +395,20 @@ published. The chatbot README describes running that external installation.
395
395
 
396
396
  The optional `@hue-run/sdk/evals` entry point supports dataset/scorer registration, frozen-version experiments, local built-in/custom scoring, upload resume, and historical rescoring. See the [evaluation guide](https://docs.hue.run/evaluations/first-evaluation) for the complete journey, content policy and checkpoint recovery contract.
397
397
 
398
+ ### App-launched local workers
399
+
400
+ The TypeScript `0.3.0` release candidate adds `runLocalAgent()` for a fixed local callback that
401
+ claims app-launched simulation work while keeping the agent, provider orchestration and debugger
402
+ in the developer's process. Until registry acceptance, test it only from the exact reviewed
403
+ `hue-run-sdk-0.3.0.tgz` archive rather than requesting `0.3.0` from npm. It shares
404
+ `runSimulation()`'s provider-aware world lifecycle, keeps scoped credentials in callback memory,
405
+ skips target/scorer execution for incomplete environments, and never reacquires or replays after
406
+ an uncertain preparation. See the
407
+ [outbound worker contract](EVALUATIONS.md#outbound-local-agent-worker) and
408
+ [simulated environment guide](ENVIRONMENTS.md). Tests exercise local control-plane fixtures, not
409
+ an issued facade endpoint or the official Gmail service, and do not claim universal provider
410
+ parity.
411
+
398
412
  ## Managed targets
399
413
 
400
414
  Start a frozen dataset run in Hue while your agent stays in your application. Expose a
@@ -1,4 +1,4 @@
1
- import type { ActionInput, ActionResult, CoverageGapInput, CoverageGapResult, CreateRunInput, Environment, EnvironmentDefinition, EnvironmentIdentity, EnvironmentPage, EnvironmentPageOptions, EnvironmentRun, EnvironmentSummary, EnvironmentVersion, EnvironmentVersionSummary, FinishRunInput, SealedRun, StepPage, StepPageOptions } from "./types.js";
1
+ import type { ActionInput, ActionResult, CoverageGapInput, CoverageGapResult, CreateRunInput, Environment, PublishableEnvironmentDefinition, EnvironmentIdentity, EnvironmentPage, EnvironmentPageOptions, EnvironmentRun, EnvironmentSummary, EnvironmentVersion, EnvironmentVersionSummary, FinishRunInput, SealedRun, StepPage, StepPageOptions } from "./types.js";
2
2
  /** Connection and retry options for {@link createEnvironmentClient}. */
3
3
  export interface EnvironmentClientOptions {
4
4
  /** Project service key sent as a bearer token; server-side only. */
@@ -37,7 +37,7 @@ export declare class EnvironmentClient {
37
37
  /** Reads one environment and its immutable version summaries. */
38
38
  getEnvironment(id: string): Promise<Environment>;
39
39
  /** Publishes an immutable definition; this non-idempotent registry write is not retried. */
40
- publishVersion(environmentId: string, definition: EnvironmentDefinition): Promise<EnvironmentVersionSummary>;
40
+ publishVersion(environmentId: string, definition: PublishableEnvironmentDefinition): Promise<EnvironmentVersionSummary>;
41
41
  /** Reads a full immutable environment version and generated action catalog. */
42
42
  getVersion(id: string): Promise<EnvironmentVersion>;
43
43
  /** Creates or recovers one fresh isolated world using a stable idempotency key. */
@@ -170,12 +170,44 @@ export interface EnvironmentDefinition {
170
170
  /** Caller-owned immutable metadata. */
171
171
  metadata?: Record<string, JsonValue>;
172
172
  }
173
+ /** The extendable legacy name remains V1. Publication and runs select their
174
+ * explicit version; provider context is validated by the authoritative server. */
175
+ export type EnvironmentDefinitionV1 = EnvironmentDefinition;
176
+ /** One synthetic Gmail principal and its world-state collection bindings. */
177
+ export interface GmailProviderInstance {
178
+ /** Stable instance key referenced by attempt provider selection. */
179
+ providerInstanceKey: string;
180
+ /** Provider discriminator for the V2 Gmail slice. */
181
+ providerId: "google.gmail";
182
+ /** Synthetic principal UUID, canonicalized to lowercase by Hue. */
183
+ syntheticPrincipalId: string;
184
+ /** Versioned mapping from Gmail concepts to authored-world collections. */
185
+ configuration: {
186
+ /** Gmail mailbox configuration discriminator. */
187
+ kind: "gmail_mailbox/v1";
188
+ /** Collection containing synthetic messages. */
189
+ messagesCollection: string;
190
+ /** Collection containing synthetic drafts. */
191
+ draftsCollection: string;
192
+ /** Synthetic mailbox address. */
193
+ mailboxAddress: string;
194
+ };
195
+ }
196
+ /** V2 authored world with immutable provider-instance bindings. */
197
+ export interface EnvironmentDefinitionV2 extends Omit<EnvironmentDefinition, "schemaVersion"> {
198
+ /** Definition schema discriminator. */
199
+ schemaVersion: 2;
200
+ /** Provider instances available to a strict attempt profile. */
201
+ providerInstances: GmailProviderInstance[];
202
+ }
203
+ /** Definition accepted by immutable environment publication. */
204
+ export type PublishableEnvironmentDefinition = EnvironmentDefinitionV1 | EnvironmentDefinitionV2;
173
205
  /** Full immutable environment version and its generated action catalog. */
174
206
  export interface EnvironmentVersion extends EnvironmentVersionSummary {
175
207
  /** Owning environment identity. */
176
208
  environmentId: string;
177
209
  /** Stored, defaulted authored definition. */
178
- definition: EnvironmentDefinition;
210
+ definition: PublishableEnvironmentDefinition;
179
211
  /** Generated agent-visible actions. */
180
212
  actions: ActionDefinition[];
181
213
  }
@@ -1,6 +1,6 @@
1
1
  import type { ProjectConnection } from "../types.js";
2
2
  import { type AttemptConnectionBundleV2, type PrepareAttemptRequestV2 } from "./attempt.js";
3
- import type { CaseWrite, CompleteExecution, Completion, Dataset, DatasetCase, DatasetVersion, EvaluationItem, EvaluationRun, EnvironmentEvidenceSnapshot, Execution, Experiment, ExperimentCase, ExperimentItem, Identity, JsonValue, JudgeBudget, JudgeJob, Page, PageOptions, RegistryPageOptions, Result, ResultSummary, Scorer, ScorerDefinition, ScorerVersion, SimulationMcpCapability, StartExecution, Subject, StoredResult } from "./types.js";
3
+ import type { CaseWrite, CompleteExecution, Completion, Dataset, DatasetCase, DatasetVersion, EvaluationItem, EvaluationRun, EnvironmentEvidenceSnapshot, Execution, Experiment, ExperimentCase, ExperimentItem, Identity, JsonValue, LocalAgentClaim, LocalAgentRegistration, RegisteredLocalAgent, JudgeBudget, JudgeJob, Page, PageOptions, RegistryPageOptions, Result, ResultSummary, Scorer, ScorerDefinition, ScorerVersion, SimulationMcpCapability, StartExecution, Subject, StoredResult } from "./types.js";
4
4
  /** Connection options for {@link createEvaluationClient}. */
5
5
  export interface EvaluationClientOptions {
6
6
  /** Project service key sent as a Bearer token; server side only. */
@@ -176,6 +176,35 @@ export declare class EvaluationClient {
176
176
  }>;
177
177
  /** Reads the project's hosted judge budget and admission controls. */
178
178
  getJudgeBudget(): Promise<JudgeBudget>;
179
+ /** Register or refresh the fixed local agent key and revision. */
180
+ registerLocalAgent(input: LocalAgentRegistration): Promise<RegisteredLocalAgent>;
181
+ /** Claim a queued run for this agent and durable worker identity. */
182
+ claimLocalAgentRun(input: {
183
+ agentId: string;
184
+ workerId: string;
185
+ }): Promise<LocalAgentClaim | null>;
186
+ /** Refresh the lease of a claimed local run. */
187
+ heartbeatLocalAgentRun(input: {
188
+ runId: string;
189
+ workerId: string;
190
+ }): Promise<{
191
+ /** Queue-run identity. */
192
+ runId: string;
193
+ /** The worker claim remains active. */
194
+ active: true;
195
+ }>;
196
+ /** Report acknowledged completion or an execution requiring attention. */
197
+ completeLocalAgentRun(input: {
198
+ runId: string;
199
+ workerId: string;
200
+ state: "completed" | "attention";
201
+ failureType?: string;
202
+ }): Promise<{
203
+ /** Queue-run identity. */
204
+ runId: string;
205
+ /** Acknowledged terminal queue state. */
206
+ state: "completed" | "attention";
207
+ }>;
179
208
  /** Creates the legacy execution-scoped generic MCP capability for one world. */
180
209
  createSimulationMcpCapability(input: {
181
210
  runId: string;
@@ -309,6 +309,22 @@ export class EvaluationClient {
309
309
  getJudgeBudget() {
310
310
  return this.request("GET", "/judge-budget");
311
311
  }
312
+ /** Register or refresh the fixed local agent key and revision. */
313
+ registerLocalAgent(input) {
314
+ return this.request("POST", "/local-agent-worker/register", input);
315
+ }
316
+ /** Claim a queued run for this agent and durable worker identity. */
317
+ claimLocalAgentRun(input) {
318
+ return this.request("POST", "/local-agent-worker/claim", input);
319
+ }
320
+ /** Refresh the lease of a claimed local run. */
321
+ heartbeatLocalAgentRun(input) {
322
+ return this.request("POST", "/local-agent-worker/runs/heartbeat", input);
323
+ }
324
+ /** Report acknowledged completion or an execution requiring attention. */
325
+ completeLocalAgentRun(input) {
326
+ return this.request("POST", "/local-agent-worker/runs/complete", input);
327
+ }
312
328
  /** Creates the legacy execution-scoped generic MCP capability for one world. */
313
329
  createSimulationMcpCapability(input) {
314
330
  return this.request("POST", "/local-agent-worker/mcp-capability", input);
@@ -0,0 +1,84 @@
1
+ import type { HueClient } from "../client.js";
2
+ import type { EnvironmentClient } from "../environment/client.js";
3
+ import { type EnvironmentTool } from "../environment/tools.js";
4
+ import type { HueSpan } from "../types.js";
5
+ import { type ActualAgentManifestInputV2, type AttemptBaselineV2, type AttemptConnectionBundleV2, type RequestedAttemptProviderV2, type SurfaceBindingV2 } from "./attempt.js";
6
+ import type { EvaluationClient } from "./client.js";
7
+ import type { ExperimentCase, JsonValue, SimulationMcpCapability } from "./types.js";
8
+ export type McpSurfaceKeyV2 = Extract<SurfaceBindingV2["surfaceKey"], `${string}/mcp`>;
9
+ export type ActualAgentManifestResolverV2 = ActualAgentManifestInputV2 | ((context: {
10
+ config: JsonValue;
11
+ item: ExperimentCase;
12
+ signal?: AbortSignal;
13
+ }) => ActualAgentManifestInputV2 | Promise<ActualAgentManifestInputV2>);
14
+ export interface ProviderAttemptOptionsV2 {
15
+ actualAgentManifest?: ActualAgentManifestResolverV2;
16
+ requestedProviders?: RequestedAttemptProviderV2[];
17
+ mcpSurface?: {
18
+ providerInstanceKey: string;
19
+ surfaceKey: McpSurfaceKeyV2;
20
+ };
21
+ }
22
+ export type RequestedAttemptV2 = {
23
+ actualAgentManifest: ActualAgentManifestResolverV2;
24
+ requestedProviders: RequestedAttemptProviderV2[];
25
+ mcpSurface: {
26
+ providerInstanceKey: string;
27
+ surfaceKey: McpSurfaceKeyV2;
28
+ };
29
+ };
30
+ export type PinnedAttemptV2 = RequestedAttemptV2 & {
31
+ expectedAgentManifestDigest: AttemptBaselineV2["expectedAgentManifestDigest"];
32
+ };
33
+ export declare function requestedAttemptV2(options: ProviderAttemptOptionsV2): RequestedAttemptV2 | undefined;
34
+ export declare function pinRequestedAttemptV2(requested: RequestedAttemptV2, config: JsonValue): PinnedAttemptV2;
35
+ export type EnvironmentTargetProgress = {
36
+ type: "world_created" | "target_started" | "world_sealed";
37
+ environmentRunId: string;
38
+ } | {
39
+ type: "attempt_prepared";
40
+ environmentRunId: string;
41
+ bindingId: string;
42
+ status: "ready" | "environment_incomplete";
43
+ findingCodes: string[];
44
+ executionManifestDigest?: AttemptConnectionBundleV2["parity"]["executionManifestDigest"];
45
+ };
46
+ export interface EnvironmentTargetContext {
47
+ config: JsonValue;
48
+ item: ExperimentCase;
49
+ executionId: string;
50
+ environmentRunId: string;
51
+ trace: {
52
+ traceId: string;
53
+ spanId: string;
54
+ };
55
+ tools: Record<string, EnvironmentTool>;
56
+ mcp: SimulationMcpCapability;
57
+ connectionBundle?: AttemptConnectionBundleV2;
58
+ signal?: AbortSignal;
59
+ }
60
+ interface RunnerTargetContext {
61
+ config: JsonValue;
62
+ item: ExperimentCase;
63
+ executionId: string;
64
+ span: HueSpan;
65
+ }
66
+ export interface RunEnvironmentTargetOptions {
67
+ client: EvaluationClient;
68
+ environmentClient: EnvironmentClient;
69
+ hue: HueClient;
70
+ inputs: JsonValue;
71
+ context: RunnerTargetContext;
72
+ requested?: PinnedAttemptV2;
73
+ maxSteps?: number;
74
+ ttlSeconds?: number;
75
+ signal?: AbortSignal;
76
+ onProgress?(event: EnvironmentTargetProgress): void | Promise<void>;
77
+ target(inputs: JsonValue, context: EnvironmentTargetContext): JsonValue | undefined | Promise<JsonValue | undefined>;
78
+ }
79
+ /** One authoritative environment/provider lifecycle shared by direct simulations and
80
+ * outbound local workers. Credential-bearing connections stay in this call frame and
81
+ * are never returned to either runner's checkpoint state.
82
+ */
83
+ export declare function runEnvironmentTarget(options: RunEnvironmentTargetOptions): Promise<JsonValue | undefined>;
84
+ export {};
@@ -0,0 +1,201 @@
1
+ import { randomUUID } from "node:crypto";
2
+ import { bindEnvironmentTools } from "../environment/tools.js";
3
+ import { actualAgentManifestV2, attemptBaselineV2, projectMcpConnectionV2, requestedAttemptProvidersV2, validateAttemptConnectionBundleV2, } from "./attempt.js";
4
+ import { TargetCancelledError, TargetOutcomeUncertainError } from "./runner.js";
5
+ export function requestedAttemptV2(options) {
6
+ const requested = options.requestedProviders !== undefined;
7
+ const selected = options.mcpSurface !== undefined;
8
+ if (!requested && !selected) {
9
+ if (options.actualAgentManifest !== undefined)
10
+ throw new TypeError("actualAgentManifest requires requestedProviders and mcpSurface");
11
+ return undefined;
12
+ }
13
+ if (!requested || !selected)
14
+ throw new TypeError("requestedProviders and mcpSurface must be supplied together");
15
+ const requestedProviders = requestedAttemptProvidersV2.parse(options.requestedProviders);
16
+ const mcpSurface = options.mcpSurface;
17
+ const provider = requestedProviders.find((candidate) => candidate.providerInstanceKey === mcpSurface.providerInstanceKey);
18
+ if (!provider?.surfaceKeys.includes(mcpSurface.surfaceKey))
19
+ throw new TypeError("mcpSurface must identify an exactly requested MCP surface");
20
+ const actualAgentManifest = typeof options.actualAgentManifest === "function"
21
+ ? options.actualAgentManifest
22
+ : actualAgentManifestV2.parse(options.actualAgentManifest);
23
+ return {
24
+ actualAgentManifest,
25
+ requestedProviders,
26
+ mcpSurface: { ...mcpSurface },
27
+ };
28
+ }
29
+ export function pinRequestedAttemptV2(requested, config) {
30
+ if (!config || typeof config !== "object" || Array.isArray(config))
31
+ throw new TypeError("Provider-profile simulations require an immutable V2 attempt baseline");
32
+ const source = config;
33
+ const baseline = attemptBaselineV2.safeParse(source.attemptBaselineV2);
34
+ if (!baseline.success) {
35
+ if (source.attemptBaselineV2 === undefined && source.attemptBaselineV1 !== undefined)
36
+ throw new TypeError("Legacy V1 attempts require a fresh experiment with a V2 baseline");
37
+ if (source.attemptBaselineV2 !== undefined)
38
+ throw new TypeError("The immutable V2 attempt baseline is invalid");
39
+ throw new TypeError("Provider-profile simulations require an immutable V2 attempt baseline");
40
+ }
41
+ return { ...requested, expectedAgentManifestDigest: baseline.data.expectedAgentManifestDigest };
42
+ }
43
+ /** Confirm a seal from the authoritative run after a lost acknowledgement. */
44
+ async function seal(client, runId, executionId, status) {
45
+ try {
46
+ await client.finishRun(runId, {
47
+ idempotencyKey: `execution:${executionId}:${status}`,
48
+ status,
49
+ });
50
+ }
51
+ catch (error) {
52
+ const recovered = await client.getRun(runId).catch(() => undefined);
53
+ if (recovered?.status !== status)
54
+ throw new TargetOutcomeUncertainError(executionId, { cause: error });
55
+ }
56
+ }
57
+ /** One authoritative environment/provider lifecycle shared by direct simulations and
58
+ * outbound local workers. Credential-bearing connections stay in this call frame and
59
+ * are never returned to either runner's checkpoint state.
60
+ */
61
+ export async function runEnvironmentTarget(options) {
62
+ const { context } = options;
63
+ const environmentVersionId = context.item.environmentVersionId;
64
+ if (!environmentVersionId)
65
+ throw new Error("The simulation case has no pinned environment version");
66
+ const run = await options.environmentClient.createRun({
67
+ idempotencyKey: `execution:${context.executionId}`,
68
+ environmentVersionId,
69
+ executionId: context.executionId,
70
+ maxSteps: options.maxSteps,
71
+ ttlSeconds: options.ttlSeconds,
72
+ });
73
+ const progress = (event) => options.onProgress?.(event);
74
+ let finalized = false;
75
+ try {
76
+ await progress({ type: "world_created", environmentRunId: run.id });
77
+ if (options.signal?.aborted)
78
+ throw new TargetCancelledError();
79
+ const tools = bindEnvironmentTools({
80
+ hue: options.hue,
81
+ client: options.environmentClient,
82
+ run,
83
+ parentContext: context.span.context,
84
+ });
85
+ let connectionBundle;
86
+ let mcp;
87
+ if (options.requested) {
88
+ const actualManifest = actualAgentManifestV2.parse(typeof options.requested.actualAgentManifest === "function"
89
+ ? await options.requested.actualAgentManifest({
90
+ config: structuredClone(context.config),
91
+ item: structuredClone(context.item),
92
+ signal: options.signal,
93
+ })
94
+ : options.requested.actualAgentManifest);
95
+ let prepared;
96
+ try {
97
+ prepared = await options.client.prepareAttempt({
98
+ schemaVersion: 2,
99
+ idempotencyKey: randomUUID(),
100
+ executionId: context.executionId,
101
+ environmentRunId: run.id,
102
+ expectedAgentManifestDigest: options.requested.expectedAgentManifestDigest,
103
+ actualManifest,
104
+ requestedProviders: options.requested.requestedProviders,
105
+ });
106
+ }
107
+ catch (error) {
108
+ // A transport failure or malformed credential-bearing response may follow a
109
+ // committed decision. The runner's running checkpoint prevents reacquisition
110
+ // and target replay on resume.
111
+ throw new TargetOutcomeUncertainError(context.executionId, { cause: error });
112
+ }
113
+ await progress({
114
+ type: "attempt_prepared",
115
+ environmentRunId: run.id,
116
+ bindingId: prepared.status === "ready" ? prepared.bundle.bindingId : prepared.bindingId,
117
+ status: prepared.status,
118
+ findingCodes: prepared.preflightReport.findings.map((finding) => finding.code),
119
+ ...(prepared.status === "ready"
120
+ ? { executionManifestDigest: prepared.bundle.parity.executionManifestDigest }
121
+ : {}),
122
+ });
123
+ if (prepared.status === "environment_incomplete") {
124
+ await seal(options.environmentClient, run.id, context.executionId, "completed");
125
+ finalized = true;
126
+ await Promise.resolve(progress({ type: "world_sealed", environmentRunId: run.id })).catch(() => undefined);
127
+ return undefined;
128
+ }
129
+ connectionBundle = validateAttemptConnectionBundleV2(prepared.bundle, {
130
+ requireFresh: true,
131
+ });
132
+ const projected = projectMcpConnectionV2(connectionBundle, options.requested.mcpSurface.providerInstanceKey);
133
+ if (!projected)
134
+ throw new TypeError("The prepared attempt has no selected MCP surface");
135
+ mcp = projected;
136
+ }
137
+ else {
138
+ mcp = await options.client.createSimulationMcpCapability({
139
+ runId: run.id,
140
+ executionId: context.executionId,
141
+ });
142
+ }
143
+ if (options.signal?.aborted)
144
+ throw new TargetCancelledError();
145
+ await progress({ type: "target_started", environmentRunId: run.id });
146
+ const output = await options.target(options.inputs, {
147
+ config: context.config,
148
+ item: context.item,
149
+ executionId: context.executionId,
150
+ environmentRunId: run.id,
151
+ trace: { traceId: context.span.traceId, spanId: context.span.spanId },
152
+ tools,
153
+ mcp,
154
+ ...(connectionBundle ? { connectionBundle } : {}),
155
+ signal: options.signal,
156
+ });
157
+ await seal(options.environmentClient, run.id, context.executionId, "completed");
158
+ finalized = true;
159
+ await Promise.resolve(progress({ type: "world_sealed", environmentRunId: run.id })).catch(() => undefined);
160
+ return output;
161
+ }
162
+ catch (error) {
163
+ if (error instanceof TargetOutcomeUncertainError || finalized)
164
+ throw error;
165
+ let environmentIncomplete;
166
+ try {
167
+ environmentIncomplete =
168
+ (await options.environmentClient.getRun(run.id)).validity === "environment_incomplete";
169
+ }
170
+ catch (inspectionError) {
171
+ throw new TargetOutcomeUncertainError(context.executionId, {
172
+ cause: new AggregateError([error, inspectionError]),
173
+ });
174
+ }
175
+ if (environmentIncomplete) {
176
+ try {
177
+ await seal(options.environmentClient, run.id, context.executionId, "completed");
178
+ }
179
+ catch (finalizationError) {
180
+ throw new TargetOutcomeUncertainError(context.executionId, {
181
+ cause: new AggregateError([error, finalizationError]),
182
+ });
183
+ }
184
+ finalized = true;
185
+ await Promise.resolve(progress({ type: "world_sealed", environmentRunId: run.id })).catch(() => undefined);
186
+ return undefined;
187
+ }
188
+ try {
189
+ await seal(options.environmentClient, run.id, context.executionId, "abandoned");
190
+ }
191
+ catch (finalizationError) {
192
+ throw new TargetOutcomeUncertainError(context.executionId, {
193
+ cause: new AggregateError([error, finalizationError]),
194
+ });
195
+ }
196
+ await Promise.resolve(progress({ type: "world_sealed", environmentRunId: run.id })).catch(() => undefined);
197
+ if (options.signal?.aborted && !(error instanceof TargetCancelledError))
198
+ throw new TargetCancelledError();
199
+ throw error;
200
+ }
201
+ }
@@ -0,0 +1,88 @@
1
+ import type { HueClient } from "../client.js";
2
+ import type { EnvironmentClient } from "../environment/client.js";
3
+ import type { EnvironmentTool } from "../environment/tools.js";
4
+ import type { ActualAgentManifestInputV2, AttemptConnectionBundleV2, RequestedAttemptProviderV2 } from "./attempt.js";
5
+ import type { EvaluationClient } from "./client.js";
6
+ import { type RunnerReport } from "./runner.js";
7
+ import type { ExperimentCase, JsonValue, LocalAgentRegistration, LocalScorer } from "./types.js";
8
+ /** Candidate-visible context for one queued local agent execution. */
9
+ export interface LocalAgentTargetContext {
10
+ /** Frozen candidate configuration, cloned before invocation. */
11
+ config: JsonValue;
12
+ /** Identity only. Expected outcomes, metadata and world definitions are evaluator-private. */
13
+ item: Pick<ExperimentCase, "id" | "externalKey">;
14
+ /** Identity of this target execution. */
15
+ executionId: string;
16
+ /** Stable world identity for adapter control operations such as coverage reporting. */
17
+ environmentRunId: string;
18
+ /** Trace identities without mutable span or grading data. */
19
+ trace: {
20
+ /** OpenTelemetry trace identifier. */
21
+ traceId: string;
22
+ /** Root execution span identifier. */
23
+ spanId: string;
24
+ };
25
+ /** Short-lived, execution-scoped hosted tools for model providers that execute MCP remotely. */
26
+ mcp: {
27
+ /** Execution-scoped MCP endpoint. */
28
+ url: string;
29
+ /** Short-lived bearer, never the project service key. */
30
+ token: string;
31
+ /** Capability expiry as an ISO timestamp. */
32
+ expiresAt: string;
33
+ };
34
+ /** Credential-bearing provider connections for this callback only. Hue never
35
+ * checkpoints, logs or adds this response to parity digests. */
36
+ connectionBundle?: AttemptConnectionBundleV2;
37
+ }
38
+ /** Fixed local callback, clients and durable queue-worker settings. */
39
+ export interface RunLocalAgentOptions {
40
+ /** Evaluation client for the worker project. */
41
+ client: EvaluationClient;
42
+ /** Environment client for the same origin and project. */
43
+ environmentClient: EnvironmentClient;
44
+ /** Application telemetry client; required trace exports are acknowledged. */
45
+ hue: HueClient;
46
+ /** Private durable directory for worker identity and execution checkpoints. */
47
+ checkpointDirectory: string;
48
+ /** Fixed agent key and revision registered for queued runs. */
49
+ agent: LocalAgentRegistration;
50
+ /** Local scorer bindings matching the published source digests. */
51
+ scorers?: LocalScorer[];
52
+ /** Cases in flight, between 1 and 16; defaults to 1. */
53
+ concurrency?: number;
54
+ /** Polling interval in milliseconds, 250–60000; defaults to 2000. */
55
+ pollIntervalMillis?: number;
56
+ /** Stops polling cooperatively; does not cancel an active callback. */
57
+ signal?: AbortSignal;
58
+ /** Useful for one-shot jobs and deterministic acceptance. Omit to keep polling. */
59
+ maxRuns?: number;
60
+ /** Opt into the experiment's immutable V2 provider profile. These three values are
61
+ * validated together before the worker polls; no endpoint or credential is supplied here. */
62
+ actualAgentManifest?: ActualAgentManifestInputV2 | ((context: {
63
+ /** Frozen experiment configuration. */
64
+ config: JsonValue;
65
+ /** Full frozen case for resolving actual nonsecret evidence before candidate projection. */
66
+ item: ExperimentCase;
67
+ /** Cooperative worker stop signal. */
68
+ signal?: AbortSignal;
69
+ }) => ActualAgentManifestInputV2 | Promise<ActualAgentManifestInputV2>);
70
+ /** Exact provider instances and ordered surfaces asserted for strict preflight. */
71
+ requestedProviders?: RequestedAttemptProviderV2[];
72
+ /** Requested MCP surface projected to the backwards-compatible `context.mcp`. */
73
+ mcpSurface?: {
74
+ /** Provider instance selected from `requestedProviders`. */
75
+ providerInstanceKey: string;
76
+ /** Selected MCP surface. */
77
+ surfaceKey: "google.gmail/mcp" | "slack/mcp";
78
+ };
79
+ /** Invokes the existing agent against isolated tools and candidate-safe context. */
80
+ target(inputs: JsonValue, tools: Record<string, EnvironmentTool>, context: LocalAgentTargetContext): JsonValue | undefined | Promise<JsonValue | undefined>;
81
+ /** Called after the experiment and queue completion are acknowledged. */
82
+ onCompleted?(report: RunnerReport): void | Promise<void>;
83
+ }
84
+ /**
85
+ * Starts an outbound-only worker for one fixed local agent entry point. Hue chooses
86
+ * only the registered key/revision; no command or source is received from the cloud.
87
+ */
88
+ export declare function runLocalAgent(options: RunLocalAgentOptions): Promise<void>;
@@ -0,0 +1,171 @@
1
+ import { join, resolve } from "node:path";
2
+ import { randomUUID } from "node:crypto";
3
+ import { CheckpointStore } from "./checkpoint.js";
4
+ import { pinRequestedAttemptV2, requestedAttemptV2, runEnvironmentTarget, } from "./environment-target.js";
5
+ import { runExperiment, OutcomeSerializationError, TargetOutcomeUncertainError, UncertainExecutionError, } from "./runner.js";
6
+ /** Allowlist the candidate surface instead of forwarding the generic evaluation context. */
7
+ function localAgentTargetContext(context) {
8
+ return {
9
+ config: structuredClone(context.config),
10
+ item: { id: context.item.id, externalKey: context.item.externalKey },
11
+ executionId: context.executionId,
12
+ environmentRunId: context.environmentRunId,
13
+ trace: { traceId: context.trace.traceId, spanId: context.trace.spanId },
14
+ mcp: {
15
+ url: context.mcp.url,
16
+ token: context.mcp.token,
17
+ expiresAt: context.mcp.expiresAt,
18
+ },
19
+ ...(context.connectionBundle
20
+ ? { connectionBundle: structuredClone(context.connectionBundle) }
21
+ : {}),
22
+ };
23
+ }
24
+ function validInterval(value) {
25
+ const interval = value ?? 2_000;
26
+ if (!Number.isInteger(interval) || interval < 250 || interval > 60_000)
27
+ throw new RangeError("pollIntervalMillis must be 250–60000");
28
+ return interval;
29
+ }
30
+ function stopReason(signal) {
31
+ return signal?.reason instanceof Error
32
+ ? signal.reason
33
+ : new Error("Local worker stopped", { cause: signal?.reason });
34
+ }
35
+ function wait(milliseconds, signal) {
36
+ if (signal?.aborted)
37
+ return Promise.reject(stopReason(signal));
38
+ return new Promise((resolve, reject) => {
39
+ const timeout = setTimeout(done, milliseconds);
40
+ const aborted = () => {
41
+ clearTimeout(timeout);
42
+ reject(stopReason(signal));
43
+ };
44
+ function done() {
45
+ signal?.removeEventListener("abort", aborted);
46
+ resolve();
47
+ }
48
+ signal?.addEventListener("abort", aborted, { once: true });
49
+ });
50
+ }
51
+ function needsAttention(error, seen = new Set()) {
52
+ if (error instanceof TargetOutcomeUncertainError ||
53
+ error instanceof UncertainExecutionError ||
54
+ error instanceof OutcomeSerializationError)
55
+ return true;
56
+ if (!(error instanceof AggregateError) || seen.has(error))
57
+ return false;
58
+ seen.add(error);
59
+ return error.errors.some((nested) => needsAttention(nested, seen));
60
+ }
61
+ /**
62
+ * Starts an outbound-only worker for one fixed local agent entry point. Hue chooses
63
+ * only the registered key/revision; no command or source is received from the cloud.
64
+ */
65
+ export async function runLocalAgent(options) {
66
+ const requestedConfiguration = requestedAttemptV2(options);
67
+ const interval = validInterval(options.pollIntervalMillis);
68
+ const maxRuns = options.maxRuns ?? Number.POSITIVE_INFINITY;
69
+ if (!(maxRuns === Number.POSITIVE_INFINITY || (Number.isInteger(maxRuns) && maxRuns > 0)))
70
+ throw new RangeError("maxRuns must be a positive integer");
71
+ if (options.environmentClient.baseUrl !== options.client.baseUrl)
72
+ throw new Error("Environments and evaluations must use the same Hue origin");
73
+ const directory = resolve(options.checkpointDirectory);
74
+ const project = await options.client.checkConnection();
75
+ const store = await CheckpointStore.acquire(directory, {
76
+ kind: "local-agent-worker",
77
+ projectId: project.id,
78
+ baseUrl: options.client.baseUrl,
79
+ agent: { key: options.agent.key, revision: options.agent.revision },
80
+ });
81
+ try {
82
+ let workerId = await store.read("worker-id");
83
+ if (workerId !== undefined &&
84
+ !/^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/.test(workerId))
85
+ throw new Error("Invalid local worker identity");
86
+ if (workerId === undefined) {
87
+ workerId = randomUUID();
88
+ await store.write("worker-id", workerId);
89
+ }
90
+ let completed = 0;
91
+ while (completed < maxRuns && !options.signal?.aborted) {
92
+ const agent = await options.client.registerLocalAgent({
93
+ ...options.agent,
94
+ capabilities: options.agent.capabilities ?? ["environment:v1"],
95
+ scorerDigests: options.agent.scorerDigests ??
96
+ options.scorers?.map((item) => item.definition.sourceDigest) ??
97
+ [],
98
+ });
99
+ const claim = await options.client.claimLocalAgentRun({ agentId: agent.id, workerId });
100
+ if (!claim) {
101
+ await wait(interval, options.signal);
102
+ continue;
103
+ }
104
+ const heartbeat = setInterval(() => {
105
+ void options.client
106
+ .heartbeatLocalAgentRun({ runId: claim.runId, workerId })
107
+ .catch(() => undefined);
108
+ }, Math.min(15_000, Math.max(1_000, interval)));
109
+ let experimentFinished = false;
110
+ try {
111
+ const requested = requestedConfiguration
112
+ ? pinRequestedAttemptV2(requestedConfiguration, (await options.client.getExperiment(claim.experimentId)).config)
113
+ : undefined;
114
+ const report = await runExperiment({
115
+ client: options.client,
116
+ hue: options.hue,
117
+ experimentId: claim.experimentId,
118
+ checkpointDirectory: join(directory, `experiment-${claim.experimentId}`),
119
+ persistResultContent: true,
120
+ environmentEvidence: "required",
121
+ traceEvidence: { mode: "required" },
122
+ scorers: options.scorers,
123
+ concurrency: options.concurrency,
124
+ target: (inputs, context) => runEnvironmentTarget({
125
+ client: options.client,
126
+ environmentClient: options.environmentClient,
127
+ hue: options.hue,
128
+ inputs,
129
+ context,
130
+ requested,
131
+ signal: options.signal,
132
+ target: (targetInputs, targetContext) => options.target(structuredClone(targetInputs), targetContext.tools, localAgentTargetContext(targetContext)),
133
+ }),
134
+ });
135
+ // From this point onward the experiment outcome is authoritative. If reporting the
136
+ // queue completion fails, leave the claim intact for checkpointed recovery instead of
137
+ // rewriting a successful experiment as an agent failure.
138
+ experimentFinished = true;
139
+ await options.client.completeLocalAgentRun({
140
+ runId: claim.runId,
141
+ workerId,
142
+ state: "completed",
143
+ });
144
+ await options.onCompleted?.(report);
145
+ completed++;
146
+ }
147
+ catch (error) {
148
+ // A durable outcome can still need completion/result uploads. Leave operational
149
+ // failures claimed so the same worker can resume them through its checkpoints.
150
+ // Attention is terminal in the queue and is reserved for explicit unsafe-to-resume
151
+ // outcomes that require operator intervention.
152
+ if (!experimentFinished && needsAttention(error))
153
+ await options.client
154
+ .completeLocalAgentRun({
155
+ runId: claim.runId,
156
+ workerId,
157
+ state: "attention",
158
+ failureType: error instanceof Error ? error.name.slice(0, 200) : "WorkerError",
159
+ })
160
+ .catch(() => undefined);
161
+ throw error;
162
+ }
163
+ finally {
164
+ clearInterval(heartbeat);
165
+ }
166
+ }
167
+ }
168
+ finally {
169
+ await store.release();
170
+ }
171
+ }
@@ -1,9 +1,9 @@
1
1
  import type { HueClient } from "../client.js";
2
2
  import { type EnvironmentClient } from "../environment/client.js";
3
- import { type EnvironmentTool } from "../environment/tools.js";
4
- import type { EnvironmentDefinition, EnvironmentIdentity } from "../environment/types.js";
3
+ import type { EnvironmentTool } from "../environment/tools.js";
4
+ import type { EnvironmentIdentity, PublishableEnvironmentDefinition } from "../environment/types.js";
5
5
  import { type EvaluationClient } from "./client.js";
6
- import { type ActualAgentManifestInputV2, type AttemptConnectionBundleV2, type RequestedAttemptProviderV2 } from "./attempt.js";
6
+ import type { ActualAgentManifestInputV2, AttemptConnectionBundleV2, RequestedAttemptProviderV2 } from "./attempt.js";
7
7
  import { type RunnerReport } from "./runner.js";
8
8
  import type { ExperimentCase, Identity, JsonValue, LocalScorer, ScorerDefinition, SimulationMcpCapability } from "./types.js";
9
9
  /** Repository-authored scorer identity and its public definition or local binding. */
@@ -40,7 +40,7 @@ export type SimulationScenario = {
40
40
  /** Environment identity and authored world definition. */
41
41
  environment: EnvironmentIdentity & {
42
42
  /** Definition normalized and published as an immutable version. */
43
- definition: EnvironmentDefinition;
43
+ definition: PublishableEnvironmentDefinition;
44
44
  };
45
45
  /** Cases published into one frozen dataset version. */
46
46
  cases: RepositorySimulationCase[];
@@ -92,8 +92,8 @@ export type SimulationProgress = {
92
92
  export interface SimulationTargetContext {
93
93
  /** Frozen experiment configuration. */
94
94
  config: JsonValue;
95
- /** Frozen case including its immutable environment-version pin. */
96
- item: ExperimentCase;
95
+ /** Candidate-visible identity. References, metadata and source pins stay with grading. */
96
+ item: Pick<ExperimentCase, "id" | "externalKey">;
97
97
  /** Target execution identity. */
98
98
  executionId: string;
99
99
  /** Stable world identity for adapter control operations such as coverage reporting. */
@@ -1,56 +1,25 @@
1
1
  import { randomUUID } from "node:crypto";
2
2
  import { join } from "node:path";
3
3
  import { HueEnvironmentError } from "../environment/client.js";
4
- import { bindEnvironmentTools } from "../environment/tools.js";
5
4
  import { HueApiError } from "./client.js";
6
5
  import { CheckpointStore } from "./checkpoint.js";
7
6
  import { MAX_ENVIRONMENT_STEPS } from "./environment-evidence.js";
8
7
  import { aggregateBounds, digest, json } from "./json.js";
9
8
  import { normalizeScorerDefinitionForPublication } from "./scorer-publication.js";
10
- import { actualAgentManifestV2, attemptBaselineV2, projectMcpConnectionV2, requestedAttemptProvidersV2, validateAttemptConnectionBundleV2, } from "./attempt.js";
11
- import { runExperiment, TargetCancelledError, TargetOutcomeUncertainError, } from "./runner.js";
9
+ import { pinRequestedAttemptV2, requestedAttemptV2, runEnvironmentTarget, } from "./environment-target.js";
10
+ import { runExperiment } from "./runner.js";
12
11
  const scorerDefinition = (entry) => "definition" in entry.scorer ? entry.scorer.definition : entry.scorer;
13
- function requestedAttempt(options) {
14
- const requested = options.requestedProviders !== undefined;
15
- const selected = options.mcpSurface !== undefined;
16
- if (!requested && !selected) {
17
- if (options.actualAgentManifest !== undefined)
18
- throw new TypeError("actualAgentManifest requires requestedProviders and mcpSurface");
19
- return undefined;
20
- }
21
- if (!requested || !selected)
22
- throw new TypeError("requestedProviders and mcpSurface must be supplied together");
23
- const requestedProviders = requestedAttemptProvidersV2.parse(options.requestedProviders);
24
- const mcpSurface = options.mcpSurface;
25
- const provider = requestedProviders.find((candidate) => candidate.providerInstanceKey === mcpSurface.providerInstanceKey);
26
- if (!provider?.surfaceKeys.includes(mcpSurface.surfaceKey))
27
- throw new TypeError("mcpSurface must identify an exactly requested MCP surface");
28
- const actualAgentManifest = typeof options.actualAgentManifest === "function"
29
- ? options.actualAgentManifest
30
- : actualAgentManifestV2.parse(options.actualAgentManifest);
31
- return {
32
- actualAgentManifest,
33
- requestedProviders,
34
- mcpSurface: { ...mcpSurface },
35
- };
36
- }
37
- function expectedManifestDigest(config) {
38
- if (!config || typeof config !== "object" || Array.isArray(config))
39
- throw new TypeError("Provider-profile simulations require an immutable V2 attempt baseline");
40
- const source = config;
41
- const baseline = attemptBaselineV2.safeParse(source.attemptBaselineV2);
42
- if (!baseline.success) {
43
- if (source.attemptBaselineV2 === undefined && source.attemptBaselineV1 !== undefined)
44
- throw new TypeError("Legacy V1 attempts require a fresh experiment with a V2 baseline");
45
- if (source.attemptBaselineV2 !== undefined)
46
- throw new TypeError("The immutable V2 attempt baseline is invalid");
47
- throw new TypeError("Provider-profile simulations require an immutable V2 attempt baseline");
48
- }
49
- return baseline.data.expectedAgentManifestDigest;
50
- }
51
12
  function normalizedEnvironmentDefinition(definition) {
52
13
  return json({
53
14
  ...definition,
15
+ ...(definition.schemaVersion === 2
16
+ ? {
17
+ providerInstances: definition.providerInstances.map((instance) => ({
18
+ ...instance,
19
+ syntheticPrincipalId: instance.syntheticPrincipalId.toLowerCase(),
20
+ })),
21
+ }
22
+ : {}),
54
23
  determinism: {
55
24
  clock: {
56
25
  startNs: definition.determinism?.clock?.startNs ?? "0",
@@ -341,24 +310,11 @@ async function resolveExperiment(options, idempotencyKey) {
341
310
  bindings: [...scorers.bindings, ...(options.localScorers ?? [])],
342
311
  };
343
312
  }
344
- async function seal(client, runId, executionId, status) {
345
- try {
346
- await client.finishRun(runId, {
347
- idempotencyKey: `execution:${executionId}:${status}`,
348
- status,
349
- });
350
- }
351
- catch (error) {
352
- const recovered = await client.getRun(runId).catch(() => undefined);
353
- if (recovered?.status !== status)
354
- throw error;
355
- }
356
- }
357
313
  /** Run an existing agent callback against one fresh hosted world per case. The helper
358
314
  * owns immutable resolution, execution linkage, finalization, scoring and resumable uploads.
359
315
  */
360
316
  export async function runSimulation(options) {
361
- const requestedConfiguration = requestedAttempt(options);
317
+ const requestedConfiguration = requestedAttemptV2(options);
362
318
  if (options.maxSteps !== undefined &&
363
319
  (!Number.isInteger(options.maxSteps) ||
364
320
  options.maxSteps < 1 ||
@@ -408,10 +364,7 @@ export async function runSimulation(options) {
408
364
  const runUrl = new URL(`/experiments/${experimentId}`, options.client.baseUrl).toString();
409
365
  await options.onProgress?.({ type: "run_created", experimentId, runUrl });
410
366
  const requested = requestedConfiguration
411
- ? {
412
- ...requestedConfiguration,
413
- expectedAgentManifestDigest: expectedManifestDigest((await options.client.getExperiment(experimentId)).config),
414
- }
367
+ ? pinRequestedAttemptV2(requestedConfiguration, (await options.client.getExperiment(experimentId)).config)
415
368
  : undefined;
416
369
  const report = await runExperiment({
417
370
  client: options.client,
@@ -424,172 +377,42 @@ export async function runSimulation(options) {
424
377
  scorers: bindings,
425
378
  concurrency: options.concurrency,
426
379
  schemaTimeoutMillis: options.schemaTimeoutMillis,
427
- target: async (inputs, context) => {
428
- const environmentVersionId = context.item.environmentVersionId;
429
- if (!environmentVersionId)
430
- throw new Error("The simulation case has no pinned environment version");
431
- const run = await options.environmentClient.createRun({
432
- idempotencyKey: `execution:${context.executionId}`,
433
- environmentVersionId,
434
- executionId: context.executionId,
435
- maxSteps: options.maxSteps,
436
- ttlSeconds: options.ttlSeconds,
437
- });
438
- const progress = (type) => options.onProgress?.({
439
- type,
380
+ target: (inputs, context) => runEnvironmentTarget({
381
+ client: options.client,
382
+ environmentClient: options.environmentClient,
383
+ hue: options.hue,
384
+ inputs,
385
+ context,
386
+ requested,
387
+ maxSteps: options.maxSteps,
388
+ ttlSeconds: options.ttlSeconds,
389
+ signal: options.signal,
390
+ onProgress: (event) => options.onProgress?.({
391
+ ...event,
440
392
  experimentId,
441
393
  executionId: context.executionId,
442
394
  caseId: context.item.id,
443
- environmentRunId: run.id,
444
- });
445
- let finalized = false;
446
- try {
447
- await progress("world_created");
448
- if (options.signal?.aborted)
449
- throw new TargetCancelledError();
450
- const tools = bindEnvironmentTools({
451
- hue: options.hue,
452
- client: options.environmentClient,
453
- run,
454
- parentContext: context.span.context,
455
- });
456
- let connectionBundle;
457
- let mcp;
458
- if (requested) {
459
- const actualManifest = actualAgentManifestV2.parse(typeof requested.actualAgentManifest === "function"
460
- ? await requested.actualAgentManifest({
461
- config: context.config,
462
- item: structuredClone(context.item),
463
- signal: options.signal,
464
- })
465
- : requested.actualAgentManifest);
466
- let prepared;
467
- try {
468
- prepared = await options.client.prepareAttempt({
469
- schemaVersion: 2,
470
- idempotencyKey: randomUUID(),
471
- executionId: context.executionId,
472
- environmentRunId: run.id,
473
- expectedAgentManifestDigest: requested.expectedAgentManifestDigest,
474
- actualManifest,
475
- requestedProviders: requested.requestedProviders,
476
- });
477
- }
478
- catch (error) {
479
- // A transport failure or malformed credential-bearing response may
480
- // follow a committed decision. Preserve the running checkpoint and
481
- // never reacquire credentials or replay the target on resume.
482
- throw new TargetOutcomeUncertainError(context.executionId, { cause: error });
483
- }
484
- await options.onProgress?.({
485
- type: "attempt_prepared",
486
- experimentId,
487
- executionId: context.executionId,
488
- caseId: context.item.id,
489
- environmentRunId: run.id,
490
- bindingId: prepared.status === "ready" ? prepared.bundle.bindingId : prepared.bindingId,
491
- status: prepared.status,
492
- findingCodes: prepared.preflightReport.findings.map((finding) => finding.code),
493
- ...(prepared.status === "ready"
494
- ? {
495
- executionManifestDigest: prepared.bundle.parity.executionManifestDigest,
496
- }
497
- : {}),
498
- });
499
- if (prepared.status === "environment_incomplete") {
500
- try {
501
- await seal(options.environmentClient, run.id, context.executionId, "completed");
502
- }
503
- catch (error) {
504
- throw new TargetOutcomeUncertainError(context.executionId, { cause: error });
505
- }
506
- finalized = true;
507
- await Promise.resolve(progress("world_sealed")).catch(() => undefined);
508
- return undefined;
509
- }
510
- connectionBundle = validateAttemptConnectionBundleV2(prepared.bundle, {
511
- requireFresh: true,
512
- });
513
- const projected = projectMcpConnectionV2(connectionBundle, requested.mcpSurface.providerInstanceKey);
514
- if (!projected)
515
- throw new TypeError("The prepared attempt has no selected MCP surface");
516
- mcp = projected;
517
- }
518
- else {
519
- mcp = await options.client.createSimulationMcpCapability({
520
- runId: run.id,
521
- executionId: context.executionId,
522
- });
523
- }
524
- if (options.signal?.aborted)
525
- throw new TargetCancelledError();
526
- await progress("target_started");
527
- const output = await options.target(inputs, {
528
- config: context.config,
529
- item: context.item,
530
- executionId: context.executionId,
531
- environmentRunId: run.id,
532
- tools,
533
- mcp,
534
- ...(connectionBundle ? { connectionBundle } : {}),
535
- signal: options.signal,
536
- });
537
- try {
538
- await seal(options.environmentClient, run.id, context.executionId, "completed");
539
- }
540
- catch (error) {
541
- throw new TargetOutcomeUncertainError(context.executionId, { cause: error });
542
- }
543
- finalized = true;
544
- await Promise.resolve(progress("world_sealed")).catch(() => undefined);
545
- return output;
546
- }
547
- catch (error) {
548
- if (error instanceof TargetOutcomeUncertainError || finalized)
549
- throw error;
550
- let environmentIncomplete;
551
- try {
552
- environmentIncomplete =
553
- (await options.environmentClient.getRun(run.id)).validity ===
554
- "environment_incomplete";
555
- }
556
- catch (inspectionError) {
557
- // A target error can be the adapter surfacing a coverage gap. If the
558
- // authoritative run cannot be read, do not guess that it was an agent
559
- // failure or replay the target on resume.
560
- throw new TargetOutcomeUncertainError(context.executionId, {
561
- cause: new AggregateError([error, inspectionError]),
562
- });
563
- }
564
- // A durable coverage gap invalidates parity independently of caller timing;
565
- // do not let a racing local abort hide it as an ordinary cancellation.
566
- if (environmentIncomplete) {
567
- try {
568
- await seal(options.environmentClient, run.id, context.executionId, "completed");
569
- }
570
- catch (finalizationError) {
571
- throw new TargetOutcomeUncertainError(context.executionId, {
572
- cause: new AggregateError([error, finalizationError]),
573
- });
574
- }
575
- finalized = true;
576
- await Promise.resolve(progress("world_sealed")).catch(() => undefined);
577
- return undefined;
578
- }
579
- try {
580
- await seal(options.environmentClient, run.id, context.executionId, "abandoned");
581
- }
582
- catch (finalizationError) {
583
- throw new TargetOutcomeUncertainError(context.executionId, {
584
- cause: new AggregateError([error, finalizationError]),
585
- });
586
- }
587
- await Promise.resolve(progress("world_sealed")).catch(() => undefined);
588
- if (options.signal?.aborted && !(error instanceof TargetCancelledError))
589
- throw new TargetCancelledError();
590
- throw error;
591
- }
592
- },
395
+ }),
396
+ target: (targetInputs, targetContext) => options.target(structuredClone(targetInputs), {
397
+ config: structuredClone(targetContext.config),
398
+ item: {
399
+ id: targetContext.item.id,
400
+ externalKey: targetContext.item.externalKey,
401
+ },
402
+ executionId: targetContext.executionId,
403
+ environmentRunId: targetContext.environmentRunId,
404
+ tools: targetContext.tools,
405
+ mcp: {
406
+ url: targetContext.mcp.url,
407
+ token: targetContext.mcp.token,
408
+ expiresAt: targetContext.mcp.expiresAt,
409
+ },
410
+ ...(targetContext.connectionBundle
411
+ ? { connectionBundle: structuredClone(targetContext.connectionBundle) }
412
+ : {}),
413
+ signal: targetContext.signal,
414
+ }),
415
+ }),
593
416
  });
594
417
  const complete = { ...report, experimentId, runUrl };
595
418
  attempt.stage = "completed";
@@ -603,3 +603,34 @@ export interface SimulationMcpCapability {
603
603
  /** Credential expiry timestamp. */
604
604
  expiresAt: string;
605
605
  }
606
+ /** Identity and capabilities of one fixed local agent entry point. */
607
+ export interface LocalAgentRegistration {
608
+ /** Stable application-selected agent key. */
609
+ key: string;
610
+ /** Display name. */
611
+ name: string;
612
+ /** Application-selected revision of the agent configuration. */
613
+ revision: string;
614
+ /** Supported execution contracts; defaults to environment:v1 in the worker. */
615
+ capabilities?: string[];
616
+ /** Local scorer source digests available in this process. */
617
+ scorerDigests?: string[];
618
+ }
619
+ /** Server registration and heartbeat timestamps for a local agent. */
620
+ export interface RegisteredLocalAgent extends Required<LocalAgentRegistration> {
621
+ /** Registered agent identity. */
622
+ id: string;
623
+ /** Whether Hue permits this registration to receive runs. */
624
+ enabled: boolean;
625
+ /** Latest registration heartbeat, as an ISO timestamp. */
626
+ lastSeenAt: string;
627
+ /** Registration creation timestamp. */
628
+ createdAt: string;
629
+ }
630
+ /** Queue claim connecting a local run to a pinned experiment. */
631
+ export interface LocalAgentClaim {
632
+ /** Claimed queue-run identity. */
633
+ runId: string;
634
+ /** Pinned experiment to execute. */
635
+ experimentId: string;
636
+ }
package/dist/evals.d.ts CHANGED
@@ -9,3 +9,5 @@ export type { ActualAgentManifestInputV2, ActualAgentManifestV2, AttemptBaseline
9
9
  export { builtins, defineLocalScorer, scoreLocally } from "./evals/scorers.js";
10
10
  export { sourceDigest } from "./evals/json.js";
11
11
  export type * from "./evals/types.js";
12
+ export { runLocalAgent } from "./evals/local-worker.js";
13
+ export type { LocalAgentTargetContext, RunLocalAgentOptions } from "./evals/local-worker.js";
package/dist/evals.js CHANGED
@@ -4,3 +4,4 @@ export { runSimulation } from "./evals/simulation.js";
4
4
  export { actualAgentManifestV2, agentManifestDigestV2, attemptBaselineV2, attemptBindingRead, attemptConnectionBundleV2, attemptIdentityV2, dependencyManifestV2, dependencyProviderV2, expectedAgentManifestV2, executionManifestDigestV2, parityEvidenceV2, preflightFindingV2, preflightReportV2, prepareAttemptInputV2, projectMcpConnectionV2, secretFreeBindingV2, surfaceBindingV2, } from "./evals/attempt.js";
5
5
  export { builtins, defineLocalScorer, scoreLocally } from "./evals/scorers.js";
6
6
  export { sourceDigest } from "./evals/json.js";
7
+ export { runLocalAgent } from "./evals/local-worker.js";
package/dist/safety.js CHANGED
@@ -166,10 +166,16 @@ export function estimateRecordBytes(value, limit) {
166
166
  bytes += item.byteLength;
167
167
  else if (item && typeof item === "object" && !seen.has(item)) {
168
168
  seen.add(item);
169
- for (const [key, child] of Object.entries(item)) {
170
- bytes += key.length * 2 + 16;
171
- visit(child, depth + 1);
172
- }
169
+ // Match admission (`Snapshot.copy`): array elements are nodes without retained index
170
+ // keys. Charging indexes here would refuse array-heavy records the queue admitted.
171
+ if (Array.isArray(item))
172
+ for (const child of item)
173
+ visit(child, depth + 1);
174
+ else
175
+ for (const [key, child] of Object.entries(item)) {
176
+ bytes += key.length * 2 + 16;
177
+ visit(child, depth + 1);
178
+ }
173
179
  }
174
180
  if (bytes > limit)
175
181
  throw new RangeError("Telemetry byte limit exceeded");
package/dist/version.d.ts CHANGED
@@ -1,2 +1,2 @@
1
1
  /** Package version shared by the instrumentation scope and the export User-Agent. */
2
- export declare const sdkVersion = "0.2.1";
2
+ export declare const sdkVersion = "0.3.0";
package/dist/version.js CHANGED
@@ -1,3 +1,3 @@
1
1
  // Generated by scripts/write-version.mjs from package.json; do not edit by hand.
2
2
  /** Package version shared by the instrumentation scope and the export User-Agent. */
3
- export const sdkVersion = "0.2.1";
3
+ export const sdkVersion = "0.3.0";
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@hue-run/sdk",
3
- "version": "0.2.1",
3
+ "version": "0.3.0",
4
4
  "private": false,
5
5
  "license": "MIT",
6
6
  "publishConfig": {