@hue-run/sdk 0.2.2 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ENVIRONMENTS.md +26 -7
- package/EVALUATIONS.md +65 -1
- package/README.md +14 -0
- package/dist/environment/client.d.ts +2 -2
- package/dist/environment/types.d.ts +33 -1
- package/dist/evals/client.d.ts +30 -1
- package/dist/evals/client.js +16 -0
- package/dist/evals/environment-target.d.ts +84 -0
- package/dist/evals/environment-target.js +201 -0
- package/dist/evals/local-worker.d.ts +88 -0
- package/dist/evals/local-worker.js +171 -0
- package/dist/evals/simulation.d.ts +6 -6
- package/dist/evals/simulation.js +45 -222
- package/dist/evals/types.d.ts +31 -0
- package/dist/evals.d.ts +2 -0
- package/dist/evals.js +1 -0
- package/dist/version.d.ts +1 -1
- package/dist/version.js +1 -1
- package/package.json +1 -1
package/ENVIRONMENTS.md
CHANGED
|
@@ -1,9 +1,10 @@
|
|
|
1
1
|
# Simulated environments
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
This guide documents the TypeScript `0.3.0` release candidate. Until registry acceptance, use
|
|
4
|
+
the exact reviewed archive rather than requesting `0.3.0` from npm:
|
|
4
5
|
|
|
5
6
|
```bash
|
|
6
|
-
npm install
|
|
7
|
+
npm install /path/to/reviewed/hue-run-sdk-0.3.0.tgz zod
|
|
7
8
|
```
|
|
8
9
|
|
|
9
10
|
Your agent runs in your process while a disposable simulated world runs in Hue. The world is
|
|
@@ -77,11 +78,12 @@ them to checkpoints or progress events, and it never mutates global `process.env
|
|
|
77
78
|
world seal cannot be confirmed, the checkpoint remains uncertain and resume neither reacquires
|
|
78
79
|
credentials nor invokes the callback again.
|
|
79
80
|
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
81
|
+
The SDK consumes the versioned connection contract but does not itself establish provider
|
|
82
|
+
fidelity. Public package acceptance exercises local control-plane responses and connection-bundle
|
|
83
|
+
handling; it does not call the issued facade endpoint. Exact installed-registry-package to hosted-
|
|
84
|
+
facade acceptance remains a post-publication Fern integration gate. No public SDK test calls the
|
|
85
|
+
official Gmail service, and passing these tests is not evidence of universal Gmail or Slack
|
|
86
|
+
parity. A matching Hue deployment and verified provider profile remain required.
|
|
85
87
|
|
|
86
88
|
## Repository-authored scenarios
|
|
87
89
|
|
|
@@ -97,6 +99,13 @@ identity-affecting defaults as Hue before resolving versions and rejects unknown
|
|
|
97
99
|
kinds. In particular, `document_verifier` is not part of this SDK contract and is rejected rather
|
|
98
100
|
than published with a guessed digest.
|
|
99
101
|
|
|
102
|
+
The extendable `EnvironmentDefinition` name remains the V1 contract and is also exported as
|
|
103
|
+
`EnvironmentDefinitionV1`. Use `EnvironmentDefinitionV2` to add immutable Gmail
|
|
104
|
+
`providerInstances`; `PublishableEnvironmentDefinition` is the publication/repository union.
|
|
105
|
+
Hue canonicalizes valid synthetic-principal UUIDs to lowercase, and repository resolution does
|
|
106
|
+
the same before comparing immutable digests, so casing-only UUID changes reuse the stored
|
|
107
|
+
version without dropping provider bindings.
|
|
108
|
+
|
|
100
109
|
```ts
|
|
101
110
|
const scenario = {
|
|
102
111
|
kind: "repository" as const,
|
|
@@ -180,3 +189,13 @@ that cannot be confirmed stays uncertain and never causes the agent to be replay
|
|
|
180
189
|
The hosted MCP connection exposes Hue's bounded native actions; it is not general Gmail or
|
|
181
190
|
Slack HTTP parity and does not proxy arbitrary provider traffic. Forking, in-place reset and
|
|
182
191
|
arbitrary-step diffs are outside this interface.
|
|
192
|
+
|
|
193
|
+
## Candidate context migration
|
|
194
|
+
|
|
195
|
+
This candidate-context restriction is part of the `@hue-run/sdk@0.3.0` release candidate.
|
|
196
|
+
|
|
197
|
+
`runSimulation` now supplies `context.item` as `{ id, externalKey }`. Read candidate inputs
|
|
198
|
+
from the callback's first argument. Expected outcomes, case metadata and environment-version
|
|
199
|
+
pins are available to evaluation and scoring code, and are omitted from the candidate callback.
|
|
200
|
+
Inputs and configuration are cloned before invocation so candidate mutations cannot change
|
|
201
|
+
pinned grading data. The generic `runExperiment` evaluator interface is unchanged.
|
package/EVALUATIONS.md
CHANGED
|
@@ -9,6 +9,8 @@ npm install @hue-run/sdk zod
|
|
|
9
9
|
|
|
10
10
|
The SDK executes targets and scorers on your machine. Hue stores pinned definitions, experiment progress and results. It does not execute uploaded source code. Follow the [installation guide](https://docs.hue.run/installation) to add `@hue-run/sdk` to your application.
|
|
11
11
|
|
|
12
|
+
Create a **Tracing and evaluations** project service key under **Settings → Integrations & API keys** and expose it to this server-side process as `HUE_API_KEY`. A **Tracing only** key cannot author datasets or evaluation runs.
|
|
13
|
+
|
|
12
14
|
```ts
|
|
13
15
|
import { randomUUID } from "node:crypto";
|
|
14
16
|
import { createHue } from "@hue-run/sdk";
|
|
@@ -150,4 +152,66 @@ The runner stops scheduling more cases after an operational failure and waits fo
|
|
|
150
152
|
|
|
151
153
|
## Verification boundaries
|
|
152
154
|
|
|
153
|
-
`scripts/verify-package.mjs` installs a real packed tarball outside the monorepo and runs HTTP contract tests against a synthetic service plus actual OpenTelemetry exporters. It checks two configurations, rescoring without target invocation, absent/null output, upload resume, uncertain execution, exclusive checkpoints, source/metric contracts, content policy and terminating schema workers. `scripts/verify-evaluation-api.mjs` is a separate opt-in acceptance against a real Hue receiver/API; it creates synthetic datasets/scorers/experiments in the project associated with the supplied development key.
|
|
155
|
+
`scripts/verify-package.mjs` installs a real packed tarball outside the monorepo and runs HTTP contract tests against a synthetic service plus actual OpenTelemetry exporters. It checks two configurations, rescoring without target invocation, absent/null output, upload resume, uncertain execution, exclusive checkpoints, source/metric contracts, content policy and terminating schema workers. It also exercises the local worker's ready, incomplete and uncertain provider-attempt control-plane paths, but does not call an issued provider facade. `scripts/verify-evaluation-api.mjs` is a separate opt-in acceptance against a real Hue receiver/API; it creates synthetic datasets/scorers/experiments in the project associated with the supplied development key.
|
|
156
|
+
|
|
157
|
+
## Outbound local agent worker
|
|
158
|
+
|
|
159
|
+
`runLocalAgent` is included in the `@hue-run/sdk@0.3.0` release candidate. Queue registration,
|
|
160
|
+
claims, scoped MCP capabilities and sealed evidence require a supporting Hue server and project
|
|
161
|
+
access; the package version alone does not establish hosted availability. Until registry
|
|
162
|
+
acceptance, install the exact reviewed `hue-run-sdk-0.3.0.tgz` archive with the optional `zod`
|
|
163
|
+
peer. `node packages/sdk-typescript/scripts/verify-package.mjs` creates and verifies that archive;
|
|
164
|
+
do not request `0.3.0` from npm yet.
|
|
165
|
+
|
|
166
|
+
`runLocalAgent` registers one fixed application callback and polls for queued runs. Hue selects
|
|
167
|
+
the registered key/revision; it does not send executable code or shell commands. Keep the
|
|
168
|
+
checkpoint directory private and durable. The worker persists result content and requires
|
|
169
|
+
acknowledged trace and sealed environment evidence.
|
|
170
|
+
|
|
171
|
+
```ts
|
|
172
|
+
import { createHue } from "@hue-run/sdk";
|
|
173
|
+
import { createEnvironmentClient } from "@hue-run/sdk/environment";
|
|
174
|
+
import { createEvaluationClient, runLocalAgent } from "@hue-run/sdk/evals";
|
|
175
|
+
import { runMyAgent } from "./agent.js"; // Your existing application entry point.
|
|
176
|
+
|
|
177
|
+
const connection = { apiKey: process.env.HUE_API_KEY! };
|
|
178
|
+
const hue = createHue({ ...connection, serviceName: "local-worker", captureContent: false });
|
|
179
|
+
try {
|
|
180
|
+
await runLocalAgent({
|
|
181
|
+
client: createEvaluationClient(connection),
|
|
182
|
+
environmentClient: createEnvironmentClient(connection),
|
|
183
|
+
hue,
|
|
184
|
+
agent: { key: "support-agent", name: "Support agent", revision: "1" },
|
|
185
|
+
checkpointDirectory: ".hue-checkpoints/support-agent",
|
|
186
|
+
target: (inputs, tools, context) => runMyAgent({ inputs, tools, config: context.config }),
|
|
187
|
+
});
|
|
188
|
+
} finally {
|
|
189
|
+
await hue.shutdownSafe();
|
|
190
|
+
}
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
The callback receives cloned inputs, local tools, and an allowlisted context containing
|
|
194
|
+
`config`, `item: {id, externalKey}`, `executionId`, `environmentRunId`,
|
|
195
|
+
`trace: {traceId,spanId}` and a short-lived `mcp` capability. Expected outcomes, case metadata
|
|
196
|
+
and original source pins remain private to grading. Pass the tools or scoped MCP capability into
|
|
197
|
+
the agent's actual tool boundary; their presence does not redirect provider calls. Capabilities
|
|
198
|
+
are not written to checkpoints. `maxRuns` limits completed runs for one-shot workers, while
|
|
199
|
+
`signal` stops polling. A stop signal does not forcibly cancel an already executing callback.
|
|
200
|
+
|
|
201
|
+
For an experiment with an immutable V2 attempt baseline, also supply `actualAgentManifest`, the
|
|
202
|
+
exact ordered `requestedProviders`, and an `mcpSurface` selected from that request. The worker
|
|
203
|
+
creates the world and prepares once before target code. A ready response exposes the memory-only
|
|
204
|
+
`connectionBundle` and keeps `context.mcp` as its selected MCP projection; it never mints the
|
|
205
|
+
legacy generic capability for that attempt. An incomplete response seals the world as completed
|
|
206
|
+
without invoking the target or scorers. A lost preparation acknowledgement remains uncertain
|
|
207
|
+
and is never recovered through binding reads, credential refresh or target replay. Synthetic
|
|
208
|
+
acceptance does not contact official Gmail or claim universal provider parity.
|
|
209
|
+
|
|
210
|
+
Completion or result-upload failures keep the run claimed by the durable worker identity.
|
|
211
|
+
Restart with the same checkpoint directory to resume saved uploads without invoking the
|
|
212
|
+
candidate again. A lost world-seal acknowledgement is recovered by reading authoritative world
|
|
213
|
+
state. If the seal or candidate outcome cannot be confirmed, or an outcome cannot be serialized,
|
|
214
|
+
the worker reports `attention` and stops; operator investigation is required. Such runs are not
|
|
215
|
+
automatically reclaimed, and presenting the same uncertain checkpoint again cannot replay the
|
|
216
|
+
candidate. Public package acceptance proves this lifecycle against local fixtures; exact
|
|
217
|
+
installed-registry-package to hosted-facade acceptance remains a post-publication Fern gate.
|
package/README.md
CHANGED
|
@@ -395,6 +395,20 @@ published. The chatbot README describes running that external installation.
|
|
|
395
395
|
|
|
396
396
|
The optional `@hue-run/sdk/evals` entry point supports dataset/scorer registration, frozen-version experiments, local built-in/custom scoring, upload resume, and historical rescoring. See the [evaluation guide](https://docs.hue.run/evaluations/first-evaluation) for the complete journey, content policy and checkpoint recovery contract.
|
|
397
397
|
|
|
398
|
+
### App-launched local workers
|
|
399
|
+
|
|
400
|
+
The TypeScript `0.3.0` release candidate adds `runLocalAgent()` for a fixed local callback that
|
|
401
|
+
claims app-launched simulation work while keeping the agent, provider orchestration and debugger
|
|
402
|
+
in the developer's process. Until registry acceptance, test it only from the exact reviewed
|
|
403
|
+
`hue-run-sdk-0.3.0.tgz` archive rather than requesting `0.3.0` from npm. It shares
|
|
404
|
+
`runSimulation()`'s provider-aware world lifecycle, keeps scoped credentials in callback memory,
|
|
405
|
+
skips target/scorer execution for incomplete environments, and never reacquires or replays after
|
|
406
|
+
an uncertain preparation. See the
|
|
407
|
+
[outbound worker contract](EVALUATIONS.md#outbound-local-agent-worker) and
|
|
408
|
+
[simulated environment guide](ENVIRONMENTS.md). Tests exercise local control-plane fixtures, not
|
|
409
|
+
an issued facade endpoint or the official Gmail service, and do not claim universal provider
|
|
410
|
+
parity.
|
|
411
|
+
|
|
398
412
|
## Managed targets
|
|
399
413
|
|
|
400
414
|
Start a frozen dataset run in Hue while your agent stays in your application. Expose a
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import type { ActionInput, ActionResult, CoverageGapInput, CoverageGapResult, CreateRunInput, Environment,
|
|
1
|
+
import type { ActionInput, ActionResult, CoverageGapInput, CoverageGapResult, CreateRunInput, Environment, PublishableEnvironmentDefinition, EnvironmentIdentity, EnvironmentPage, EnvironmentPageOptions, EnvironmentRun, EnvironmentSummary, EnvironmentVersion, EnvironmentVersionSummary, FinishRunInput, SealedRun, StepPage, StepPageOptions } from "./types.js";
|
|
2
2
|
/** Connection and retry options for {@link createEnvironmentClient}. */
|
|
3
3
|
export interface EnvironmentClientOptions {
|
|
4
4
|
/** Project service key sent as a bearer token; server-side only. */
|
|
@@ -37,7 +37,7 @@ export declare class EnvironmentClient {
|
|
|
37
37
|
/** Reads one environment and its immutable version summaries. */
|
|
38
38
|
getEnvironment(id: string): Promise<Environment>;
|
|
39
39
|
/** Publishes an immutable definition; this non-idempotent registry write is not retried. */
|
|
40
|
-
publishVersion(environmentId: string, definition:
|
|
40
|
+
publishVersion(environmentId: string, definition: PublishableEnvironmentDefinition): Promise<EnvironmentVersionSummary>;
|
|
41
41
|
/** Reads a full immutable environment version and generated action catalog. */
|
|
42
42
|
getVersion(id: string): Promise<EnvironmentVersion>;
|
|
43
43
|
/** Creates or recovers one fresh isolated world using a stable idempotency key. */
|
|
@@ -170,12 +170,44 @@ export interface EnvironmentDefinition {
|
|
|
170
170
|
/** Caller-owned immutable metadata. */
|
|
171
171
|
metadata?: Record<string, JsonValue>;
|
|
172
172
|
}
|
|
173
|
+
/** The extendable legacy name remains V1. Publication and runs select their
|
|
174
|
+
* explicit version; provider context is validated by the authoritative server. */
|
|
175
|
+
export type EnvironmentDefinitionV1 = EnvironmentDefinition;
|
|
176
|
+
/** One synthetic Gmail principal and its world-state collection bindings. */
|
|
177
|
+
export interface GmailProviderInstance {
|
|
178
|
+
/** Stable instance key referenced by attempt provider selection. */
|
|
179
|
+
providerInstanceKey: string;
|
|
180
|
+
/** Provider discriminator for the V2 Gmail slice. */
|
|
181
|
+
providerId: "google.gmail";
|
|
182
|
+
/** Synthetic principal UUID, canonicalized to lowercase by Hue. */
|
|
183
|
+
syntheticPrincipalId: string;
|
|
184
|
+
/** Versioned mapping from Gmail concepts to authored-world collections. */
|
|
185
|
+
configuration: {
|
|
186
|
+
/** Gmail mailbox configuration discriminator. */
|
|
187
|
+
kind: "gmail_mailbox/v1";
|
|
188
|
+
/** Collection containing synthetic messages. */
|
|
189
|
+
messagesCollection: string;
|
|
190
|
+
/** Collection containing synthetic drafts. */
|
|
191
|
+
draftsCollection: string;
|
|
192
|
+
/** Synthetic mailbox address. */
|
|
193
|
+
mailboxAddress: string;
|
|
194
|
+
};
|
|
195
|
+
}
|
|
196
|
+
/** V2 authored world with immutable provider-instance bindings. */
|
|
197
|
+
export interface EnvironmentDefinitionV2 extends Omit<EnvironmentDefinition, "schemaVersion"> {
|
|
198
|
+
/** Definition schema discriminator. */
|
|
199
|
+
schemaVersion: 2;
|
|
200
|
+
/** Provider instances available to a strict attempt profile. */
|
|
201
|
+
providerInstances: GmailProviderInstance[];
|
|
202
|
+
}
|
|
203
|
+
/** Definition accepted by immutable environment publication. */
|
|
204
|
+
export type PublishableEnvironmentDefinition = EnvironmentDefinitionV1 | EnvironmentDefinitionV2;
|
|
173
205
|
/** Full immutable environment version and its generated action catalog. */
|
|
174
206
|
export interface EnvironmentVersion extends EnvironmentVersionSummary {
|
|
175
207
|
/** Owning environment identity. */
|
|
176
208
|
environmentId: string;
|
|
177
209
|
/** Stored, defaulted authored definition. */
|
|
178
|
-
definition:
|
|
210
|
+
definition: PublishableEnvironmentDefinition;
|
|
179
211
|
/** Generated agent-visible actions. */
|
|
180
212
|
actions: ActionDefinition[];
|
|
181
213
|
}
|
package/dist/evals/client.d.ts
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import type { ProjectConnection } from "../types.js";
|
|
2
2
|
import { type AttemptConnectionBundleV2, type PrepareAttemptRequestV2 } from "./attempt.js";
|
|
3
|
-
import type { CaseWrite, CompleteExecution, Completion, Dataset, DatasetCase, DatasetVersion, EvaluationItem, EvaluationRun, EnvironmentEvidenceSnapshot, Execution, Experiment, ExperimentCase, ExperimentItem, Identity, JsonValue, JudgeBudget, JudgeJob, Page, PageOptions, RegistryPageOptions, Result, ResultSummary, Scorer, ScorerDefinition, ScorerVersion, SimulationMcpCapability, StartExecution, Subject, StoredResult } from "./types.js";
|
|
3
|
+
import type { CaseWrite, CompleteExecution, Completion, Dataset, DatasetCase, DatasetVersion, EvaluationItem, EvaluationRun, EnvironmentEvidenceSnapshot, Execution, Experiment, ExperimentCase, ExperimentItem, Identity, JsonValue, LocalAgentClaim, LocalAgentRegistration, RegisteredLocalAgent, JudgeBudget, JudgeJob, Page, PageOptions, RegistryPageOptions, Result, ResultSummary, Scorer, ScorerDefinition, ScorerVersion, SimulationMcpCapability, StartExecution, Subject, StoredResult } from "./types.js";
|
|
4
4
|
/** Connection options for {@link createEvaluationClient}. */
|
|
5
5
|
export interface EvaluationClientOptions {
|
|
6
6
|
/** Project service key sent as a Bearer token; server side only. */
|
|
@@ -176,6 +176,35 @@ export declare class EvaluationClient {
|
|
|
176
176
|
}>;
|
|
177
177
|
/** Reads the project's hosted judge budget and admission controls. */
|
|
178
178
|
getJudgeBudget(): Promise<JudgeBudget>;
|
|
179
|
+
/** Register or refresh the fixed local agent key and revision. */
|
|
180
|
+
registerLocalAgent(input: LocalAgentRegistration): Promise<RegisteredLocalAgent>;
|
|
181
|
+
/** Claim a queued run for this agent and durable worker identity. */
|
|
182
|
+
claimLocalAgentRun(input: {
|
|
183
|
+
agentId: string;
|
|
184
|
+
workerId: string;
|
|
185
|
+
}): Promise<LocalAgentClaim | null>;
|
|
186
|
+
/** Refresh the lease of a claimed local run. */
|
|
187
|
+
heartbeatLocalAgentRun(input: {
|
|
188
|
+
runId: string;
|
|
189
|
+
workerId: string;
|
|
190
|
+
}): Promise<{
|
|
191
|
+
/** Queue-run identity. */
|
|
192
|
+
runId: string;
|
|
193
|
+
/** The worker claim remains active. */
|
|
194
|
+
active: true;
|
|
195
|
+
}>;
|
|
196
|
+
/** Report acknowledged completion or an execution requiring attention. */
|
|
197
|
+
completeLocalAgentRun(input: {
|
|
198
|
+
runId: string;
|
|
199
|
+
workerId: string;
|
|
200
|
+
state: "completed" | "attention";
|
|
201
|
+
failureType?: string;
|
|
202
|
+
}): Promise<{
|
|
203
|
+
/** Queue-run identity. */
|
|
204
|
+
runId: string;
|
|
205
|
+
/** Acknowledged terminal queue state. */
|
|
206
|
+
state: "completed" | "attention";
|
|
207
|
+
}>;
|
|
179
208
|
/** Creates the legacy execution-scoped generic MCP capability for one world. */
|
|
180
209
|
createSimulationMcpCapability(input: {
|
|
181
210
|
runId: string;
|
package/dist/evals/client.js
CHANGED
|
@@ -309,6 +309,22 @@ export class EvaluationClient {
|
|
|
309
309
|
getJudgeBudget() {
|
|
310
310
|
return this.request("GET", "/judge-budget");
|
|
311
311
|
}
|
|
312
|
+
/** Register or refresh the fixed local agent key and revision. */
|
|
313
|
+
registerLocalAgent(input) {
|
|
314
|
+
return this.request("POST", "/local-agent-worker/register", input);
|
|
315
|
+
}
|
|
316
|
+
/** Claim a queued run for this agent and durable worker identity. */
|
|
317
|
+
claimLocalAgentRun(input) {
|
|
318
|
+
return this.request("POST", "/local-agent-worker/claim", input);
|
|
319
|
+
}
|
|
320
|
+
/** Refresh the lease of a claimed local run. */
|
|
321
|
+
heartbeatLocalAgentRun(input) {
|
|
322
|
+
return this.request("POST", "/local-agent-worker/runs/heartbeat", input);
|
|
323
|
+
}
|
|
324
|
+
/** Report acknowledged completion or an execution requiring attention. */
|
|
325
|
+
completeLocalAgentRun(input) {
|
|
326
|
+
return this.request("POST", "/local-agent-worker/runs/complete", input);
|
|
327
|
+
}
|
|
312
328
|
/** Creates the legacy execution-scoped generic MCP capability for one world. */
|
|
313
329
|
createSimulationMcpCapability(input) {
|
|
314
330
|
return this.request("POST", "/local-agent-worker/mcp-capability", input);
|
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
import type { HueClient } from "../client.js";
|
|
2
|
+
import type { EnvironmentClient } from "../environment/client.js";
|
|
3
|
+
import { type EnvironmentTool } from "../environment/tools.js";
|
|
4
|
+
import type { HueSpan } from "../types.js";
|
|
5
|
+
import { type ActualAgentManifestInputV2, type AttemptBaselineV2, type AttemptConnectionBundleV2, type RequestedAttemptProviderV2, type SurfaceBindingV2 } from "./attempt.js";
|
|
6
|
+
import type { EvaluationClient } from "./client.js";
|
|
7
|
+
import type { ExperimentCase, JsonValue, SimulationMcpCapability } from "./types.js";
|
|
8
|
+
export type McpSurfaceKeyV2 = Extract<SurfaceBindingV2["surfaceKey"], `${string}/mcp`>;
|
|
9
|
+
export type ActualAgentManifestResolverV2 = ActualAgentManifestInputV2 | ((context: {
|
|
10
|
+
config: JsonValue;
|
|
11
|
+
item: ExperimentCase;
|
|
12
|
+
signal?: AbortSignal;
|
|
13
|
+
}) => ActualAgentManifestInputV2 | Promise<ActualAgentManifestInputV2>);
|
|
14
|
+
export interface ProviderAttemptOptionsV2 {
|
|
15
|
+
actualAgentManifest?: ActualAgentManifestResolverV2;
|
|
16
|
+
requestedProviders?: RequestedAttemptProviderV2[];
|
|
17
|
+
mcpSurface?: {
|
|
18
|
+
providerInstanceKey: string;
|
|
19
|
+
surfaceKey: McpSurfaceKeyV2;
|
|
20
|
+
};
|
|
21
|
+
}
|
|
22
|
+
export type RequestedAttemptV2 = {
|
|
23
|
+
actualAgentManifest: ActualAgentManifestResolverV2;
|
|
24
|
+
requestedProviders: RequestedAttemptProviderV2[];
|
|
25
|
+
mcpSurface: {
|
|
26
|
+
providerInstanceKey: string;
|
|
27
|
+
surfaceKey: McpSurfaceKeyV2;
|
|
28
|
+
};
|
|
29
|
+
};
|
|
30
|
+
export type PinnedAttemptV2 = RequestedAttemptV2 & {
|
|
31
|
+
expectedAgentManifestDigest: AttemptBaselineV2["expectedAgentManifestDigest"];
|
|
32
|
+
};
|
|
33
|
+
export declare function requestedAttemptV2(options: ProviderAttemptOptionsV2): RequestedAttemptV2 | undefined;
|
|
34
|
+
export declare function pinRequestedAttemptV2(requested: RequestedAttemptV2, config: JsonValue): PinnedAttemptV2;
|
|
35
|
+
export type EnvironmentTargetProgress = {
|
|
36
|
+
type: "world_created" | "target_started" | "world_sealed";
|
|
37
|
+
environmentRunId: string;
|
|
38
|
+
} | {
|
|
39
|
+
type: "attempt_prepared";
|
|
40
|
+
environmentRunId: string;
|
|
41
|
+
bindingId: string;
|
|
42
|
+
status: "ready" | "environment_incomplete";
|
|
43
|
+
findingCodes: string[];
|
|
44
|
+
executionManifestDigest?: AttemptConnectionBundleV2["parity"]["executionManifestDigest"];
|
|
45
|
+
};
|
|
46
|
+
export interface EnvironmentTargetContext {
|
|
47
|
+
config: JsonValue;
|
|
48
|
+
item: ExperimentCase;
|
|
49
|
+
executionId: string;
|
|
50
|
+
environmentRunId: string;
|
|
51
|
+
trace: {
|
|
52
|
+
traceId: string;
|
|
53
|
+
spanId: string;
|
|
54
|
+
};
|
|
55
|
+
tools: Record<string, EnvironmentTool>;
|
|
56
|
+
mcp: SimulationMcpCapability;
|
|
57
|
+
connectionBundle?: AttemptConnectionBundleV2;
|
|
58
|
+
signal?: AbortSignal;
|
|
59
|
+
}
|
|
60
|
+
interface RunnerTargetContext {
|
|
61
|
+
config: JsonValue;
|
|
62
|
+
item: ExperimentCase;
|
|
63
|
+
executionId: string;
|
|
64
|
+
span: HueSpan;
|
|
65
|
+
}
|
|
66
|
+
export interface RunEnvironmentTargetOptions {
|
|
67
|
+
client: EvaluationClient;
|
|
68
|
+
environmentClient: EnvironmentClient;
|
|
69
|
+
hue: HueClient;
|
|
70
|
+
inputs: JsonValue;
|
|
71
|
+
context: RunnerTargetContext;
|
|
72
|
+
requested?: PinnedAttemptV2;
|
|
73
|
+
maxSteps?: number;
|
|
74
|
+
ttlSeconds?: number;
|
|
75
|
+
signal?: AbortSignal;
|
|
76
|
+
onProgress?(event: EnvironmentTargetProgress): void | Promise<void>;
|
|
77
|
+
target(inputs: JsonValue, context: EnvironmentTargetContext): JsonValue | undefined | Promise<JsonValue | undefined>;
|
|
78
|
+
}
|
|
79
|
+
/** One authoritative environment/provider lifecycle shared by direct simulations and
|
|
80
|
+
* outbound local workers. Credential-bearing connections stay in this call frame and
|
|
81
|
+
* are never returned to either runner's checkpoint state.
|
|
82
|
+
*/
|
|
83
|
+
export declare function runEnvironmentTarget(options: RunEnvironmentTargetOptions): Promise<JsonValue | undefined>;
|
|
84
|
+
export {};
|
|
@@ -0,0 +1,201 @@
|
|
|
1
|
+
import { randomUUID } from "node:crypto";
|
|
2
|
+
import { bindEnvironmentTools } from "../environment/tools.js";
|
|
3
|
+
import { actualAgentManifestV2, attemptBaselineV2, projectMcpConnectionV2, requestedAttemptProvidersV2, validateAttemptConnectionBundleV2, } from "./attempt.js";
|
|
4
|
+
import { TargetCancelledError, TargetOutcomeUncertainError } from "./runner.js";
|
|
5
|
+
export function requestedAttemptV2(options) {
|
|
6
|
+
const requested = options.requestedProviders !== undefined;
|
|
7
|
+
const selected = options.mcpSurface !== undefined;
|
|
8
|
+
if (!requested && !selected) {
|
|
9
|
+
if (options.actualAgentManifest !== undefined)
|
|
10
|
+
throw new TypeError("actualAgentManifest requires requestedProviders and mcpSurface");
|
|
11
|
+
return undefined;
|
|
12
|
+
}
|
|
13
|
+
if (!requested || !selected)
|
|
14
|
+
throw new TypeError("requestedProviders and mcpSurface must be supplied together");
|
|
15
|
+
const requestedProviders = requestedAttemptProvidersV2.parse(options.requestedProviders);
|
|
16
|
+
const mcpSurface = options.mcpSurface;
|
|
17
|
+
const provider = requestedProviders.find((candidate) => candidate.providerInstanceKey === mcpSurface.providerInstanceKey);
|
|
18
|
+
if (!provider?.surfaceKeys.includes(mcpSurface.surfaceKey))
|
|
19
|
+
throw new TypeError("mcpSurface must identify an exactly requested MCP surface");
|
|
20
|
+
const actualAgentManifest = typeof options.actualAgentManifest === "function"
|
|
21
|
+
? options.actualAgentManifest
|
|
22
|
+
: actualAgentManifestV2.parse(options.actualAgentManifest);
|
|
23
|
+
return {
|
|
24
|
+
actualAgentManifest,
|
|
25
|
+
requestedProviders,
|
|
26
|
+
mcpSurface: { ...mcpSurface },
|
|
27
|
+
};
|
|
28
|
+
}
|
|
29
|
+
export function pinRequestedAttemptV2(requested, config) {
|
|
30
|
+
if (!config || typeof config !== "object" || Array.isArray(config))
|
|
31
|
+
throw new TypeError("Provider-profile simulations require an immutable V2 attempt baseline");
|
|
32
|
+
const source = config;
|
|
33
|
+
const baseline = attemptBaselineV2.safeParse(source.attemptBaselineV2);
|
|
34
|
+
if (!baseline.success) {
|
|
35
|
+
if (source.attemptBaselineV2 === undefined && source.attemptBaselineV1 !== undefined)
|
|
36
|
+
throw new TypeError("Legacy V1 attempts require a fresh experiment with a V2 baseline");
|
|
37
|
+
if (source.attemptBaselineV2 !== undefined)
|
|
38
|
+
throw new TypeError("The immutable V2 attempt baseline is invalid");
|
|
39
|
+
throw new TypeError("Provider-profile simulations require an immutable V2 attempt baseline");
|
|
40
|
+
}
|
|
41
|
+
return { ...requested, expectedAgentManifestDigest: baseline.data.expectedAgentManifestDigest };
|
|
42
|
+
}
|
|
43
|
+
/** Confirm a seal from the authoritative run after a lost acknowledgement. */
|
|
44
|
+
async function seal(client, runId, executionId, status) {
|
|
45
|
+
try {
|
|
46
|
+
await client.finishRun(runId, {
|
|
47
|
+
idempotencyKey: `execution:${executionId}:${status}`,
|
|
48
|
+
status,
|
|
49
|
+
});
|
|
50
|
+
}
|
|
51
|
+
catch (error) {
|
|
52
|
+
const recovered = await client.getRun(runId).catch(() => undefined);
|
|
53
|
+
if (recovered?.status !== status)
|
|
54
|
+
throw new TargetOutcomeUncertainError(executionId, { cause: error });
|
|
55
|
+
}
|
|
56
|
+
}
|
|
57
|
+
/** One authoritative environment/provider lifecycle shared by direct simulations and
|
|
58
|
+
* outbound local workers. Credential-bearing connections stay in this call frame and
|
|
59
|
+
* are never returned to either runner's checkpoint state.
|
|
60
|
+
*/
|
|
61
|
+
export async function runEnvironmentTarget(options) {
|
|
62
|
+
const { context } = options;
|
|
63
|
+
const environmentVersionId = context.item.environmentVersionId;
|
|
64
|
+
if (!environmentVersionId)
|
|
65
|
+
throw new Error("The simulation case has no pinned environment version");
|
|
66
|
+
const run = await options.environmentClient.createRun({
|
|
67
|
+
idempotencyKey: `execution:${context.executionId}`,
|
|
68
|
+
environmentVersionId,
|
|
69
|
+
executionId: context.executionId,
|
|
70
|
+
maxSteps: options.maxSteps,
|
|
71
|
+
ttlSeconds: options.ttlSeconds,
|
|
72
|
+
});
|
|
73
|
+
const progress = (event) => options.onProgress?.(event);
|
|
74
|
+
let finalized = false;
|
|
75
|
+
try {
|
|
76
|
+
await progress({ type: "world_created", environmentRunId: run.id });
|
|
77
|
+
if (options.signal?.aborted)
|
|
78
|
+
throw new TargetCancelledError();
|
|
79
|
+
const tools = bindEnvironmentTools({
|
|
80
|
+
hue: options.hue,
|
|
81
|
+
client: options.environmentClient,
|
|
82
|
+
run,
|
|
83
|
+
parentContext: context.span.context,
|
|
84
|
+
});
|
|
85
|
+
let connectionBundle;
|
|
86
|
+
let mcp;
|
|
87
|
+
if (options.requested) {
|
|
88
|
+
const actualManifest = actualAgentManifestV2.parse(typeof options.requested.actualAgentManifest === "function"
|
|
89
|
+
? await options.requested.actualAgentManifest({
|
|
90
|
+
config: structuredClone(context.config),
|
|
91
|
+
item: structuredClone(context.item),
|
|
92
|
+
signal: options.signal,
|
|
93
|
+
})
|
|
94
|
+
: options.requested.actualAgentManifest);
|
|
95
|
+
let prepared;
|
|
96
|
+
try {
|
|
97
|
+
prepared = await options.client.prepareAttempt({
|
|
98
|
+
schemaVersion: 2,
|
|
99
|
+
idempotencyKey: randomUUID(),
|
|
100
|
+
executionId: context.executionId,
|
|
101
|
+
environmentRunId: run.id,
|
|
102
|
+
expectedAgentManifestDigest: options.requested.expectedAgentManifestDigest,
|
|
103
|
+
actualManifest,
|
|
104
|
+
requestedProviders: options.requested.requestedProviders,
|
|
105
|
+
});
|
|
106
|
+
}
|
|
107
|
+
catch (error) {
|
|
108
|
+
// A transport failure or malformed credential-bearing response may follow a
|
|
109
|
+
// committed decision. The runner's running checkpoint prevents reacquisition
|
|
110
|
+
// and target replay on resume.
|
|
111
|
+
throw new TargetOutcomeUncertainError(context.executionId, { cause: error });
|
|
112
|
+
}
|
|
113
|
+
await progress({
|
|
114
|
+
type: "attempt_prepared",
|
|
115
|
+
environmentRunId: run.id,
|
|
116
|
+
bindingId: prepared.status === "ready" ? prepared.bundle.bindingId : prepared.bindingId,
|
|
117
|
+
status: prepared.status,
|
|
118
|
+
findingCodes: prepared.preflightReport.findings.map((finding) => finding.code),
|
|
119
|
+
...(prepared.status === "ready"
|
|
120
|
+
? { executionManifestDigest: prepared.bundle.parity.executionManifestDigest }
|
|
121
|
+
: {}),
|
|
122
|
+
});
|
|
123
|
+
if (prepared.status === "environment_incomplete") {
|
|
124
|
+
await seal(options.environmentClient, run.id, context.executionId, "completed");
|
|
125
|
+
finalized = true;
|
|
126
|
+
await Promise.resolve(progress({ type: "world_sealed", environmentRunId: run.id })).catch(() => undefined);
|
|
127
|
+
return undefined;
|
|
128
|
+
}
|
|
129
|
+
connectionBundle = validateAttemptConnectionBundleV2(prepared.bundle, {
|
|
130
|
+
requireFresh: true,
|
|
131
|
+
});
|
|
132
|
+
const projected = projectMcpConnectionV2(connectionBundle, options.requested.mcpSurface.providerInstanceKey);
|
|
133
|
+
if (!projected)
|
|
134
|
+
throw new TypeError("The prepared attempt has no selected MCP surface");
|
|
135
|
+
mcp = projected;
|
|
136
|
+
}
|
|
137
|
+
else {
|
|
138
|
+
mcp = await options.client.createSimulationMcpCapability({
|
|
139
|
+
runId: run.id,
|
|
140
|
+
executionId: context.executionId,
|
|
141
|
+
});
|
|
142
|
+
}
|
|
143
|
+
if (options.signal?.aborted)
|
|
144
|
+
throw new TargetCancelledError();
|
|
145
|
+
await progress({ type: "target_started", environmentRunId: run.id });
|
|
146
|
+
const output = await options.target(options.inputs, {
|
|
147
|
+
config: context.config,
|
|
148
|
+
item: context.item,
|
|
149
|
+
executionId: context.executionId,
|
|
150
|
+
environmentRunId: run.id,
|
|
151
|
+
trace: { traceId: context.span.traceId, spanId: context.span.spanId },
|
|
152
|
+
tools,
|
|
153
|
+
mcp,
|
|
154
|
+
...(connectionBundle ? { connectionBundle } : {}),
|
|
155
|
+
signal: options.signal,
|
|
156
|
+
});
|
|
157
|
+
await seal(options.environmentClient, run.id, context.executionId, "completed");
|
|
158
|
+
finalized = true;
|
|
159
|
+
await Promise.resolve(progress({ type: "world_sealed", environmentRunId: run.id })).catch(() => undefined);
|
|
160
|
+
return output;
|
|
161
|
+
}
|
|
162
|
+
catch (error) {
|
|
163
|
+
if (error instanceof TargetOutcomeUncertainError || finalized)
|
|
164
|
+
throw error;
|
|
165
|
+
let environmentIncomplete;
|
|
166
|
+
try {
|
|
167
|
+
environmentIncomplete =
|
|
168
|
+
(await options.environmentClient.getRun(run.id)).validity === "environment_incomplete";
|
|
169
|
+
}
|
|
170
|
+
catch (inspectionError) {
|
|
171
|
+
throw new TargetOutcomeUncertainError(context.executionId, {
|
|
172
|
+
cause: new AggregateError([error, inspectionError]),
|
|
173
|
+
});
|
|
174
|
+
}
|
|
175
|
+
if (environmentIncomplete) {
|
|
176
|
+
try {
|
|
177
|
+
await seal(options.environmentClient, run.id, context.executionId, "completed");
|
|
178
|
+
}
|
|
179
|
+
catch (finalizationError) {
|
|
180
|
+
throw new TargetOutcomeUncertainError(context.executionId, {
|
|
181
|
+
cause: new AggregateError([error, finalizationError]),
|
|
182
|
+
});
|
|
183
|
+
}
|
|
184
|
+
finalized = true;
|
|
185
|
+
await Promise.resolve(progress({ type: "world_sealed", environmentRunId: run.id })).catch(() => undefined);
|
|
186
|
+
return undefined;
|
|
187
|
+
}
|
|
188
|
+
try {
|
|
189
|
+
await seal(options.environmentClient, run.id, context.executionId, "abandoned");
|
|
190
|
+
}
|
|
191
|
+
catch (finalizationError) {
|
|
192
|
+
throw new TargetOutcomeUncertainError(context.executionId, {
|
|
193
|
+
cause: new AggregateError([error, finalizationError]),
|
|
194
|
+
});
|
|
195
|
+
}
|
|
196
|
+
await Promise.resolve(progress({ type: "world_sealed", environmentRunId: run.id })).catch(() => undefined);
|
|
197
|
+
if (options.signal?.aborted && !(error instanceof TargetCancelledError))
|
|
198
|
+
throw new TargetCancelledError();
|
|
199
|
+
throw error;
|
|
200
|
+
}
|
|
201
|
+
}
|
|
@@ -0,0 +1,88 @@
|
|
|
1
|
+
import type { HueClient } from "../client.js";
|
|
2
|
+
import type { EnvironmentClient } from "../environment/client.js";
|
|
3
|
+
import type { EnvironmentTool } from "../environment/tools.js";
|
|
4
|
+
import type { ActualAgentManifestInputV2, AttemptConnectionBundleV2, RequestedAttemptProviderV2 } from "./attempt.js";
|
|
5
|
+
import type { EvaluationClient } from "./client.js";
|
|
6
|
+
import { type RunnerReport } from "./runner.js";
|
|
7
|
+
import type { ExperimentCase, JsonValue, LocalAgentRegistration, LocalScorer } from "./types.js";
|
|
8
|
+
/** Candidate-visible context for one queued local agent execution. */
|
|
9
|
+
export interface LocalAgentTargetContext {
|
|
10
|
+
/** Frozen candidate configuration, cloned before invocation. */
|
|
11
|
+
config: JsonValue;
|
|
12
|
+
/** Identity only. Expected outcomes, metadata and world definitions are evaluator-private. */
|
|
13
|
+
item: Pick<ExperimentCase, "id" | "externalKey">;
|
|
14
|
+
/** Identity of this target execution. */
|
|
15
|
+
executionId: string;
|
|
16
|
+
/** Stable world identity for adapter control operations such as coverage reporting. */
|
|
17
|
+
environmentRunId: string;
|
|
18
|
+
/** Trace identities without mutable span or grading data. */
|
|
19
|
+
trace: {
|
|
20
|
+
/** OpenTelemetry trace identifier. */
|
|
21
|
+
traceId: string;
|
|
22
|
+
/** Root execution span identifier. */
|
|
23
|
+
spanId: string;
|
|
24
|
+
};
|
|
25
|
+
/** Short-lived, execution-scoped hosted tools for model providers that execute MCP remotely. */
|
|
26
|
+
mcp: {
|
|
27
|
+
/** Execution-scoped MCP endpoint. */
|
|
28
|
+
url: string;
|
|
29
|
+
/** Short-lived bearer, never the project service key. */
|
|
30
|
+
token: string;
|
|
31
|
+
/** Capability expiry as an ISO timestamp. */
|
|
32
|
+
expiresAt: string;
|
|
33
|
+
};
|
|
34
|
+
/** Credential-bearing provider connections for this callback only. Hue never
|
|
35
|
+
* checkpoints, logs or adds this response to parity digests. */
|
|
36
|
+
connectionBundle?: AttemptConnectionBundleV2;
|
|
37
|
+
}
|
|
38
|
+
/** Fixed local callback, clients and durable queue-worker settings. */
|
|
39
|
+
export interface RunLocalAgentOptions {
|
|
40
|
+
/** Evaluation client for the worker project. */
|
|
41
|
+
client: EvaluationClient;
|
|
42
|
+
/** Environment client for the same origin and project. */
|
|
43
|
+
environmentClient: EnvironmentClient;
|
|
44
|
+
/** Application telemetry client; required trace exports are acknowledged. */
|
|
45
|
+
hue: HueClient;
|
|
46
|
+
/** Private durable directory for worker identity and execution checkpoints. */
|
|
47
|
+
checkpointDirectory: string;
|
|
48
|
+
/** Fixed agent key and revision registered for queued runs. */
|
|
49
|
+
agent: LocalAgentRegistration;
|
|
50
|
+
/** Local scorer bindings matching the published source digests. */
|
|
51
|
+
scorers?: LocalScorer[];
|
|
52
|
+
/** Cases in flight, between 1 and 16; defaults to 1. */
|
|
53
|
+
concurrency?: number;
|
|
54
|
+
/** Polling interval in milliseconds, 250–60000; defaults to 2000. */
|
|
55
|
+
pollIntervalMillis?: number;
|
|
56
|
+
/** Stops polling cooperatively; does not cancel an active callback. */
|
|
57
|
+
signal?: AbortSignal;
|
|
58
|
+
/** Useful for one-shot jobs and deterministic acceptance. Omit to keep polling. */
|
|
59
|
+
maxRuns?: number;
|
|
60
|
+
/** Opt into the experiment's immutable V2 provider profile. These three values are
|
|
61
|
+
* validated together before the worker polls; no endpoint or credential is supplied here. */
|
|
62
|
+
actualAgentManifest?: ActualAgentManifestInputV2 | ((context: {
|
|
63
|
+
/** Frozen experiment configuration. */
|
|
64
|
+
config: JsonValue;
|
|
65
|
+
/** Full frozen case for resolving actual nonsecret evidence before candidate projection. */
|
|
66
|
+
item: ExperimentCase;
|
|
67
|
+
/** Cooperative worker stop signal. */
|
|
68
|
+
signal?: AbortSignal;
|
|
69
|
+
}) => ActualAgentManifestInputV2 | Promise<ActualAgentManifestInputV2>);
|
|
70
|
+
/** Exact provider instances and ordered surfaces asserted for strict preflight. */
|
|
71
|
+
requestedProviders?: RequestedAttemptProviderV2[];
|
|
72
|
+
/** Requested MCP surface projected to the backwards-compatible `context.mcp`. */
|
|
73
|
+
mcpSurface?: {
|
|
74
|
+
/** Provider instance selected from `requestedProviders`. */
|
|
75
|
+
providerInstanceKey: string;
|
|
76
|
+
/** Selected MCP surface. */
|
|
77
|
+
surfaceKey: "google.gmail/mcp" | "slack/mcp";
|
|
78
|
+
};
|
|
79
|
+
/** Invokes the existing agent against isolated tools and candidate-safe context. */
|
|
80
|
+
target(inputs: JsonValue, tools: Record<string, EnvironmentTool>, context: LocalAgentTargetContext): JsonValue | undefined | Promise<JsonValue | undefined>;
|
|
81
|
+
/** Called after the experiment and queue completion are acknowledged. */
|
|
82
|
+
onCompleted?(report: RunnerReport): void | Promise<void>;
|
|
83
|
+
}
|
|
84
|
+
/**
|
|
85
|
+
* Starts an outbound-only worker for one fixed local agent entry point. Hue chooses
|
|
86
|
+
* only the registered key/revision; no command or source is received from the cloud.
|
|
87
|
+
*/
|
|
88
|
+
export declare function runLocalAgent(options: RunLocalAgentOptions): Promise<void>;
|
|
@@ -0,0 +1,171 @@
|
|
|
1
|
+
import { join, resolve } from "node:path";
|
|
2
|
+
import { randomUUID } from "node:crypto";
|
|
3
|
+
import { CheckpointStore } from "./checkpoint.js";
|
|
4
|
+
import { pinRequestedAttemptV2, requestedAttemptV2, runEnvironmentTarget, } from "./environment-target.js";
|
|
5
|
+
import { runExperiment, OutcomeSerializationError, TargetOutcomeUncertainError, UncertainExecutionError, } from "./runner.js";
|
|
6
|
+
/** Allowlist the candidate surface instead of forwarding the generic evaluation context. */
|
|
7
|
+
function localAgentTargetContext(context) {
|
|
8
|
+
return {
|
|
9
|
+
config: structuredClone(context.config),
|
|
10
|
+
item: { id: context.item.id, externalKey: context.item.externalKey },
|
|
11
|
+
executionId: context.executionId,
|
|
12
|
+
environmentRunId: context.environmentRunId,
|
|
13
|
+
trace: { traceId: context.trace.traceId, spanId: context.trace.spanId },
|
|
14
|
+
mcp: {
|
|
15
|
+
url: context.mcp.url,
|
|
16
|
+
token: context.mcp.token,
|
|
17
|
+
expiresAt: context.mcp.expiresAt,
|
|
18
|
+
},
|
|
19
|
+
...(context.connectionBundle
|
|
20
|
+
? { connectionBundle: structuredClone(context.connectionBundle) }
|
|
21
|
+
: {}),
|
|
22
|
+
};
|
|
23
|
+
}
|
|
24
|
+
function validInterval(value) {
|
|
25
|
+
const interval = value ?? 2_000;
|
|
26
|
+
if (!Number.isInteger(interval) || interval < 250 || interval > 60_000)
|
|
27
|
+
throw new RangeError("pollIntervalMillis must be 250–60000");
|
|
28
|
+
return interval;
|
|
29
|
+
}
|
|
30
|
+
function stopReason(signal) {
|
|
31
|
+
return signal?.reason instanceof Error
|
|
32
|
+
? signal.reason
|
|
33
|
+
: new Error("Local worker stopped", { cause: signal?.reason });
|
|
34
|
+
}
|
|
35
|
+
function wait(milliseconds, signal) {
|
|
36
|
+
if (signal?.aborted)
|
|
37
|
+
return Promise.reject(stopReason(signal));
|
|
38
|
+
return new Promise((resolve, reject) => {
|
|
39
|
+
const timeout = setTimeout(done, milliseconds);
|
|
40
|
+
const aborted = () => {
|
|
41
|
+
clearTimeout(timeout);
|
|
42
|
+
reject(stopReason(signal));
|
|
43
|
+
};
|
|
44
|
+
function done() {
|
|
45
|
+
signal?.removeEventListener("abort", aborted);
|
|
46
|
+
resolve();
|
|
47
|
+
}
|
|
48
|
+
signal?.addEventListener("abort", aborted, { once: true });
|
|
49
|
+
});
|
|
50
|
+
}
|
|
51
|
+
function needsAttention(error, seen = new Set()) {
|
|
52
|
+
if (error instanceof TargetOutcomeUncertainError ||
|
|
53
|
+
error instanceof UncertainExecutionError ||
|
|
54
|
+
error instanceof OutcomeSerializationError)
|
|
55
|
+
return true;
|
|
56
|
+
if (!(error instanceof AggregateError) || seen.has(error))
|
|
57
|
+
return false;
|
|
58
|
+
seen.add(error);
|
|
59
|
+
return error.errors.some((nested) => needsAttention(nested, seen));
|
|
60
|
+
}
|
|
61
|
+
/**
|
|
62
|
+
* Starts an outbound-only worker for one fixed local agent entry point. Hue chooses
|
|
63
|
+
* only the registered key/revision; no command or source is received from the cloud.
|
|
64
|
+
*/
|
|
65
|
+
export async function runLocalAgent(options) {
|
|
66
|
+
const requestedConfiguration = requestedAttemptV2(options);
|
|
67
|
+
const interval = validInterval(options.pollIntervalMillis);
|
|
68
|
+
const maxRuns = options.maxRuns ?? Number.POSITIVE_INFINITY;
|
|
69
|
+
if (!(maxRuns === Number.POSITIVE_INFINITY || (Number.isInteger(maxRuns) && maxRuns > 0)))
|
|
70
|
+
throw new RangeError("maxRuns must be a positive integer");
|
|
71
|
+
if (options.environmentClient.baseUrl !== options.client.baseUrl)
|
|
72
|
+
throw new Error("Environments and evaluations must use the same Hue origin");
|
|
73
|
+
const directory = resolve(options.checkpointDirectory);
|
|
74
|
+
const project = await options.client.checkConnection();
|
|
75
|
+
const store = await CheckpointStore.acquire(directory, {
|
|
76
|
+
kind: "local-agent-worker",
|
|
77
|
+
projectId: project.id,
|
|
78
|
+
baseUrl: options.client.baseUrl,
|
|
79
|
+
agent: { key: options.agent.key, revision: options.agent.revision },
|
|
80
|
+
});
|
|
81
|
+
try {
|
|
82
|
+
let workerId = await store.read("worker-id");
|
|
83
|
+
if (workerId !== undefined &&
|
|
84
|
+
!/^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/.test(workerId))
|
|
85
|
+
throw new Error("Invalid local worker identity");
|
|
86
|
+
if (workerId === undefined) {
|
|
87
|
+
workerId = randomUUID();
|
|
88
|
+
await store.write("worker-id", workerId);
|
|
89
|
+
}
|
|
90
|
+
let completed = 0;
|
|
91
|
+
while (completed < maxRuns && !options.signal?.aborted) {
|
|
92
|
+
const agent = await options.client.registerLocalAgent({
|
|
93
|
+
...options.agent,
|
|
94
|
+
capabilities: options.agent.capabilities ?? ["environment:v1"],
|
|
95
|
+
scorerDigests: options.agent.scorerDigests ??
|
|
96
|
+
options.scorers?.map((item) => item.definition.sourceDigest) ??
|
|
97
|
+
[],
|
|
98
|
+
});
|
|
99
|
+
const claim = await options.client.claimLocalAgentRun({ agentId: agent.id, workerId });
|
|
100
|
+
if (!claim) {
|
|
101
|
+
await wait(interval, options.signal);
|
|
102
|
+
continue;
|
|
103
|
+
}
|
|
104
|
+
const heartbeat = setInterval(() => {
|
|
105
|
+
void options.client
|
|
106
|
+
.heartbeatLocalAgentRun({ runId: claim.runId, workerId })
|
|
107
|
+
.catch(() => undefined);
|
|
108
|
+
}, Math.min(15_000, Math.max(1_000, interval)));
|
|
109
|
+
let experimentFinished = false;
|
|
110
|
+
try {
|
|
111
|
+
const requested = requestedConfiguration
|
|
112
|
+
? pinRequestedAttemptV2(requestedConfiguration, (await options.client.getExperiment(claim.experimentId)).config)
|
|
113
|
+
: undefined;
|
|
114
|
+
const report = await runExperiment({
|
|
115
|
+
client: options.client,
|
|
116
|
+
hue: options.hue,
|
|
117
|
+
experimentId: claim.experimentId,
|
|
118
|
+
checkpointDirectory: join(directory, `experiment-${claim.experimentId}`),
|
|
119
|
+
persistResultContent: true,
|
|
120
|
+
environmentEvidence: "required",
|
|
121
|
+
traceEvidence: { mode: "required" },
|
|
122
|
+
scorers: options.scorers,
|
|
123
|
+
concurrency: options.concurrency,
|
|
124
|
+
target: (inputs, context) => runEnvironmentTarget({
|
|
125
|
+
client: options.client,
|
|
126
|
+
environmentClient: options.environmentClient,
|
|
127
|
+
hue: options.hue,
|
|
128
|
+
inputs,
|
|
129
|
+
context,
|
|
130
|
+
requested,
|
|
131
|
+
signal: options.signal,
|
|
132
|
+
target: (targetInputs, targetContext) => options.target(structuredClone(targetInputs), targetContext.tools, localAgentTargetContext(targetContext)),
|
|
133
|
+
}),
|
|
134
|
+
});
|
|
135
|
+
// From this point onward the experiment outcome is authoritative. If reporting the
|
|
136
|
+
// queue completion fails, leave the claim intact for checkpointed recovery instead of
|
|
137
|
+
// rewriting a successful experiment as an agent failure.
|
|
138
|
+
experimentFinished = true;
|
|
139
|
+
await options.client.completeLocalAgentRun({
|
|
140
|
+
runId: claim.runId,
|
|
141
|
+
workerId,
|
|
142
|
+
state: "completed",
|
|
143
|
+
});
|
|
144
|
+
await options.onCompleted?.(report);
|
|
145
|
+
completed++;
|
|
146
|
+
}
|
|
147
|
+
catch (error) {
|
|
148
|
+
// A durable outcome can still need completion/result uploads. Leave operational
|
|
149
|
+
// failures claimed so the same worker can resume them through its checkpoints.
|
|
150
|
+
// Attention is terminal in the queue and is reserved for explicit unsafe-to-resume
|
|
151
|
+
// outcomes that require operator intervention.
|
|
152
|
+
if (!experimentFinished && needsAttention(error))
|
|
153
|
+
await options.client
|
|
154
|
+
.completeLocalAgentRun({
|
|
155
|
+
runId: claim.runId,
|
|
156
|
+
workerId,
|
|
157
|
+
state: "attention",
|
|
158
|
+
failureType: error instanceof Error ? error.name.slice(0, 200) : "WorkerError",
|
|
159
|
+
})
|
|
160
|
+
.catch(() => undefined);
|
|
161
|
+
throw error;
|
|
162
|
+
}
|
|
163
|
+
finally {
|
|
164
|
+
clearInterval(heartbeat);
|
|
165
|
+
}
|
|
166
|
+
}
|
|
167
|
+
}
|
|
168
|
+
finally {
|
|
169
|
+
await store.release();
|
|
170
|
+
}
|
|
171
|
+
}
|
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
import type { HueClient } from "../client.js";
|
|
2
2
|
import { type EnvironmentClient } from "../environment/client.js";
|
|
3
|
-
import {
|
|
4
|
-
import type {
|
|
3
|
+
import type { EnvironmentTool } from "../environment/tools.js";
|
|
4
|
+
import type { EnvironmentIdentity, PublishableEnvironmentDefinition } from "../environment/types.js";
|
|
5
5
|
import { type EvaluationClient } from "./client.js";
|
|
6
|
-
import {
|
|
6
|
+
import type { ActualAgentManifestInputV2, AttemptConnectionBundleV2, RequestedAttemptProviderV2 } from "./attempt.js";
|
|
7
7
|
import { type RunnerReport } from "./runner.js";
|
|
8
8
|
import type { ExperimentCase, Identity, JsonValue, LocalScorer, ScorerDefinition, SimulationMcpCapability } from "./types.js";
|
|
9
9
|
/** Repository-authored scorer identity and its public definition or local binding. */
|
|
@@ -40,7 +40,7 @@ export type SimulationScenario = {
|
|
|
40
40
|
/** Environment identity and authored world definition. */
|
|
41
41
|
environment: EnvironmentIdentity & {
|
|
42
42
|
/** Definition normalized and published as an immutable version. */
|
|
43
|
-
definition:
|
|
43
|
+
definition: PublishableEnvironmentDefinition;
|
|
44
44
|
};
|
|
45
45
|
/** Cases published into one frozen dataset version. */
|
|
46
46
|
cases: RepositorySimulationCase[];
|
|
@@ -92,8 +92,8 @@ export type SimulationProgress = {
|
|
|
92
92
|
export interface SimulationTargetContext {
|
|
93
93
|
/** Frozen experiment configuration. */
|
|
94
94
|
config: JsonValue;
|
|
95
|
-
/**
|
|
96
|
-
item: ExperimentCase
|
|
95
|
+
/** Candidate-visible identity. References, metadata and source pins stay with grading. */
|
|
96
|
+
item: Pick<ExperimentCase, "id" | "externalKey">;
|
|
97
97
|
/** Target execution identity. */
|
|
98
98
|
executionId: string;
|
|
99
99
|
/** Stable world identity for adapter control operations such as coverage reporting. */
|
package/dist/evals/simulation.js
CHANGED
|
@@ -1,56 +1,25 @@
|
|
|
1
1
|
import { randomUUID } from "node:crypto";
|
|
2
2
|
import { join } from "node:path";
|
|
3
3
|
import { HueEnvironmentError } from "../environment/client.js";
|
|
4
|
-
import { bindEnvironmentTools } from "../environment/tools.js";
|
|
5
4
|
import { HueApiError } from "./client.js";
|
|
6
5
|
import { CheckpointStore } from "./checkpoint.js";
|
|
7
6
|
import { MAX_ENVIRONMENT_STEPS } from "./environment-evidence.js";
|
|
8
7
|
import { aggregateBounds, digest, json } from "./json.js";
|
|
9
8
|
import { normalizeScorerDefinitionForPublication } from "./scorer-publication.js";
|
|
10
|
-
import {
|
|
11
|
-
import { runExperiment
|
|
9
|
+
import { pinRequestedAttemptV2, requestedAttemptV2, runEnvironmentTarget, } from "./environment-target.js";
|
|
10
|
+
import { runExperiment } from "./runner.js";
|
|
12
11
|
const scorerDefinition = (entry) => "definition" in entry.scorer ? entry.scorer.definition : entry.scorer;
|
|
13
|
-
function requestedAttempt(options) {
|
|
14
|
-
const requested = options.requestedProviders !== undefined;
|
|
15
|
-
const selected = options.mcpSurface !== undefined;
|
|
16
|
-
if (!requested && !selected) {
|
|
17
|
-
if (options.actualAgentManifest !== undefined)
|
|
18
|
-
throw new TypeError("actualAgentManifest requires requestedProviders and mcpSurface");
|
|
19
|
-
return undefined;
|
|
20
|
-
}
|
|
21
|
-
if (!requested || !selected)
|
|
22
|
-
throw new TypeError("requestedProviders and mcpSurface must be supplied together");
|
|
23
|
-
const requestedProviders = requestedAttemptProvidersV2.parse(options.requestedProviders);
|
|
24
|
-
const mcpSurface = options.mcpSurface;
|
|
25
|
-
const provider = requestedProviders.find((candidate) => candidate.providerInstanceKey === mcpSurface.providerInstanceKey);
|
|
26
|
-
if (!provider?.surfaceKeys.includes(mcpSurface.surfaceKey))
|
|
27
|
-
throw new TypeError("mcpSurface must identify an exactly requested MCP surface");
|
|
28
|
-
const actualAgentManifest = typeof options.actualAgentManifest === "function"
|
|
29
|
-
? options.actualAgentManifest
|
|
30
|
-
: actualAgentManifestV2.parse(options.actualAgentManifest);
|
|
31
|
-
return {
|
|
32
|
-
actualAgentManifest,
|
|
33
|
-
requestedProviders,
|
|
34
|
-
mcpSurface: { ...mcpSurface },
|
|
35
|
-
};
|
|
36
|
-
}
|
|
37
|
-
function expectedManifestDigest(config) {
|
|
38
|
-
if (!config || typeof config !== "object" || Array.isArray(config))
|
|
39
|
-
throw new TypeError("Provider-profile simulations require an immutable V2 attempt baseline");
|
|
40
|
-
const source = config;
|
|
41
|
-
const baseline = attemptBaselineV2.safeParse(source.attemptBaselineV2);
|
|
42
|
-
if (!baseline.success) {
|
|
43
|
-
if (source.attemptBaselineV2 === undefined && source.attemptBaselineV1 !== undefined)
|
|
44
|
-
throw new TypeError("Legacy V1 attempts require a fresh experiment with a V2 baseline");
|
|
45
|
-
if (source.attemptBaselineV2 !== undefined)
|
|
46
|
-
throw new TypeError("The immutable V2 attempt baseline is invalid");
|
|
47
|
-
throw new TypeError("Provider-profile simulations require an immutable V2 attempt baseline");
|
|
48
|
-
}
|
|
49
|
-
return baseline.data.expectedAgentManifestDigest;
|
|
50
|
-
}
|
|
51
12
|
function normalizedEnvironmentDefinition(definition) {
|
|
52
13
|
return json({
|
|
53
14
|
...definition,
|
|
15
|
+
...(definition.schemaVersion === 2
|
|
16
|
+
? {
|
|
17
|
+
providerInstances: definition.providerInstances.map((instance) => ({
|
|
18
|
+
...instance,
|
|
19
|
+
syntheticPrincipalId: instance.syntheticPrincipalId.toLowerCase(),
|
|
20
|
+
})),
|
|
21
|
+
}
|
|
22
|
+
: {}),
|
|
54
23
|
determinism: {
|
|
55
24
|
clock: {
|
|
56
25
|
startNs: definition.determinism?.clock?.startNs ?? "0",
|
|
@@ -341,24 +310,11 @@ async function resolveExperiment(options, idempotencyKey) {
|
|
|
341
310
|
bindings: [...scorers.bindings, ...(options.localScorers ?? [])],
|
|
342
311
|
};
|
|
343
312
|
}
|
|
344
|
-
async function seal(client, runId, executionId, status) {
|
|
345
|
-
try {
|
|
346
|
-
await client.finishRun(runId, {
|
|
347
|
-
idempotencyKey: `execution:${executionId}:${status}`,
|
|
348
|
-
status,
|
|
349
|
-
});
|
|
350
|
-
}
|
|
351
|
-
catch (error) {
|
|
352
|
-
const recovered = await client.getRun(runId).catch(() => undefined);
|
|
353
|
-
if (recovered?.status !== status)
|
|
354
|
-
throw error;
|
|
355
|
-
}
|
|
356
|
-
}
|
|
357
313
|
/** Run an existing agent callback against one fresh hosted world per case. The helper
|
|
358
314
|
* owns immutable resolution, execution linkage, finalization, scoring and resumable uploads.
|
|
359
315
|
*/
|
|
360
316
|
export async function runSimulation(options) {
|
|
361
|
-
const requestedConfiguration =
|
|
317
|
+
const requestedConfiguration = requestedAttemptV2(options);
|
|
362
318
|
if (options.maxSteps !== undefined &&
|
|
363
319
|
(!Number.isInteger(options.maxSteps) ||
|
|
364
320
|
options.maxSteps < 1 ||
|
|
@@ -408,10 +364,7 @@ export async function runSimulation(options) {
|
|
|
408
364
|
const runUrl = new URL(`/experiments/${experimentId}`, options.client.baseUrl).toString();
|
|
409
365
|
await options.onProgress?.({ type: "run_created", experimentId, runUrl });
|
|
410
366
|
const requested = requestedConfiguration
|
|
411
|
-
?
|
|
412
|
-
...requestedConfiguration,
|
|
413
|
-
expectedAgentManifestDigest: expectedManifestDigest((await options.client.getExperiment(experimentId)).config),
|
|
414
|
-
}
|
|
367
|
+
? pinRequestedAttemptV2(requestedConfiguration, (await options.client.getExperiment(experimentId)).config)
|
|
415
368
|
: undefined;
|
|
416
369
|
const report = await runExperiment({
|
|
417
370
|
client: options.client,
|
|
@@ -424,172 +377,42 @@ export async function runSimulation(options) {
|
|
|
424
377
|
scorers: bindings,
|
|
425
378
|
concurrency: options.concurrency,
|
|
426
379
|
schemaTimeoutMillis: options.schemaTimeoutMillis,
|
|
427
|
-
target:
|
|
428
|
-
|
|
429
|
-
|
|
430
|
-
|
|
431
|
-
|
|
432
|
-
|
|
433
|
-
|
|
434
|
-
|
|
435
|
-
|
|
436
|
-
|
|
437
|
-
|
|
438
|
-
|
|
439
|
-
type,
|
|
380
|
+
target: (inputs, context) => runEnvironmentTarget({
|
|
381
|
+
client: options.client,
|
|
382
|
+
environmentClient: options.environmentClient,
|
|
383
|
+
hue: options.hue,
|
|
384
|
+
inputs,
|
|
385
|
+
context,
|
|
386
|
+
requested,
|
|
387
|
+
maxSteps: options.maxSteps,
|
|
388
|
+
ttlSeconds: options.ttlSeconds,
|
|
389
|
+
signal: options.signal,
|
|
390
|
+
onProgress: (event) => options.onProgress?.({
|
|
391
|
+
...event,
|
|
440
392
|
experimentId,
|
|
441
393
|
executionId: context.executionId,
|
|
442
394
|
caseId: context.item.id,
|
|
443
|
-
|
|
444
|
-
|
|
445
|
-
|
|
446
|
-
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
|
|
450
|
-
|
|
451
|
-
|
|
452
|
-
|
|
453
|
-
|
|
454
|
-
|
|
455
|
-
|
|
456
|
-
|
|
457
|
-
|
|
458
|
-
|
|
459
|
-
|
|
460
|
-
|
|
461
|
-
|
|
462
|
-
|
|
463
|
-
|
|
464
|
-
})
|
|
465
|
-
: requested.actualAgentManifest);
|
|
466
|
-
let prepared;
|
|
467
|
-
try {
|
|
468
|
-
prepared = await options.client.prepareAttempt({
|
|
469
|
-
schemaVersion: 2,
|
|
470
|
-
idempotencyKey: randomUUID(),
|
|
471
|
-
executionId: context.executionId,
|
|
472
|
-
environmentRunId: run.id,
|
|
473
|
-
expectedAgentManifestDigest: requested.expectedAgentManifestDigest,
|
|
474
|
-
actualManifest,
|
|
475
|
-
requestedProviders: requested.requestedProviders,
|
|
476
|
-
});
|
|
477
|
-
}
|
|
478
|
-
catch (error) {
|
|
479
|
-
// A transport failure or malformed credential-bearing response may
|
|
480
|
-
// follow a committed decision. Preserve the running checkpoint and
|
|
481
|
-
// never reacquire credentials or replay the target on resume.
|
|
482
|
-
throw new TargetOutcomeUncertainError(context.executionId, { cause: error });
|
|
483
|
-
}
|
|
484
|
-
await options.onProgress?.({
|
|
485
|
-
type: "attempt_prepared",
|
|
486
|
-
experimentId,
|
|
487
|
-
executionId: context.executionId,
|
|
488
|
-
caseId: context.item.id,
|
|
489
|
-
environmentRunId: run.id,
|
|
490
|
-
bindingId: prepared.status === "ready" ? prepared.bundle.bindingId : prepared.bindingId,
|
|
491
|
-
status: prepared.status,
|
|
492
|
-
findingCodes: prepared.preflightReport.findings.map((finding) => finding.code),
|
|
493
|
-
...(prepared.status === "ready"
|
|
494
|
-
? {
|
|
495
|
-
executionManifestDigest: prepared.bundle.parity.executionManifestDigest,
|
|
496
|
-
}
|
|
497
|
-
: {}),
|
|
498
|
-
});
|
|
499
|
-
if (prepared.status === "environment_incomplete") {
|
|
500
|
-
try {
|
|
501
|
-
await seal(options.environmentClient, run.id, context.executionId, "completed");
|
|
502
|
-
}
|
|
503
|
-
catch (error) {
|
|
504
|
-
throw new TargetOutcomeUncertainError(context.executionId, { cause: error });
|
|
505
|
-
}
|
|
506
|
-
finalized = true;
|
|
507
|
-
await Promise.resolve(progress("world_sealed")).catch(() => undefined);
|
|
508
|
-
return undefined;
|
|
509
|
-
}
|
|
510
|
-
connectionBundle = validateAttemptConnectionBundleV2(prepared.bundle, {
|
|
511
|
-
requireFresh: true,
|
|
512
|
-
});
|
|
513
|
-
const projected = projectMcpConnectionV2(connectionBundle, requested.mcpSurface.providerInstanceKey);
|
|
514
|
-
if (!projected)
|
|
515
|
-
throw new TypeError("The prepared attempt has no selected MCP surface");
|
|
516
|
-
mcp = projected;
|
|
517
|
-
}
|
|
518
|
-
else {
|
|
519
|
-
mcp = await options.client.createSimulationMcpCapability({
|
|
520
|
-
runId: run.id,
|
|
521
|
-
executionId: context.executionId,
|
|
522
|
-
});
|
|
523
|
-
}
|
|
524
|
-
if (options.signal?.aborted)
|
|
525
|
-
throw new TargetCancelledError();
|
|
526
|
-
await progress("target_started");
|
|
527
|
-
const output = await options.target(inputs, {
|
|
528
|
-
config: context.config,
|
|
529
|
-
item: context.item,
|
|
530
|
-
executionId: context.executionId,
|
|
531
|
-
environmentRunId: run.id,
|
|
532
|
-
tools,
|
|
533
|
-
mcp,
|
|
534
|
-
...(connectionBundle ? { connectionBundle } : {}),
|
|
535
|
-
signal: options.signal,
|
|
536
|
-
});
|
|
537
|
-
try {
|
|
538
|
-
await seal(options.environmentClient, run.id, context.executionId, "completed");
|
|
539
|
-
}
|
|
540
|
-
catch (error) {
|
|
541
|
-
throw new TargetOutcomeUncertainError(context.executionId, { cause: error });
|
|
542
|
-
}
|
|
543
|
-
finalized = true;
|
|
544
|
-
await Promise.resolve(progress("world_sealed")).catch(() => undefined);
|
|
545
|
-
return output;
|
|
546
|
-
}
|
|
547
|
-
catch (error) {
|
|
548
|
-
if (error instanceof TargetOutcomeUncertainError || finalized)
|
|
549
|
-
throw error;
|
|
550
|
-
let environmentIncomplete;
|
|
551
|
-
try {
|
|
552
|
-
environmentIncomplete =
|
|
553
|
-
(await options.environmentClient.getRun(run.id)).validity ===
|
|
554
|
-
"environment_incomplete";
|
|
555
|
-
}
|
|
556
|
-
catch (inspectionError) {
|
|
557
|
-
// A target error can be the adapter surfacing a coverage gap. If the
|
|
558
|
-
// authoritative run cannot be read, do not guess that it was an agent
|
|
559
|
-
// failure or replay the target on resume.
|
|
560
|
-
throw new TargetOutcomeUncertainError(context.executionId, {
|
|
561
|
-
cause: new AggregateError([error, inspectionError]),
|
|
562
|
-
});
|
|
563
|
-
}
|
|
564
|
-
// A durable coverage gap invalidates parity independently of caller timing;
|
|
565
|
-
// do not let a racing local abort hide it as an ordinary cancellation.
|
|
566
|
-
if (environmentIncomplete) {
|
|
567
|
-
try {
|
|
568
|
-
await seal(options.environmentClient, run.id, context.executionId, "completed");
|
|
569
|
-
}
|
|
570
|
-
catch (finalizationError) {
|
|
571
|
-
throw new TargetOutcomeUncertainError(context.executionId, {
|
|
572
|
-
cause: new AggregateError([error, finalizationError]),
|
|
573
|
-
});
|
|
574
|
-
}
|
|
575
|
-
finalized = true;
|
|
576
|
-
await Promise.resolve(progress("world_sealed")).catch(() => undefined);
|
|
577
|
-
return undefined;
|
|
578
|
-
}
|
|
579
|
-
try {
|
|
580
|
-
await seal(options.environmentClient, run.id, context.executionId, "abandoned");
|
|
581
|
-
}
|
|
582
|
-
catch (finalizationError) {
|
|
583
|
-
throw new TargetOutcomeUncertainError(context.executionId, {
|
|
584
|
-
cause: new AggregateError([error, finalizationError]),
|
|
585
|
-
});
|
|
586
|
-
}
|
|
587
|
-
await Promise.resolve(progress("world_sealed")).catch(() => undefined);
|
|
588
|
-
if (options.signal?.aborted && !(error instanceof TargetCancelledError))
|
|
589
|
-
throw new TargetCancelledError();
|
|
590
|
-
throw error;
|
|
591
|
-
}
|
|
592
|
-
},
|
|
395
|
+
}),
|
|
396
|
+
target: (targetInputs, targetContext) => options.target(structuredClone(targetInputs), {
|
|
397
|
+
config: structuredClone(targetContext.config),
|
|
398
|
+
item: {
|
|
399
|
+
id: targetContext.item.id,
|
|
400
|
+
externalKey: targetContext.item.externalKey,
|
|
401
|
+
},
|
|
402
|
+
executionId: targetContext.executionId,
|
|
403
|
+
environmentRunId: targetContext.environmentRunId,
|
|
404
|
+
tools: targetContext.tools,
|
|
405
|
+
mcp: {
|
|
406
|
+
url: targetContext.mcp.url,
|
|
407
|
+
token: targetContext.mcp.token,
|
|
408
|
+
expiresAt: targetContext.mcp.expiresAt,
|
|
409
|
+
},
|
|
410
|
+
...(targetContext.connectionBundle
|
|
411
|
+
? { connectionBundle: structuredClone(targetContext.connectionBundle) }
|
|
412
|
+
: {}),
|
|
413
|
+
signal: targetContext.signal,
|
|
414
|
+
}),
|
|
415
|
+
}),
|
|
593
416
|
});
|
|
594
417
|
const complete = { ...report, experimentId, runUrl };
|
|
595
418
|
attempt.stage = "completed";
|
package/dist/evals/types.d.ts
CHANGED
|
@@ -603,3 +603,34 @@ export interface SimulationMcpCapability {
|
|
|
603
603
|
/** Credential expiry timestamp. */
|
|
604
604
|
expiresAt: string;
|
|
605
605
|
}
|
|
606
|
+
/** Identity and capabilities of one fixed local agent entry point. */
|
|
607
|
+
export interface LocalAgentRegistration {
|
|
608
|
+
/** Stable application-selected agent key. */
|
|
609
|
+
key: string;
|
|
610
|
+
/** Display name. */
|
|
611
|
+
name: string;
|
|
612
|
+
/** Application-selected revision of the agent configuration. */
|
|
613
|
+
revision: string;
|
|
614
|
+
/** Supported execution contracts; defaults to environment:v1 in the worker. */
|
|
615
|
+
capabilities?: string[];
|
|
616
|
+
/** Local scorer source digests available in this process. */
|
|
617
|
+
scorerDigests?: string[];
|
|
618
|
+
}
|
|
619
|
+
/** Server registration and heartbeat timestamps for a local agent. */
|
|
620
|
+
export interface RegisteredLocalAgent extends Required<LocalAgentRegistration> {
|
|
621
|
+
/** Registered agent identity. */
|
|
622
|
+
id: string;
|
|
623
|
+
/** Whether Hue permits this registration to receive runs. */
|
|
624
|
+
enabled: boolean;
|
|
625
|
+
/** Latest registration heartbeat, as an ISO timestamp. */
|
|
626
|
+
lastSeenAt: string;
|
|
627
|
+
/** Registration creation timestamp. */
|
|
628
|
+
createdAt: string;
|
|
629
|
+
}
|
|
630
|
+
/** Queue claim connecting a local run to a pinned experiment. */
|
|
631
|
+
export interface LocalAgentClaim {
|
|
632
|
+
/** Claimed queue-run identity. */
|
|
633
|
+
runId: string;
|
|
634
|
+
/** Pinned experiment to execute. */
|
|
635
|
+
experimentId: string;
|
|
636
|
+
}
|
package/dist/evals.d.ts
CHANGED
|
@@ -9,3 +9,5 @@ export type { ActualAgentManifestInputV2, ActualAgentManifestV2, AttemptBaseline
|
|
|
9
9
|
export { builtins, defineLocalScorer, scoreLocally } from "./evals/scorers.js";
|
|
10
10
|
export { sourceDigest } from "./evals/json.js";
|
|
11
11
|
export type * from "./evals/types.js";
|
|
12
|
+
export { runLocalAgent } from "./evals/local-worker.js";
|
|
13
|
+
export type { LocalAgentTargetContext, RunLocalAgentOptions } from "./evals/local-worker.js";
|
package/dist/evals.js
CHANGED
|
@@ -4,3 +4,4 @@ export { runSimulation } from "./evals/simulation.js";
|
|
|
4
4
|
export { actualAgentManifestV2, agentManifestDigestV2, attemptBaselineV2, attemptBindingRead, attemptConnectionBundleV2, attemptIdentityV2, dependencyManifestV2, dependencyProviderV2, expectedAgentManifestV2, executionManifestDigestV2, parityEvidenceV2, preflightFindingV2, preflightReportV2, prepareAttemptInputV2, projectMcpConnectionV2, secretFreeBindingV2, surfaceBindingV2, } from "./evals/attempt.js";
|
|
5
5
|
export { builtins, defineLocalScorer, scoreLocally } from "./evals/scorers.js";
|
|
6
6
|
export { sourceDigest } from "./evals/json.js";
|
|
7
|
+
export { runLocalAgent } from "./evals/local-worker.js";
|
package/dist/version.d.ts
CHANGED
|
@@ -1,2 +1,2 @@
|
|
|
1
1
|
/** Package version shared by the instrumentation scope and the export User-Agent. */
|
|
2
|
-
export declare const sdkVersion = "0.
|
|
2
|
+
export declare const sdkVersion = "0.3.0";
|
package/dist/version.js
CHANGED