@hue-run/sdk 0.9.0 → 0.11.0

package/EVALUATIONS.md

@@ -18,7 +18,8 @@ import { builtins, createEvaluationClient, runExperiment, rescore } from "@hue-r
 
 const connection = { apiKey: process.env.HUE_API_KEY! };
 const client = createEvaluationClient(connection);
-const hue = createHue({ ...connection, serviceName: "evaluation-demo", captureContent: false });
+// captureContent: false sends metadata only.
+const hue = createHue({ ...connection, serviceName: "evaluation-demo", captureContent: true });
 
 // Registry writes use optimistic revisions; they do not have automatic retries.
 const dataset = await client.createDataset({ name: "Greetings", slug: "greetings" });
@@ -115,7 +116,7 @@ Both choices are required and independent:
 
 Frozen dataset inputs/references already exist on Hue. Their presence is independent of these switches. No output is inferred from telemetry. JavaScript `undefined` means unavailable output; JSON `null`, false, zero and empty string remain present values. Historical subjects with unavailable output produce skipped results without executing a scorer callback. A failed quality metric remains `state:"scored"` with `passed:false`. Target errors and scorer errors remain separate. Target error types are generalized to `TargetError`; when content is enabled, bounded messages may be stored. Never put credentials in error messages, output, custom metric values, names or metadata.
 
-`traceEvidence:{mode:"required"}` waits for trace and log export acknowledgement after the root span ends. The explicit alternative `{mode:"omit",reason:"..."}` stores the omission reason and declared trace ID without a fake snapshot. It can complete despite an export failure; export diagnostics remain available on the Hue client. There is no automatic fallback to omission.
+`traceEvidence:{mode:"required"}` waits for trace and log export acknowledgement after the root span ends. The explicit alternative `{mode:"omit",reason:"..."}` stores the omission reason and declared trace ID without a fake snapshot. It can complete despite an export failure; export diagnostics remain available on the Hue client. There is no automatic fallback to omission. When required telemetry is not accepted, the runner by default keeps the saved outcome, leaves the execution started and rejects (`traceNotAccepted: "stop"`). With `traceNotAccepted: "fail_case"` it instead completes that case as failed (a `TelemetryNotAccepted` error, the evidence omitted with a reason starting `telemetry_not_accepted`, and no output or generated files attached, so no scorer can pass it), calls `onTelemetryNotAccepted(entry)` once, as soon as it completes the case (a resumed call only lists it), lists it in `report.telemetryNotAccepted` with sanitized issue counts, and goes on; `hue eval` uses this. A scorer that grades only the world can still score such a case, so treat a listed case as failed whatever its results say, as `hue eval` does.
 
 ## Local scorers
 
@@ -158,7 +159,7 @@ built-ins keep their usual missing-output behavior. See
 
 Scorer deferral shipped in TypeScript `0.3.1`: the local runner executes only the three known built-in entries and bound `local_code` scorers. It leaves every other pin pending and reports its ID in `deferredScorerVersionIds`, including kinds and built-in entries introduced by a newer server. It never uploads a placeholder result that would occupy the immutable result slot, including placeholders already saved in an older SDK's checkpoint. Direct `scoreLocally()` calls reject pins that require another executor.
 
-`world_outcome` pins run inside Hue and need no local callback or executable source digest. A supporting server owns their execution from saved world evidence. Legacy `local_code` pins still require the exact registered callback; changing the worker cannot convert those immutable pins into hosted ones.
+`world_outcome` pins (`hue.conversion_outcome.v1` and `.v2`, and the `hue.outcome_assertions.v2` and `.v3` case-outcome evaluators, `.v3` with its pinned judge in `config.judge`) run inside Hue and need no local callback or executable source digest. A supporting server owns their execution from saved world evidence. Legacy `local_code` pins still require the exact registered callback; changing the worker cannot convert those immutable pins into hosted ones.
 
 Manual results require a human session. Hosted model-judge dispatch is an explicit separate API operation: inspect `getJudgeBudget()`, then call `createJudgeJobs(runId,{idempotencyKey,jobs:[{evaluationItemId,scorerVersionId}]})`. `listJudgeJobs`, `getJudgeJob` and `cancelJudgeJob` expose job progress and cancellation requests. These methods never claim that local execution has hosted provenance. Hosted job endpoints are covered by HTTP contract tests here; live hosted model execution is a separate platform acceptance phase. `listResults` and `getResult` read recorded local or hosted results.
 
@@ -182,7 +183,7 @@ An exclusive `.lock` prevents two processes from invoking targets through the sa
 
 The runner saves a starting marker before `start`, and a running marker before invoking a target. If an outcome is not durably saved, resume throws `UncertainExecutionError` and never reruns the target. Replaying an original start key only recovers its execution ID. The low-level `startExecution` API requires an explicit `previousExecutionId`; replacing a still-started attempt additionally requires `allowUncertainRetry:true`. The convenience runner does not automatically adopt externally created attempts. Create a fresh experiment for a fresh target run after investigating side effects.
 
-After a target completes and local scoring finishes, the runner saves the allowed completion/result payloads before uploading. Network/API errors leave those payloads and stable keys available for another call to `runExperiment` with the same directory. Saved receipt IDs prevent duplicate result writes. Serialization failure raises `OutcomeSerializationError` with the execution ID; it does not relabel the target as failed or invoke it again. Scoring/upload failures never change target state.
+After a target completes and local scoring finishes, the runner saves the allowed completion/result payloads before uploading. Network/API errors leave those payloads and stable keys available for another call to `runExperiment` with the same directory. Saved receipt IDs prevent duplicate result writes. An output beyond what Hue stores for one case (200,000 bytes of JSON, 20,000 values or 32 levels of nesting) is that case's own failure: the case completes as `error` with the error type `OutputTooLarge` (and, when result content is persisted, a message naming the bound), no output is stored and the other cases keep running; return a large result as a generated file instead. Output that is not JSON (a cycle, a class instance, a non-finite number) raises `OutcomeSerializationError` with the execution ID. The output is checked as it is read, and the first value that is not JSON or past the value or nesting bound decides which; the byte bound is checked last, so an output past it that also holds a value that is not JSON raises `OutcomeSerializationError`, and one too long for the runtime to serialize is refused before it is serialized. `OutcomeSerializationError` never relabels the target as failed or invokes it again. Scoring/upload failures never change target state.
 
 A crash between target completion and saving its permitted result still leaves an uncertain outcome; metadata-only mode intentionally cannot reconstruct discarded output. Export acknowledgement is saved separately. If required telemetry export was not acknowledged, a fresh empty exporter is not evidence of prior receipt: automatic completion is refused. Inspect/export the original trace or use the low-level completion API with an explicit omission policy; do not rerun a known completed target to manufacture telemetry. API completion/results retain their own idempotency guarantees for explicit recovery.
 
@@ -192,7 +193,7 @@ The runner stops scheduling more cases after an operational failure and waits fo
 
 ## Verification boundaries
 
-`scripts/verify-package.mjs` installs a real packed tarball outside the monorepo and runs HTTP contract tests against a synthetic service plus actual OpenTelemetry exporters. It checks two configurations, rescoring without target invocation, absent/null output, upload resume, uncertain execution, exclusive checkpoints, source/metric contracts, content policy and terminating schema workers. It also exercises the local worker's ready, incomplete and uncertain provider-attempt control-plane paths, but does not call an issued provider facade. `scripts/verify-file-cases.mjs` runs against the same installed consumer: pinned input download and verification, artifact publication of a generated document, file-aware local scoring during the run and again on a later rescore, and a declared file the target could not deliver recorded as the target's own error. `scripts/verify-evaluation-api.mjs` is a separate opt-in acceptance against a real Hue receiver/API; it creates synthetic datasets/scorers/experiments in the project associated with the supplied development key.
+`scripts/verify-package.mjs` installs a real packed tarball outside the monorepo and runs HTTP contract tests against a synthetic service plus actual OpenTelemetry exporters. It checks two configurations, rescoring without target invocation, absent/null output, upload resume, uncertain execution, exclusive checkpoints, source/metric contracts, content policy and terminating schema workers. It also exercises the local worker's ready, incomplete and uncertain provider-attempt control-plane paths, but does not call an issued provider facade. `scripts/verify-file-cases.mjs` runs against the same installed consumer: pinned input download and verification, artifact publication of a generated document, file-aware local scoring during the run and again on a later rescore, a declared file the target could not deliver recorded as the target's own error, and a worker world case that receives its agent-visible files with the world, uploads the returned document and refuses altered bytes as `case_file_mismatch`. `scripts/verify-evaluation-api.mjs` is a separate opt-in acceptance against a real Hue receiver/API; it creates synthetic datasets/scorers/experiments in the project associated with the supplied development key.
 
 ## Outbound local agent worker
 
@@ -215,7 +216,8 @@ import { createEvaluationClient, runLocalAgent } from "@hue-run/sdk/evals";
 import { runMyAgent } from "./agent.js"; // Your existing application entry point.
 
 const connection = { apiKey: process.env.HUE_API_KEY! };
-const hue = createHue({ ...connection, serviceName: "local-worker", captureContent: false });
+// captureContent: false sends metadata only.
+const hue = createHue({ ...connection, serviceName: "local-worker", captureContent: true });
 try {
   await runLocalAgent({
     client: createEvaluationClient(connection),
@@ -233,7 +235,9 @@ try {
 
 The callback receives cloned inputs, local tools, and an allowlisted context containing
 `config`, `item: {id, externalKey}`, `executionId`, `environmentRunId`,
-`trace: {traceId,spanId}` and a short-lived `mcp` capability. Expected outcomes, case metadata
+`trace: {traceId,spanId}`, a short-lived `mcp` capability, `files` (verified copies of the case's
+agent-visible input files, empty when it has none) and a private `outputDirectory`; return
+`withFiles(output, files)` to upload generated files, as for direct cases. Expected outcomes, case metadata
 and original source pins remain private to grading. Pass the tools or scoped MCP capability into
 the agent's actual tool boundary; their presence does not redirect provider calls. Capabilities
 are not written to checkpoints. `maxRuns` limits completed runs for one-shot workers, while
@@ -260,8 +264,8 @@ installed-registry-package to hosted-facade acceptance remains a post-publicatio
 ## Direct cases and files
 
 Cases without a simulated world — document workflows whose inputs are a task plus pinned files —
-run on the same runner under the same checkpoint rules. This is unreleased and ships in the next
-`@hue-run/sdk` release. It requires a Hue deployment that serves case `inputFiles` on experiment
+run on the same runner under the same checkpoint rules. They shipped in `@hue-run/sdk` `0.5.0`
+and require a Hue deployment that serves case `inputFiles` on experiment
 items, subject `files`, and the artifact reservation, upload, completion and download APIs. The
 Python SDK has no equivalent.
 
@@ -270,7 +274,14 @@ On the outbound worker, supply `directTarget` beside or instead of `target`. Sup
 `environment:v1` (both strings are exported as `localAgentCapabilities`). Declare the file
 capabilities the agent accepts and returns — `input:docx`, `input:pdf`, `output:docx` and so on — so
 Hue matches them against each case's `hue.requiredCapabilities` metadata and offers only matching
-cases. `registeredCapabilities` refuses a registration that names a capability without its callback.
+cases. A case pinned to a world whose manifest also holds agent-visible files is offered only to a
+registration that declares `environment-files:v1` (`localAgentCapabilities.environmentFiles`)
+and `input:<extension>` for each file; `target` then receives them as `context.files` with the
+world, as described in
+[ENVIRONMENTS.md](ENVIRONMENTS.md#worlds-served-by-the-simulation-gateway). The worker never adds
+`environment-files:v1` itself: Hue keeps each registered revision's capabilities fixed, so declare
+it under a new revision. `registeredCapabilities` refuses a registration that names a capability
+without its callback.
 
 ```ts
 import { basename } from "node:path";
@@ -312,16 +323,23 @@ The direct callback receives cloned inputs and an allowlisted context of `config
 plus the worker's `signal` when one was supplied. Expected outcomes and case metadata stay with
 grading, as they do for world cases.
 
-Before an execution exists, `runExperiment` downloads every pinned input file named by the frozen
+Before an execution exists, `runExperiment` downloads the pinned input files named by the frozen
 case's `inputFiles` and verifies byte count and SHA-256. A download failure is an SDK failure and
-consumes no execution slot; a file already saved with the pinned identity is reused instead of
-downloaded again. The target sees only the agent-visible roles — `source`, `attached_template`,
-`attached_reference` and `original` — as `context.files`; evaluator-only
+consumes no execution slot; bytes that differ from the manifest raise `CaseFileError` with the
+stable code `case_file_mismatch`, and a download stops one byte past the pinned size. A file
+already saved with the pinned identity is reused instead of downloaded again. The target sees only the agent-visible roles — `source`,
+`attached_template`, `attached_reference` and `original` — as `context.files`; evaluator-only
 `org_template` and `evaluator_reference` files (an organization's template, a legal corpus, an
-answer key) reach scorers but not the agent, and are not even downloaded when no bound code
-evaluator runs in this process. Verified copies live under `filesDirectory` (default
-`<checkpointDirectory>/files`, created mode 0700), and each case gets its own private
-`context.outputDirectory` to write into.
+answer key) reach scorers but not the agent. They are fetched only when a bound code evaluator
+runs in this process: checked with the others before the execution starts but not kept, then
+downloaded again and saved after the target finished, so they are not on disk while the agent
+runs. An agent running as the same operating-system user
+as the runner can still read whatever the runner can; run it under another account when local
+evaluator material must stay out of its reach. Verified copies live under `filesDirectory` (default
+`<checkpointDirectory>/files`, created mode 0700), evaluator-only files apart from the agent's, and
+each case gets its own private `context.outputDirectory` to write into. The files of a case pinned
+to a world are removed when it ends, except staged outputs an interrupted upload resumes from;
+those of a direct case stay for inspection.
 
 Return generated documents by wrapping the output in `withFiles(output, files)`. Each entry names a
 `path` or in-memory `bytes`, a `filename`, a Hue-accepted `contentType` and at most one `primary`.
@@ -329,9 +347,11 @@ The runner copies them next to its checkpoint, publishes them through the artifa
 upload and verified completion APIs with stable per-execution keys, and completes the execution with
 `artifactIds` and `primaryArtifactId`, so the subject's frozen manifest holds inputs and outputs
 together. Generated files are always uploaded regardless of `persistResultContent`: they are the
-execution's evidence. A declared file that cannot be read, exceeds 25 MiB, repeats a filename or has
-an unsupported content type is saved as the target's error (`TargetError`), not as an uncertain
-execution. A crash after the target finished resumes from the staged files without invoking the
+execution's evidence. A declared `path` is read once, as a regular file (a symlink, FIFO or device
+is refused) whose size is checked before reading, and staged owner-only from those bytes. A
+declared file that cannot be read, exceeds 25 MiB, repeats a filename or has an unsupported
+content type is saved as the target's error (`TargetError`), not as an uncertain execution. A
+files directory must be owned by the current user and closed to everyone else (mode 0700). A crash after the target finished resumes from the staged files without invoking the
 target again; if result content is not persisted, the JSON output cannot be reconstructed and the
 case is reported as uncertain.
 

package/README.md

@@ -35,7 +35,7 @@ import { createHue, HueExportError } from "@hue-run/sdk";
 const hue = createHue({
   apiKey: process.env.HUE_API_KEY!, // a project service key, on the server only
   serviceName: "my-agent",
-  captureContent: false, // required: explicitly choose true or false
+  captureContent: true, // required: true records content; false sends metadata only
   onExportIssue: (issue) => console.error(issue), // sanitized counts, never server bodies
 });
 
@@ -58,6 +58,11 @@ try {
 await hue.shutdown(); // flushes and releases providers owned by this client
 ```
 
+`captureContent` is required. The example sets `true`, the recommended choice: trace inspection,
+evaluations and judges read the prompts, responses and tool inputs/outputs it records. Choose
+`false` for metadata only when a policy forbids sending that content; see
+[Privacy and content](#privacy-and-content).
+
 The default destination is `https://app.hue.run`. Data goes to
 `/api/v1/otlp/v1/traces` and `/api/v1/otlp/v1/logs` with a Bearer project key.
 Set `baseUrl` only for another Hue deployment. It must be an origin without an API path; a trailing slash is accepted.
@@ -78,8 +83,9 @@ claim. Setup never enables content capture or creates a simulation, Hue Run, eva
 or remote execution. See the [setup CLI contract](./CLI.md) for the supported shapes and release gates.
 
 For an existing account, `hue login` validates keys created in Hue Settings and stores them in
-`.env.hue` without printing them, and `hue mcp install --client claude-code` (or `cursor`, `codex`,
-`vscode`, `windsurf`, `gemini`) writes the Hue MCP configuration that references `HUE_MCP_KEY`. See
+`.env.hue` without printing them, and `hue mcp install --client claude-code` (or `codex`,
+`conductor`, `cursor`, `vscode`, `windsurf`, `gemini`) writes the Hue MCP configuration that
+references `HUE_MCP_KEY`, or with `--auth oauth` only the URL, for signing in with Hue. See
 [Sign in and store keys](./CLI.md#sign-in-and-store-keys) and
 [Install the MCP for your coding agent](./CLI.md#install-the-mcp-for-your-coding-agent).
 
@@ -150,12 +156,22 @@ await hue.model(
 
 Each OpenAI Responses `mcp_call`, `web_search_call`, `file_search_call` or `code_interpreter_call`
 item, and each Anthropic `mcp_tool_use` or `server_tool_use` block with its result, becomes an
-`execute_tool {name}` child span with `gen_ai.tool.type` `extension`, `gen_ai.tool.call.id` and,
-for MCP calls, `mcp.server.name` (the provider's label). Arguments and results follow
-`captureContent`; a failed call carries `error.type` (`mcp_error`, the provider's status or error
-code) and ERROR status. An `mcp_list_tools` item becomes a `tools/list` child span with that
-server's `gen_ai.tool.definitions`. Pass the request so each server's host is recorded as
-`server.address` (only server URLs are read, never credentials), and `servers` to record a
+`execute_tool {name}` child span with `gen_ai.tool.type` `extension`, `gen_ai.tool.call.id`,
+`hue.tool.call.position` (the 0-based position of its item in the response: the OpenAI `output`
+index or the Anthropic `content` block index, in both capture modes) and, for MCP calls,
+`mcp.server.name` (the provider's label). Arguments and results follow `captureContent`; a failed
+call carries `error.type` (`mcp_error`, the provider's status or error code) and ERROR status. With
+`captureContent: true`, a failed OpenAI MCP call's status description is the provider's error
+text, credentials scrubbed (URL userinfo, query values and fragments, or the whole URL for a scheme
+other than `http(s)`, `ws(s)` and `ftp`; tokens with a known credential prefix such as `hue_sk_`,
+`sk-` or `xoxb-`; authorization-scheme credentials and an `Authorization` header's whole value;
+the value of a credential-named `key=value` or `key: value` pair, quoted or not) and cut to 1,024
+characters. An `mcp_list_tools`
+item becomes a `tools/list` child span with that server's `gen_ai.tool.definitions`; with
+`captureContent: false` it carries only `hue.tool.names` and `hue.tool.definitions.sha256`, the
+summary described below. Pass the request so each server's host is recorded as `server.address`
+(only server URLs are read, never credentials; a host name keeps its underscores, as WHATWG URL
+parsing does), and `servers` to record a
 server's real `name`, `version`, Hue `provider` and `surface` under its label. `provider` defaults
 to the enclosing `model()` call's provider (`openai` or `anthropic`). The spans have no duration of
 their own: the provider ran the tools inside the model request.
@@ -287,6 +303,12 @@ outputs and exports through Hue's attached processors keeps the label.
 
 ## Privacy and content
 
+Set `captureContent: true` for full traces: prompts and messages, responses and tool inputs/outputs,
+alongside model, usage, timing and errors. Trace inspection, evaluations and judges read that
+content. Choose `false`, metadata-only mode, when your users decline or an application or data
+policy forbids sending that content to another service. `redact` and the credential filtering
+described below apply in both modes.
+
 `captureContent: false` disables manual input/output/messages/tool content and
 removes recognized GenAI, Vercel, OpenInference and OpenLLMetry content attributes,
 legacy GenAI content events, log bodies, status messages and exception text before
@@ -306,12 +328,18 @@ an OpenAI hosted MCP tool. Before export, and before `redact`, Hue replaces the
 credential-like fields including `authorization`, `authorization_token`, `headers`, `api_key`,
 `access_token`, `x-api-key`, and keys ending in `token`, `secret`, `password`, `apikey` or
 `credential` (case-insensitively, ignoring `-` and `_`) with `"[redacted]"` in recorded tool definitions
-(`gen_ai.tool.definitions`, `ai.prompt.tools`, `llm.tools.*.tool.json_schema`) and in the `tools`
-and `mcp_servers` entries of a raw provider request or response recorded as `input.value`,
-`output.value` or `llm.invocation_parameters`. Parameters named in a JSON Schema `properties`
+(`gen_ai.tool.definitions`, `ai.prompt.tools`, `llm.tools.*.tool.json_schema`) and in the
+top-level `tools` and `mcp_servers` entries of any JSON object recorded as `input.value`,
+`output.value` or `llm.invocation_parameters`, whether or not it is a provider request; nothing
+else in that object changes. Parameters named in a JSON Schema `properties`
 object keep their schemas, so a tool that takes a `headers` argument is still described. A
 definition nested more than 256 levels deep rejects its record. Sensitive `default`, `const`,
-`examples` and `enum` values under credential-named schema parameters are redacted too.
+`examples` and `enum` values under credential-named schema parameters are redacted too. A `url`
+or `server_url` field (named like the credential fields, so `serverUrl` too) that has userinfo, a
+query or a fragment loses its userinfo and fragment, and each query value becomes `[redacted]`
+under its original name; the result is serialized as WHATWG `URL` does (lowercase scheme and
+host, IDN hosts in Punycode, default ports dropped, dot segments resolved). A URL with none of
+them is exported as recorded, and a value `URL` cannot parse becomes `"[redacted]"`.
 
 With `captureContent: false`, export removes tool definitions but keeps a summary of them on the
 same record: `hue.tool.names` lists each definition's `name` (Chat Completions `function.name`,
@@ -324,12 +352,18 @@ rotates. Only definitions another integration recorded can be summarized: `hueTe
 inputs and exports through Hue's attached processors gets the summary.
 Recorded messages can inline files: GenAI `blob` parts in `gen_ai.input.messages` /
 `gen_ai.output.messages` (what the AI SDK 7 adapter records for a file part) and AI SDK 6 `file`
-parts in `ai.prompt.messages`. A span whose messages exceed 256 KiB would be rejected, so before
-export Hue replaces the `content`/`data` of any such part longer than 64 KiB with the file's
-`sha256` (of the decoded bytes for base64 and `data:` URLs, of the UTF-8 text otherwise) and
-`size`, keeping the part's other fields such as `type`, `mime_type` and `mediaType`. Smaller inline
+parts in `ai.prompt.messages`. A span whose messages exceed 256 KiB would be rejected, so when a
+record is queued Hue replaces the `content`/`data` of any such part longer than 64 KiB with the
+file's `sha256` and `size`, both of the file's own bytes whatever its media type (base64 content
+and `;base64` `data:` URLs decoded, other `data:` URLs percent-decoded, anything else as UTF-8 text), keeping the part's other fields such as `type`, `mime_type` and `mediaType`. Smaller inline
 files are exported as recorded. The digest matches `hue.recordFile`'s `hue.file.sha256` for the same
-bytes, so a file can be recognized wherever it appears.
+bytes, so a file can be recognized wherever it appears. The replacement happens before the record is
+charged to the queue budget, so a large file does not drop its span. It is bounded: a message
+attribute longer than 8 MiB of text is left unchanged, and the default 8 MiB `maxQueueBytes` then
+drops the span. Base64 grows a file by a third, so that ceiling is an inline file of about 6 MiB.
+The same message text is hashed once per record, and a record inspects at most 16 MiB of message
+text in all; messages past that are charged as recorded. With `captureContent: false` the messages,
+which export removes, are neither hashed nor charged.
 
 Manual helpers encode JSON values without converting null into absence. Unknown
 outputs and usage remain absent. This SDK does not estimate tokens or cost. A thrown
@@ -361,7 +395,7 @@ import { createHue, createHueTransport } from "@hue-run/sdk";
 const transport = createHueTransport({
   apiKey: process.env.HUE_API_KEY!,
   serviceName: "existing-app",
-  captureContent: false,
+  captureContent: true, // false sends metadata only
 });
 const tracerProvider = new TracerProvider({ spanProcessors: [transport.spanProcessor] });
 const loggerProvider = new LoggerProvider({ processors: [transport.logRecordProcessor] });
@@ -576,9 +610,9 @@ an uncertain preparation. See the
 an issued facade endpoint or the official Gmail service, and do not claim universal provider
 parity.
 
-An unreleased `directTarget` callback extends the same worker to cases without a world: the runner
-verifies the case's pinned input files, hands them to the agent, and uploads the documents it
-returns as verified Hue artifacts for scoring. See
+The `directTarget` callback, shipped in TypeScript `0.5.0`, extends the same worker to cases
+without a world: the runner verifies the case's pinned input files, hands them to the agent, and
+uploads the documents it returns as verified Hue artifacts for scoring. See
 [direct cases and files](EVALUATIONS.md#direct-cases-and-files).
 
 Scorer deferral shipped in TypeScript `0.3.1`. Only built-ins
@@ -591,17 +625,23 @@ their agents with `runLocalAgent()`; setup does not register workers or launch s
 
 ### Command-line evaluation
 
-The unreleased `hue eval` command wraps `runSimulation()` and `runLocalAgent()` for an adapter
-file or a shell command: `hue eval --case "<name>" ./hue-agent.ts` creates a fresh run
-from a published case's immutable pins, runs the agent in one isolated world per case, waits
+The `hue eval` command, shipped in TypeScript `0.5.0` (`--case` since `0.6.0`, `--scenario`
+before), wraps `runSimulation()` and `runLocalAgent()` for an adapter file or a shell command:
+`hue eval --case "<name>" ./hue-agent.ts --content` creates a fresh run from a published case's
+immutable pins, runs the agent in one isolated world per case, waits
 for Hue's outcome checks and prints the run URL and per-case PASS/FAIL verdicts with an exit code;
 `--worker` registers the same adapter for runs launched from Hue. It needs a Read and
-write key in `HUE_API_KEY` (never printed) and keeps content capture off unless `--content`
-is passed. See [Evaluate an agent against a case](CLI.md#evaluate-an-agent-against-a-case).
+write key in `HUE_API_KEY` (never printed). Content capture stays off unless `--content` is
+passed; the example passes it so case spans carry content. Case outputs, error messages and
+explanations are stored in Hue either way (`--no-output` keeps them out of a one-shot run's
+results); a command's stdout is its stored answer, with the credentials `hue eval` handed the case
+replaced by `[redacted]`.
+Instrument the agent itself for its model and tool spans. See
+[Evaluate an agent against a case](CLI.md#evaluate-an-agent-against-a-case).
 Eval sets whose cases pin files instead of a world run as direct cases through `runExperiment()`:
-`hue eval --set <slug> --scorer <slug> --command "…"` hands the agent each case's pinned files in a
-private directory, uploads the documents it writes and waits for Hue's grading executor to score
-them. See [Evaluate a document eval set](CLI.md#evaluate-a-document-eval-set).
+`hue eval --set <slug> --scorer <slug> --command "…" --content` hands the agent each case's
+pinned files in a private directory, uploads the documents it writes and waits for Hue's grading
+executor to score them. See [Evaluate a document eval set](CLI.md#evaluate-a-document-eval-set).
 
 ## Managed targets
 

package/dist/cli/eval-direct.d.ts

@@ -38,10 +38,21 @@ export declare function stageDirectCase(root: string, input: {
     executionId: string;
     files: LocalFile[];
 }): Promise<DirectCaseLayout>;
+/** Test seams: `beforeEntry` runs before each listed entry is examined, `afterListing` between
+ * the listing and the reads. */
+export interface CollectionSeams {
+    beforeEntry?: (name: string) => Promise<void>;
+    afterListing?: () => Promise<void>;
+}
 /**
  * Turns the output folder into the target's result. Every regular file, in subdirectories too,
  * becomes a generated file; an unsupported extension is the agent's error rather than a silently
  * dropped document. `fallbackOutput` (for example the command's stdout) is used when no JSON
  * output or summary file was written.
+ *
+ * Nothing here trusts a path the agent could have changed: the output directory must be a real
+ * directory, every file is opened without following a final symlink or blocking on a FIFO, must
+ * be the regular file the listing saw and within its size limit, and the result carries the bytes
+ * read from that open file. `seams` are for tests only.
  */
-export declare function collectDirectOutputs(outputDirectory: string, fallbackOutput?: JsonValue): Promise<TargetResult>;
+export declare function collectDirectOutputs(outputDirectory: string, fallbackOutput?: JsonValue, seams?: CollectionSeams): Promise<TargetResult>;

package/dist/cli/eval-direct.js

@@ -1,6 +1,7 @@
-import { copyFile, mkdir, readdir, readFile, stat, writeFile } from "node:fs/promises";
+import { constants } from "node:fs";
+import { copyFile, lstat, mkdir, opendir, writeFile } from "node:fs/promises";
 import { basename, extname, join } from "node:path";
-import { outputContentTypes } from "../evals/files.js";
+import { outputContentTypes, outputFileLimits, readAgentFile, safeFilename, } from "../evals/files.js";
 import { withFiles } from "../evals/types.js";
 /**
  * Direct (file) cases for `hue eval`: a case is a task plus pinned input files, and the agent's
@@ -43,6 +44,11 @@ for (const type of Object.values(outputExtensions))
         throw new Error(`outputExtensions maps to a content type Hue does not accept: ${type}`);
 const HELPER_FILES = new Set(["manifest.json", "result.json", "output.json"]);
 const SUMMARY_FILES = ["summary.txt", "summary.md", "final.txt", "resumen.txt"];
+/** The most a helper (`manifest.json`, `result.json`, `summary.txt`, …) may hold: the same bound
+ * as the command's stdout. */
+const HELPER_BYTES = 4 * 1024 * 1024;
+/** Entries a collection visits at most, hidden and skipped ones included. */
+const MAX_ENTRIES = 1024;
 /** Writes the case directory: inputs, descriptor, verified file copies and an empty output folder. */
 export async function stageDirectCase(root, input) {
     const caseDirectory = join(root, "case");
@@ -50,21 +56,25 @@ export async function stageDirectCase(root, input) {
     await mkdir(outputDirectory, { recursive: true, mode: 0o700 });
     const inputsPath = join(caseDirectory, "inputs.json");
     await writeFile(inputsPath, JSON.stringify(input.inputs, null, 2), { mode: 0o600 });
+    // Names that differ only in case or Unicode normalization are the same file on macOS and
+    // Windows filesystems; keep both inputs under distinct names, and never overwrite one.
     const used = new Set();
+    const key = (folder, name) => `${folder}/${name.normalize("NFC").toLowerCase()}`;
     const files = [];
     for (const file of input.files) {
         const folder = join(caseDirectory, "files", file.role);
         await mkdir(folder, { recursive: true, mode: 0o700 });
         // Two inputs may share a filename (two incident reports, say); keep both.
-        const name = basename(file.filename);
+        const name = safeFilename(basename(file.filename));
         const extension = extname(name);
         const stem = name.slice(0, name.length - extension.length);
-        let target = join(folder, name);
-        for (let copy = 2; used.has(target); copy++)
-            target = join(folder, `${stem} (${copy})${extension}`);
-        used.add(target);
-        await copyFile(file.path, target);
-        files.push({ role: file.role, filename: basename(target), path: target });
+        let target = name;
+        for (let copy = 2; used.has(key(folder, target)); copy++)
+            target = `${stem} (${copy})${extension}`;
+        used.add(key(folder, target));
+        const path = join(folder, target);
+        await copyFile(file.path, path, constants.COPYFILE_EXCL);
+        files.push({ role: file.role, filename: target, path });
     }
     await writeFile(join(caseDirectory, "case.json"), JSON.stringify({
         id: input.item.id,
@@ -76,33 +86,119 @@ export async function stageDirectCase(root, input) {
     }, null, 2), { mode: 0o600 });
     return { caseDirectory, inputsPath, outputDirectory, files };
 }
-async function readJsonFile(path) {
+const identityOf = (info) => ({
+    dev: info.dev,
+    ino: info.ino,
+});
+const sameIdentity = (a, b) => a.dev === b.dev && a.ino === b.ino;
+/** A real directory (not a symlink) and its identity, or undefined when `path` is absent. A
+ * parent that is no longer a directory (ENOTDIR) is a change, never an absent directory: a case
+ * directory replaced by a file must not read as "wrote nothing". */
+async function directoryIdentity(path, label) {
+    let info;
     try {
-        return JSON.parse(await readFile(path, "utf8"));
+        info = await lstat(path);
     }
     catch (error) {
         if (error.code === "ENOENT")
             return undefined;
-        throw new Error(`${basename(path)} in the output directory is not valid JSON`);
+        throw unexamined(label, error);
     }
+    if (info.isSymbolicLink() || !info.isDirectory())
+        throw new Error(`${label} is not a directory; a symbolic link or file is never collected`);
+    return identityOf(info);
 }
-/** Regular files anywhere under the output directory, named by their `/`-joined relative path.
- * Hidden and lock entries (dot names, `~$…`) and symlinks are skipped at every level. */
-async function listOutputFiles(directory, prefix = "") {
-    const entries = await readdir(directory, { withFileTypes: true }).catch(() => []);
+/** The error for an entry that vanished or changed kind while it was being collected, or could
+ * not be examined at all. */
+function unexamined(label, error) {
+    const code = error.code;
+    return new Error(code === "ENOENT" || code === "ENOTDIR" || code === "ELOOP"
+        ? `${label} changed while it was collected`
+        : `${label} could not be read`);
+}
+/**
+ * Regular files anywhere under the output directory, named by their `/`-joined relative path,
+ * with the device and inode each had when listed. Hidden and lock entries (dot names, `~$…`),
+ * symlinks and anything that is neither a regular file nor a directory are skipped at every
+ * level, except that a top-level helper name must be a regular file when present. The walk stops
+ * past 32 documents or 1024 entries, and an entry that vanished or a directory that changed while
+ * it was listed is refused.
+ */
+async function listOutputFiles(root, rootIdentity, seams) {
     const found = [];
-    for (const entry of entries) {
-        if (entry.name.startsWith(".") || entry.name.startsWith("~$"))
-            continue;
-        const path = join(directory, entry.name);
-        const name = `${prefix}${entry.name}`;
-        if (entry.isDirectory())
-            found.push(...(await listOutputFiles(path, `${name}/`)));
-        else if (entry.isFile())
-            found.push({ name, path });
+    const directories = [
+        { path: root, label: "output/", identity: rootIdentity },
+    ];
+    let visited = 0;
+    let documents = 0;
+    for (let index = 0; index < directories.length; index++) {
+        const directory = directories[index];
+        const prefix = index === 0 ? "" : `${directory.label.slice("output/".length)}`;
+        let entries;
+        try {
+            entries = await opendir(directory.path);
+        }
+        catch (error) {
+            throw unexamined(directory.label, error);
+        }
+        for await (const entry of entries) {
+            if (++visited > MAX_ENTRIES)
+                throw new Error(`The output directory holds more than ${MAX_ENTRIES} entries`);
+            const name = `${prefix}${entry.name}`;
+            const helper = index === 0 && (HELPER_FILES.has(name) || SUMMARY_FILES.includes(name));
+            if (!helper && (entry.name.startsWith(".") || entry.name.startsWith("~$")))
+                continue;
+            const path = join(directory.path, entry.name);
+            if (helper && !entry.isFile())
+                throw new Error(`output/${name} is not a regular file; the agent must write it itself`);
+            if (entry.isDirectory() || entry.isFile())
+                await seams.beforeEntry?.(name);
+            if (entry.isDirectory()) {
+                const identity = await directoryIdentity(path, `output/${name}`);
+                if (!identity)
+                    throw new Error(`output/${name} changed while it was collected`);
+                directories.push({ path, label: `output/${name}/`, identity });
+            }
+            else if (entry.isFile()) {
+                let info;
+                try {
+                    info = await lstat(path);
+                }
+                catch (error) {
+                    throw unexamined(`output/${name}`, error);
+                }
+                if (!info.isFile())
+                    throw new Error(`output/${name} changed while it was collected`);
+                found.push({ name, path, identity: identityOf(info) });
+                if (!helper && ++documents > outputFileLimits.count)
+                    throw new Error(`The agent wrote more than ${outputFileLimits.count} files`);
+            }
+        }
+    }
+    // Every directory must still be the one listed: none was swapped for a link meanwhile.
+    for (const directory of directories) {
+        const now = await directoryIdentity(directory.path, directory.label);
+        if (!now || !sameIdentity(now, directory.identity))
+            throw new Error(`${directory.label} changed while it was collected`);
     }
     return found;
 }
+async function readHelper(listed) {
+    if (!listed)
+        return undefined;
+    const bytes = await readAgentFile(listed.path, `output/${listed.name}`, HELPER_BYTES, listed.identity);
+    return Buffer.from(bytes).toString("utf8");
+}
+function parseHelper(listed, text) {
+    if (text === undefined)
+        return undefined;
+    try {
+        return JSON.parse(text);
+    }
+    catch {
+        throw new Error(`${listed.name} in the output directory is not valid JSON`);
+    }
+}
 /** Artifact filenames cannot contain path separators or controls. Encode only those forbidden
  * characters plus `%`, then represent `/` as `%2F`; distinct output paths stay distinct. */
 function artifactFilename(relativePath) {
@@ -120,9 +216,18 @@ function artifactFilename(relativePath) {
  * becomes a generated file; an unsupported extension is the agent's error rather than a silently
  * dropped document. `fallbackOutput` (for example the command's stdout) is used when no JSON
  * output or summary file was written.
+ *
+ * Nothing here trusts a path the agent could have changed: the output directory must be a real
+ * directory, every file is opened without following a final symlink or blocking on a FIFO, must
+ * be the regular file the listing saw and within its size limit, and the result carries the bytes
+ * read from that open file. `seams` are for tests only.
  */
-export async function collectDirectOutputs(outputDirectory, fallbackOutput) {
-    const manifest = await readJsonFile(join(outputDirectory, "manifest.json"));
+export async function collectDirectOutputs(outputDirectory, fallbackOutput, seams = {}) {
+    const rootIdentity = await directoryIdentity(outputDirectory, "The output directory");
+    const listed = rootIdentity ? await listOutputFiles(outputDirectory, rootIdentity, seams) : [];
+    await seams.afterListing?.();
+    const top = (name) => listed.find((entry) => entry.name === name);
+    const manifest = parseHelper(top("manifest.json"), await readHelper(top("manifest.json")));
     const declaredPrimary = manifest && typeof manifest === "object" && !Array.isArray(manifest)
         ? manifest.primary
         : undefined;
@@ -130,11 +235,11 @@ export async function collectDirectOutputs(outputDirectory, fallbackOutput) {
         ? manifest.output
         : undefined;
     let output = declaredOutput ??
-        (await readJsonFile(join(outputDirectory, "result.json"))) ??
-        (await readJsonFile(join(outputDirectory, "output.json")));
+        parseHelper(top("result.json"), await readHelper(top("result.json"))) ??
+        parseHelper(top("output.json"), await readHelper(top("output.json")));
     if (output === undefined)
         for (const name of SUMMARY_FILES) {
-            const text = await readFile(join(outputDirectory, name), "utf8").catch(() => undefined);
+            const text = await readHelper(top(name));
             if (text !== undefined) {
                 output = { summary: text };
                 break;
@@ -143,7 +248,7 @@ export async function collectDirectOutputs(outputDirectory, fallbackOutput) {
     if (output === undefined)
         output = fallbackOutput;
     // Helper and summary names are reserved at the top level only; a nested one is a document.
-    const entries = (await listOutputFiles(outputDirectory)).filter((entry) => !HELPER_FILES.has(entry.name) && !SUMMARY_FILES.includes(entry.name));
+    const entries = listed.filter((entry) => !HELPER_FILES.has(entry.name) && !SUMMARY_FILES.includes(entry.name));
     // Directory order differs between filesystems; keep uploads stable.
     entries.sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0));
     const unsupported = entries.filter((entry) => !outputExtensions[extname(entry.name).toLowerCase()]);
@@ -151,10 +256,11 @@ export async function collectDirectOutputs(outputDirectory, fallbackOutput) {
         throw new Error(`The agent wrote files Hue does not accept as generated documents: ${unsupported.map((entry) => entry.name).join(", ")} (accepted: ${Object.keys(outputExtensions).join(" ")})`);
     const files = [];
     for (const entry of entries) {
-        if ((await stat(entry.path)).size === 0)
+        const bytes = (await readAgentFile(entry.path, `Generated file ${entry.name}`, outputFileLimits.bytes, entry.identity));
+        if (!bytes.byteLength)
             throw new Error(`The agent wrote an empty file: ${entry.name}`);
         files.push({
-            path: entry.path,
+            bytes,
             filename: artifactFilename(entry.name),
             contentType: outputExtensions[extname(entry.name).toLowerCase()],
         });
@@ -167,5 +273,11 @@ export async function collectDirectOutputs(outputDirectory, fallbackOutput) {
     }
     else if (files.length === 1)
         files[0].primary = true;
+    // The directory must still be the one listed when every byte has been read.
+    if (rootIdentity) {
+        const now = await directoryIdentity(outputDirectory, "The output directory");
+        if (!now || !sameIdentity(now, rootIdentity))
+            throw new Error("The output directory changed while it was collected");
+    }
     return withFiles(output, files);
 }