@hue-run/sdk 0.9.0 → 0.11.0

package/CLI.md

@@ -42,7 +42,7 @@ project manifest are refused because managers can update ancestor locks; Python
 The generated `hue.setup.mjs` or `hue_setup.py` always selects `captureContent: false` /
 `capture_content=False`. For a supported application, setup installs the dependency and adds the
 managed import and middleware registration to the existing entrypoint; an unreferenced helper is
-not a completed integration. TypeScript uses `@hue-run/sdk@0.9.0`, `@opentelemetry/api@1.9.1` and
+not a completed integration. TypeScript uses `@hue-run/sdk@0.11.0`, `@opentelemetry/api@1.9.1` and
 `@opentelemetry/context-async-hooks@2.11.0`; Python setup uses its separately tested package pin.
 Content capture requires an ordinary account-managed key and a later explicit application decision.
 
@@ -113,6 +113,7 @@ hue setup --format human  # explicit append-only terminal rendering
 hue setup --origin http://127.0.0.1:PORT # isolated loopback tests only
 hue login                 # validate keys created in Hue and store them in ./.env.hue
 hue mcp install --client claude-code # write the Hue MCP configuration for a coding agent
+hue listen --subscription <id> --forward-to http://localhost:3000/slack/events # deliver events to a local bot
 ```
 
 Hosted setup accepts HTTPS origins only. HTTP is accepted solely for `localhost`, `127.0.0.1` and
@@ -176,30 +177,130 @@ warns; `--gitignore` appends the file name to the `.gitignore` next to it. Exit
 
 ## Install the MCP for your coding agent
 
-`hue mcp install --client <name>` writes, runs or prints the configuration for Hue's MCP server.
-Every shape matches the snippet Hue shows in Settings: server name `hue`, the endpoint (default
-`https://mcp.hue.run/mcp`; `--url https://mcp.staging.hue.run/mcp` for staging) and a reference to
-the `HUE_MCP_KEY` environment variable. A key value is never written.
-
-| Client | Result |
-| --- | --- |
-| `claude-code` | Merges `mcpServers.hue` into `./.mcp.json`. `--scope user` runs `claude mcp add --transport http --scope user hue URL --header 'Authorization: Bearer ${HUE_MCP_KEY}'` when `claude` is on `PATH`, otherwise prints it. |
-| `cursor` | Merges `mcpServers.hue` into `./.cursor/mcp.json` with `${env:HUE_MCP_KEY}`. |
-| `codex` | Runs `codex mcp add hue --url URL --bearer-token-env-var HUE_MCP_KEY`, or prints the `[mcp_servers.hue]` TOML block for `~/.codex/config.toml`. |
-| `vscode` | Merges `servers.hue` and the `hue-mcp-key` password input into `./.vscode/mcp.json`. |
-| `windsurf` | Prints the `serverUrl` snippet for `~/.codeium/windsurf/mcp_config.json`; nothing is written to the home directory. |
-| `gemini` | Runs `gemini mcp add --transport http hue URL -H 'Authorization: Bearer $HUE_MCP_KEY'`, or prints it. |
+`hue mcp install --client <name>` writes, runs or prints the configuration for Hue's MCP server,
+in the shapes of the [connection guide](https://docs.hue.run/agents/mcp-server): server name
+`hue` and the endpoint (default `https://mcp.hue.run/mcp`; `--url https://mcp.staging.hue.run/mcp`
+for staging). Hue's Settings page does not show these snippets; this command keeps its own copy.
+
+`--auth` chooses how the client authenticates:
+
+- `key` (the default, except for `conductor`) references the `HUE_MCP_KEY` environment variable,
+  which `hue login` stores in `.env.hue`. A key value is never written.
+- `oauth` configures the URL only. The client opens Hue in a browser, where you sign in and
+  approve the connection; no key or header is involved. A new sign-in connection has **Read and
+  write** access to every active project in the organization you choose, within your role, so an
+  agent first calls `list_projects` and passes the chosen project's id as `project_id`. It is available
+  for `claude-code`, `codex` and `conductor`. Hue does not yet accept Cursor's sign-in callback, so
+  `cursor` and the other clients use a key.
+
+`--read-only` adds `?read_only=true` to a key configuration's URL, so Hue hides and rejects write
+tools whatever the key allows. A project key, with or without it, keeps reaching its one project.
+With `--auth oauth` it is refused: a sign-in connection has **Read and write** access, so use a
+**Read** project key for read-only access.
+
+`--toolsets <names>` lists only the named tools, for clients with a tool-count limit or agents that
+only read production: `observe` (the production reads: projects, traces, spans and their recorded
+content, sessions, trace checks, intents and documentation), `all`, or the groups `project`, `traces`, `evals`, `environments`,
+`intents` and `docs`, comma-separated. A key configuration adds `?toolsets=<names>` to the URL. A
+sign-in configuration keeps the URL bare and sends the `X-Hue-MCP-Toolsets` header instead,
+because toolsets are not part of the connection's identity: Claude Code stores the header, and for
+Codex the command prints the `http_headers` line to add to `~/.codex/config.toml`, since
+`codex mcp add` stores none. `--toolsets` replaces a selection already in `--url`, and a selection in `--url` is kept without
+it; otherwise `cursor` selects `observe`. `--toolsets all` lists every tool. Unknown names are refused rather than passed on, because Hue
+ignores them and would list every tool. `get_project_context` is always listed and reports the
+selection.
+
+| Client | Key (`--auth key`) | Sign-in (`--auth oauth`) |
+| --- | --- | --- |
+| `claude-code` | Merges `mcpServers.hue` into `./.mcp.json`. `--scope user` runs `claude mcp add --transport http --scope user hue URL --header 'Authorization: Bearer ${HUE_MCP_KEY}'` when `claude` is on `PATH`, otherwise prints it. | Merges `mcpServers.hue` with only `type` and `url` into `./.mcp.json`; `--scope user` runs `claude mcp add --transport http --scope user hue URL`. Then run `/mcp` in Claude Code, select `hue` and choose **Authenticate**. |
+| `codex` | Runs `codex mcp add hue --url URL --bearer-token-env-var HUE_MCP_KEY`, or prints the `[mcp_servers.hue]` TOML block for `~/.codex/config.toml`. | Runs `codex mcp add hue --url URL`, which can start sign-in right away; `codex mcp login hue` starts or repeats it. |
+| `conductor` | Runs the Claude Code user-scope and Codex key commands above. | The default: runs `claude mcp add --transport http --scope user hue URL` and `codex mcp add hue --url URL`. Then use `hue`'s authentication action in Conductor's MCP status (the plug icon or `/mcp-status`). |
+| `cursor` | Merges `mcpServers.hue` into `./.cursor/mcp.json` with `${env:HUE_MCP_KEY}`. | Refused. |
+| `vscode` | Merges `servers.hue` and the `hue-mcp-key` password input into `./.vscode/mcp.json`. | Refused. |
+| `windsurf` | Prints the `serverUrl` snippet for `~/.codeium/windsurf/mcp_config.json`; nothing is written to the home directory. | Refused. |
+| `gemini` | Runs `gemini mcp add --scope user --transport http hue URL --header 'Authorization: Bearer ${HUE_MCP_KEY}'`, or prints it. | Refused. |
+
+Conductor has no MCP configuration of its own: its Claude Code and Codex agents read their user
+configuration in every workspace, so `conductor` registers the server at user scope with each CLI
+on `PATH` and prints the command for a CLI that is missing. A failing CLI does not stop the other;
+the command then exits `1`. Hue has not yet verified the complete signed-in Conductor flow (consent
+through a trace read) end to end; if Hue's tools do not load after signing in, run the command
+again with `--auth key`.
 
 JSON files are parsed and merged: other servers, inputs and top-level fields are kept, only the
 `hue` entry is replaced, and invalid JSON (including comments) is refused together with the snippet
 to add by hand. Files are written with mode `0644` through a temporary file and an atomic rename;
-symlinks are refused. `--dry-run` prints the resulting file content or command without writing or
+symlinks are refused. `--dry-run` prints the resulting file content or commands without writing or
 running; `--print` prints only the snippet. Client CLIs run without a shell, so the
-`${HUE_MCP_KEY}` and `$HUE_MCP_KEY` references reach them literally; the printed commands use
-single quotes for the same reason. After installation the command reminds you to export
-`HUE_MCP_KEY` in the shell that starts the client (VS Code prompts for the key instead) and prints
-the verification prompt: `Use the Hue MCP: call get_project_context, then show my 5 most recent
-error traces with links.` Exit codes: `0` done or printed, `1` failed, `2` usage error.
+`${HUE_MCP_KEY}` reference reaches them literally; the printed commands use single quotes for the
+same reason, and quote a URL with a query such as `?read_only=true`, whose `?` is a zsh glob.
+
+After a key installation the command reminds you to export `HUE_MCP_KEY` in the shell that starts
+the client (VS Code prompts for the key instead). A desktop app started from the Dock or a launcher
+does not see that shell's variables, and Conductor's agents read the login-shell environment that
+Conductor captures, so sign-in is the simpler choice there. After a sign-in installation it names
+the client's authentication action. Both end with the prompt that verifies the connection:
+`Use the Hue MCP: call list_projects and confirm which project to inspect. Then call get_project_context and show that project's traces from the last 24 hours that need attention or have errors, with links; if there are none, show its 5 most recent traces. For an organization connection, pass the project's id as project_id on each call after list_projects.` `list_projects` returns a project key's one project too. Exit codes:
+`0` done or printed, `1` failed, `2` usage error.
+
+## Deliver simulated events to a local bot
+
+`hue listen` delivers a simulated world's events to a receiver on your machine, so an event-driven
+agent such as a Slack bot can be tested without a public URL, the way `stripe listen --forward-to`
+delivers webhooks. Hue signs each request as the provider does (Slack Events API requests today);
+the command pulls them, forwards each one unchanged to `--forward-to` and reports the bot's answer
+to Hue, which settles or retries the event on the provider's schedule. It needs a **listen** event
+subscription on the world or on a connection key; the subscription's id and signing secret are shown
+once when it is created. Configure the bot with that signing secret where `SLACK_SIGNING_SECRET`
+went, then start the command: Hue offers the URL verification first, and events follow once the bot
+has answered it. Event subscriptions are not yet available on `https://app.hue.run`; until the
+origin offers them, the command stops with Hue's refusal.
+
+```sh
+hue listen --subscription <id> --forward-to http://localhost:3000/slack/events --env-path .env.world
+```
+The credential is the subscription's own, read from the environment (or `--env-path`, loaded first;
+a variable already set keeps its value): `HUE_WORLD_TOKEN` for a world's subscription or
+`HUE_CONNECTION_KEY` for a connection key's, chosen with `--credential world-token|connection-key`
+when both are set. It is never taken from the command line, where other local processes and shell
+history can read it. A project key is refused: a value equal to `HUE_API_KEY` or `HUE_MCP_KEY` exits
+2 before any request, and Hue answers any credential that is not the subscription's with `401`.
+`--origin` (default `HUE_BASE_URL`, then `https://app.hue.run`) must be HTTPS, or plain HTTP on a
+loopback test server.
+
+Each request reaches the bot with its body bytes and headers unchanged, so the provider's signature
+verifies as it would in production; only `Host`, `Content-Length` and connection headers are the
+local request's own, and Hue's credential is never forwarded. No redirect is followed: a `3xx` is
+reported as the answer. An answer counts only within three seconds, Slack's acknowledgement window;
+a later one is a timeout. A failing answer with `x-slack-no-retry: 1` is reported so Hue stops
+retrying it. Only a successful URL verification answer's body (at most 4 KiB, for its challenge) is
+sent to Hue; any other answer body stays on this machine. Each pull waits for at most 20 seconds and
+leases up to `--max` deliveries (1 to 10, default 10), which are forwarded concurrently; the next
+pull starts once they are acknowledged, and a delivery beyond `--max` is never forwarded. A provider
+retry of an event is a new delivery and is forwarded again with its `X-Slack-Retry-Num` and
+`X-Slack-Retry-Reason`, so the bot deduplicates by `event_id` as it does with Slack. A delivery Hue
+hands out a second time is not sent again; its recorded answer is repeated. An acknowledgement Hue
+did not answer is repeated until the delivery's lease ends, as Hue stated it (30 seconds from the
+lease).
+
+`--forward-to` must name this machine: `localhost`, a `.localhost` name or a loopback address, and a
+name is refused unless every address it resolves to is a loopback address. `--allow-remote-forward`
+permits another host over HTTPS only, because every request carries the subscription's verification
+token and a valid signature; plain HTTP still reaches only loopback addresses. Credentials and
+fragments in the URL are refused. Output is one line per delivery (time, event id, retry number, the
+bot's status and duration, and the state Hue recorded), never a body, a signature or a credential;
+credentials in error messages are replaced with `[redacted]`.
+
+Ctrl+C or `SIGTERM` stops pulling (a waiting pull is cancelled), finishes the deliveries in flight
+and acknowledges them, then exits `0`; a second Ctrl+C abandons the forwards and acknowledgements
+still in flight and exits `130`. Nothing is acknowledged before the bot answered or its window
+closed, and a lease left unacknowledged lapses into a timeout that Hue retries on the provider's
+schedule, so no event is lost and none is marked delivered unanswered. Network errors, `429`, `5xx`
+and another open pull for the same subscription (a second `hue listen`) are retried with backoff up
+to 30 seconds; a `Retry-After` can lengthen a wait to at most 60 seconds, never shorten it. Exit
+codes: `0` stopped, `1` Hue refused the credential, the subscription (unknown, revoked, or one that
+delivers to a request URL) or the pull itself (a redirect, another refusal or an unreadable answer),
+`2` usage error, `130` interrupted twice.
 
 ## Local state and conflicts
 
@@ -305,7 +406,7 @@ node packages/sdk-typescript/scripts/verify-package.mjs --artifacts-dir .artifac
 # Set project to an existing supported fixture; use the same directory on resume.
 project=/absolute/path/to/supported-fixture
 node packages/sdk-typescript/scripts/verify-setup-live.mjs \
-  --archive .artifacts/typescript/hue-run-sdk-0.9.0.tgz \
+  --archive .artifacts/typescript/hue-run-sdk-0.11.0.tgz \
   --origin https://STAGING_ORIGIN \
   --project "$project" --command setup \
   --evidence .context/setup-staging-before-claim.json
@@ -316,7 +417,7 @@ the private local handoff and finish the real browser claim, then reconcile the
 
 ```sh
 node packages/sdk-typescript/scripts/verify-setup-live.mjs \
-  --archive .artifacts/typescript/hue-run-sdk-0.9.0.tgz \
+  --archive .artifacts/typescript/hue-run-sdk-0.11.0.tgz \
   --origin https://STAGING_ORIGIN \
   --project "$project" --command claim \
   --evidence .context/setup-staging-after-claim.json
@@ -351,8 +452,11 @@ the CLI never prints it. The optional `zod` peer of `@hue-run/sdk/evals` must be
 Write an adapter module that hands the case inputs and the world's tools or MCP connection to
 the real agent. The module exports `default` or `runMyAgent`; `context` is the SDK's
 `SimulationTargetContext` (`world`, `mcp`, `tools`, `config`, `item`, `executionId`,
-`environmentRunId`, `signal`; `world` carries the provider mirror URLs, the world token, `env` and
-`mcpConfig` where Hue's simulation gateway serves the world):
+`environmentRunId`, `files`, `outputDirectory`, `signal`; `world` carries the provider mirror URLs,
+the world token, `env` and `mcpConfig` where Hue's simulation gateway serves the world). When the
+case also pins input files, `files` holds verified copies of its agent-visible ones (see
+[Evaluate a document eval set](#evaluate-a-document-eval-set)); return
+`withFiles(output, files)` from `@hue-run/sdk/evals` to upload documents the agent produced:
 
 ```ts
 // hue-agent.ts: erasable TypeScript only; Node.js 24 and Bun strip the types natively.
@@ -368,66 +472,136 @@ export default function runMyAgent(inputs: JsonValue, context: SimulationTargetC
 ```
 
 ```sh
-hue eval --case "Refund an eligible charge" ./hue-agent.ts --env-file .env.hue
-hue eval --case https://app.hue.run/projects/demo/scenarios/<id> ./hue-agent.ts --baseline <run id>
-hue eval --set "Billing regressions" --scorer-version <id> ./hue-agent.ts --save-version
+hue eval --case "Refund an eligible charge" ./hue-agent.ts --content --env-file .env.hue
+hue eval --case https://app.hue.run/projects/demo/scenarios/<id> ./hue-agent.ts --content --baseline <run id>
+hue eval --set "Billing regressions" --scorer-version <id> ./hue-agent.ts --content --save-version
 hue eval --case "Refund an eligible charge" --command "python agent.py" --timeout 120 --content
-hue eval --worker ./hue-agent.ts --agent-key support-agent --env-file .env.hue
+hue eval --worker ./hue-agent.ts --agent-key support-agent --content --env-file .env.hue
 ```
 
 `--scenario` remains an alias for `--case` for existing scripts. Pass one selection flag.
 
-The one-shot mode resolves the selection (`--case` by name, ID or URL; `--set` by name, ID or
-URL with explicit `--scorer-version` pins; or `--dataset-version` with `--scorer-version`),
+The one-shot mode resolves the selection (`--case` by name, or by the ID or URL of the published
+case or of the case conversion that published it, among the project's first 1,000 Scenarios;
+`--set` by name, ID or URL with explicit `--scorer-version` pins; or `--dataset-version` with `--scorer-version`),
 creates a fresh run from those immutable pins named `<agent key> @ <revision>` (a commit hash is
 shortened to 7 characters)
 (`--name` overrides), prints `Run: <url>` and `Experiment: <id>` as soon as the experiment exists,
 one line per case event (world created, agent started, world sealed), then
 `Waiting for Hue checks...` and a table with one row per case: boolean metrics as `PASS`/`FAIL`,
 numbers as values, text and category metrics as-is, an overall result per case and a pass count.
-Failing cases print the scorer explanation. `--baseline <experiment id|url>` adds improvement,
+Failing cases print the scorer explanation. An evaluator Hue records as not applicable to a case
+(its outcome scoring has no outcome criteria or conversion rubric to grade there) shows `n/a` in
+that case's columns and neither passes nor fails it, so in a mixed eval set a case is decided by
+the evaluators that apply to it; the pass line counts the not-applicable results. A case that no
+pinned evaluator applies to is an error that names what they need. A result of a pinned Hue judge
+(an evaluator of kind `world_judge`) that Hue marks advisory, as it does every judge's today, is
+shown in its column, marked `(advisory)`, but never decides a case or the exit code; the pass line
+counts advisory failures as not counted, and a case only advisory results ran for is an error. Any
+other evaluator's result, an error, or a metric that carries `passed` is never advisory, whatever
+its evidence says. A metric name that
+several evaluators report, such as two judges' `verdict`, gets a column per evaluator, labelled
+with the version's first eight characters. `--baseline <experiment id|url>` adds improvement,
 regression and unchanged counts with per-case deltas. `--json` prints one JSON document
 (`experimentId`, `runId`, `runUrl`, `complete`, `cases`, `totals`, optional `baseline`) on stdout
-and sends progress to stderr. `--wait <seconds>` (default 300) bounds the verdict wait because
+and sends progress to stderr; each case lists its not-applicable evaluator versions in
+`notApplicable` and its advisory ones in `advisory`, and `totals.notApplicable` counts the
+former; a case failed for its telemetry has `state: "error"`,
+`passed: false` and
+`telemetry: { code: "telemetry_not_accepted", issues: [{ signal, kind, status?, count }] }`. `--wait <seconds>` (default 300) bounds the verdict wait because
 Hue-owned `world_outcome` checks are graded after the world seals. An experiment always covers
 every case of the saved version; there is no case subset.
 
 `--command "<shell command>"` spawns the command once per case with the world's environment
 (`HUE_WORLD_ID`, `HUE_WORLD_TOKEN`, one `HUE_SIM_<SURFACE ID>_URL` per provider mirror,
 `HUE_MCP_CONFIG` naming an owner-only `mcpServers` file that is removed after the case, and
-`HUE_MCP_URL`, `HUE_MCP_TOKEN`, `HUE_MCP_EXPIRES_AT` for the first MCP mirror), plus
+`HUE_MCP_URL`, `HUE_MCP_TOKEN`, `HUE_MCP_EXPIRES_AT` for the first MCP mirror; these name the
+world's simulated MCP server and replace a `HUE_MCP_URL` that `hue login` stored for Hue's project
+MCP server), plus
 `HUE_EXECUTION_ID`, `HUE_ENVIRONMENT_RUN_ID`, `HUE_CASE_ID` and `HUE_CASE_KEY`, and
 `{"inputs": ..., "config": ...}` on stdin. The child does not receive `HUE_API_KEY`, `HUE_MCP_KEY`
 or any other Hue control-plane credential unless `--allow-hue-credentials` is passed; the rest of
-the parent environment (model keys, application settings) is inherited. Its stdout is the answer
+the parent environment (model keys, application settings) is inherited. `HUE_CASE_DIR`,
+`HUE_CASE_INPUTS` and `HUE_CASE_OUTPUT_DIR` name a private case directory with the layout direct
+cases use: the case inputs, verified copies of the case's agent-visible pinned files under
+`files/<role>/`, and `output/`, whose files are uploaded after the world seals (the command still
+starts in the current directory). The case directory never holds the world token and is removed
+after the case. Its stdout is the answer
 (JSON when it parses, otherwise trimmed text; empty means no output); a non-zero exit or the
-per-case `--timeout` (default 600 seconds) is a target failure. The world token is never logged.
+per-case `--timeout` (default 600 seconds) is a target failure. A timed-out or interrupted command
+is stopped as a whole process group on macOS and Linux: SIGTERM, then SIGKILL for anything still
+running 5 seconds later, including an agent a compound command started after its shell exited;
+the case settles only once nothing in the group is left. A command that exits normally but leaves
+processes in its group has them stopped the same way before its answer and files are read. A
+second Ctrl+C during that grace kills the group at once, removes the world case's files, the MCP
+configuration file and the checkpoint locks, and exits with 130. The world token is never logged.
 A world created while the deployment's simulation gateway is off gets the legacy `hue_sim_`
 capability under the same `HUE_MCP_*` names. The agent key defaults to the slug of the command's
 script name; `--revision` is sent to Hue as the agent revision of every world.
 
 `--worker` registers the adapter through `runLocalAgent()` with key `--agent-key` (default: the
 adapter filename slug), name `--agent-name` (default: the key), revision `--revision` (default:
-`AGENT_REVISION`, then the Git `HEAD` short hash, then `dev`) and capability `environment:v1`,
-prints the registration and each claimed run with its URL, executes runs launched from Hue until
-Ctrl+C or `--max-runs <n>`, and prints the verdict table after each run. Selection flags do not
-apply; Hue chooses the pinned experiment. The worker exits 0 when it stops normally.
+`AGENT_REVISION`, then the Git `HEAD` short hash, then `dev`) and capability `environment:v1`
+plus any repeated `--capability <value>`, prints the registration and each claimed run with its
+URL, executes runs launched from Hue until Ctrl+C or `--max-runs <n>`, and prints the verdict table
+after each run. Selection flags do not apply; Hue chooses the pinned experiment. The worker exits 0
+when it stops normally. Hue offers a case whose world comes with agent-visible files only to a
+registration that declares `environment-files:v1` and `input:<extension>` for each file type, for
+example `--capability environment-files:v1 --capability input:pdf`; the adapter or command then
+receives the files as described above. Hue keeps each registered revision's capabilities fixed,
+so declare new ones under a new `--revision`. A world pinned to provider profiles, a Gmail world
+for example, needs a reviewed `attemptBaselineV2` (see
+[Pinned provider-profile preflight](ENVIRONMENTS.md#pinned-provider-profile-preflight)); `--worker`
+registers none, so put `"attemptBaselineV2"` in the run's configuration JSON when you launch the
+run from Hue.
 
 A Scenario or eval set whose dataset version is not saved cannot back an experiment: the command
 exits 1 and asks for **Save eval-set version** in Hue or `--save-version`, which freezes that
 version at its current revision. Connection settings are `HUE_API_KEY` and `HUE_BASE_URL`
 (default `https://app.hue.run`), loaded from `--env-file <path>` (or its alias `--env-path`) first when given; `--origin`
-overrides the origin. Telemetry content capture stays off unless `--content` is passed; in
-one-shot mode it also decides whether case outputs, error messages and explanations are persisted
-to Hue. `--worker` always persists them, because a run launched from Hue is read on its run page:
-that is `runLocalAgent()`'s contract and `--content` does not change it. Trace evidence is required
-for every case.
+overrides the origin. Telemetry content capture stays off unless `--content` is passed; the
+examples pass it so the run's case spans carry content. Model and tool spans inside the agent come
+only from the agent's own instrumentation. Case outputs, error messages and explanations are
+stored in Hue whether or not `--content` is passed, so a case's answer can be graded and read on
+its run page. A `--command`'s stdout is its answer unless it writes a result file, and is stored,
+so it must not print credentials or debug logs there; before storing, `hue eval` replaces the
+credentials it handed the case (the world token and MCP headers, a legacy MCP token, attempt
+bearers) and every Hue control-plane credential in its environment with `[redacted]`, in the
+answer, an adapter's thrown message and name, generated file names and the local checkpoint.
+`--no-output` keeps outputs, error messages and explanations out of a one-shot run's stored
+results; with `--content` the case span still carries the output. `--worker` always stores them, because a run launched from Hue is read
+on its run page (that is `runLocalAgent()`'s contract), and refuses `--no-output`. An interrupted
+one-shot run keeps the choice it started with, so one run never mixes stored and unstored outputs:
+rerunning it with other `--no-output` or `--content` flags is refused with the flags it started
+with. Resume a run that an earlier SDK started without `--content` by passing `--no-output`.
+
+Files the agent writes to `output/` are uploaded as generated documents with or without
+`--no-output`, and they are not redacted the way the answer is: `hue eval` replaces the same
+credentials only in the UTF-8 `.txt`, `.csv` and `.json` documents it collects, while PDF, Office
+and image documents, and files an adapter returns by `path`, are uploaded as written. A direct
+case also keeps the agent's raw `output/` files on disk under its checkpoint directory
+(`.hue/eval/<agent-key>/<project>/direct/<experiment>/files/`). Never write credentials to
+`output/`.
+
+Trace evidence is required for every case: when Hue does not accept a case's traces or logs, the
+case is completed as failed (error `TelemetryNotAccepted`, evidence omitted as
+`telemetry_not_accepted`, no output or generated files attached) instead of being left started,
+and the run goes on. Each export of the CLI's own telemetry may take 30 seconds, retries
+included: a slow acknowledgement is waited for, and a 429, 502, 503, 504 or network error is
+retried within that time, honouring `Retry-After`; any other response, or rejected records, fails
+the case at once. Stderr names the case as it completes, with the export
+issue counts, for example
+`[refund] telemetry not accepted, case failed: telemetry_not_accepted: traces failed 1 (HTTP 400)`;
+the case counts as an error in the table and JSON whatever its scores, and the command exits 1.
+Counts carry signals, kinds, HTTP statuses and record numbers only, never content or credentials.
 Resumable checkpoints live in `.hue/eval/<agent-key>/<project id>/` (a `.gitignore` is written
 inside `.hue/eval/`); `--checkpoint-dir` overrides the root. Rerunning the same selection resumes
 an interrupted run without invoking the agent again; a different selection is refused until the
 unfinished one is resumed or its directory is removed.
 
-Exit codes: `0` every case passed, `1` a case failed, errored, was skipped or Hue's checks were
+Exit codes: `0` every case passed the evaluators that apply to it (advisory ones never count),
+`1` a case failed, errored (including one no pinned evaluator applies to or decides), was skipped
+or Hue's checks were
 still pending at `--wait`, `2` usage or configuration error (including a missing key), `130`
 interrupted. On Node.js 22, load TypeScript adapters with `NODE_OPTIONS=--experimental-strip-types`;
 non-erasable syntax (enums, parameter properties, namespaces) needs a loader such as `--import tsx`
@@ -445,8 +619,8 @@ published version, beside or instead of explicit `--scorer-version` IDs.
 ```sh
 hue eval --set gia-d1-citation --scorer gia-d1-citation \
   --command "pnpm --filter @august/frontend run hue:gia-agent" \
-  --revision prompt-v10 --wait 1800 --json --env-file .env.hue
-hue eval --set gia-d1-citation --set-version 1 --scorer gia-d1-citation ./hue-agent.ts --baseline <experiment id>
+  --revision prompt-v10 --wait 1800 --content --json --env-file .env.hue
+hue eval --set gia-d1-citation --set-version 1 --scorer gia-d1-citation ./hue-agent.ts --content --baseline <experiment id>
 ```
 
 For each case the command is spawned once **inside a private case directory** with
@@ -460,6 +634,11 @@ For each case the command is spawned once **inside a private case directory** wi
 <case dir>/output/                write the generated documents here
 ```
 
+Hue verifies each generated document before the case completes. Once a document is reserved and
+its bytes are uploaded, a verification that outlasts a request is waited out for up to three
+minutes, so a completion that cannot reach Hue takes that long to fail; a reservation or upload
+that cannot reach Hue still fails at once.
+
 Every regular file the command leaves under `output/` is uploaded as a generated document
 (accepted: `.pdf .docx .pptx .xlsx .json .txt .csv .png .jpg .jpeg .webp`; another extension or an
 empty file is the case's error). Optional helpers: `manifest.json` (`{"primary": "<filename>",
@@ -469,6 +648,17 @@ the command's stdout is the output (JSON when it parses). A single generated fil
 document by default. Model and provider credentials stay in the command's own environment; the
 scoped case files are copies under `--checkpoint-dir` (default `.hue/eval/<agent-key>/<project>/direct/<experiment>`).
 
+The same collection serves world cases, and it does not trust paths the agent controls. `output/`
+must be a real directory, and a helper name, when present, a regular file; a symlink, FIFO or
+directory in either place is the case's error. Symlinks and other special files among the
+documents are skipped, as are hidden and lock entries. Every file is opened without following a
+final symlink or blocking, must be the regular file the listing saw and within its limit (4 MiB
+for a helper, 25 MiB for a document, checked before reading), and its upload is staged from the
+bytes read. A file or directory that vanished or changed while it was collected, more than 32
+documents or more than 1024 entries is the case's error too. A hard link is an ordinary file and
+is collected like one: an agent running as your user could copy the file just as well, so run an
+agent that must not reach your files under another account or in a sandbox.
+
 An adapter file works too: it is called with `(inputs, context)` where `context.mode` is
 `"direct"` and `context` carries `config`, `item`, `executionId`, `files` (agent-visible pinned
 files on disk), `outputDirectory` and `signal`. Return a JSON value, or `withFiles(output, files)`
@@ -478,5 +668,6 @@ Code evaluators pinned to the run are **not** executed on your machine: the CLI
 deferred (`deferUnboundLocalScorers`) and Hue's grading executor scores the uploaded documents;
 evaluator-only pinned files such as a legal corpus are never downloaded. `Waiting for Hue
 checks...` then covers that grading, so size `--wait` to the evaluator's runtime. The
-`--json` document gains `"mode": "direct"` and `"deferredScorerVersionIds"`. Exit codes, `--baseline`
-and checkpoints behave as for Scenarios.
+`--json` document gains `"mode": "direct"` and `"deferredScorerVersionIds"`. Exit codes, `--baseline`,
+`--content`, `--no-output` and checkpoints behave as for Scenarios; generated documents are
+uploaded with or without them.

package/ENVIRONMENTS.md

@@ -62,7 +62,7 @@ to the callback and is never written to checkpoints.
 
 ### Published pins and verdicts
 
-`scenario: { kind: "pins", datasetVersionId, scorerVersionIds, config?, name? }` runs already
+`definition: { kind: "pins", datasetVersionId, scorerVersionIds, config?, name? }` runs already
 published immutable pins, such as a Scenario's frozen case and Hue-owned outcome checks or a saved
 eval set with explicitly chosen scorer versions. `resolveScenarioPins(client, selector)` reads
 those pins from a Scenario ID, its Hue URL or its name (`listScenarios` and `getScenario` expose
@@ -76,9 +76,15 @@ bound `local_code` pins.
 Hue grades `world_outcome` pins after the world seals, so the runner's report precedes the
 verdicts. `waitForResults(client, { runId, scorerVersionIds, subjectIds, timeoutMillis })` polls
 until every item has a terminal result for every pin or the budget elapses (`complete: false`);
-`summarizeVerdicts` turns results into per-case rows with `passed` and totals; `compareVerdicts`
-diffs two summaries by case key; and `collectExperimentVerdicts` combines those reads for one
-experiment. The `hue eval` command uses the same path; see
+`summarizeVerdicts` turns results into per-case rows with `passed` and totals; a result Hue marks
+`notApplicable` (the evaluator has no outcome criteria or conversion rubric to grade in that case)
+neither passes nor fails its case and is listed in the row's `notApplicable`, and a case no pinned
+evaluator applies to is an error. A result of a Hue judge listed in `waitForResults`'
+`judgeScorerVersionIds` (`collectExperimentVerdicts` passes the experiment's `world_judge` pins)
+whose evidence says `advisory: true` keeps its metrics in the row and is listed in its
+`advisory`, but never decides the case; a case only advisory results ran for is an error. Any
+other result, an error, or one with a metric carrying `passed` is never advisory. `compareVerdicts` diffs two summaries by case key, and
+`collectExperimentVerdicts` combines those reads for one experiment. The `hue eval` command uses the same path; see
 [Evaluate an agent against a case](CLI.md#evaluate-an-agent-against-a-case).
 
 ### Pinned provider-profile preflight
@@ -223,6 +229,8 @@ its own Gmail MCP or REST client, the mirror URL, the world token where the Goog
 import {
   agentEnvironment,
   createEnvironmentClient,
+  HueEnvironmentError,
+  isTransientEnvironmentError,
   worldHandoff,
   writeMcpConfig,
 } from "@hue-run/sdk/environment";
@@ -252,12 +260,22 @@ try {
   );
   const sealDeadline = Date.now() + 30_000;
   let sealed = false;
-  while (Date.now() < sealDeadline) {
-    if ((await environmentClient.getRun(run.id)).status !== "open") {
-      sealed = true;
-      break;
+  while (!sealed && Date.now() < sealDeadline) {
+    let wait = 250;
+    try {
+      // The deadline also ends a read that hangs, with the client's retries inside it.
+      const state = await environmentClient.getRun(run.id, {
+        signal: AbortSignal.timeout(Math.max(0, sealDeadline - Date.now())),
+      });
+      sealed = state.status !== "open";
+    } catch (error) {
+      if (!isTransientEnvironmentError(error)) throw error; // a refusal such as 404 is final
+      wait = Math.max(wait, (error as HueEnvironmentError).retryAfterMs ?? 0);
     }
-    await new Promise((resolve) => setTimeout(resolve, 250));
+    if (!sealed)
+      await new Promise((resolve) =>
+        setTimeout(resolve, Math.min(wait, Math.max(0, sealDeadline - Date.now()))),
+      );
   }
   if (!sealed) throw new Error("World was not sealed after its completion grace");
 }
@@ -272,6 +290,27 @@ world token, so an adapter that read `HUE_MCP_URL` and `HUE_MCP_TOKEN` keeps wor
 no Hue-native `tools` (Hue refuses them); a world created while the gateway is off keeps its tools
 and the `hue_sim_` capability and emits a one-time `DeprecationWarning`.
 
+A case pinned to a world may also pin input files. Before the case's execution starts, the
+helpers download each agent-visible file (`source`, `attached_template`, `attached_reference`,
+`original`), verify its size and SHA-256 against the case manifest and hand the verified copies to
+the callback as `context.files` beside `context.world`, in a private directory (mode 0700, files
+0600), with a private `context.outputDirectory`. Evaluator-only files (`org_template`,
+`evaluator_reference`) never reach the agent; a local code evaluator's copies are checked before
+the execution starts but saved only after the agent finished. Bytes that differ from the manifest raise
+`CaseFileError` with code `case_file_mismatch`; a name that is not one safe file name (a path
+separator, `.` or `..`, a C0 or C1 control character, a character Windows reserves such as `:` or
+`?`, a Windows device name such as `CON`, a trailing dot or space, more than 200 bytes) raises
+`case_file_name_refused`. Either stops the run before an
+execution or world exists for the case. Return `withFiles(output, files)` to upload what the agent
+produced: the files are published and linked to the execution as `artifactIds` and
+`primaryArtifactId`, within the limits of direct cases (at most 32 files, 25 MiB each, the
+accepted document types). The world token is never written into the case directory, which is
+removed when the case ends; only staged outputs an interrupted upload resumes from are kept until
+it does. `hue eval --command` receives the same files through
+`HUE_CASE_DIR`, `HUE_CASE_INPUTS` and `HUE_CASE_OUTPUT_DIR`. A `runLocalAgent` worker is offered
+such cases only when it declares `environment-files:v1` and `input:<extension>` for each file
+type; see [EVALUATIONS.md](EVALUATIONS.md#direct-cases-and-files).
+
 A refusal such as 409 `simulation_gateway_required` exposes its validated server code as `HueEnvironmentError.diagnostic`.
 
 `agentEnvironment` removes Hue control-plane credentials from the child by default: `HUE_API_KEY`,