@hue-run/sdk 0.4.2 → 0.5.1

package/CLI.md

@@ -42,7 +42,7 @@ project manifest are refused because managers can update ancestor locks; Python
 The generated `hue.setup.mjs` or `hue_setup.py` always selects `captureContent: false` /
 `capture_content=False`. For a supported application, setup installs the dependency and adds the
 managed import and middleware registration to the existing entrypoint; an unreferenced helper is
-not a completed integration. TypeScript uses `@hue-run/sdk@0.4.2`, `@opentelemetry/api@1.9.1` and
+not a completed integration. TypeScript uses `@hue-run/sdk@0.5.1`, `@opentelemetry/api@1.9.1` and
 `@opentelemetry/context-async-hooks@2.11.0`; Python uses published `hue-run==0.2.2`.
 Content capture requires an ordinary account-managed key and a later explicit application decision.
 
@@ -79,7 +79,7 @@ events, checkpoints or receipts. Ordinary app starts have no handshake and retai
 listener behavior. This prevents accidental requests to an unrelated listener, not access by
 malicious code running as the same local user.
 
-Setup creates no Scenario, Hue Run, evaluation, source capture, worker or remote execution. Package
+Setup creates no simulation, Hue Run, evaluation, source capture, worker or remote execution. Package
 manager lifecycle scripts are disabled. The supported existing application entrypoint is executed
 directly with fixed argv solely for its bounded local HTTP verification; no shell command is accepted.
 
@@ -111,6 +111,8 @@ hue claim --restart       # explicit owner recovery; requires an interactive loc
 hue setup --agent         # noninteractive version-2 JSONL events
 hue setup --format human  # explicit append-only terminal rendering
 hue setup --origin http://127.0.0.1:PORT # isolated loopback tests only
+hue login                 # validate keys created in Hue and store them in ./.env.hue
+hue mcp install --client claude-code # write the Hue MCP configuration for a coding agent
 ```
 
 Hosted setup accepts HTTPS origins only. HTTP is accepted solely for `localhost`, `127.0.0.1` and
@@ -138,6 +140,63 @@ uses the observed predecessor for compare-and-swap. A conflict refreshes status
 At most 32 distinct handoff IDs exist per installation; `SETUP_HANDOFF_LIMIT` is terminal. Replacing a
 handoff does not create a trial, reset quota or rerun business work.
 
+## Sign in and store keys
+
+`hue login` stores keys that a person creates in Hue; it never mints one, because setup
+credentials are deliberately isolated from ordinary project keys. It prints the key settings page
+(`<origin>/settings/integrations`, opened in a browser only when a terminal is attached and
+`--no-browser` is absent), then reads one **Read and write** key from stdin without echo. By
+default that single key serves both uses: it is checked with `GET /api/v1/projects/current` and
+`GET /api/v1/datasets` (a **Read** or **Tracing only** key is refused because it cannot use
+evaluations) and with an MCP `tools/list` request, then stored as `HUE_API_KEY` with
+`HUE_BASE_URL` and `HUE_MCP_KEY` with `HUE_MCP_URL` in one write. A rejected key (`401` or `403`)
+exits `1` and stores nothing. `--keys evaluations` stores only `HUE_API_KEY`; `--keys coding-agent`
+stores only `HUE_MCP_KEY` and also accepts a **Read** key, for an agent that should only inspect the
+project. Run them separately with different keys if you want to revoke either use alone.
+
+```sh
+hue login                                  # one key for both uses into ./.env.hue
+hue login --keys coding-agent --gitignore  # only HUE_MCP_KEY; add .env.hue to .gitignore
+hue login --origin https://staging.hue.run --env-file .env.staging
+```
+
+The env file is written with mode `0600` through a temporary file and an atomic rename. Other
+lines are preserved; a symlink or a non-regular file is refused; an existing different value is
+replaced only with `--force`. Empty values, whitespace and URLs are refused before any request.
+The MCP endpoint is `https://mcp.hue.run/mcp` for `https://app.hue.run`,
+`https://mcp.staging.hue.run/mcp` for `https://staging.hue.run` and `<origin>/api/mcp` otherwise;
+plain HTTP origins are accepted for loopback test servers only. Output names variables and lengths
+(`Stored the key (NN chars) as HUE_API_KEY and HUE_MCP_KEY`), never values. When git does not ignore the env file, the command
+warns; `--gitignore` appends the file name to the `.gitignore` next to it. Exit codes: `0` stored,
+`1` failed, `2` usage error, `130` interrupted.
+
+## Install the MCP for your coding agent
+
+`hue mcp install --client <name>` writes, runs or prints the configuration for Hue's MCP server.
+Every shape matches the snippet Hue shows in Settings: server name `hue`, the endpoint (default
+`https://mcp.hue.run/mcp`; `--url https://mcp.staging.hue.run/mcp` for staging) and a reference to
+the `HUE_MCP_KEY` environment variable. A key value is never written.
+
+| Client | Result |
+| --- | --- |
+| `claude-code` | Merges `mcpServers.hue` into `./.mcp.json`. `--scope user` runs `claude mcp add --transport http --scope user hue URL --header 'Authorization: Bearer ${HUE_MCP_KEY}'` when `claude` is on `PATH`, otherwise prints it. |
+| `cursor` | Merges `mcpServers.hue` into `./.cursor/mcp.json` with `${env:HUE_MCP_KEY}`. |
+| `codex` | Runs `codex mcp add hue --url URL --bearer-token-env-var HUE_MCP_KEY`, or prints the `[mcp_servers.hue]` TOML block for `~/.codex/config.toml`. |
+| `vscode` | Merges `servers.hue` and the `hue-mcp-key` password input into `./.vscode/mcp.json`. |
+| `windsurf` | Prints the `serverUrl` snippet for `~/.codeium/windsurf/mcp_config.json`; nothing is written to the home directory. |
+| `gemini` | Runs `gemini mcp add --transport http hue URL -H 'Authorization: Bearer $HUE_MCP_KEY'`, or prints it. |
+
+JSON files are parsed and merged: other servers, inputs and top-level fields are kept, only the
+`hue` entry is replaced, and invalid JSON (including comments) is refused together with the snippet
+to add by hand. Files are written with mode `0644` through a temporary file and an atomic rename;
+symlinks are refused. `--dry-run` prints the resulting file content or command without writing or
+running; `--print` prints only the snippet. Client CLIs run without a shell, so the
+`${HUE_MCP_KEY}` and `$HUE_MCP_KEY` references reach them literally; the printed commands use
+single quotes for the same reason. After installation the command reminds you to export
+`HUE_MCP_KEY` in the shell that starts the client (VS Code prompts for the key instead) and prints
+the verification prompt: `Use the Hue MCP: call get_project_context, then show my 5 most recent
+error traces with links.` Exit codes: `0` done or printed, `1` failed, `2` usage error.
+
 ## Local state and conflicts
 
 Before its first network write, setup creates a lowercase UUIDv4 and a 32-byte random installation
@@ -241,7 +300,7 @@ node packages/sdk-typescript/scripts/verify-package.mjs --artifacts-dir .artifac
 # Set project to an existing supported fixture; use the same directory on resume.
 project=/absolute/path/to/supported-fixture
 node packages/sdk-typescript/scripts/verify-setup-live.mjs \
-  --archive .artifacts/typescript/hue-run-sdk-0.4.2.tgz \
+  --archive .artifacts/typescript/hue-run-sdk-0.5.1.tgz \
   --origin https://STAGING_ORIGIN \
   --project "$project" --command setup \
   --evidence .context/setup-staging-before-claim.json
@@ -252,7 +311,7 @@ the private local handoff and finish the real browser claim, then reconcile the
 
 ```sh
 node packages/sdk-typescript/scripts/verify-setup-live.mjs \
-  --archive .artifacts/typescript/hue-run-sdk-0.4.2.tgz \
+  --archive .artifacts/typescript/hue-run-sdk-0.5.1.tgz \
   --origin https://STAGING_ORIGIN \
   --project "$project" --command claim \
   --evidence .context/setup-staging-after-claim.json
@@ -273,3 +332,135 @@ only the private local file and produces no secret-bearing screenshots, traces,
 Publication under a candidate dist-tag, registry acceptance, production activation, promotion of
 the same version to `latest`, and clean-project literal `@latest` smoke remain separate gates; see
 [RELEASING.md](../../RELEASING.md).
+
+## Evaluate an agent against a Scenario
+
+`hue eval` runs a developer's local agent against a published Hue Scenario or a saved eval set
+and prints Hue's verdicts. It is the command-line form of `runSimulation()` (one-shot) and
+`runLocalAgent()` (worker): the agent, its prompts and its provider credentials stay in the local
+process, Hue creates one isolated simulated world per case, and Hue-owned outcome checks grade
+the sealed world. Hue never executes the agent. `HUE_API_KEY` must be a **Read and
+write** project key (a **Read** or **Tracing only** key cannot read Scenarios or create experiments);
+the CLI never prints it. The optional `zod` peer of `@hue-run/sdk/evals` must be installed.
+
+Write an adapter module that hands the case inputs and the world's tools or MCP connection to
+the real agent. The module exports `default` or `runMyAgent`; `context` is the SDK's
+`SimulationTargetContext` (`tools`, `mcp`, `config`, `item`, `executionId`, `environmentRunId`,
+`signal`):
+
+```ts
+// hue-agent.ts: erasable TypeScript only; Node.js 24 and Bun strip the types natively.
+import type { JsonValue, SimulationTargetContext } from "@hue-run/sdk/evals";
+import { runAgent } from "./src/agent.js"; // The application's existing entry point.
+
+export default function runMyAgent(inputs: JsonValue, context: SimulationTargetContext) {
+  // Hand context.tools or context.mcp to the agent's real tool boundary; configuration alone
+  // does not redirect provider calls. Pass context.signal for cooperative cancellation.
+  const { config, tools, mcp, signal } = context;
+  return runAgent({ inputs, config, tools, mcp, signal });
+}
+```
+
+```sh
+hue eval --scenario "Refund an eligible charge" ./hue-agent.ts --env-file .env.hue
+hue eval --scenario https://app.hue.run/projects/demo/scenarios/<id> ./hue-agent.ts --baseline <experiment id>
+hue eval --set "Billing regressions" --scorer-version <id> ./hue-agent.ts --save-version
+hue eval --scenario "Refund an eligible charge" --command "python agent.py" --timeout 120 --content
+hue eval --worker ./hue-agent.ts --agent-key support-agent --env-file .env.hue
+```
+
+The one-shot mode resolves the selection (`--scenario` by name, ID or URL; `--set` by name, ID or
+URL with explicit `--scorer-version` pins; or `--dataset-version` with `--scorer-version`),
+creates a fresh experiment from those immutable pins named `<scenario> · <agent key> · <revision>`
+(`--name` overrides), prints `Run: <url>` and `Experiment: <id>` as soon as the experiment exists,
+one line per case event (world created, agent started, world sealed), then
+`Waiting for Hue checks...` and a table with one row per case: boolean metrics as `PASS`/`FAIL`,
+numbers as values, text and category metrics as-is, an overall result per case and a pass count.
+Failing cases print the scorer explanation. `--baseline <experiment id|url>` adds improvement,
+regression and unchanged counts with per-case deltas. `--json` prints one JSON document
+(`experimentId`, `runId`, `runUrl`, `complete`, `cases`, `totals`, optional `baseline`) on stdout
+and sends progress to stderr. `--wait <seconds>` (default 300) bounds the verdict wait because
+Hue-owned `world_outcome` checks are graded after the world seals. An experiment always covers
+every case of the saved version; there is no case subset.
+
+`--command "<shell command>"` spawns the command once per case with `HUE_MCP_URL`,
+`HUE_MCP_TOKEN`, `HUE_MCP_EXPIRES_AT`, `HUE_EXECUTION_ID`, `HUE_ENVIRONMENT_RUN_ID`, `HUE_CASE_ID`
+and `HUE_CASE_KEY` in its environment and `{"inputs": ..., "config": ...}` on stdin. Its stdout is
+the answer (JSON when it parses, otherwise trimmed text; empty means no output); a non-zero exit
+or the per-case `--timeout` (default 600 seconds) is a target failure. The scoped MCP token is
+never logged. The agent key defaults to the slug of the command's script name.
+
+`--worker` registers the adapter through `runLocalAgent()` with key `--agent-key` (default: the
+adapter filename slug), name `--agent-name` (default: the key), revision `--revision` (default:
+`AGENT_REVISION`, then the Git `HEAD` short hash, then `dev`) and capability `environment:v1`,
+prints the registration and each claimed run with its URL, executes runs launched from Hue until
+Ctrl+C or `--max-runs <n>`, and prints the verdict table after each run. Selection flags do not
+apply; Hue chooses the pinned experiment. The worker exits 0 when it stops normally.
+
+A Scenario or eval set whose dataset version is not saved cannot back an experiment: the command
+exits 1 and asks for **Save eval-set version** in Hue or `--save-version`, which freezes that
+version at its current revision. Connection settings are `HUE_API_KEY` and `HUE_BASE_URL`
+(default `https://app.hue.run`), loaded from `--env-file <path>` first when given; `--origin`
+overrides the origin. Telemetry content capture stays off unless `--content` is passed; in
+one-shot mode it also decides whether case outputs, error messages and explanations are persisted
+to Hue. `--worker` always persists them, because a run launched from Hue is read on its run page:
+that is `runLocalAgent()`'s contract and `--content` does not change it. Trace evidence is required
+for every case.
+Resumable checkpoints live in `.hue/eval/<agent-key>/<project id>/` (a `.gitignore` is written
+inside `.hue/eval/`); `--checkpoint-dir` overrides the root. Rerunning the same selection resumes
+an interrupted run without invoking the agent again; a different selection is refused until the
+unfinished one is resumed or its directory is removed.
+
+Exit codes: `0` every case passed, `1` a case failed, errored, was skipped or Hue's checks were
+still pending at `--wait`, `2` usage or configuration error (including a missing key), `130`
+interrupted. On Node.js 22, load TypeScript adapters with `NODE_OPTIONS=--experimental-strip-types`;
+non-erasable syntax (enums, parameter properties, namespaces) needs a loader such as `--import tsx`
+on any Node.js version.
+
+## Evaluate a document eval set
+
+Eval sets whose cases pin no simulated world — a task plus pinned input files, answered with
+generated documents — run as **direct** cases through `runExperiment()`. `hue eval` detects this
+from the saved version (`--mode direct|simulation` overrides the detection; `--scenario` is always
+a simulation). `--set` accepts the eval set's slug, name, ID or URL; `--set-version <n>` pins a
+saved version other than the latest; `--scorer <slug|name|id>` pins an evaluator at its newest
+published version, beside or instead of explicit `--scorer-version` IDs.
+
+```sh
+hue eval --set gia-d1-citation --scorer gia-d1-citation \
+  --command "pnpm --filter @august/frontend run hue:gia-agent" \
+  --revision prompt-v10 --wait 1800 --json --env-file .env.hue
+hue eval --set gia-d1-citation --set-version 1 --scorer gia-d1-citation ./hue-agent.ts --baseline <experiment id>
+```
+
+For each case the command is spawned once **inside a private case directory** with
+`HUE_CASE_DIR`, `HUE_CASE_INPUTS`, `HUE_CASE_OUTPUT_DIR`, `HUE_CASE_ID`, `HUE_CASE_KEY` and
+`HUE_EXECUTION_ID` in its environment (`{"inputs","config"}` is also written to stdin):
+
+```text
+<case dir>/inputs.json            the case inputs (for example {"query": "...", "tipo_diligencia": "Virtual"})
+<case dir>/case.json              id, external key, execution id, run config, staged file list
+<case dir>/files/<role>/<name>    verified copies of the agent-visible pinned files
+<case dir>/output/                write the generated documents here
+```
+
+Every regular file the command leaves under `output/` is uploaded as a generated document
+(accepted: `.pdf .docx .pptx .xlsx .json .txt .csv .png .jpg .jpeg .webp`; another extension or an
+empty file is the case's error). Optional helpers: `manifest.json` (`{"primary": "<filename>",
+"output": <json>}`) names the primary document and the JSON output; `result.json` is the JSON
+output; `summary.txt` or `summary.md` is recorded as `{"summary": "..."}`. When none is written,
+the command's stdout is the output (JSON when it parses). A single generated file is the primary
+document by default. Model and provider credentials stay in the command's own environment; the
+scoped case files are copies under `--checkpoint-dir` (default `.hue/eval/<agent-key>/<project>/direct/<experiment>`).
+
+An adapter file works too: it is called with `(inputs, context)` where `context.mode` is
+`"direct"` and `context` carries `config`, `item`, `executionId`, `files` (agent-visible pinned
+files on disk), `outputDirectory` and `signal`. Return a JSON value, or `withFiles(output, files)`
+from `@hue-run/sdk/evals` to attach generated documents.
+
+Code evaluators pinned to the run are **not** executed on your machine: the CLI leaves them
+deferred (`deferUnboundLocalScorers`) and Hue's grading executor scores the uploaded documents;
+evaluator-only pinned files such as a legal corpus are never downloaded. `Waiting for Hue
+checks...` then covers that grading, so size `--wait` to the evaluator's runtime. The
+`--json` document gains `"mode": "direct"` and `"deferredScorerVersionIds"`. Exit codes, `--baseline`
+and checkpoints behave as for Scenarios.

package/ENVIRONMENTS.md

@@ -10,7 +10,7 @@ Your agent runs in your process while a disposable simulated world runs in Hue.
 authoritative and records an ordered journal; Hue does not execute your agent code or provider
 credentials.
 
-## Run a scenario like a test
+## Run a definition like a test
 
 `runSimulation` is the one-shot developer path. It calls your existing callback directly, so
 IDE breakpoints and cooperative cancellation work. It does not register a worker, poll for jobs,
@@ -29,7 +29,7 @@ try {
     client: createEvaluationClient(connection),
     environmentClient: createEnvironmentClient(connection),
     hue,
-    checkpointDirectory: ".hue-checkpoints/refund-scenario",
+    checkpointDirectory: ".hue-checkpoints/refund-definition",
     definition: { kind: "experiment", experimentId: process.env.HUE_EXPERIMENT_ID! },
     persistResultContent: true,
     traceEvidence: { mode: "required" },
@@ -60,6 +60,27 @@ identifies the same world for adapter control operations such as
 `environmentClient.recordCoverageGap`; it is not a credential. The MCP token is delivered only
 to the callback and is never written to checkpoints.
 
+### Published pins and verdicts
+
+`scenario: { kind: "pins", datasetVersionId, scorerVersionIds, config?, name? }` runs already
+published immutable pins, such as a Scenario's frozen case and Hue-owned outcome checks or a saved
+eval set with explicitly chosen scorer versions. `resolveScenarioPins(client, selector)` reads
+those pins from a Scenario ID, its Hue URL or its name (`listScenarios` and `getScenario` expose
+the underlying reads; a Read and write key is required), and `resolveEvalSetPins`
+resolves an eval set to its latest saved version. `runSimulation` creates the experiment directly
+(`name` defaults to the dataset name, `config` to `{}`) and binds the pins and configuration into
+the checkpoint identity, so resuming with different pins is refused like the other kinds.
+Hue-executed pins such as `world_outcome` need no local callback; pass `localScorers` only for
+bound `local_code` pins.
+
+Hue grades `world_outcome` pins after the world seals, so the runner's report precedes the
+verdicts. `waitForResults(client, { runId, scorerVersionIds, subjectIds, timeoutMillis })` polls
+until every item has a terminal result for every pin or the budget elapses (`complete: false`);
+`summarizeVerdicts` turns results into per-case rows with `passed` and totals; `compareVerdicts`
+diffs two summaries by case key; and `collectExperimentVerdicts` combines those reads for one
+experiment. The `hue eval` command uses the same path; see
+[Evaluate an agent against a Scenario](CLI.md#evaluate-an-agent-against-a-scenario).
+
 ### Pinned provider-profile preflight
 
 An experiment with an immutable `attemptBaselineV2` can require the local process to describe
@@ -84,10 +105,10 @@ facade acceptance remains a post-publication Fern integration gate. No public SD
 official Gmail service, and passing these tests is not evidence of universal Gmail or Slack
 parity. A matching Hue deployment and verified provider profile remain required.
 
-## Repository-authored scenarios
+## Repository-authored definitions
 
-Repository scenarios publish through the same validated environment, dataset, scorer and
-experiment APIs as Hue-authored scenarios. Stable slugs reuse matching immutable content
+Repository definitions publish through the same validated environment, dataset, scorer and
+experiment APIs as definitions authored in Hue. Stable slugs reuse matching immutable content
 digests; changed definitions, tasks or scorers publish new versions. Hue does not synchronize
 files back from its UI, and the helper refuses an unrelated mutable dataset draft instead of
 overwriting it.
@@ -106,7 +127,7 @@ the same before comparing immutable digests, so casing-only UUID changes reuse t
 version without dropping provider bindings.
 
 ```ts
-const scenario = {
+const definition = {
   kind: "repository" as const,
   name: "Refund an eligible charge",
   slug: "refund-eligible-charge",
@@ -130,8 +151,8 @@ await runSimulation({
   client,
   environmentClient,
   hue,
-  checkpointDirectory: ".hue-checkpoints/refund-scenario",
-  scenario,
+  checkpointDirectory: ".hue-checkpoints/refund-definition",
+  definition,
   persistResultContent: false,
   traceEvidence: { mode: "required" },
   target: (inputs, context) => runMyExistingAgent({ inputs, ...context }),

package/EVALUATIONS.md

@@ -9,7 +9,7 @@ npm install @hue-run/sdk zod
 
 The SDK executes targets and scorers on your machine. Hue stores pinned definitions, experiment progress and results. It does not execute uploaded source code. Follow the [installation guide](https://docs.hue.run/installation) to add `@hue-run/sdk` to your application.
 
-Create a **Tracing and evaluations** project service key under **Settings → Integrations & API keys** and expose it to this server-side process as `HUE_API_KEY`. A **Tracing only** key cannot author datasets or evaluation runs.
+Create a **Read and write** project service key under **Settings → Integrations & API keys** and expose it to this server-side process as `HUE_API_KEY`. A **Tracing only** key cannot author datasets or evaluation runs.
 
 ```ts
 import { randomUUID } from "node:crypto";
@@ -72,10 +72,20 @@ try {
 
 Create another experiment with the same frozen version and different `config` to compare configurations. The runner reads the exact experiment case/version and scorer definitions; it never resolves a mutable latest version. `rescore` accepts an existing evaluation-run ID and has no target callback. Subject IDs refer to immutable saved outputs and trace evidence.
 
+`rescore` preserves terminal scores already recorded for each item and evaluator version,
+including built-in checks scheduled by Hue's **Grade again** flow. It computes only the missing
+local scores and reports the existing receipts alongside the new ones. If another executor
+finishes the same score while this one uploads, the runner accepts that exact item/version
+receipt; an unrelated conflict still fails. To evaluate the same saved output again, create a new
+scoring run.
+
 For the shorter agent-against-a-hosted-world workflow, use `runSimulation`. It owns immutable
 resolution, a fresh linked world per case, local and hosted MCP tools, finalization, sealed
 evidence and scoring while retaining this runner's checkpoint guarantees. See
-[Simulated environments](ENVIRONMENTS.md#run-a-scenario-like-a-test).
+[Simulated environments](ENVIRONMENTS.md#run-a-definition-like-a-test).
+
+Cases whose inputs are a task plus pinned documents, and rescoring over documents a run already
+saved, use the same runner without a world. See [Direct cases and files](#direct-cases-and-files).
 
 ## Content and result states
 
@@ -110,7 +120,7 @@ const published = await client.publishScorerVersion(identity.id, local.definitio
 // Pin published.id in the experiment and pass scorers:[local] to the runner.
 ```
 
-A callback receives `{inputs,hasOutput,output?,hasExpected,expected?,metadata,executionState}` and returns one of:
+A callback receives `{inputs,hasOutput,output?,hasExpected,expected?,metadata,executionState,files?}` and returns one of:
 
 - `{state:"scored",metrics:[{name,value,passed?}],explanation?,evidence?}` (explanation or evidence required).
 - `{state:"error",error:{type,message?}}`.
@@ -118,6 +128,13 @@ A callback receives `{inputs,hasOutput,output?,hasExpected,expected?,metadata,ex
 
 All declared metrics must appear exactly once and satisfy pinned types, bounds and categories. The binding must match language, entrypoint, SHA-256 source digest and metric definitions. The digest is an authenticated caller declaration; it does not attest closures, dependency versions or actual execution. Callback source is never downloaded or evaluated. Callbacks are trusted local code; they have **no execution timeout or side-effect cancellation**. Concurrency limits active cases to 1–16 (default 1), with scorers evaluated sequentially within each case.
 
+`files` is present only when the runner handled files for that execution: every pinned input file
+(all roles, including the evaluator-only `org_template`) and every generated output, each with its
+verified local `path`, `role`, `filename`, `contentType`, `byteSize`, `sha256`, `artifactId` and
+`primary` flag. A `local_code` callback runs when generated files exist even without a JSON output;
+built-ins keep their usual missing-output behavior. See
+[Direct cases and files](#direct-cases-and-files).
+
 ### Hosted and manual scorer pins
 
 Scorer deferral shipped in TypeScript `0.3.1`: the local runner executes only the three known built-in entries and bound `local_code` scorers. It leaves every other pin pending and reports its ID in `deferredScorerVersionIds`, including kinds and built-in entries introduced by a newer server. It never uploads a placeholder result that would occupy the immutable result slot, including placeholders already saved in an older SDK's checkpoint. Direct `scoreLocally()` calls reject pins that require another executor.
@@ -156,7 +173,7 @@ The runner stops scheduling more cases after an operational failure and waits fo
 
 ## Verification boundaries
 
-`scripts/verify-package.mjs` installs a real packed tarball outside the monorepo and runs HTTP contract tests against a synthetic service plus actual OpenTelemetry exporters. It checks two configurations, rescoring without target invocation, absent/null output, upload resume, uncertain execution, exclusive checkpoints, source/metric contracts, content policy and terminating schema workers. It also exercises the local worker's ready, incomplete and uncertain provider-attempt control-plane paths, but does not call an issued provider facade. `scripts/verify-evaluation-api.mjs` is a separate opt-in acceptance against a real Hue receiver/API; it creates synthetic datasets/scorers/experiments in the project associated with the supplied development key.
+`scripts/verify-package.mjs` installs a real packed tarball outside the monorepo and runs HTTP contract tests against a synthetic service plus actual OpenTelemetry exporters. It checks two configurations, rescoring without target invocation, absent/null output, upload resume, uncertain execution, exclusive checkpoints, source/metric contracts, content policy and terminating schema workers. It also exercises the local worker's ready, incomplete and uncertain provider-attempt control-plane paths, but does not call an issued provider facade. `scripts/verify-file-cases.mjs` runs against the same installed consumer: pinned input download and verification, artifact publication of a generated document, file-aware local scoring during the run and again on a later rescore, and a declared file the target could not deliver recorded as the target's own error. `scripts/verify-evaluation-api.mjs` is a separate opt-in acceptance against a real Hue receiver/API; it creates synthetic datasets/scorers/experiments in the project associated with the supplied development key.
 
 ## Outbound local agent worker
 
@@ -168,7 +185,9 @@ establish hosted provider availability.
 `runLocalAgent` registers one fixed application callback and polls for queued runs. Hue selects
 the registered key/revision; it does not send executable code or shell commands. Keep the
 checkpoint directory private and durable. The worker persists result content and requires
-acknowledged trace and sealed environment evidence.
+acknowledged trace evidence; a case pinned to a world also requires sealed environment evidence.
+A case without a world runs through `directTarget` instead, described in
+[Direct cases and files](#direct-cases-and-files).
 
 ```ts
 import { createHue } from "@hue-run/sdk";
@@ -218,3 +237,105 @@ the worker reports `attention` and stops; operator investigation is required. Su
 automatically reclaimed, and presenting the same uncertain checkpoint again cannot replay the
 candidate. Public package acceptance proves this lifecycle against local fixtures; exact
 installed-registry-package to hosted-facade acceptance remains a post-publication Fern gate.
+
+## Direct cases and files
+
+Cases without a simulated world — document workflows whose inputs are a task plus pinned files —
+run on the same runner under the same checkpoint rules. This is unreleased and ships in the next
+`@hue-run/sdk` release. It requires a Hue deployment that serves case `inputFiles` on experiment
+items, subject `files`, and the artifact reservation, upload, completion and download APIs. The
+Python SDK has no equivalent.
+
+On the outbound worker, supply `directTarget` beside or instead of `target`. Supplying
+`directTarget` registers the `direct:v1` capability and supplying `target` registers
+`environment:v1` (both strings are exported as `localAgentCapabilities`). Declare the file
+capabilities the agent accepts and returns — `input:docx`, `input:pdf`, `output:docx` and so on — so
+Hue matches them against each case's `hue.requiredCapabilities` metadata and offers only matching
+cases. `registeredCapabilities` refuses a registration that names a capability without its callback.
+
+```ts
+import { basename } from "node:path";
+import { createEnvironmentClient } from "@hue-run/sdk/environment";
+import { createEvaluationClient, runLocalAgent, withFiles } from "@hue-run/sdk/evals";
+import { runMyAgent } from "./document-agent.js"; // Your existing application entry point.
+import { letterGrader } from "./letter-grader.js"; // A defineLocalScorer binding.
+
+await runLocalAgent({
+  client: createEvaluationClient(connection),
+  environmentClient: createEnvironmentClient(connection),
+  hue,
+  checkpointDirectory: ".hue-checkpoints/document-worker",
+  agent: {
+    key: "letter-agent",
+    name: "Letter agent",
+    revision: process.env.GIT_COMMIT!,
+    capabilities: ["input:docx", "input:pdf", "output:docx"], // direct:v1 is added for directTarget
+  },
+  scorers: [letterGrader], // Its callback reads context.files.
+  directTarget: async (inputs, { config, files, outputDirectory }) => {
+    // `files` are verified copies of the case's agent-visible input files on disk.
+    const { summary, documents } = await runMyAgent({ inputs, config, files, outputDirectory });
+    return withFiles(
+      { summary },
+      documents.map((path, index) => ({
+        path,
+        filename: basename(path),
+        contentType: "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
+        primary: index === 0,
+      })),
+    );
+  },
+});
+```
+
+The direct callback receives cloned inputs and an allowlisted context of `config`,
+`item: {id, externalKey}`, `executionId`, `trace: {traceId,spanId}`, `files` and `outputDirectory`,
+plus the worker's `signal` when one was supplied. Expected outcomes and case metadata stay with
+grading, as they do for world cases.
+
+Before an execution exists, `runExperiment` downloads every pinned input file named by the frozen
+case's `inputFiles` and verifies byte count and SHA-256. A download failure is an SDK failure and
+consumes no execution slot; a file already saved with the pinned identity is reused instead of
+downloaded again. The target sees only the agent-visible roles — `source`, `attached_template`,
+`attached_reference` and `original` — as `context.files`; evaluator-only
+`org_template` and `evaluator_reference` files (an organization's template, a legal corpus, an
+answer key) reach scorers but not the agent, and are not even downloaded when no bound code
+evaluator runs in this process. Verified copies live under `filesDirectory` (default
+`<checkpointDirectory>/files`, created mode 0700), and each case gets its own private
+`context.outputDirectory` to write into.
+
+Return generated documents by wrapping the output in `withFiles(output, files)`. Each entry names a
+`path` or in-memory `bytes`, a `filename`, a Hue-accepted `contentType` and at most one `primary`.
+The runner copies them next to its checkpoint, publishes them through the artifact reservation,
+upload and verified completion APIs with stable per-execution keys, and completes the execution with
+`artifactIds` and `primaryArtifactId`, so the subject's frozen manifest holds inputs and outputs
+together. Generated files are always uploaded regardless of `persistResultContent`: they are the
+execution's evidence. A declared file that cannot be read, exceeds 25 MiB, repeats a filename or has
+an unsupported content type is saved as the target's error (`TargetError`), not as an uncertain
+execution. A crash after the target finished resumes from the staged files without invoking the
+target again; if result content is not persisted, the JSON output cannot be reconstructed and the
+case is reported as uncertain.
+
+`rescore` downloads a subject's frozen `files` — the pinned inputs and the generated outputs, with
+`role: "output"` for the documents a run produced — so a code evaluator can grade saved documents
+without invoking an agent. Local scorers receive them as `context.files` exactly as they do during
+an experiment.
+
+`runExperiment` and `rescore` accept `environmentEvidence: "required" | "when_pinned"`. A connected
+worker that supplies `directTarget` uses `when_pinned`: direct cases never contact the environment
+evidence endpoint, while cases pinned to a world still require sealed evidence. Use the same policy
+for file-only regrading, and keep `required` for a generic target that attaches a world
+independently of the case pin.
+
+### Grading on Hue's side
+
+By default a pinned `local_code` version without a matching local scorer refuses the run: nobody
+would produce its result. `deferUnboundLocalScorers: true` changes that for both `runExperiment`
+and `rescore`: unbound code-evaluator pins are reported in `deferredScorerVersionIds` and left to
+the executor that owns their source, typically a grading worker Hue operates with its own provider
+credentials. The customer's process then runs only the agent and uploads its documents; nothing
+grader-related is installed there and evaluator-only files never reach it. The grading worker
+calls `rescore` on the same run with the evaluator bound and the same option, so pins belonging to
+other evaluators are left alone rather than refused. `EvaluationClient.listEvaluationRuns` pages
+the project's runs so such a worker can find the ones that still owe it results. `hue eval` uses
+this mode for eval sets whose cases pin no simulated world.

package/README.md

@@ -74,9 +74,15 @@ preflight checks availability and presents the published [privacy notice](https:
 and [security information](https://trust.hue.run/) before telemetry. Anonymous trials last 24 hours
 and are limited to 100 traces, 1,000 spans and 2 MiB. A private owner-only browser handoff supports
 account linkage; the original request evidence is retained and business work is never replayed on
-claim. Setup never enables content capture or creates a Scenario, Hue Run, evaluation, source capture
+claim. Setup never enables content capture or creates a simulation, Hue Run, evaluation, source capture
 or remote execution. See the [setup CLI contract](./CLI.md) for the supported shapes and release gates.
 
+For an existing account, `hue login` validates keys created in Hue Settings and stores them in
+`.env.hue` without printing them, and `hue mcp install --client claude-code` (or `cursor`, `codex`,
+`vscode`, `windsurf`, `gemini`) writes the Hue MCP configuration that references `HUE_MCP_KEY`. See
+[Sign in and store keys](./CLI.md#sign-in-and-store-keys) and
+[Install the MCP for your coding agent](./CLI.md#install-the-mcp-for-your-coding-agent).
+
 `checkConnection()` rejects with `HueConnectionError`: its fixed message is safe to log, `status`
 carries the HTTP status when Hue answered, and `cause` carries the underlying network, timeout or
 parsing error. `serviceVersion` and `resourceAttributes` (for example
@@ -423,13 +429,32 @@ an uncertain preparation. See the
 an issued facade endpoint or the official Gmail service, and do not claim universal provider
 parity.
 
+An unreleased `directTarget` callback extends the same worker to cases without a world: the runner
+verifies the case's pinned input files, hands them to the agent, and uploads the documents it
+returns as verified Hue artifacts for scoring. See
+[direct cases and files](EVALUATIONS.md#direct-cases-and-files).
+
 Scorer deferral shipped in TypeScript `0.3.1`. Only built-ins
 and bound `local_code` callbacks run here; other pins remain pending for their authorized executor.
 See [scorer execution](EVALUATIONS.md#hosted-and-manual-scorer-pins).
 
 The published [setup CLI](CLI.md) is a resumable local inspection core; the unreleased `0.4.0`
 candidate adds the bounded application onboarding flow described above. Existing customers connect
-their agents with `runLocalAgent()`; setup does not register workers or launch Scenarios.
+their agents with `runLocalAgent()`; setup does not register workers or launch simulations.
+
+### Command-line evaluation
+
+The unreleased `hue eval` command wraps `runSimulation()` and `runLocalAgent()` for an adapter
+file or a shell command: `hue eval --scenario "<name>" ./hue-agent.ts` creates a fresh experiment
+from a published Scenario's immutable pins, runs the agent in one isolated world per case, waits
+for Hue's outcome checks and prints the run URL and per-case PASS/FAIL verdicts with an exit code;
+`--worker` registers the same adapter for runs launched from Hue. It needs a Read and
+write key in `HUE_API_KEY` (never printed) and keeps content capture off unless `--content`
+is passed. See [Evaluate an agent against a Scenario](CLI.md#evaluate-an-agent-against-a-scenario).
+Eval sets whose cases pin files instead of a world run as direct cases through `runExperiment()`:
+`hue eval --set <slug> --scorer <slug> --command "…"` hands the agent each case's pinned files in a
+private directory, uploads the documents it writes and waits for Hue's grading executor to score
+them. See [Evaluate a document eval set](CLI.md#evaluate-a-document-eval-set).
 
 ## Managed targets
 

package/dist/cli/eval-direct.d.ts

@@ -0,0 +1,47 @@
+import type { LocalFile, TargetResult } from "../evals/types.js";
+import type { JsonValue } from "../types.js";
+/**
+ * Direct (file) cases for `hue eval`: a case is a task plus pinned input files, and the agent's
+ * answer is one or more generated documents. The agent command receives a private case directory
+ * instead of stdin/stdout JSON, the same layout Hue's document workers use:
+ *
+ *   <case dir>/inputs.json            the case inputs
+ *   <case dir>/case.json              id, external key, run config and the staged file list
+ *   <case dir>/files/<role>/<name>    verified copies of the agent-visible pinned files
+ *   <case dir>/output/                where the command writes what it produced
+ *
+ * Everything under `output/` becomes the execution's generated files. Optional helpers there:
+ * `manifest.json` (`{ "primary": "<filename>", "output": <json> }`), `result.json` (the JSON
+ * output) and `summary.txt` / `summary.md` (recorded as `{ "summary": "…" }`).
+ */
+export declare const directCaseEnvironment: readonly ["HUE_CASE_DIR", "HUE_CASE_INPUTS", "HUE_CASE_OUTPUT_DIR", "HUE_CASE_ID", "HUE_CASE_KEY", "HUE_EXECUTION_ID"];
+/** Generated-file extensions Hue accepts and the content type recorded for each. */
+export declare const outputExtensions: Readonly<Record<string, string>>;
+export interface DirectCaseLayout {
+    caseDirectory: string;
+    inputsPath: string;
+    outputDirectory: string;
+    files: {
+        role: string;
+        filename: string;
+        path: string;
+    }[];
+}
+/** Writes the case directory: inputs, descriptor, verified file copies and an empty output folder. */
+export declare function stageDirectCase(root: string, input: {
+    inputs: JsonValue;
+    config: JsonValue;
+    item: {
+        id: string;
+        externalKey: string;
+    };
+    executionId: string;
+    files: LocalFile[];
+}): Promise<DirectCaseLayout>;
+/**
+ * Turns the output folder into the target's result. Every regular file, in subdirectories too,
+ * becomes a generated file; an unsupported extension is the agent's error rather than a silently
+ * dropped document. `fallbackOutput` (for example the command's stdout) is used when no JSON
+ * output or summary file was written.
+ */
+export declare function collectDirectOutputs(outputDirectory: string, fallbackOutput?: JsonValue): Promise<TargetResult>;