coder-eval 0.12.10__tar.gz → 0.12.12__tar.gz

{coder_eval-0.12.10 → coder_eval-0.12.12}/.claude/notes/agents.md

@@ -434,10 +434,13 @@ agent had produced; bounding it only by a fixed cycle count disconnected from `t
 path unreachable — the watchdog always wins, and the same spurious-orphan turn burns the
 full turn timeout before crashing with zero criteria evaluated.
 
-The cycle cap applies alongside that deadline, whichever comes first, and is the SOLE
-bound when a task sets no timeout at all. Under a long `turn_timeout` (1800 s gives a
-1440 s deadline) the deadline alone let a never-ending command (a dev server, an
-unanswered prompt) idle the turn for 24 minutes. It is deliberately not
+The cycle cap (120 × 5 s) bounds the wait only when a task sets no timeout at all. With
+one, the deadline alone bounds it, so a never-ending command (a dev server, an unanswered
+prompt) can idle up to 80% of the turn, but a slow job gets the time the task author
+budgeted for it. Applying the cap under a timeout as well force-closed real solvers and
+simulations at 10 minutes: on the SkillsBench Gemini 4-arm campaign it ended 22 of 348
+rows, 4 of which had passed with the deadline alone (an exam-scheduling MIP solve that
+passed at 1211 s among them). It is deliberately not
 "break after N consecutive empty polls": `receive_steps()` returns identically empty
 whether a backgrounded job is still running or will never resolve, and there is no signal
 that tells the two apart except waiting. A count small enough to matter would abort real
@@ -464,10 +467,32 @@ after the model had already finished. `finalize` force-closes them as unresolved
 way it confines Claude Code. Without it Antigravity ran with every builtin, including
 `start_subagent` and `search_web`, under a list that gave Claude Code neither. `Skill` and
 `TodoWrite` have no builtin (skills load through `skills_paths`) and are skipped. `finish`
-stays on under any allowlist: it returns structured output, not a capability.
-`enable_subagents` is a separate switch from the toolset and follows whether
-`start_subagent` survives. The SDK takes an allowlist OR a denylist, so with both set the
-denied tools are removed from the allowlist.
+and `schedule` stay on under any allowlist: `finish` returns structured output, and
+`schedule` is the timer the model sleeps on while a backgrounded `run_command` finishes.
+Without `schedule`, Gemini re-checks the background task every few seconds and each check
+is a model call against `max_turns`: on the SkillsBench Gemini 4-arm campaign mean turns
+rose 36.8 → 50.2 and max-turn rows 4 → 16 on the same 110 rows, and the skill arms were hit
+2-3x harder than baseline. With `schedule` on they came back to 35.4 and 1.
+
+An allowlist only narrows the harness's default toolset (`BuiltinTools.default()`). `Glob`,
+`Grep` and `LS` map to `find_file`, `search_directory` and `list_directory`, which the
+harness ships off, so an allowlist naming them leaves them off. Turned on, Gemini used them
+in place of shell scripts (1,429 calls in 333 of 348 SkillsBench rows, none before), often
+one search per turn.
+
+`start_subagent` also stays on under any allowlist, because the harness builds its system
+prompt from the toolset: with `start_subagent` off it drops the whole subagents section
+(4,801 → 2,783 characters), and with it the prompt's only "you do NOT need to poll ... you
+will be notified" guidance. With it on, the system prompt is byte-identical to 0.12.9's.
+The section alone does not stop Gemini polling a long job with status checks: in a 48-row
+SkillsBench A/B it made 13.4 checks per row with the section and 13.7 without (8.5 on
+0.12.9). The tool descriptions are identical either way, and
+`enable_subagents` alone does not restore the section. Subagents the tool lists withhold
+(no `Task`) are denied by policy at `invoke_subagent` instead, which covers both the
+built-in `research` subagent and one the model defines with `define_subagent`; both get
+`search_web` regardless of the parent's toolset, and a parent policy on `search_web` does
+not reach a model-defined one. The SDK takes an allowlist OR a denylist, so with both set
+the denied tools are removed from the allowlist, except the always-enabled ones.
 
 ## Antigravity non-interactive commands
 
@@ -478,7 +503,7 @@ behind keeps the turn waiting on a prompt no one answers. `_NONINTERACTIVE_ENV`
 (`CI`, `npm_config_yes`, `GIT_TERMINAL_PROMPT=0`, `DEBIAN_FRONTEND`, `PIP_NO_INPUT`,
 `PAGER`/`GIT_PAGER=cat`) rides the per-agent `env` seam, each variable only where the
 environment does not already set it, so those commands answer themselves or fail fast. It
-cannot close the terminal: a bare `read` or a server still runs until the poll cap.
+cannot close the terminal: a bare `read` or a server still runs until the poll deadline.
 
 ## The receive_steps re-entrancy window
 
@@ -533,7 +558,7 @@ shell-aware `parameters["command"]` extraction in `criteria/command_executed.py`
 to raw-JSON matching — so the same task scores differently per harness. Unknown names pass
 through unchanged.
 
-Three cases are worth knowing:
+Four cases are worth knowing:
 
 - **OpenCode's tool set varies by MODEL within the one harness.** A live 174-task run
   showed DeepSeek using `write`/`edit` 199 times and `apply_patch` 0, while GPT-5.6 used
@@ -546,6 +571,11 @@ Three cases are worth knowing:
 - **Pi's search tool is `find`** (glob-by-pattern), not `glob`; there is no `glob` tool in
   its built-in set, so mapping `find` to the canonical `Glob` is what keeps
   `command_executed` and `commands_efficiency` comparable.
+- **Delegate's host maps most names itself, but not `LoadSkill`.** `delegate-stdio` reports
+  `ReadFile` as `Read` and `ExecuteSkillApi` as `Skill`, so `DelegateAgent` maps only
+  `LoadSkill {name}` to `Skill {skill}`; without it `skill_triggered` never sees a Delegate skill
+  load. The rename is keyed by the host's name: keyed by `Skill`, it would also turn
+  ExecuteSkillApi's `name`, an API call, into a skill engagement.
 
 Antigravity additionally strips the result payload out of a tool call's arguments: the
 harness folds result fields into the same `args` dict at DONE. Beyond a static key list,
@@ -776,89 +806,117 @@ release adding an unclassified field fails loudly instead of silently passing th
 
 ## Delegate agent
 
-`DelegateAgent` drives UiPath Autopilot's Delegate agent — reasoning in the UiPath backend,
-tools executing locally through the SDK's bundled interop process. Unlike every other CLI-driven
-agent in this file, there is no vendor-provided stdio protocol host to spawn: `@uipath/delegate-stdio`,
-the internal package a UiPath-only sibling plugin (`coder_eval_uipath`) drives, is not public. Only
-`@uipath/delegate-sdk` (a programmatic library exposing a `DelegateAgent` class with
-`initialize()`/`onEvent()`/`sendMessage()`/`destroy()`) and `@uipath/delegate-cli` (a terminal wrapper
-around it) are. So this agent ships its own first-party Node host, `agents/delegate/delegate_host.mjs`,
-which wraps the SDK class in a newline-JSON stdio protocol this framework designed, not one it had to
-reverse-engineer — verified live against a real `@uipath/delegate-sdk@0.1.12` install with no token,
-confirming both the plain-public-install story and that the SDK's own event vocabulary
-(`session_start`/`thinking`/`message`/`tool_call`/`tool_result`/`error`/`done`/`step`) substantially
-matches the internal sibling's protocol.
-
-**No multi-generation transcript splitting.** The internal sibling reconstructs one `AssistantMessage`
-per backend round-trip from an `isStepStart` flag and a `turnUsages` array on the host's terminal
-message. Neither is present on `DelegateAgent.onEvent`'s payloads (confirmed absent by grepping the
-installed SDK bundle's string literals), so there is no reliable per-round-trip boundary signal at this
-API layer. `DelegateAgent` builds exactly ONE `AssistantMessage` per `communicate()` call instead — a
-deliberate simplification, not an oversight; richer segmentation can be added once a real boundary
-signal is confirmed against a live backend. `timing.close_window` still opens that one window from the
-turn's own start (there is nothing to tile from), so the head and tail both measure ~0.
-
-**Deferred, not ported speculatively.** The internal sibling's module docstring documents several
-hard-won failure-signature-specific recoveries: a Cloudflare WAF-block-page rewrite, an SSE-connect-
-timeout rewrite, session-conflict fresh-host recovery, and first-response stall-timeout+resend. None
-are ported here — porting a marker tuned to the internal sibling's own observed failures risks matching
-nothing (or the wrong thing) against the public SDK/backend's actual error surface. A crash still ends
-the turn correctly as a retryable `AgentCrashError`; it is just not specially diagnosed. Port these once
-the same failures are actually observed running this agent for real.
-
-**The process handle must be cleared on every path that leaves the host dead or dying** — EOF, a host
-`fatal` message, a timeout (both the top-of-loop pre-check AND a timeout elapsing while blocked inside
-`asyncio.wait_for`), cooperative stop, and `max_turns` exhaustion. A real bug shipped once during this
-agent's own development: a timeout elapsing mid-read fell through to the generic crash path instead of
-`TurnTimeoutError`, and left the process handle set, so the NEXT `communicate()` call reused a host with
-a `"send"` still nominally in flight instead of respawning — risking a stale response being consumed as
-the new turn's. `tests/test_delegate_agent.py`'s `test_timeout_elapsing_mid_read_still_raises_turn_timeout_error`
-pins the fix.
-
-**Skills mapping is deliberately NOT the shared `agents/_skills.py` resolver.** That resolver enumerates
-individual skill directories for a repeated `--skill <dir>`-style CLI argument (OpenCode/Pi's shape).
-The Delegate SDK's `bundledSkillsPath` wants exactly ONE parent directory whose children are skill
-folders — a genuinely different shape — so `_resolve_bundled_skills_path` mirrors the internal sibling's
-own mapping (`<plugin.path>/skills`, first plugin wins) instead of force-fitting the shared helper.
-
-**Registered unconditionally**, exactly like `codex`/`antigravity` — there is no conditional-registration
-gate keyed on an optional-dependency extra anywhere in this codebase; `opencode`/`pi` declare an EMPTY
-extra purely as packaging-metadata documentation, and `delegate` follows that same shape (no new pip
-package — Node/`@uipath/delegate-sdk` is the real prerequisite, resolved lazily in `start()`).
-
-**`environment` resolution needs org/tenant SLUGS, not just IDs — confirmed by reading the installed
-SDK's own bundle, not by guessing from its error message.** When `DELEGATE_BACKEND_URL` is absent and
-`DELEGATE_ENV` (-> `environment` init option) is used instead, the SDK resolves the backend URL from
-`organizationName`/`orgLogicalName` and `tenantName` fields on the `auth` object passed to
-`initialize()` — NOT from `ORG_SLUG`/`TENANT_SLUG` read off `process.env` directly, despite that being
-exactly what the SDK's own thrown error suggests ("set ORG_SLUG and TENANT_SLUG env vars alongside
-ORG_ID/TENANT_ID"). That advice describes the separate `@uipath/delegate-cli`/`delegate-stdio` wrapper's
-own env-var-driven bootstrapping, not the `DelegateAgent` class itself, which we drive directly. So
-`_build_init_options` forwards `ORG_SLUG`/`TENANT_SLUG` (when set) into `auth.organizationName`/
-`auth.tenantName` itself — confirmed against `node_modules/@uipath/delegate-sdk/dist/index.mjs`'s own
-`yie()` resolver function live in CI (`pyright`/tests can't catch this class of bug; it only surfaces
-against a real backend). `tests/test_delegate_agent.py::TestStart::test_org_and_tenant_slug_forwarded_into_auth`
-pins it. Passing `DELEGATE_BACKEND_URL` directly instead of `DELEGATE_ENV` skips this whole path.
-
-**Token usage comes from two getters called after `sendMessage()` resolves, not from any event or the
-resolved value itself — confirmed by reading the installed SDK's bundle, not by guessing.** A first
-pass guessed at a flat `usage` dict carried on the resolved `sendMessage()` value or on a forwarded
-event, tried several plausible snake_case/camelCase bucket-name spellings, and silently returned zero
-tokens every turn against a real backend (`tests/test_delegate_agent_live.py::test_delegate_live_token_usage_populated`
-failed live: `record.crashed is False`, real text/tool output, `token_usage=None`). Reading
-`node_modules/@uipath/delegate-sdk/dist/index.mjs` directly settled it: `sendMessage()` always resolves
-to a plain string (never an object), and no event this host forwards via `agent.onEvent()` ever carries
-a `usage` field — the SDK's own internal event vocabulary has no `"usage"` member (that name IS used
-internally, but only inside the SDK's own Zustand store reducer that updates its "Token usage" UI
-panel, never re-emitted through the public `onEvent` bus). The only way to reach it is
-`DelegateAgent.getLastTurnUsage(sessionId?)` (defaults to the just-used session), so
-`delegate_host.mjs`'s `handleSend` calls it — and `getSessionId()` — right after `sendMessage()`
-resolves, and attaches both to the `send_ok` message itself. That getter's shape, read straight off the
-SDK's own `setUsage` store action, is `{promptTokens, completionTokens, promptTokensCached,
-cacheCreationTokens, turnTokenUnits, contextBreakdown}` — `promptTokens` is the TOTAL input token count
-(OpenAI-style, cached + uncached), `promptTokensCached` the cache-READ subset of it, so
-`_parse_usage` computes `uncached_input_tokens = promptTokens - promptTokensCached`. This is now
-CONFIRMED, not a guess, so `_parse_usage`'s docstring no longer marks it `# UNVERIFIED`.
+`DelegateAgent` drives UiPath Autopilot's Delegate agent: reasoning in the UiPath backend, tools run
+locally. It spawns the host from the public `@uipath/delegate-stdio` npm package
+(`dist/delegate_stdio.mjs`), which pulls in `@uipath/delegate-sdk` and the interop binaries. The
+package README is the wire-protocol SSOT. `agents/delegate/package.json` sets the floor at `1.203.0`, the
+first release with the `auth` init option and the `usage` frame (UiPath/Autopilot#6656). The other frame
+shapes were confirmed against a live `1.202.1` transcript; the frame builders in
+`tests/_fixtures/golden_streams/delegate_fixtures.py` replay those shapes. (An earlier first-party
+`delegate_host.mjs` wrapper is gone; see git history.)
+
+**Auth goes to the host as the `auth` init option, not through its env.** `_env` reads the
+`DELEGATE_*` name first, then the bare `AUTH_TOKEN` / `TENANT_ID` / `ORG_ID` / `ORG_SLUG` /
+`TENANT_SLUG`. `DELEGATE_BACKEND_URL` → `backendUrl`, `DELEGATE_ENV` → `env`. We do not adopt the
+host's own names (`ORG_LOGICAL_NAME`, `BACKEND_URL`, …): bare names collide with other tooling, and CI
+secrets use the `DELEGATE_*` names. `_HOST_ENV_REMOVED` strips the host's names and the token from the
+host env, because the agent's shells inherit it (token leak to code under test) and a stray
+`BACKEND_URL` would reroute the host. A refresh source still writes the fresh token into the host's
+own `process.env.AUTH_TOKEN`. `_INIT_CONFIG_ERROR_HINT` names our variables beside the host's message,
+which names the host's own. `env=<slug>` without org/tenant
+slugs fails init.
+
+**`LLMGW_*` leaves the host env only when a token file is set.** Without a token file, the host's
+`selectTokenSource` uses that S2S pair to refresh, so stripping it always would end refresh after an
+hour. `_strip_redundant_gateway_creds` mirrors the host's lookup exactly
+(`DELEGATE_AUTH_TOKEN_FILE ?? AUTH_TOKEN_FILE`, path-delimiter split, blanks ignored).
+
+**Effort rides `sdk_options.effort`** — the same key as Claude Code, so one `-D` drives both.
+`get_sdk_options()` returns the whole `init` dict (so the reports show Model and Effort), with
+credentials redacted: `auth` → its field names, `backendUrl` → its host (`None` when `urlparse` cannot
+parse the URL: the orchestrator reads it while it persists `task.json`, so a raise there would lose
+the file and hide the original error). It is persisted in
+`task.json`.
+
+**`enableSkills` must be sent explicitly**: the host default is `false`, so `bundledSkillsPath` alone
+loads nothing. `_resolve_bundled_skills_path` maps `<plugin.path>/skills` (first plugin wins), not the
+shared `agents/_skills.py` resolver: the SDK wants ONE parent directory, not a list of skill dirs.
+
+**`max_turns` stays client-side.** The host's `maxSteps` does not stop the turn (live: `maxSteps: 2`
+ran 7 steps). The adapter counts calls from the event stream and abandons the host when call N+1
+opens; `len(turnUsages)` from the `result` then replaces the estimate as `num_turns`.
+
+**A cut turn keeps the usage of its finished calls.** A turn cut by `max_turns` or a stop gets no
+`result`, so the adapter sums the per-round-trip `usage` frames; the `result`'s total replaces the sum
+when it arrives. Ordering (from SDK/backend source, not a live transcript): call N's `usage` frame
+always precedes the `tool_result` that opens call N+1. The frame is NOT a call boundary — it arrives
+before its call's tools run — so the cap stays on "the previous call's tools have all returned". The
+adapter warns when a cut turn had finished calls but no usage.
+
+**One `AssistantMessage` per `communicate()`.** `isStepStart` and `turnUsages` would allow a
+per-round-trip split; today `isStepStart` only merges streamed deltas. `close_window` opens the one
+window at turn start, so head and tail measure ~0.
+
+**Known host failures are recategorized** in `_describe_host_error` / `_crash_on_host_error`, ported
+from the out-of-tree `delegate-sdk` adapter that drove the same host and backend:
+
+- **Cloudflare WAF block page** (`<title>Continue with UiPath Platform</title>` or "not available in
+  your country"): a WAF rule matched shell-like text in the request, not a geo/auth block. A retry
+  is blocked again, so the reason carries "content filter" (`AGENT_INVALID_OUTPUT`). Two markers,
+  because the host truncates near 50 KB and the country sentence sits after ~48 KB of CSS.
+- **SSE connect timeout**: no response headers, so the backend never started the turn. Rewritten to
+  "connection" (`AGENT_API_ERROR`, retried) with "timeout" removed, which would be `AGENT_TIMEOUT`
+  (not retried).
+- **Session conflict** ("A reply is already being generated", 409): `_session_id` is dropped so the
+  retry starts a new conversation, but only while no turn of that conversation has finished
+  (`_conversation_has_history`). On a later turn a new conversation would continue the task without
+  the earlier turns, and the row would still be graded as one trajectory. So the retry keeps the
+  session. The 409 is short-lived: the backend cancels the reply when the killed host's stream
+  disconnects, then releases its turn claim, so the 5 s and 10 s `AGENT_CRASH` retries usually
+  succeed. A claim it fails to release expires after `TURN_CLAIM_TTL_SECONDS` (900 s), and the task
+  ends as an error. A `session_id` pinned in config counts as history, and dropping it would not
+  help anyway: each respawn sends it in `init`, and the SDK falls back to it when `send` carries no
+  `sessionId` (`sessionId || this.options.sessionId` in the SDK's `sendMessage`).
+
+Not ported: stall-timeout+resend and the S2S token-file refresher. Port them when the need shows.
+
+**No crash reason carries the stderr tail.** Categorization matches substrings of the reason, and the
+tail holds incidental text (the sandbox path, so the task id). In build 13599116 "guardrail" in a task
+path turned a transient error into non-retryable `AGENT_INVALID_OUTPUT`. `_log_stderr_tail` logs the
+tail at WARNING instead.
+
+**Only an init error that a retry cannot fix is non-retryable.** `AgentConfigError` only on
+`_INIT_CONFIG_ERROR_MARKERS` (auth missing/rejected, missing slugs, unknown env) or on a whole-word
+401/403. A bare `"401"` substring also matched ports and GUIDs, such as `127.0.0.1:54013`, and ended a
+transient failure with no retry. Any other init error and the 60 s init deadline raise a retryable
+`AgentCrashError`. The shared categorizer matches status codes as whole words too
+(`errors/categorization.py::_mentions_status`): the `AgentCrashError` goes through `categorize_error`,
+whose raw `"401"` substring made the same port and GUID cases a non-retryable auth error. A mid-run
+respawn keeps the same classification: it once made every `AgentConfigError` retryable, so an expired
+token ended the task at `start()` but was retried later.
+
+**Clear the process handle on every path that leaves the host dead or dying** — EOF, `error` frame,
+timeout (pre-check AND mid-`wait_for`), stop, `max_turns`. A shipped bug left it set after a mid-read
+timeout, so the next turn reused a host with a `send` in flight
+(`test_timeout_elapsing_mid_read_still_raises_turn_timeout_error` pins it). `_force_kill_host` clears
+the handle before its reap. The adapter never reuses a host after a failed `send`, so no late event
+leaks into the retry.
+
+**Registered unconditionally**, like `codex`/`antigravity`; its extra is empty. Node and
+`@uipath/delegate-stdio` are the real prerequisite, resolved lazily in `start()`.
+
+**Host resolution: `DELEGATE_STDIO_PATH`, then local installs, then a global one.** Local means
+the cwd, its ancestors and `agents/delegate/`; a local install wins, as in Node. A global install
+is found through the package's `delegate-stdio` bin shim on `PATH`, not `npm root -g`, so the
+resolver runs no npm subprocess. On POSIX the shim is a symlink to the bundle. On Windows it is a
+`.cmd` file beside the prefix's `node_modules`. We never execute the `.cmd`: that would make
+`cmd.exe` parse the arguments again. The home probe and `DELEGATE_SDK_NODE_MODULES` are gone. The
+global search covers "install once, run from anywhere", and the root override only repeated the
+path override. The explicit path stays for CI: a source build has no install root, and an install
+in a temporary directory for each run keeps the runs on a shared host on their own versions.
+
+**Windows path length**: an install root that pushes the interop binary path past 260 characters makes
+the spawn fail with `ENOENT`. The docs tell users to install into a short path.
 
 ### Delegate agent pricing
 
@@ -888,16 +946,13 @@ un-mirrored gap, with the same caveat comment). The two tables can price the SAM
 differently until someone reconciles them — a known, pre-existing state, not something this port
 introduced or should silently "fix" by picking one number over the other.
 
-### Delegate agent golden-master and timing-identity coverage
+### Delegate agent golden-master coverage
 
-`AgentKind.DELEGATE` is excluded from `tests/test_agent_golden_master.py`'s `_NO_GOLDEN_COVERAGE`
-allowlist rather than given fixture scenarios: a golden snapshot pins a byte-identical `TurnRecord`
-for a scripted event stream, and every event field name below the confirmed `type`/`content`/
-`toolName`/`sessionId`/`model`/`usage` set is a guess (see this file's own "Delegate agent" section
-above and `delegate_agent.py`'s module docstring). Snapshotting a guess would make it look verified.
-Add real scenarios once the fields are confirmed against a live backend.
-
-This does NOT extend to `tests/test_timing_identity_contract.py::test_delegate_buckets_tile_the_turn`,
-which IS a real, unexempted case: the ms-exact four-bucket identity depends only on this agent's own
-`close_window`/`_finalize_turn` code (a single window per turn, opened at `_TurnState.__init__` and
-closed at finalize), never on the SDK's field names, so there is nothing unverified for it to guess at.
+`tests/_fixtures/golden_streams/delegate_fixtures.py` scripts the host's stdout frames for four
+scenarios: a tool call, a `max_turns` cut, an `error`-frame crash and an orphaned tool. The `event`,
+`result` and `error` frames mirror the live `1.202.1` transcript. The `usage` frame and the `result`
+fields new in `1.203.0` follow the host source of UiPath/Autopilot#6656 (`src/usage.ts` and
+`handleSend`): every usage object carries all four buckets with `input_tokens` exclusive of cache,
+the frames flush before each event and before the `result` or `error`, and `result.usage` is the sum
+of `turnUsages`. No live `1.203.0` transcript has been compared yet. If one differs, change the
+builders and regenerate the snapshots with `GOLDEN_REGEN=1`.

{coder_eval-0.12.10 → coder_eval-0.12.12}/.env.example

@@ -89,36 +89,33 @@ LOG_TO_FILE=false  # Set to true to enable file logging
 # CODER_EVAL_REMEDIATE_HOME_PLUGINS=0
 
 # Delegate agent settings (UiPath Autopilot's Delegate agent, requires Node.js
-# on PATH plus `npm install @uipath/delegate-sdk` — a genuinely public package,
-# no token needed). See docs/agents/DELEGATE.md.
+# on PATH plus `npm install -g @uipath/delegate-stdio`, a release that accepts the
+# `auth` init option). coder_eval finds the install itself. See docs/agents/DELEGATE.md.
 #
-# DELEGATE_SDK_NODE_MODULES / DELEGATE_SDK_PATH — override the auto-located
-# install (ancestor-walk from cwd, plus home); set at most one.
-# DELEGATE_SDK_NODE_MODULES="/path/to/install/root"
-# DELEGATE_SDK_PATH="/path/to/node_modules/@uipath/delegate-sdk/dist/index.mjs"
-#
-# DELEGATE_ENV — cloud environment slug (alpha/staging/production/localhost),
-# used to derive the backend URL from the auth record's org/tenant slugs.
+# DELEGATE_ENV — cloud environment slug (alpha/staging/production),
+# used to derive the backend URL from the org/tenant slugs.
 # DELEGATE_ENV="alpha"
 #
 # DELEGATE_BACKEND_URL — pin the full agent-service URL directly instead
 # (wins over DELEGATE_ENV when both are set).
 # DELEGATE_BACKEND_URL="https://cloud.uipath.com/<org>/<tenant>/delegate_"
 #
-# Auth: either the token set below, or a prior `delegate-sdk`/`delegate-cli`
-# login that wrote ~/.aria/sdk-auth.json (read by the Node host itself, not
-# by coder_eval). Each var also accepts a DELEGATE_-namespaced spelling
-# (DELEGATE_AUTH_TOKEN, DELEGATE_TENANT_ID, ...), checked first -- prefer that
-# spelling in a shared environment, since the bare names collide with what
-# other tooling (npm, Vault, Terraform) commonly exports.
-# AUTH_TOKEN="..."
-# TENANT_ID="..."
-# ORG_ID="..."
+# Auth: either the token set below, or a prior `delegate-cli login`
+# that wrote ~/.aria/sdk-auth.json (read by the Node host itself, not
+# by coder_eval). coder_eval sends these values to the host as its `auth`
+# init option, so the token stays out of the environment that the agent's
+# shell tools inherit. Each var also accepts the bare spelling (AUTH_TOKEN,
+# TENANT_ID, ORG_ID, ORG_SLUG, TENANT_SLUG) as a fallback -- prefer the
+# DELEGATE_ spelling in a shared environment, since the bare names collide
+# with what other tooling (npm, Vault, Terraform) commonly exports.
+# DELEGATE_AUTH_TOKEN="..."
+# DELEGATE_TENANT_ID="..."
+# DELEGATE_ORG_ID="..."
 # With DELEGATE_ENV (not DELEGATE_BACKEND_URL) also set these two -- the
 # human-readable org/tenant names, not the GUIDs above -- or init fails with
-# "--env alpha needs org/tenant slugs". See docs/agents/DELEGATE.md.
-# ORG_SLUG="..."
-# TENANT_SLUG="..."
+# 'env="alpha" requires org/tenant slugs'. See docs/agents/DELEGATE.md.
+# DELEGATE_ORG_SLUG="..."
+# DELEGATE_TENANT_SLUG="..."
 
 # Usage Telemetry (anonymous; on by default).
 # Disable entirely:

{coder_eval-0.12.10 → coder_eval-0.12.12}/.github/workflows/pr-checks.yml

@@ -1175,8 +1175,8 @@ jobs:
           retention-days: 7
 
   delegate-live-tests:
-    # Live Delegate e2e: spawns the real Node host (agents/delegate/delegate_host.mjs)
-    # wrapping the public @uipath/delegate-sdk against a real UiPath backend. Runs on
+    # Live Delegate e2e: spawns the real Node host from the public
+    # @uipath/delegate-stdio package against a real UiPath backend. Runs on
     # every PR/push like the other live-test jobs (codex-live-tests, byoa-live-tests)
     # now that the DELEGATE_ROPC_* secrets are provisioned; the presence-check gate
     # below skips cleanly ONLY on a fork PR (no secrets available there) and fails
@@ -1187,10 +1187,10 @@ jobs:
     # and implemented in coder_eval_uipath/eval_runner/scripts/ci/refresh-auth.sh —
     # rather than a long-lived AUTH_TOKEN secret, since a minted token's ~1h TTL
     # would make a static secret stale between runs. A dedicated bot user's
-    # username/password is the durable credential; the Delegate SDK reads the
-    # minted token straight from AUTH_TOKEN/TENANT_ID/ORG_ID (see
-    # docs/agents/DELEGATE.md), so no sdk-auth.json or client-secret refresh chain
-    # is needed here.
+    # username/password is the durable credential; the agent sends the minted
+    # token (DELEGATE_AUTH_TOKEN, with DELEGATE_TENANT_ID / DELEGATE_ORG_ID) to the
+    # host as its `auth` init option (see docs/agents/DELEGATE.md), so no
+    # sdk-auth.json or client-secret refresh chain is needed here.
     name: Delegate Live E2E (ROPC)
     runs-on: uipath-ubuntu-24.04
     timeout-minutes: 15
@@ -1260,7 +1260,7 @@ jobs:
         if: steps.gate.outputs.present == 'true'
         run: uv sync --frozen --extra dev
 
-      - name: Install @uipath/delegate-sdk (public npm, no token needed)
+      - name: Install @uipath/delegate-stdio (public npm, no token needed)
         if: steps.gate.outputs.present == 'true'
         working-directory: src/coder_eval/agents/delegate
         # --safe-chain-skip-minimum-package-age (not a growing exclusion-list entry
@@ -1372,17 +1372,15 @@ jobs:
       - name: Run fizzbuzz delegate task (assert PASS)
         if: steps.gate.outputs.present == 'true'
         env:
-          DELEGATE_SDK_NODE_MODULES: ${{ github.workspace }}/src/coder_eval/agents/delegate
           DELEGATE_ENV: ${{ github.event.inputs.delegate_uipath_env || 'alpha' }}
-          AUTH_TOKEN: ${{ steps.ropc.outputs.access_token }}
-          TENANT_ID: ${{ secrets.DELEGATE_TENANT_ID }}
-          ORG_ID: ${{ secrets.DELEGATE_ORG_ID }}
-          # Forwarded into `auth.organizationName` / `auth.tenantName` by
-          # DelegateAgent._build_init_options (see .claude/notes/agents.md §
-          # Delegate agent) — needed alongside the GUIDs above whenever
-          # `environment` (rather than `backendUrl`) resolves the backend.
-          ORG_SLUG: ${{ secrets.DELEGATE_ORG_SLUG }}
-          TENANT_SLUG: ${{ secrets.DELEGATE_TENANT_SLUG }}
+          DELEGATE_AUTH_TOKEN: ${{ steps.ropc.outputs.access_token }}
+          DELEGATE_TENANT_ID: ${{ secrets.DELEGATE_TENANT_ID }}
+          DELEGATE_ORG_ID: ${{ secrets.DELEGATE_ORG_ID }}
+          # Needed alongside the GUIDs above whenever DELEGATE_ENV (rather than
+          # DELEGATE_BACKEND_URL) resolves the backend; the agent sends all five to
+          # the host as its `auth` init option (see .claude/notes/agents.md § Delegate agent).
+          DELEGATE_ORG_SLUG: ${{ secrets.DELEGATE_ORG_SLUG }}
+          DELEGATE_TENANT_SLUG: ${{ secrets.DELEGATE_TENANT_SLUG }}
         run: .venv/bin/coder-eval run tasks/delegate/fizzbuzz_delegate.yaml --run-dir runs/delegate-live
 
       - name: Verify Delegate live run PASSED
@@ -1405,17 +1403,16 @@ jobs:
       - name: Run Delegate live unit tests (assert PASS)
         if: steps.gate.outputs.present == 'true'
         env:
-          DELEGATE_SDK_NODE_MODULES: ${{ github.workspace }}/src/coder_eval/agents/delegate
           DELEGATE_ENV: ${{ github.event.inputs.delegate_uipath_env || 'alpha' }}
-          AUTH_TOKEN: ${{ steps.ropc.outputs.access_token }}
-          TENANT_ID: ${{ secrets.DELEGATE_TENANT_ID }}
-          ORG_ID: ${{ secrets.DELEGATE_ORG_ID }}
-          ORG_SLUG: ${{ secrets.DELEGATE_ORG_SLUG }}
-          TENANT_SLUG: ${{ secrets.DELEGATE_TENANT_SLUG }}
+          DELEGATE_AUTH_TOKEN: ${{ steps.ropc.outputs.access_token }}
+          DELEGATE_TENANT_ID: ${{ secrets.DELEGATE_TENANT_ID }}
+          DELEGATE_ORG_ID: ${{ secrets.DELEGATE_ORG_ID }}
+          DELEGATE_ORG_SLUG: ${{ secrets.DELEGATE_ORG_SLUG }}
+          DELEGATE_TENANT_SLUG: ${{ secrets.DELEGATE_TENANT_SLUG }}
         run: |
           mkdir -p tmp
           .venv/bin/pytest tests/test_delegate_agent_live.py \
-            -m live -n0 -v --tb=short --strict-markers -ra \
+            -m live -n0 -v --tb=short --strict-markers -ra --log-level=DEBUG \
             --junit-xml=tmp/junit-delegate-live.xml
 
       - name: Assert Delegate live tests actually ran (not silently skipped)