coder-eval 0.12.11__tar.gz → 0.12.12__tar.gz

{coder_eval-0.12.11 → coder_eval-0.12.12}/.claude/notes/agents.md

@@ -558,7 +558,7 @@ shell-aware `parameters["command"]` extraction in `criteria/command_executed.py`
 to raw-JSON matching — so the same task scores differently per harness. Unknown names pass
 through unchanged.
 
-Three cases are worth knowing:
+Four cases are worth knowing:
 
 - **OpenCode's tool set varies by MODEL within the one harness.** A live 174-task run
   showed DeepSeek using `write`/`edit` 199 times and `apply_patch` 0, while GPT-5.6 used
@@ -571,6 +571,11 @@ Three cases are worth knowing:
 - **Pi's search tool is `find`** (glob-by-pattern), not `glob`; there is no `glob` tool in
   its built-in set, so mapping `find` to the canonical `Glob` is what keeps
   `command_executed` and `commands_efficiency` comparable.
+- **Delegate's host maps most names itself, but not `LoadSkill`.** `delegate-stdio` reports
+  `ReadFile` as `Read` and `ExecuteSkillApi` as `Skill`, so `DelegateAgent` maps only
+  `LoadSkill {name}` to `Skill {skill}`; without it `skill_triggered` never sees a Delegate skill
+  load. The rename is keyed by the host's name: keyed by `Skill`, it would also turn
+  ExecuteSkillApi's `name`, an API call, into a skill engagement.
 
 Antigravity additionally strips the result payload out of a tool call's arguments: the
 harness folds result fields into the same `args` dict at DONE. Beyond a static key list,
@@ -801,89 +806,117 @@ release adding an unclassified field fails loudly instead of silently passing th
 
 ## Delegate agent
 
-`DelegateAgent` drives UiPath Autopilot's Delegate agent — reasoning in the UiPath backend,
-tools executing locally through the SDK's bundled interop process. Unlike every other CLI-driven
-agent in this file, there is no vendor-provided stdio protocol host to spawn: `@uipath/delegate-stdio`,
-the internal package a UiPath-only sibling plugin (`coder_eval_uipath`) drives, is not public. Only
-`@uipath/delegate-sdk` (a programmatic library exposing a `DelegateAgent` class with
-`initialize()`/`onEvent()`/`sendMessage()`/`destroy()`) and `@uipath/delegate-cli` (a terminal wrapper
-around it) are. So this agent ships its own first-party Node host, `agents/delegate/delegate_host.mjs`,
-which wraps the SDK class in a newline-JSON stdio protocol this framework designed, not one it had to
-reverse-engineer — verified live against a real `@uipath/delegate-sdk@0.1.12` install with no token,
-confirming both the plain-public-install story and that the SDK's own event vocabulary
-(`session_start`/`thinking`/`message`/`tool_call`/`tool_result`/`error`/`done`/`step`) substantially
-matches the internal sibling's protocol.
-
-**No multi-generation transcript splitting.** The internal sibling reconstructs one `AssistantMessage`
-per backend round-trip from an `isStepStart` flag and a `turnUsages` array on the host's terminal
-message. Neither is present on `DelegateAgent.onEvent`'s payloads (confirmed absent by grepping the
-installed SDK bundle's string literals), so there is no reliable per-round-trip boundary signal at this
-API layer. `DelegateAgent` builds exactly ONE `AssistantMessage` per `communicate()` call instead — a
-deliberate simplification, not an oversight; richer segmentation can be added once a real boundary
-signal is confirmed against a live backend. `timing.close_window` still opens that one window from the
-turn's own start (there is nothing to tile from), so the head and tail both measure ~0.
-
-**Deferred, not ported speculatively.** The internal sibling's module docstring documents several
-hard-won failure-signature-specific recoveries: a Cloudflare WAF-block-page rewrite, an SSE-connect-
-timeout rewrite, session-conflict fresh-host recovery, and first-response stall-timeout+resend. None
-are ported here — porting a marker tuned to the internal sibling's own observed failures risks matching
-nothing (or the wrong thing) against the public SDK/backend's actual error surface. A crash still ends
-the turn correctly as a retryable `AgentCrashError`; it is just not specially diagnosed. Port these once
-the same failures are actually observed running this agent for real.
-
-**The process handle must be cleared on every path that leaves the host dead or dying** — EOF, a host
-`fatal` message, a timeout (both the top-of-loop pre-check AND a timeout elapsing while blocked inside
-`asyncio.wait_for`), cooperative stop, and `max_turns` exhaustion. A real bug shipped once during this
-agent's own development: a timeout elapsing mid-read fell through to the generic crash path instead of
-`TurnTimeoutError`, and left the process handle set, so the NEXT `communicate()` call reused a host with
-a `"send"` still nominally in flight instead of respawning — risking a stale response being consumed as
-the new turn's. `tests/test_delegate_agent.py`'s `test_timeout_elapsing_mid_read_still_raises_turn_timeout_error`
-pins the fix.
-
-**Skills mapping is deliberately NOT the shared `agents/_skills.py` resolver.** That resolver enumerates
-individual skill directories for a repeated `--skill <dir>`-style CLI argument (OpenCode/Pi's shape).
-The Delegate SDK's `bundledSkillsPath` wants exactly ONE parent directory whose children are skill
-folders — a genuinely different shape — so `_resolve_bundled_skills_path` mirrors the internal sibling's
-own mapping (`<plugin.path>/skills`, first plugin wins) instead of force-fitting the shared helper.
-
-**Registered unconditionally**, exactly like `codex`/`antigravity` — there is no conditional-registration
-gate keyed on an optional-dependency extra anywhere in this codebase; `opencode`/`pi` declare an EMPTY
-extra purely as packaging-metadata documentation, and `delegate` follows that same shape (no new pip
-package — Node/`@uipath/delegate-sdk` is the real prerequisite, resolved lazily in `start()`).
-
-**`environment` resolution needs org/tenant SLUGS, not just IDs — confirmed by reading the installed
-SDK's own bundle, not by guessing from its error message.** When `DELEGATE_BACKEND_URL` is absent and
-`DELEGATE_ENV` (-> `environment` init option) is used instead, the SDK resolves the backend URL from
-`organizationName`/`orgLogicalName` and `tenantName` fields on the `auth` object passed to
-`initialize()` — NOT from `ORG_SLUG`/`TENANT_SLUG` read off `process.env` directly, despite that being
-exactly what the SDK's own thrown error suggests ("set ORG_SLUG and TENANT_SLUG env vars alongside
-ORG_ID/TENANT_ID"). That advice describes the separate `@uipath/delegate-cli`/`delegate-stdio` wrapper's
-own env-var-driven bootstrapping, not the `DelegateAgent` class itself, which we drive directly. So
-`_build_init_options` forwards `ORG_SLUG`/`TENANT_SLUG` (when set) into `auth.organizationName`/
-`auth.tenantName` itself — confirmed against `node_modules/@uipath/delegate-sdk/dist/index.mjs`'s own
-`yie()` resolver function live in CI (`pyright`/tests can't catch this class of bug; it only surfaces
-against a real backend). `tests/test_delegate_agent.py::TestStart::test_org_and_tenant_slug_forwarded_into_auth`
-pins it. Passing `DELEGATE_BACKEND_URL` directly instead of `DELEGATE_ENV` skips this whole path.
-
-**Token usage comes from two getters called after `sendMessage()` resolves, not from any event or the
-resolved value itself — confirmed by reading the installed SDK's bundle, not by guessing.** A first
-pass guessed at a flat `usage` dict carried on the resolved `sendMessage()` value or on a forwarded
-event, tried several plausible snake_case/camelCase bucket-name spellings, and silently returned zero
-tokens every turn against a real backend (`tests/test_delegate_agent_live.py::test_delegate_live_token_usage_populated`
-failed live: `record.crashed is False`, real text/tool output, `token_usage=None`). Reading
-`node_modules/@uipath/delegate-sdk/dist/index.mjs` directly settled it: `sendMessage()` always resolves
-to a plain string (never an object), and no event this host forwards via `agent.onEvent()` ever carries
-a `usage` field — the SDK's own internal event vocabulary has no `"usage"` member (that name IS used
-internally, but only inside the SDK's own Zustand store reducer that updates its "Token usage" UI
-panel, never re-emitted through the public `onEvent` bus). The only way to reach it is
-`DelegateAgent.getLastTurnUsage(sessionId?)` (defaults to the just-used session), so
-`delegate_host.mjs`'s `handleSend` calls it — and `getSessionId()` — right after `sendMessage()`
-resolves, and attaches both to the `send_ok` message itself. That getter's shape, read straight off the
-SDK's own `setUsage` store action, is `{promptTokens, completionTokens, promptTokensCached,
-cacheCreationTokens, turnTokenUnits, contextBreakdown}` — `promptTokens` is the TOTAL input token count
-(OpenAI-style, cached + uncached), `promptTokensCached` the cache-READ subset of it, so
-`_parse_usage` computes `uncached_input_tokens = promptTokens - promptTokensCached`. This is now
-CONFIRMED, not a guess, so `_parse_usage`'s docstring no longer marks it `# UNVERIFIED`.
+`DelegateAgent` drives UiPath Autopilot's Delegate agent: reasoning in the UiPath backend, tools run
+locally. It spawns the host from the public `@uipath/delegate-stdio` npm package
+(`dist/delegate_stdio.mjs`), which pulls in `@uipath/delegate-sdk` and the interop binaries. The
+package README is the wire-protocol SSOT. `agents/delegate/package.json` sets the floor at `1.203.0`, the
+first release with the `auth` init option and the `usage` frame (UiPath/Autopilot#6656). The other frame
+shapes were confirmed against a live `1.202.1` transcript; the frame builders in
+`tests/_fixtures/golden_streams/delegate_fixtures.py` replay those shapes. (An earlier first-party
+`delegate_host.mjs` wrapper is gone; see git history.)
+
+**Auth goes to the host as the `auth` init option, not through its env.** `_env` reads the
+`DELEGATE_*` name first, then the bare `AUTH_TOKEN` / `TENANT_ID` / `ORG_ID` / `ORG_SLUG` /
+`TENANT_SLUG`. `DELEGATE_BACKEND_URL` → `backendUrl`, `DELEGATE_ENV` → `env`. We do not adopt the
+host's own names (`ORG_LOGICAL_NAME`, `BACKEND_URL`, …): bare names collide with other tooling, and CI
+secrets use the `DELEGATE_*` names. `_HOST_ENV_REMOVED` strips the host's names and the token from the
+host env, because the agent's shells inherit it (token leak to code under test) and a stray
+`BACKEND_URL` would reroute the host. A refresh source still writes the fresh token into the host's
+own `process.env.AUTH_TOKEN`. `_INIT_CONFIG_ERROR_HINT` names our variables beside the host's message,
+which names the host's own. `env=<slug>` without org/tenant
+slugs fails init.
+
+**`LLMGW_*` leaves the host env only when a token file is set.** Without a token file, the host's
+`selectTokenSource` uses that S2S pair to refresh, so stripping it always would end refresh after an
+hour. `_strip_redundant_gateway_creds` mirrors the host's lookup exactly
+(`DELEGATE_AUTH_TOKEN_FILE ?? AUTH_TOKEN_FILE`, path-delimiter split, blanks ignored).
+
+**Effort rides `sdk_options.effort`** — the same key as Claude Code, so one `-D` drives both.
+`get_sdk_options()` returns the whole `init` dict (so the reports show Model and Effort), with
+credentials redacted: `auth` → its field names, `backendUrl` → its host (`None` when `urlparse` cannot
+parse the URL: the orchestrator reads it while it persists `task.json`, so a raise there would lose
+the file and hide the original error). It is persisted in
+`task.json`.
+
+**`enableSkills` must be sent explicitly**: the host default is `false`, so `bundledSkillsPath` alone
+loads nothing. `_resolve_bundled_skills_path` maps `<plugin.path>/skills` (first plugin wins), not the
+shared `agents/_skills.py` resolver: the SDK wants ONE parent directory, not a list of skill dirs.
+
+**`max_turns` stays client-side.** The host's `maxSteps` does not stop the turn (live: `maxSteps: 2`
+ran 7 steps). The adapter counts calls from the event stream and abandons the host when call N+1
+opens; `len(turnUsages)` from the `result` then replaces the estimate as `num_turns`.
+
+**A cut turn keeps the usage of its finished calls.** A turn cut by `max_turns` or a stop gets no
+`result`, so the adapter sums the per-round-trip `usage` frames; the `result`'s total replaces the sum
+when it arrives. Ordering (from SDK/backend source, not a live transcript): call N's `usage` frame
+always precedes the `tool_result` that opens call N+1. The frame is NOT a call boundary — it arrives
+before its call's tools run — so the cap stays on "the previous call's tools have all returned". The
+adapter warns when a cut turn had finished calls but no usage.
+
+**One `AssistantMessage` per `communicate()`.** `isStepStart` and `turnUsages` would allow a
+per-round-trip split; today `isStepStart` only merges streamed deltas. `close_window` opens the one
+window at turn start, so head and tail measure ~0.
+
+**Known host failures are recategorized** in `_describe_host_error` / `_crash_on_host_error`, ported
+from the out-of-tree `delegate-sdk` adapter that drove the same host and backend:
+
+- **Cloudflare WAF block page** (`<title>Continue with UiPath Platform</title>` or "not available in
+  your country"): a WAF rule matched shell-like text in the request, not a geo/auth block. A retry
+  is blocked again, so the reason carries "content filter" (`AGENT_INVALID_OUTPUT`). Two markers,
+  because the host truncates near 50 KB and the country sentence sits after ~48 KB of CSS.
+- **SSE connect timeout**: no response headers, so the backend never started the turn. Rewritten to
+  "connection" (`AGENT_API_ERROR`, retried) with "timeout" removed, which would be `AGENT_TIMEOUT`
+  (not retried).
+- **Session conflict** ("A reply is already being generated", 409): `_session_id` is dropped so the
+  retry starts a new conversation, but only while no turn of that conversation has finished
+  (`_conversation_has_history`). On a later turn a new conversation would continue the task without
+  the earlier turns, and the row would still be graded as one trajectory. So the retry keeps the
+  session. The 409 is short-lived: the backend cancels the reply when the killed host's stream
+  disconnects, then releases its turn claim, so the 5 s and 10 s `AGENT_CRASH` retries usually
+  succeed. A claim it fails to release expires after `TURN_CLAIM_TTL_SECONDS` (900 s), and the task
+  ends as an error. A `session_id` pinned in config counts as history, and dropping it would not
+  help anyway: each respawn sends it in `init`, and the SDK falls back to it when `send` carries no
+  `sessionId` (`sessionId || this.options.sessionId` in the SDK's `sendMessage`).
+
+Not ported: stall-timeout+resend and the S2S token-file refresher. Port them when the need shows.
+
+**No crash reason carries the stderr tail.** Categorization matches substrings of the reason, and the
+tail holds incidental text (the sandbox path, so the task id). In build 13599116 "guardrail" in a task
+path turned a transient error into non-retryable `AGENT_INVALID_OUTPUT`. `_log_stderr_tail` logs the
+tail at WARNING instead.
+
+**Only an init error that a retry cannot fix is non-retryable.** `AgentConfigError` only on
+`_INIT_CONFIG_ERROR_MARKERS` (auth missing/rejected, missing slugs, unknown env) or on a whole-word
+401/403. A bare `"401"` substring also matched ports and GUIDs, such as `127.0.0.1:54013`, and ended a
+transient failure with no retry. Any other init error and the 60 s init deadline raise a retryable
+`AgentCrashError`. The shared categorizer matches status codes as whole words too
+(`errors/categorization.py::_mentions_status`): the `AgentCrashError` goes through `categorize_error`,
+whose raw `"401"` substring made the same port and GUID cases a non-retryable auth error. A mid-run
+respawn keeps the same classification: it once made every `AgentConfigError` retryable, so an expired
+token ended the task at `start()` but was retried later.
+
+**Clear the process handle on every path that leaves the host dead or dying** — EOF, `error` frame,
+timeout (pre-check AND mid-`wait_for`), stop, `max_turns`. A shipped bug left it set after a mid-read
+timeout, so the next turn reused a host with a `send` in flight
+(`test_timeout_elapsing_mid_read_still_raises_turn_timeout_error` pins it). `_force_kill_host` clears
+the handle before its reap. The adapter never reuses a host after a failed `send`, so no late event
+leaks into the retry.
+
+**Registered unconditionally**, like `codex`/`antigravity`; its extra is empty. Node and
+`@uipath/delegate-stdio` are the real prerequisite, resolved lazily in `start()`.
+
+**Host resolution: `DELEGATE_STDIO_PATH`, then local installs, then a global one.** Local means
+the cwd, its ancestors and `agents/delegate/`; a local install wins, as in Node. A global install
+is found through the package's `delegate-stdio` bin shim on `PATH`, not `npm root -g`, so the
+resolver runs no npm subprocess. On POSIX the shim is a symlink to the bundle. On Windows it is a
+`.cmd` file beside the prefix's `node_modules`. We never execute the `.cmd`: that would make
+`cmd.exe` parse the arguments again. The home probe and `DELEGATE_SDK_NODE_MODULES` are gone. The
+global search covers "install once, run from anywhere", and the root override only repeated the
+path override. The explicit path stays for CI: a source build has no install root, and an install
+in a temporary directory for each run keeps the runs on a shared host on their own versions.
+
+**Windows path length**: an install root that pushes the interop binary path past 260 characters makes
+the spawn fail with `ENOENT`. The docs tell users to install into a short path.
 
 ### Delegate agent pricing
 
@@ -913,16 +946,13 @@ un-mirrored gap, with the same caveat comment). The two tables can price the SAM
 differently until someone reconciles them — a known, pre-existing state, not something this port
 introduced or should silently "fix" by picking one number over the other.
 
-### Delegate agent golden-master and timing-identity coverage
+### Delegate agent golden-master coverage
 
-`AgentKind.DELEGATE` is excluded from `tests/test_agent_golden_master.py`'s `_NO_GOLDEN_COVERAGE`
-allowlist rather than given fixture scenarios: a golden snapshot pins a byte-identical `TurnRecord`
-for a scripted event stream, and every event field name below the confirmed `type`/`content`/
-`toolName`/`sessionId`/`model`/`usage` set is a guess (see this file's own "Delegate agent" section
-above and `delegate_agent.py`'s module docstring). Snapshotting a guess would make it look verified.
-Add real scenarios once the fields are confirmed against a live backend.
-
-This does NOT extend to `tests/test_timing_identity_contract.py::test_delegate_buckets_tile_the_turn`,
-which IS a real, unexempted case: the ms-exact four-bucket identity depends only on this agent's own
-`close_window`/`_finalize_turn` code (a single window per turn, opened at `_TurnState.__init__` and
-closed at finalize), never on the SDK's field names, so there is nothing unverified for it to guess at.
+`tests/_fixtures/golden_streams/delegate_fixtures.py` scripts the host's stdout frames for four
+scenarios: a tool call, a `max_turns` cut, an `error`-frame crash and an orphaned tool. The `event`,
+`result` and `error` frames mirror the live `1.202.1` transcript. The `usage` frame and the `result`
+fields new in `1.203.0` follow the host source of UiPath/Autopilot#6656 (`src/usage.ts` and
+`handleSend`): every usage object carries all four buckets with `input_tokens` exclusive of cache,
+the frames flush before each event and before the `result` or `error`, and `result.usage` is the sum
+of `turnUsages`. No live `1.203.0` transcript has been compared yet. If one differs, change the
+builders and regenerate the snapshots with `GOLDEN_REGEN=1`.

{coder_eval-0.12.11 → coder_eval-0.12.12}/.env.example

@@ -89,36 +89,33 @@ LOG_TO_FILE=false  # Set to true to enable file logging
 # CODER_EVAL_REMEDIATE_HOME_PLUGINS=0
 
 # Delegate agent settings (UiPath Autopilot's Delegate agent, requires Node.js
-# on PATH plus `npm install @uipath/delegate-sdk` — a genuinely public package,
-# no token needed). See docs/agents/DELEGATE.md.
+# on PATH plus `npm install -g @uipath/delegate-stdio`, a release that accepts the
+# `auth` init option). coder_eval finds the install itself. See docs/agents/DELEGATE.md.
 #
-# DELEGATE_SDK_NODE_MODULES / DELEGATE_SDK_PATH — override the auto-located
-# install (ancestor-walk from cwd, plus home); set at most one.
-# DELEGATE_SDK_NODE_MODULES="/path/to/install/root"
-# DELEGATE_SDK_PATH="/path/to/node_modules/@uipath/delegate-sdk/dist/index.mjs"
-#
-# DELEGATE_ENV — cloud environment slug (alpha/staging/production/localhost),
-# used to derive the backend URL from the auth record's org/tenant slugs.
+# DELEGATE_ENV — cloud environment slug (alpha/staging/production),
+# used to derive the backend URL from the org/tenant slugs.
 # DELEGATE_ENV="alpha"
 #
 # DELEGATE_BACKEND_URL — pin the full agent-service URL directly instead
 # (wins over DELEGATE_ENV when both are set).
 # DELEGATE_BACKEND_URL="https://cloud.uipath.com/<org>/<tenant>/delegate_"
 #
-# Auth: either the token set below, or a prior `delegate-sdk`/`delegate-cli`
-# login that wrote ~/.aria/sdk-auth.json (read by the Node host itself, not
-# by coder_eval). Each var also accepts a DELEGATE_-namespaced spelling
-# (DELEGATE_AUTH_TOKEN, DELEGATE_TENANT_ID, ...), checked first -- prefer that
-# spelling in a shared environment, since the bare names collide with what
-# other tooling (npm, Vault, Terraform) commonly exports.
-# AUTH_TOKEN="..."
-# TENANT_ID="..."
-# ORG_ID="..."
+# Auth: either the token set below, or a prior `delegate-cli login`
+# that wrote ~/.aria/sdk-auth.json (read by the Node host itself, not
+# by coder_eval). coder_eval sends these values to the host as its `auth`
+# init option, so the token stays out of the environment that the agent's
+# shell tools inherit. Each var also accepts the bare spelling (AUTH_TOKEN,
+# TENANT_ID, ORG_ID, ORG_SLUG, TENANT_SLUG) as a fallback -- prefer the
+# DELEGATE_ spelling in a shared environment, since the bare names collide
+# with what other tooling (npm, Vault, Terraform) commonly exports.
+# DELEGATE_AUTH_TOKEN="..."
+# DELEGATE_TENANT_ID="..."
+# DELEGATE_ORG_ID="..."
 # With DELEGATE_ENV (not DELEGATE_BACKEND_URL) also set these two -- the
 # human-readable org/tenant names, not the GUIDs above -- or init fails with
-# "--env alpha needs org/tenant slugs". See docs/agents/DELEGATE.md.
-# ORG_SLUG="..."
-# TENANT_SLUG="..."
+# 'env="alpha" requires org/tenant slugs'. See docs/agents/DELEGATE.md.
+# DELEGATE_ORG_SLUG="..."
+# DELEGATE_TENANT_SLUG="..."
 
 # Usage Telemetry (anonymous; on by default).
 # Disable entirely:

{coder_eval-0.12.11 → coder_eval-0.12.12}/.github/workflows/pr-checks.yml

@@ -1175,8 +1175,8 @@ jobs:
           retention-days: 7
 
   delegate-live-tests:
-    # Live Delegate e2e: spawns the real Node host (agents/delegate/delegate_host.mjs)
-    # wrapping the public @uipath/delegate-sdk against a real UiPath backend. Runs on
+    # Live Delegate e2e: spawns the real Node host from the public
+    # @uipath/delegate-stdio package against a real UiPath backend. Runs on
     # every PR/push like the other live-test jobs (codex-live-tests, byoa-live-tests)
     # now that the DELEGATE_ROPC_* secrets are provisioned; the presence-check gate
     # below skips cleanly ONLY on a fork PR (no secrets available there) and fails
@@ -1187,10 +1187,10 @@ jobs:
     # and implemented in coder_eval_uipath/eval_runner/scripts/ci/refresh-auth.sh —
     # rather than a long-lived AUTH_TOKEN secret, since a minted token's ~1h TTL
     # would make a static secret stale between runs. A dedicated bot user's
-    # username/password is the durable credential; the Delegate SDK reads the
-    # minted token straight from AUTH_TOKEN/TENANT_ID/ORG_ID (see
-    # docs/agents/DELEGATE.md), so no sdk-auth.json or client-secret refresh chain
-    # is needed here.
+    # username/password is the durable credential; the agent sends the minted
+    # token (DELEGATE_AUTH_TOKEN, with DELEGATE_TENANT_ID / DELEGATE_ORG_ID) to the
+    # host as its `auth` init option (see docs/agents/DELEGATE.md), so no
+    # sdk-auth.json or client-secret refresh chain is needed here.
     name: Delegate Live E2E (ROPC)
     runs-on: uipath-ubuntu-24.04
     timeout-minutes: 15
@@ -1260,7 +1260,7 @@ jobs:
         if: steps.gate.outputs.present == 'true'
         run: uv sync --frozen --extra dev
 
-      - name: Install @uipath/delegate-sdk (public npm, no token needed)
+      - name: Install @uipath/delegate-stdio (public npm, no token needed)
         if: steps.gate.outputs.present == 'true'
         working-directory: src/coder_eval/agents/delegate
         # --safe-chain-skip-minimum-package-age (not a growing exclusion-list entry
@@ -1372,17 +1372,15 @@ jobs:
       - name: Run fizzbuzz delegate task (assert PASS)
         if: steps.gate.outputs.present == 'true'
         env:
-          DELEGATE_SDK_NODE_MODULES: ${{ github.workspace }}/src/coder_eval/agents/delegate
           DELEGATE_ENV: ${{ github.event.inputs.delegate_uipath_env || 'alpha' }}
-          AUTH_TOKEN: ${{ steps.ropc.outputs.access_token }}
-          TENANT_ID: ${{ secrets.DELEGATE_TENANT_ID }}
-          ORG_ID: ${{ secrets.DELEGATE_ORG_ID }}
-          # Forwarded into `auth.organizationName` / `auth.tenantName` by
-          # DelegateAgent._build_init_options (see .claude/notes/agents.md §
-          # Delegate agent) — needed alongside the GUIDs above whenever
-          # `environment` (rather than `backendUrl`) resolves the backend.
-          ORG_SLUG: ${{ secrets.DELEGATE_ORG_SLUG }}
-          TENANT_SLUG: ${{ secrets.DELEGATE_TENANT_SLUG }}
+          DELEGATE_AUTH_TOKEN: ${{ steps.ropc.outputs.access_token }}
+          DELEGATE_TENANT_ID: ${{ secrets.DELEGATE_TENANT_ID }}
+          DELEGATE_ORG_ID: ${{ secrets.DELEGATE_ORG_ID }}
+          # Needed alongside the GUIDs above whenever DELEGATE_ENV (rather than
+          # DELEGATE_BACKEND_URL) resolves the backend; the agent sends all five to
+          # the host as its `auth` init option (see .claude/notes/agents.md § Delegate agent).
+          DELEGATE_ORG_SLUG: ${{ secrets.DELEGATE_ORG_SLUG }}
+          DELEGATE_TENANT_SLUG: ${{ secrets.DELEGATE_TENANT_SLUG }}
         run: .venv/bin/coder-eval run tasks/delegate/fizzbuzz_delegate.yaml --run-dir runs/delegate-live
 
       - name: Verify Delegate live run PASSED
@@ -1405,17 +1403,16 @@ jobs:
       - name: Run Delegate live unit tests (assert PASS)
         if: steps.gate.outputs.present == 'true'
         env:
-          DELEGATE_SDK_NODE_MODULES: ${{ github.workspace }}/src/coder_eval/agents/delegate
           DELEGATE_ENV: ${{ github.event.inputs.delegate_uipath_env || 'alpha' }}
-          AUTH_TOKEN: ${{ steps.ropc.outputs.access_token }}
-          TENANT_ID: ${{ secrets.DELEGATE_TENANT_ID }}
-          ORG_ID: ${{ secrets.DELEGATE_ORG_ID }}
-          ORG_SLUG: ${{ secrets.DELEGATE_ORG_SLUG }}
-          TENANT_SLUG: ${{ secrets.DELEGATE_TENANT_SLUG }}
+          DELEGATE_AUTH_TOKEN: ${{ steps.ropc.outputs.access_token }}
+          DELEGATE_TENANT_ID: ${{ secrets.DELEGATE_TENANT_ID }}
+          DELEGATE_ORG_ID: ${{ secrets.DELEGATE_ORG_ID }}
+          DELEGATE_ORG_SLUG: ${{ secrets.DELEGATE_ORG_SLUG }}
+          DELEGATE_TENANT_SLUG: ${{ secrets.DELEGATE_TENANT_SLUG }}
         run: |
           mkdir -p tmp
           .venv/bin/pytest tests/test_delegate_agent_live.py \
-            -m live -n0 -v --tb=short --strict-markers -ra \
+            -m live -n0 -v --tb=short --strict-markers -ra --log-level=DEBUG \
             --junit-xml=tmp/junit-delegate-live.xml
 
       - name: Assert Delegate live tests actually ran (not silently skipped)

{coder_eval-0.12.11 → coder_eval-0.12.12}/CHANGELOG.md

@@ -2,6 +2,149 @@
 
 <!-- version list -->
 
+## v0.12.12 (2026-10-06)
+
+### Bug Fixes
+
+- **delegate**: Categorize WAF blocks, SSE connect timeouts and session conflicts
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **delegate**: Keep a non-retryable init error terminal on a respawn
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **delegate**: Keep the conversation on a session conflict after a finished turn
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **delegate**: Keep the LLMGW_* secret out of agent shells when a token file is set
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **delegate**: Keep the usage of the calls under a max_turns cut
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **delegate**: Keep tool rows whose result has no open call
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **delegate**: Let the init keys coder_eval owns win over sdk_options
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **delegate**: Make crash retry independent of the stderr tail
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **delegate**: Match an init 401/403 as a whole word
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **delegate**: Record an interrupted tool as an error, not a success
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **delegate**: Record no host for a DELEGATE_BACKEND_URL that does not parse
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **delegate**: Record the SDK's LoadSkill call as the canonical Skill call
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **delegate**: Reject an sdk_options.effort that is not a string
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **deps**: Bump multidict 6.7.1 -> 6.9.1 for CVE-2026-104874
+  ([#218](https://github.com/UiPath/coder_eval/pull/218),
+  [`2fc1ae0`](https://github.com/UiPath/coder_eval/commit/2fc1ae0bc5013da6f9d4f4215d756b27f69c0d80))
+
+- **errors**: Match an HTTP status code in an error message as a whole word
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+### Chores
+
+- **delegate**: Require @uipath/delegate-stdio 1.203.0
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+### Continuous Integration
+
+- **delegate**: Add temporary shell probes to the Delegate live job
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **delegate**: Remove the temporary shell probes
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **delegate**: Replace the shell probes with an auth/host A/B/C probe
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+### Documentation
+
+- Say that the sdk_options keys and record depend on the agent type
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **delegate**: Describe the host env scrub as defense in depth
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **notes**: Shorten the Delegate agent section
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+### Features
+
+- **delegate**: Drive the public @uipath/delegate-stdio host
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **delegate**: Drive the public @uipath/delegate-stdio host [PILOT-7854]
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **delegate**: Find a global delegate-stdio install; rename DELEGATE_SDK_PATH to
+  DELEGATE_STDIO_PATH ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **delegate**: Read the delegate-stdio host's own env var names; route effort through sdk_options
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **delegate**: Send auth to the host as an init option; read the DELEGATE_* names again
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **reports**: Report agent wall and grading separately from end-to-end latency
+  ([#213](https://github.com/UiPath/coder_eval/pull/213),
+  [`b806791`](https://github.com/UiPath/coder_eval/commit/b806791cd4ebd5e3e026d20ee5acac925b750106))
+
+### Testing
+
+- **delegate**: Show the host's results when the live shell check fails
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **golden**: Add Delegate golden-master scenarios
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **golden**: Give the Delegate golden-coverage exemption a true reason
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+- **overrides**: Pin the sdk_options guard on the registry, not its contents
+  ([#207](https://github.com/UiPath/coder_eval/pull/207),
+  [`abc24e8`](https://github.com/UiPath/coder_eval/commit/abc24e8d518cacd09dac36b43f63fb0bb3d3fc21))
+
+
 ## v0.12.11 (2026-10-05)
 
 ### Bug Fixes

{coder_eval-0.12.11 → coder_eval-0.12.12}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: coder-eval
-Version: 0.12.11
+Version: 0.12.12
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity, OpenCode, Pi, Delegate) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval

{coder_eval-0.12.11 → coder_eval-0.12.12}/action.yml

@@ -25,7 +25,7 @@ inputs:
   version:
     description: coder-eval version to install from PyPI, or "local" to install from the action checkout
     required: false
-    default: "0.12.11"   # <-- kept in sync with releases by release.yml
+    default: "0.12.12"   # <-- kept in sync with releases by release.yml
   extras:
     description: >-
       Comma-separated coder-eval extras (`codex`, `antigravity,litellm`), composed