coder-eval 0.12.16__tar.gz → 0.12.17__tar.gz

{coder_eval-0.12.16 → coder_eval-0.12.17}/.claude/harness-candidates.md

@@ -545,18 +545,10 @@ divergences, so the deferred-work record is one place. Measurements in
   as tool execution. Pinned by `unbounded_cases` in
   `tests/_fixtures/timing_union_cases.json`, which both suites replay.
 
-- [ ] **Pre-existing, surfaced by this work's final review: `TokenUsage._adopt_legacy_input_tokens`
-  double-counts the cache buckets.** The validator copies a legacy record's
-  full-prompt `input_tokens` straight into `uncached_input_tokens`, and the
-  computed `input_tokens` then adds `cache_creation` + `cache_read` again. A
-  legacy record with `input_tokens=1000`, `cache_creation=200`, `cache_read=150`
-  reloads as 1350 prompt tokens and bills 1000 at the uncached rate instead of
-  650. Affects every report, budget check and detached regrade over a
-  pre-split run that used prompt caching. NOT touched by the timing work
-  (token accounting was explicitly out of its scope) and not a guardrail
-  candidate — a real bug needing its own change, with a decision about
-  whether legacy records can be distinguished from current ones at all.
-  Caught in: timing-capture final review (gpt-5.6-sol).
+- [x] ~~**Pre-existing, surfaced by this work's final review: `TokenUsage._adopt_legacy_input_tokens`
+  double-counts the cache buckets.**~~ **DONE.** The validator is deleted. Every run
+  uploaded in the last 30 days stores `uncached_input_tokens`, so no pre-split record
+  is read. Caught in: timing-capture final review (gpt-5.6-sol).
 
 - [x] ~~No golden-corpus assertion of the four-bucket identity.~~ **DONE.** The
   fixture clocks were unified (`_rebase_notifications` / `_rebase_lines` shift
@@ -1032,3 +1024,7 @@ re-derive from scratch.
   stayed at 1 so `max_turns` never tripped — all silently, with `make verify` green.
   A static rule cannot see an SDK field go dead; a live-telemetry smoke (one real turn,
   assert `token_usage` is non-empty) in the harness-bump checklist would have.
+- [ ] Validation regexes anchored with `$` and applied with `.match` (not `.fullmatch`) accept a trailing newline — `"evil.com\n:443"` passed `normalize_egress_target` until it moved to `fullmatch`. A lint rule needs to tell a validation regex from a search regex; not cheap — caught in the network: llm_only Phase 1 review.
+- [ ] `raise X(...) from exc` where `exc` came from parsing an env/URL value leaks the value through `__cause__` in a traceback (urlsplit put a URL password in its port error). Needs data-flow from env reads; not cheap — caught in the network: llm_only Phase 3 review.
+- [ ] `await asyncio.to_thread(subprocess.run, ...)` that CREATES a resource inside a try whose `finally` removes it: a cancel leaves the worker thread running, so teardown races the create and leaks it (fixed in `isolation/egress.py` by `_run_to_completion`). Detecting "creates a resource" is not mechanical — caught in the network: llm_only Phase 4 review.
+- [ ] The single-`asyncio.shield` join in `fs_permissions.set_permissions` does not survive a SECOND cancel of the caller; `isolation/egress._run_to_completion` loops on the shield. Promote that helper to a shared module and use it in both places — caught in the network: llm_only Phase 4 review.

{coder_eval-0.12.16 → coder_eval-0.12.17}/.claude/notes/agents.md

@@ -92,6 +92,14 @@ The record is BUILT before `_end_turn_ok()` on every harness: a failure inside t
 reduction is a failed turn, and `_end_turn_ok` would already have cleared the rollback
 flag `discard_pending_turn` needs.
 
+`run_limits.max_turn_retries` bounds replay of the entire `communicate()` call at
+the shared orchestrator wrapper. Benchmarks can set it to zero when replay would
+change the measured trajectory. Unset preserves the existing error-category
+policy. The executor's total-attempt cap includes the first call and is checked
+before backoff, after the failure callback drains the partial. A larger cap cannot
+make a rejected error category retryable. Agent/provider internal retries and
+setup retries keep their own policies.
+
 Three exit paths converge on `finalize`, and it is idempotent on all of them, because the
 protocol allows EXACTLY ONE `AgentEndEvent` per `communicate()`: the clean return, the
 crash/timeout kernel, and the outer `except` that can fire *after* a normal finalize (a

{coder_eval-0.12.16 → coder_eval-0.12.17}/.claude/notes/isolation.md

@@ -23,9 +23,10 @@
   written and never read. `tests/test_seed_from_prior_result.py` partitions every
   `EvaluationResult` field as CARRIED or RECOMPUTED and fails closed on a new one, and
   asserts the two `_select_gate()` call sites. A prior status that
-  `FinalStatus.is_execution_fact` (TIMEOUT / ERROR / BUILD_FAILED / the budget stops) is
-  **preserved**, never overwritten: grading may only move `NOT_GRADED` to
-  SUCCESS/FAILURE, since it neither repeated nor observed the agent phase.
+  `EvaluationResult.has_execution_failure` (TIMEOUT / execution or legacy ERROR /
+  BUILD_FAILED / the budget stops) is **preserved**, never overwritten. An explicitly
+  grading-origin ERROR can instead be repaired, with its incident retained in history;
+  see orchestration.md § Grading incidents and execution facts.
   **`MAX_TURNS_EXHAUSTED` is deliberately NOT one of them — anywhere**.
   `_EXECUTION_FACT_STATUSES` maps it to `False`, and the table and the chain that reads
   it must agree: it shipped as `True` while `_terminal_status`'s own docstring argued
@@ -516,9 +517,10 @@ the `~/.claude` copy the target can be an arbitrary path outside the staging tre
 `writable=False` is not cosmetic. The container only ever *reads* and `chmod`s the
 reference copy, so withholding the write bit keeps `_verify_reference_integrity` from being
 the sole guard against tampering during the gaps between windows. The task-dir copy is
-read-only for the same reason: criteria read fixtures there, nothing legitimately writes
-them, and withholding `o+w` keeps an agent from rewriting the expectations it is graded
-against. The `~/.claude` copy IS writable — the CLI rewrites settings and state in place,
+read-only too: criteria read fixtures there and nothing legitimately writes them. That
+does not stop an agent from rewriting the expectations it is graded against: root keeps
+`CAP_FOWNER` and can chmod the copy back, and `_verify_reference_integrity` does not cover
+it. The `~/.claude` copy IS writable — the CLI rewrites settings and state in place,
 `copytree` preserves host modes, and `~/.claude` is routinely 0700 with 0600 files, which
 without DAC_OVERRIDE is unreadable to the container, so the agent cannot authenticate.
 
@@ -1036,3 +1038,192 @@ separator a value beginning with `-` is parsed as an OPTION rather than a reposi
 (`--upload-pack=…` runs a command of the caller's choosing). That URL is task-authored, and
 since `evaluate <run_dir>` rebuilds the task from a shareable run directory it is no longer
 necessarily the operator's own string.
+
+## The egress sidecar (network: llm_only)
+
+`network: llm_only` puts the task container on a per-task `--internal` network whose only exit
+is a proxy sidecar that forwards to an exact `host:port` allowlist. The plan and its spike log
+are in `c/2026-10-01-docker-no-internet-egress.md` (not committed); the facts that shaped the
+design are below.
+
+### Why an explicit proxy, not a transparent one
+
+Harbor's design (a `gost` sidecar, nftables `redirect`, SNI sniffing, the task in the sidecar's
+netns with `NET_ADMIN`) also catches tools that ignore proxy env. Phase 0 found no built-in
+harness that needs it: Claude Code (Bun native binary), Codex (Rust reqwest), Antigravity (Go
+`ProxyFromEnvironment`) and Pi (undici `EnvHttpProxyAgent` with `NODE_USE_ENV_PROXY=1`) all
+honour `HTTPS_PROXY`. An explicit proxy needs no capability, no third-party image and no SNI
+parser, and a tool that ignores it has no route, so it fails closed. iptables inside the task
+container was rejected because it needs `NET_ADMIN` in the agent's own container, which lets the
+agent undo it.
+
+### Why a per-task network
+
+A shared internal network lets task A use task B's sidecar (and its allowlist) and reach task
+B's ports. One network per task costs one address pool each: a default Docker Desktop ran out at
+the 30th network (`all predefined address pools have been fully subnetted`), which is why the
+error names `--max-parallel` and `default-address-pools`.
+
+### Why inhibit_ipv4 and ip_forward=0
+
+On native Linux a default `--internal` bridge still holds a gateway IP on the host, and the task
+container reached every host service bound on `0.0.0.0` through it (dockerd 20.10, 24 and 29,
+both firewall backends; on 20.10 and 24 also `docker0` and the host NIC). The
+`com.docker.network.bridge.inhibit_ipv4=true` option removes the gateway; the sidecar then takes
+the `.1` address and the embedded DNS alias still works. `gateway_mode_ipv4=isolated` was
+rejected: Docker 24 ignores it silently. Docker Desktop never showed the hole.
+`--sysctl net.ipv4.ip_forward=0` on the sidecar is defence in depth: Docker enables forwarding in
+the sidecar's netns, so a task container with `NET_ADMIN` could try to route through it. No
+config field grants the task container run-time capabilities, and the measured route did not
+reach the internet, but the sysctl costs nothing.
+
+### Why a black-hole DNS server and --ipv6=false
+
+The spikes ran on dind, which has no loopback resolver, so they could not show CVE-2024-29018:
+on daemons before 26.0 / 25.0.5 / 23.0.11 the embedded DNS forwards an internal network's
+queries from the host namespace when the host resolver is loopback (systemd-resolved), which is a
+DNS tunnel out. The task container's `--dns 192.0.2.1` (TEST-NET-1, never routed) keeps the
+embedded DNS answering the sidecar alias while every external forward goes nowhere.
+`inhibit_ipv4` removes only the IPv4 gateway, so `--ipv6=false` keeps a daemon whose
+`default-network-opts` enable IPv6 from giving the internal bridge an IPv6 gateway. IPv6
+link-local reach to a host service was not measured.
+
+`NET_RAW` is dropped from the task container for the same reason: with raw sockets an agent could
+write frames for the bridge's own MAC address and reach a host service over UDP without the
+sidecar. That path was not measured; the cap drop removes it at no cost, since no tool needs raw
+sockets under `llm_only`.
+
+### Why the sidecar log is a separate, atomically written file
+
+The first version appended the sidecar log to `docker.log` after the container exited. That file
+is in the run directory, which is bind-mounted writable into the container at the SAME path, so
+the agent could replace it with a symlink to a host file (for example a shell rc file) and get a
+line it chose (`DENY $(cmd):443 CONNECT` passes the visible-ASCII check) appended there by the
+host. Container stdout could also forge `ALLOW` / `DENY` lines in it. The log is now `egress.log`,
+written once by `write_text_atomic`, whose `os.replace` replaces a planted symlink instead of
+following it. `BAD` lines carry only the method and length, because a refused target can hold
+credentials in its userinfo or query.
+
+### Why the framework image and a bind-mounted module
+
+The sidecar image is the framework image, never the task image: a task image is task-authored and
+may be a runtime-kit image with a different Python. The proxy code is NOT taken from the image:
+the host bind-mounts its own `egress_proxy.py` read-only, so the code under test is the boundary
+that runs and image skew cannot change it. That is also why the module is stdlib-only (five
+imports, guarded by a test): the image only supplies `python3`.
+
+### Why the sidecar watches the heartbeat
+
+A host SIGKILL skips every `finally`, and so does a SIGTERM (a CI cancel, `timeout`, systemd):
+coder-eval has no SIGTERM-to-cancel bridge, and `fs_permissions` re-kills with the default action.
+The task container already exits on a stale host heartbeat;
+the sidecar reads the same file through a single-file `:ro` bind (Docker Desktop VirtioFS saw
+every in-place counter write) and stops itself with the same counter-or-mtime rule as
+`heartbeat_is_alive`. It has no `--rm`, so a crashed or stale-stopped sidecar keeps its log until
+teardown reads it; `docker network rm` works with a stopped container still attached, so a later
+prune needs no ordering. The watchdog returns from `serve` instead of calling `os._exit`, because
+CE052 gates process-lethal calls on `IN_CONTAINER_ENV`, which a stdlib-only module cannot import.
+
+The heartbeat stops the sidecar but cannot remove it or its network, and each leaked network holds
+one of the daemon's address pools (about 29 on a default Docker Desktop). A few cancelled parallel
+runs then fail every `network create`, for coder-eval and for anything else on that daemon. So every
+egress setup first prunes egress-labelled stopped containers and unused networks older than
+`RECLAIM_MIN_AGE`. Prune never removes a running container or a network in use, and the age spares a
+concurrent setup between `network create` and `start`. A SIGTERM-to-cancel bridge was not added: the
+prune also covers SIGKILL, and a process-wide handler would change shutdown for every driver.
+
+### Why a refused grading host does not change the status
+
+A grader that cannot reach a host it needs (an `apt-get` mirror, PyPI) fails its criterion, and
+the row reads as an ordinary FAILURE: an agent miss that is really the grader's. Only `egress.log`
+showed it. So after a graded run the host reads the sidecar log from Docker (not the run
+directory's copy, which the container can write), takes the `DENY` lines after `PHASE grading` (or
+all of them for a container that only grades), and records the hosts in
+`environment_info["egress_denied_in_grading"]` as one comma-separated string, which the report
+environment tables render as-is.
+
+Turning such a FAILURE into an ERROR does not work. A DENY does not prove the grader needed the
+host: the probe task's criteria and any no-internet task's criteria are refused on purpose, so a
+real leak would read as a grader error with advice to allowlist `example.com`; an `agent_judge` can
+use a network tool on one run and not the next; and a process the agent left running is refused
+under the same rule. An ERROR written by the host is also an execution fact decided outside the
+orchestrator, so `evaluate` would not re-grade it and `task.html` would keep the FAILURE. Errors
+count as misses in the pass rate, so the rewrite would buy only an error count. The row keeps its
+status, and the host writes the hosts through the same `TASK_JSON_TRANSCRIPT_EXCLUDE` serializer
+the orchestrator uses.
+
+`GRADING_DENIAL_NOISE` drops the hosts the judge CLIs call and do not need (§ Expected DENY lines
+in docs/DOCKER_ISOLATION.md), so they are not recorded on every judged row.
+
+### Why every docker call runs to completion
+
+`_docker` runs `subprocess.run` in a worker thread, and cancelling the await does not stop the
+thread. A Ctrl-C during `docker create` once let teardown run `docker rm -f` before the create
+finished, leaking a created-but-never-started sidecar that no heartbeat would ever stop. So every
+call is joined across any number of cancels (`_run_to_completion`) before the cancel propagates,
+and teardown is joined the same way.
+
+### The log line format is a security boundary
+
+`docker.log` is the only evidence the boundary leaves, and operators grep it for
+`^(ALLOW|DENY|FAIL)`. A request line with a control or non-ASCII byte is refused before anything is
+logged, because a lone `\n` in a method once forged a separate `ALLOW` line.
+
+### Why the allowlist follows the model callers, not the forwarded env
+
+The first version allowed the API backend's hosts for every agent and the host of every forwarded
+`*_URL` variable. The real runs showed the cost: a Claude task could reach the Azure
+`CODEX_BASE_URL` host, and a Codex, Antigravity or Pi task could reach `api.anthropic.com`,
+because those variables are on the default passthrough list and the default backend is `direct`.
+Each host was a model provider, but none was needed. So the backend's hosts are added only for a
+component that uses `API_BACKEND` (`uses_api_backend` on the agent config, an enabled simulator,
+an `llm_judge` / `agent_judge` criterion), and a URL variable counts only where its owner reads it
+(`LITELLM_BASE_URL` for the litellm backend, `CODEX_BASE_URL` in `CodexAgentConfig.egress_hosts`).
+Any other host is explicit in `egress_allowlist`.
+
+### Why grading switches the sidecar in place
+
+Graders often install their own test dependencies (SkillsBench test scripts run `apt-get`, the
+`astral.sh` uv installer and `uvx` from PyPI). With one allowlist for both phases those hosts are
+open to the agent as well, and `github.com`, which the uv installer needs, is where public
+benchmark solutions live. So grading may widen the allowlist, never narrow it.
+
+Grading in a second container (the detached-grade path) was rejected: only the workspace crosses
+over, and the known equivalence gaps in § Two known equivalence gaps already score real rows 0.000
+for state outside it. The switch therefore happens in the same container, as in Harbor, at the
+two grading sites that run after the agent is finished (the single-shot loop and the post-failure
+path). The dialog loop is refused at load, because its criteria run between turns and a one-way
+switch would leave the agent the grading network for the rest of the dialog.
+
+The phase boundary is known only inside the container, so the in-container run asks through a
+request file in the run directory and waits for the host's answer in an ack file. The host, not
+the container, talks to the proxy: `docker exec` into the sidecar reaches a control listener on
+the sidecar's own loopback, which no other netns can reach, and the proxy confirms before the
+host answers. The ack is written with `write_text_atomic` because the run directory is writable
+from the container.
+
+This is not a boundary against an agent that targets it. The agent is root in the same
+container, so it can write the request file early, and a background process it started keeps
+running into grading. Both are the same class as the deliberate re-chmod the anti-cheat window
+leaves open. The `PHASE grading` line in `egress.log` separates the two phases' traffic.
+
+Every graded `llm_only` run switches, including one with no grading allowlist: there the switch
+adds nothing and only writes the `PHASE grading` line that § Why a refused grading host makes the
+row ERROR reads. Such a switch is best-effort, because a failure only loses that report.
+
+Grading-only hosts are checked at setup with `probe --direct` (a TCP connection from the sidecar),
+not through the proxy, because the proxy does not allow them yet.
+
+### What Phase 0 measured
+
+- Every built-in harness in the image and every judge route (`llm_judge`, `agent_judge`,
+  litellm with aiohttp) works through the proxy with an exact host set.
+- Claude Code on Bedrock calls the control plane `bedrock.<region>.amazonaws.com` at every start;
+  it is allowed so the CLI behaves as under `bridge`. `CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC`
+  does not remove that call, and on direct its Datadog `DENY` did not slow the CLI, so it is not set.
+- Claude Code OAuth refresh goes to `platform.claude.com` (binary string `TOKEN_URL`), which is on
+  the direct backend list because `agent_judge` runs the CLI too.
+- `LITELLM_LOCAL_MODEL_COST_MAP=True` costs no latency either way (the `403` is immediate); it is set
+  to remove one `DENY` per process and a network-dependent cost map.
+- busybox `wget` sends absolute-form `GET https://…` to a proxy instead of `CONNECT`; the proxy
+  answers `400` with a `BAD … absolute-form https (use CONNECT)` line rather than originate TLS.

{coder_eval-0.12.16 → coder_eval-0.12.17}/.claude/notes/lint-rules.md

@@ -714,16 +714,15 @@ None else self.open_start_ms)` collapse to one instant whenever `open_end_ms` is
 
 An identity invariant, which is why it is its own id rather than a second arm of
 CE058/CE059, both of which are about timing. Antigravity omitted the kwarg, so the field
-defaulted to `None` on every message it ever recorded, and the evalboard — which groups
-assistant emissions by `message_id` and falls back to a `SAME_EMISSION_GAP_MS` wall-clock
-gap when either side lacks one — collapsed a whole turn's generations into ONE timeline
-row as soon as the harness's generation windows became contiguous (the gap is then exactly
-0 ms, always). Nothing failed: the consumer SUMS the group, so the totals and the
-reconciliation invariant stayed right, and the golden snapshots had ratified the `null` on
-the day they were written — a snapshot is regenerated from whatever the code currently
-does, so it catches a later change and never an initial omission. The damage was not
-confined to the timeline, which is why "only granularity is lost" was the wrong way to
-describe it: a grouped emission is one API call to the evalboard's thinking-cost
+defaulted to `None` on every message it ever recorded, and the evalboard — which then
+grouped assistant emissions by `message_id`, else by a 100 ms wall-clock gap — collapsed a
+whole turn's generations into ONE timeline row as soon as the harness's generation windows
+became contiguous (the gap is then exactly 0 ms, always). Nothing failed: the consumer
+SUMS the group, so the totals and the reconciliation invariant stayed right, and the
+golden snapshots had ratified the `null` on the day they were written — a snapshot is
+regenerated from whatever the code currently does, so it catches a later change and never
+an initial omission. The damage was not confined to the timeline, which is why "only
+granularity is lost" was the wrong way to describe it: a grouped emission is one API call to the evalboard's thinking-cost
 simulator, whose prompt-cache cascade is quadratic in that count, so a single-shot
 Antigravity run had every cascade coefficient pinned at zero; the `Messages` count and the
 10 s slow-generation bar were per-turn too. Deriving the constructor set from each

{coder_eval-0.12.16 → coder_eval-0.12.17}/.claude/notes/orchestration.md

@@ -50,9 +50,10 @@
   path. Three things are refused rather than degraded: `--junit-xml` (a report of
   verdicts, and there are none — though `reports/junit.py` still emits `<skipped>` for
   an ungraded row it encounters), `--allow-host-grading` (it decides how an ungraded row
-  is GRADED, and `execute` grades nothing), and simulation tasks (their
-  turn-continuation logic reads criteria results, so an ungraded dialog would silently
-  change its own stopping behavior). `stop_early:` blocks are inert under `execute` for
+  is GRADED, and `execute` grades nothing), and simulation tasks with an LLM-simulated
+  user (their turn-continuation logic reads criteria results, so an ungraded dialog would
+  silently change its own stopping behavior). A scripted dialog reads none, so `execute`
+  accepts it. `stop_early:` blocks are inert under `execute` for
   the same reason the kill switch exists: the full trajectory is the deliverable.
   Motivating consumer: an external harness (Harbor / Terminal-Bench 2.0) that builds its
   own container, calls coder-eval as the agent, and grades with its own tests.
@@ -86,6 +87,40 @@ produces is under isolation.md § Detached grading and `Sandbox.adopt`).
 `tests/test_seed_from_prior_result.py` pins the status in a test of its own, apart from
 the loop over the real execution facts.
 
+### Grading incidents and execution facts
+
+`FinalStatus.is_execution_fact` is deliberately conservative and context-free.
+`EvaluationResult.has_execution_failure` is the result-aware predicate: only an
+explicit `ERROR` with `error_phase: grading` can be replaced by a new verdict.
+Unknown legacy ERROR, TIMEOUT, build and budget outcomes remain execution facts.
+No error-message matching or transcript inference is used.
+
+Execution limits are settled immediately after each recorded turn, before grading.
+The limit error is raised after grading returns, preserving partial-credit results.
+A grading crash cannot hide the already-settled execution fact. A real task watchdog
+keeps its phase: a fresh run times out execution, while a detached grading attempt
+times out grading. A checker merely raising a timeout or budget exception is not a
+watchdog or an agent limit.
+
+Grading sites wrap failures in one typed error. Reference-integrity failures pass
+through unchanged. A per-turn dialog grade that prevents the remaining execution
+writes an execution-phase ERROR; a completed single-shot or end-only grade writes
+a repairable grading-phase ERROR. No dialog telemetry heuristic is needed to decide
+which one a new record means.
+
+Current failure details remain in the primary error fields. Secondary and repaired
+grading incidents belong in `error_history`. A failed detached grade does not write
+an authoritative result, transcript or HTML verdict: its caller appends the incident
+to a deep copy of the immediate prior row and writes that selected row once. The
+first execute audit copy remains immutable and is never used as a rollback source.
+The Docker path may serialize a failed attempt to its private scratch transport;
+its explicit incident marker is consumed before any authoritative write or fold-back.
+
+Explicit grading ERROR rows owe a grade even if a partial vector or zero score was
+recorded. Otherwise resume eligibility and scoring keep their existing semantics:
+a graded empty vector scores zero, and execute leaves the score unset. No additional
+execution-error rows are automatically scheduled because their vector is empty.
+
 ### The four grading sites
 
 `grade=False` is checked in exactly four places, and they do not behave alike:
@@ -95,11 +130,13 @@ the loop over the real execution facts.
 2. **The single-shot loop** stops after capturing the trajectory. Returning False keeps
    `FinalStatus` off SUCCESS and the status chain turns it into NOT_GRADED. The
    reference-integrity check is skipped too — it protects a grade that is not happening.
-3. **The dialog loop** RAISES, and is unreachable today because `execute` rejects
-   simulation tasks at the CLI: the dialog reads criteria results to decide whether to
-   keep talking, so an ungraded dialog would silently change its own stopping behavior.
-   An empty-list "defensive no-op" here would be worse than a refusal — both callers go
-   straight on to the gate, which treats an empty criteria list as a vacuous pass.
+3. **The dialog loop** checks criteria only on a graded run. Its end-of-dialog and
+   budget-gate checks are skipped when ungraded, so a scripted dialog finalizes
+   NOT_GRADED. The per-turn check still RAISES, and is unreachable today because
+   `execute` rejects LLM-simulated tasks at the CLI: that dialog reads criteria results to
+   decide whether to keep talking, so an ungraded run would silently change its own
+   stopping behavior. An empty-list "defensive no-op" there would be worse than a
+   refusal — the gate treats an empty criteria list as a vacuous pass.
 4. **The diagnostics path** records nothing: a `not_evaluated` vector would imply criteria
    we were supposed to run and could not.
 
@@ -632,10 +669,37 @@ unchanged.
 § Gate selection is fired-only — so `result.early_stop` is never assigned here and an armed
 simulation task gates strict-AND on a possibly truncated trajectory.
 
-`execute` rejects simulation tasks at the CLI, because the loop reads criteria results to
+`execute` rejects LLM-simulated tasks at the CLI, because the loop reads criteria results to
 decide whether to keep talking: an ungraded dialog would silently change its own stopping
 behaviour rather than merely skipping the score.
 
+### Scripted dialogs
+
+`simulation.turns` swaps the LLM user for `ScriptedUser`, which answers the loop's own
+`next_user_message` call. `_evaluation_loop` composes it and hands it to the loop as
+`user=`; the loop never branches on the kind afterwards, so the stop, telemetry and logging
+paths are shared.
+
+- **The cursor is the loop's history.** `ScriptedUser` returns `turns[len(dialog_pairs)]`
+  and keeps no counter, so it cannot drift from the exchanges the loop actually completed.
+  The dialog ends in `evaluate_stop` (`script_exhausted`) before the user is asked past the
+  last turn.
+- **`turns[0]` is the pinned opener.** `_evaluation_loop` hands it to the loop as
+  `initial_prompt`, so turn 1 is recorded like any pinned opener (no simulator timing, an
+  accurate `conversation.log` label) and the task carries one source of truth for its
+  first prompt: `initial_prompt` beside `turns` is rejected.
+- **The fields a script ignores are compared by value, not by `model_fields_set`.**
+  `evaluate` rebuilds the task from a full `model_dump()`, which marks every field as set,
+  so a "was it set" test would reject every recorded scripted task.
+- **Script length owns the dialog length.** Derive `max_turns` after simulation config
+  validation, including merged variant scripts, so a replacement cannot inherit a stale cap.
+  Prompt overrides are rejected before mutation because a script owns every user turn.
+- **No criteria steer a script.** `check_criteria` and `stop_on_criteria_pass` stay at their
+  defaults, so the one criteria check is the end-of-dialog verdict. That is also why
+  `execute` can accept it: nothing the dialog does depends on a verdict it lacks.
+- **No simulator, no simulator egress.** `isolation/egress.py` adds the simulator's model
+  route only for a simulated user.
+
 ## Embedded commands
 
 A recorded config is UNTRUSTED INPUT. `evaluate <run_dir>` rebuilds the task from the

{coder_eval-0.12.16 → coder_eval-0.12.17}/.claude/notes/persistence.md

@@ -167,8 +167,7 @@ render still sees it; the JSON dump excludes it.
 
 The sibling is YAML rather than JSON because the transcript carries multi-line text, which
 YAML's literal block scalar renders as readable paragraphs instead of one line full of `\n`
-escapes. Its consumers are humans. The reader accepts `.json` too, so previously-spilled runs
-keep rendering, and a row with no path at all is a no-op, so old inline records keep working.
+escapes. Its consumers are humans. A row with no path at all is a no-op.
 
 FILE ORDER IS LOAD-BEARING. Each filename is keyed off the criterion's position in its result
 list, and the reader resolves the stored path, so each list must retain its order through

{coder_eval-0.12.16 → coder_eval-0.12.17}/.github/workflows/pr-checks.yml

@@ -123,7 +123,7 @@ jobs:
 
       # PHASE 3: Security scanning
       - name: Security - Dependency vulnerabilities (pip-audit)
-        run: .venv/bin/pip-audit --desc --skip-editable --ignore-vuln CVE-2026-4539 --ignore-vuln CVE-2026-3219 --ignore-vuln CVE-2026-49265 --ignore-vuln CVE-2026-49264 # pygments 2.19.2 ReDoS + pip 26.0.1 tar/ZIP ambiguity; no fixes available on PyPI yet — revisit quarterly. oauthlib 3.3.1 PKCE timing side channel + RevocationEndpoint JSONP callback injection: both server-side OAuth endpoints, unused here (transitive via azure-monitor exporter -> msrest); fix 4.0.0 is a major bump still inside the safe-chain minimum package age — revisit next month
+        run: .venv/bin/pip-audit --desc --skip-editable
 
       - name: Security - OSV vulnerability scan (osv-scanner)
         # Complements pip-audit: pip-audit queries the PyPI advisory DB;