coder-eval 0.9.2__tar.gz → 0.9.4__tar.gz

{coder_eval-0.9.2 → coder_eval-0.9.4}/CHANGELOG.md

@@ -2,6 +2,57 @@
 
 <!-- version list -->
 
+## v0.9.4 (2026-08-04)
+
+### Bug Fixes
+
+- **litellm**: Pin litellm[proxy]==1.95.0 + fastapi==0.140.0 for proxy startup
+  ([#76](https://github.com/UiPath/coder_eval/pull/76),
+  [`f2f8580`](https://github.com/UiPath/coder_eval/commit/f2f85807cff3376b7201d5d5b2f9e1c4874220d9))
+
+- **litellm**: Pin proxy deps (litellm 1.95.0 + fastapi 0.140.0) to fix startup crash
+  ([#76](https://github.com/UiPath/coder_eval/pull/76),
+  [`f2f8580`](https://github.com/UiPath/coder_eval/commit/f2f85807cff3376b7201d5d5b2f9e1c4874220d9))
+
+### Chores
+
+- **action**: Rename Marketplace listing to coder_eval, add author
+  ([`a9c274d`](https://github.com/UiPath/coder_eval/commit/a9c274d918114df5229b20e9c65a4ce620b9f9ed))
+
+### Documentation
+
+- **litellm**: Surface the proxy dep-pin override vars in start script
+  ([#76](https://github.com/UiPath/coder_eval/pull/76),
+  [`f2f8580`](https://github.com/UiPath/coder_eval/commit/f2f85807cff3376b7201d5d5b2f9e1c4874220d9))
+
+### Refactoring
+
+- **litellm**: Address PR review — pin SSOT guard, rename, doc ripple
+  ([#76](https://github.com/UiPath/coder_eval/pull/76),
+  [`f2f8580`](https://github.com/UiPath/coder_eval/commit/f2f85807cff3376b7201d5d5b2f9e1c4874220d9))
+
+
+## v0.9.3 (2026-08-04)
+
+### Bug Fixes
+
+- **early-stop**: Address PR review — polarity-blind budget, pass_threshold displacement,
+  gate-semantic split ([#74](https://github.com/UiPath/coder_eval/pull/74),
+  [`800ac77`](https://github.com/UiPath/coder_eval/commit/800ac7730e53a0ce9e9128b42ea91f0030d01f89))
+
+### Chores
+
+- **deps**: Bump aiohttp 3.14.1→3.14.3, cryptography 49.0.0→50.0.0
+  ([#74](https://github.com/UiPath/coder_eval/pull/74),
+  [`800ac77`](https://github.com/UiPath/coder_eval/commit/800ac7730e53a0ce9e9128b42ea91f0030d01f89))
+
+### Features
+
+- **early-stop**: Weighted ceiling/floor bounds + decision-step budget
+  ([#74](https://github.com/UiPath/coder_eval/pull/74),
+  [`800ac77`](https://github.com/UiPath/coder_eval/commit/800ac7730e53a0ce9e9128b42ea91f0030d01f89))
+
+
 ## v0.9.2 (2026-07-31)
 
 ### Bug Fixes

{coder_eval-0.9.2 → coder_eval-0.9.4}/CLAUDE.md

@@ -141,8 +141,8 @@ action.yml                         # Published composite GitHub Action (coder-ev
 - **Sub-agent token accounting**: There is NO separate per-sub-agent field. Every sub-agent generation is captured as a `parent_tool_use_id`-tagged `AssistantMessage` in the turn transcript, so per-sub-agent usage is derived by grouping those messages on that id (the evalboard's `aggregateSubAgentUsage` does exactly this). Claude bubbles its sub-agent's intermediate generations into the parent stream natively, and the **terminal** generation (delivered as the Agent tool result, never streamed) is synthesized into one via `_synthesize_subagent_terminal_message` from `tool_use_result.usage`. Codex reconstructs all child generations from the child rollout (`_recover_subagent_tool_calls`). The turn total already includes sub-agent cost — Claude via the SDK's cumulative `model_usage`; Codex via `_fold_subagent_tokens`, which folds the child messages (their real per-generation tokens) into the parent total. `CommandTelemetry.result_summary` is stored **untruncated** (no 200-char cap) so sub-agent returns are preserved whole. Set `CODER_EVAL_RAW_SDK_LOG=1` to dump every raw SDK event to the task log for inspection.
 - **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The LiteLLM open-weight actual-cost join (`litellm_cost.apply_actual_cost`) deliberately writes cost at the TURN level only (`token_usage.total_cost_usd` = the real OpenRouter bill) plus the per-call `TurnRecord.provider_call_costs` audit record; it does NOT touch the message token buckets, so `EventCollector` stays the single writer and this invariant holds on every backend. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports.
 - **sandbox isolation**: Tasks that don't need MCP servers should set `setting_sources: []` in their `agent:` block to isolate the sandbox from the host project's CLAUDE.md and settings. Without this, the host project's CLAUDE.md (often 20 KB+) is injected into every API call, inflating cache-creation tokens and cost significantly.
-- **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block.
-- **Early stop on criterion (opt-in)**: `run_limits.stop_early` (default off) ends a single-shot Claude run early once the run's **armed** criteria are decided, so a raised `max_turns` isn't wasted on the smoke flavor. A criterion is armed by `stop_when: pass|fail|decided|auto`; only criteria that can decide from a partial trajectory may arm (non-empty `live_stop_polarities` ClassVar + `live_verdict` override — currently `skill_triggered`, `command_executed`; CE025 keeps the two consistent). `decided` arms **both** polarities; `auto` arms whichever polarities **this instance** can decide — the value for dataset-fanned criteria whose positive/distractor role flips per row. Stop rule: the pass-stop fires when every **pass-armed** criterion live-passes (fail-armed distractors are not required to pass; zero pass-armed ⇒ never pass-stops); the fail-stop fires on the first fail-armed live-fail but is **deferred while any pass-armed criterion is undecided** — a distractor misfire must not truncate a positive row's recall signal, so the latched misfire fires once the positives resolve (or the run continues to the cap). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a `stop_early: false` run. Driven by `orchestration/early_stop.py::EarlyStopWatcher` through the Claude agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all_async` on the frozen trajectory is authoritative. An early-stopped run gates on the **armed subset** (`EvaluationResult.armed_criteria_passed`); a completed run gates on the full set. Every unsupported use is a hard error at resolution (plan *and* run), and a runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo`, report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. Defaults off ⇒ behavior byte-for-behavior unchanged.
+- **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all *task-level* run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block. The one *per-criterion* cap, `max_steps_to_decide`, deliberately lives on `LiveSuccessCriterion` instead (see below) — the watcher must attribute a decision-step-budget breach to a specific criterion, which `RunLimits` (task-scoped, criterion-agnostic) cannot express.
+- **Early stop on criterion (opt-in)**: `run_limits.stop_early` (default off) ends a single-shot Claude run early once the run's **armed** criteria are decided, so a raised `max_turns` isn't wasted on the smoke flavor. A criterion is armed by `stop_when: pass|fail|decided|auto`; only criteria that can decide from a partial trajectory may arm — "is this criterion type live-observable" is `models.LiveSuccessCriterion` subclassing (currently `skill_triggered`, `command_executed`), the single source of truth `validate_early_stop`/`EarlyStopWatcher` check directly via `isinstance`; each subclass implements the abstract, checker-independent `live_decidable_polarities()` (a pure function of its own fields) alongside the checker's `live_verdict` override, and lint rule CE025 (`tests/test_custom_lint.py::TestCE025LiveVerdictConsistency`, a registry-based whole-tree check, not a per-file AST rule) keeps the two paired. `decided` arms **both** polarities; `auto` arms whichever polarities **this instance** can decide — the value for dataset-fanned criteria whose positive/distractor role flips per row. Stop rule is weighted, not strict-boolean: `run_limits.stop_early_gate_threshold` (default `1.0`, reproducing pre-weighting behavior byte-for-byte) is the minimum weighted score (`Σ weight·score / Σ weight` over the armed subset) required to pass; a fail-stop fires once the armed subset's **ceiling** (best case for everything still undecided) can no longer reach the threshold, a pass-stop once the pass-armed subset's **floor** (worst case) already meets it — both **deferred while any pass-armed criterion is undecided**, so a distractor misfire never truncates a positive row's recall signal before the positives resolve (or the run continues to the cap). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a `stop_early: false` run. A per-criterion `max_steps_to_decide` (on `LiveSuccessCriterion` only, requires `stop_when`) caps tool-call steps spent still undecided — cumulative across retry attempts of the same turn — before `EarlyStopReason.DECISION_BUDGET_EXCEEDED` force-fails the run outright, bypassing the weighted gate (nothing to weigh a criterion that never decided against). Driven by `orchestration/early_stop.py::EarlyStopWatcher` through the Claude agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all_async` on the frozen trajectory is authoritative. An early-stopped run gates on the **armed subset** via the weighted `EvaluationResult.armed_criteria_passed`; a run that completes naturally (even with `stop_early: true`) gates on the full set via the strict-AND `all_criteria_passed` — weight magnitude only forgives under the former, so the weighted gate is contingent on the watcher itself firing, not solely on the configured threshold. Every unsupported use is a hard error at resolution (plan *and* run), and a runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo` (incl. `gate_threshold` at stop time), report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. Defaults off ⇒ behavior byte-for-behavior unchanged.
 
 ## Success Criteria (14 types)
 

{coder_eval-0.9.2 → coder_eval-0.9.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: coder-eval
-Version: 0.9.2
+Version: 0.9.4
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval

{coder_eval-0.9.2 → coder_eval-0.9.4}/action.yml

@@ -1,4 +1,12 @@
-name: coder-eval
+# `name` is the GitHub Marketplace listing title and must be globally unique
+# across Marketplace actions, users, AND organizations. `coder-eval` is taken by
+# an unrelated squatted org (github.com/coder-eval), so the listing uses the
+# underscored repo name instead. This value is display-only: consumers reference
+# the action by repo path (`uses: UiPath/coder_eval@v0`), never by this name.
+name: coder_eval
+# Matches the authorship the project already declares in pyproject.toml
+# (`authors = [{ name = "UiPath", ... }]`) and NOTICE (`© 2026 UiPath`).
+author: UiPath
 description: Run coder-eval evaluation tasks as a CI gate, with JUnit XML output and a job-summary report.
 branding:
   icon: check-circle
@@ -33,7 +41,7 @@ inputs:
   version:
     description: coder-eval version to install from PyPI, or "local" to install from the action checkout
     required: false
-    default: "0.9.2"   # <-- kept in sync with releases by release.yml
+    default: "0.9.4"   # <-- kept in sync with releases by release.yml
   run-dir:
     description: Run directory (--run-dir)
     required: false

{coder_eval-0.9.2 → coder_eval-0.9.4}/docs/EXTENDING.md

@@ -210,8 +210,16 @@ Notes:
   `count/mean/median/std/min/max`, so your criterion is suite-thresholdable for free.
   Classification-style criteria return a `ClassificationCriterionResult` and layer
   accuracy / precision / recall / F1 / confusion on top.
-- For **early stop**, implement `live_verdict(...)` and declare
-  `live_stop_polarities` — a lint rule keeps the two consistent.
+- For **early stop**, make your criterion model subclass `LiveSuccessCriterion`
+  (`models/criteria.py`) instead of `BaseSuccessCriterion`, implement its
+  abstract `live_decidable_polarities()` (a pure function of the criterion's
+  own fields — no `turn_records`, no checker instance), and override the
+  checker's `live_verdict(...)`. `LiveSuccessCriterion` subclassing is the
+  single source of truth for "is this criterion type live-observable" —
+  `validate_early_stop`/`EarlyStopWatcher` check `isinstance(c,
+  LiveSuccessCriterion)` directly, no separate checker-side flag. A lint rule
+  (`tests/test_custom_lint.py::TestCE025LiveVerdictConsistency`) keeps the
+  model subclassing and the checker's `live_verdict` override paired.
 
 > A duplicate `criterion_type` **overwrites** the earlier checker with a warning (not
 > a hard error, unlike agents) — keep type strings unique.

{coder_eval-0.9.2 → coder_eval-0.9.4}/docs/REPORT_SCHEMA.md

@@ -188,10 +188,13 @@ backend), `num_turns`, `max_turns_exhausted`,
 ### EarlyStopInfo
 
 Present (non-`null`) iff the run stopped early — there is no separate boolean.
-Fields: `reason` (`criterion_passed` / `criterion_failed`),
+Fields: `reason` (`criterion_passed` / `criterion_failed` /
+`decision_budget_exceeded` — the last forces `FinalStatus.FAILURE` outright,
+bypassing the weighted gate),
 `deciding_criterion_type`, `deciding_criterion_description`, `armed_criteria`,
 `sdk_turn_index`, `tool_call_index` (1-based, includes the in-flight call),
-`elapsed_seconds`, `turns_remaining_at_stop`.
+`elapsed_seconds`, `turns_remaining_at_stop`, `gate_threshold` (the
+`run_limits.stop_early_gate_threshold` in effect for this stop; default `1.0`).
 
 ---
 

{coder_eval-0.9.2 → coder_eval-0.9.4}/docs/TASK_DEFINITION_GUIDE.md

@@ -260,6 +260,7 @@ run_limits:
 | `count_cached_input` | `false` | — | Count `cache_read_input_tokens` toward the input/total budgets. Off by default — cached reads are typically free. |
 | `count_cache_creation` | `false` | — | Count `cache_creation_input_tokens` toward the input/total budgets. Off by default. |
 | `stop_early` | `false` | — | Opt-in master switch for early-stop-on-criterion. See [`stop_early`](#stop_early-opt-in-early-stop). |
+| `stop_early_gate_threshold` | `1.0` | `[0.0, 1.0]` (but `> 0.0` is enforced at resolution when `stop_early: true`) | Minimum weighted score over the armed subset required to gate as a pass. See [`stop_early`](#stop_early-opt-in-early-stop). |
 
 The authoritative source is `src/coder_eval/models/limits.py`. A lint rule (CE030) fails the build if
 a field defined there goes undocumented in this guide, so the table can't quietly fall behind the
@@ -376,11 +377,14 @@ Semantics:
   (e.g. `stop_when: pass` alongside a `max_count`, or `auto` on an instance that
   can decide neither) is likewise a hard error at resolution, not a silent full
   run.
-- **Verdict.** An early-stopped run is gated on the **armed subset only**; the
-  non-armed criteria become **advisory** and are clearly marked (report badge +
-  per-criterion note + `stopped_early` row). A run that completes naturally is
-  gated on the **full** set, as always. This is what lets one file serve both a
-  `smoke` flavor (`stop_early: true`) and an `e2e` flavor (`stop_early: false`) —
+- **Verdict.** Any task armed for early-stop (`stop_early: true`) is gated on
+  the **armed subset only** — the non-armed criteria become **advisory** and
+  are clearly marked (report badge + per-criterion note + `stopped_early`
+  row when the watcher actually fired) — whether or not the watcher actually
+  cut the run short; one task config maps to one gate semantic. Only a task
+  that never armed `stop_early` at all is gated on the **full** set, as
+  always. This is what lets one file serve both a `smoke` flavor
+  (`stop_early: true`) and an `e2e` flavor (`stop_early: false`) —
   see [AB_EXPERIMENTS.md](AB_EXPERIMENTS.md). Verdict parity between the flavors
   is one-sided: a **fail-stop** is verdict-preserving (the deferral above
   guarantees every pass-armed signal was allowed to resolve first), but a
@@ -390,6 +394,57 @@ Semantics:
   authoritative precision/recall belongs on the `stop_early: false` run.
 - **Fail-safe.** A live-verdict bug **fails open** to a full run (logged loudly) —
   it can never silently disable a criterion or cause a false early stop.
+- **Weighting.** `run_limits.stop_early_gate_threshold` (default `1.0`) is the
+  minimum weighted score (`Σ weight·score / Σ weight`, over the armed subset)
+  required to gate as a pass — both for the post-hoc verdict and for the live
+  stop rule itself. A fail-stop fires once the armed subset's **ceiling** (best
+  case: every still-undecided or already-passed criterion ends up scoring 1.0,
+  every live-failed one scores 0) can no longer reach the threshold — the gate
+  is mathematically guaranteed to fail regardless of how the trajectory
+  continues. A pass-stop fires once the pass-armed subset's **floor** (worst
+  case: every still-undecided one scores 0) already meets it. At the default
+  `1.0` both bounds collapse to the pre-weighting rules above exactly (any
+  single armed criterion's live-fail already drops the ceiling below 1.0, and
+  the floor only reaches 1.0 once every pass-armed criterion has actually
+  passed) — lowering it lets a low-weight armed criterion's failure be absorbed
+  without truncating the run, at the cost of the gate becoming a genuine
+  weighted average rather than a strict AND. **The armed weighted gate applies
+  whenever `stop_early: true` is set — one task config, one gate semantic —
+  regardless of whether the watcher actually fired a stop.** A task armed for
+  early-stop that instead completes naturally (the agent finishes, or
+  `max_turns` is hit, before the bound ever trips) is gated on the *same*
+  weighted armed-subset formula as an actual early stop, not the full-run
+  `all_criteria_passed`; only a task that never armed `stop_early` at all uses
+  the strict full-set gate. Each armed criterion's own `pass_threshold` still
+  decides whether it individually passed (converted to a binary 1.0/0.0
+  before weighting) — only the combination rule (weighted average vs strict
+  AND) changes, which is what makes the `gate_threshold=1.0` default an exact
+  equivalence with the pre-weighting `all(...)` rule.
+- **Decision-step budget.** `max_steps_to_decide` (per armed criterion, only
+  on `skill_triggered` / `command_executed`, requires `stop_when`) caps how
+  many tool-call steps that criterion may spend still **undecided** before the
+  run gives up on it:
+
+  ```yaml
+  success_criteria:
+    - type: skill_triggered
+      description: "date-teller must activate within 5 steps"
+      skill_name: date-teller
+      expected_skill: date-teller
+      stop_when: pass
+      max_steps_to_decide: 5
+  ```
+
+  Once the cap is exceeded (checked AFTER the normal fail-/pass-stop checks
+  each round, so a criterion that decides on that very step is never
+  penalized), the watcher fires `reason: decision_budget_exceeded` and the run
+  is forced to `FinalStatus.FAILURE` outright — bypassing
+  `stop_early_gate_threshold`'s weighted gate entirely, since a criterion that
+  never reached a verdict has nothing meaningful to weigh against the others.
+  `None` (default) = no cap; the run relies solely on `run_limits.max_turns`.
+  The step count is **cumulative across every retry attempt** of the turn —
+  including an attempt that crashed or timed out before this criterion's own
+  investigation even began — so size the budget with that headroom in mind.
 
 Observability (every early-stopped run is flagged everywhere so analysis never
 compares a truncated run against a full one):
@@ -535,6 +590,7 @@ All criteria share these fields:
 | `weight` | 1.0 | Relative importance for weighted score. `0` = **informational**: excluded from both the score and the pass/fail gate |
 | `pass_threshold` | 0.9 | Minimum score (0.0–1.0) to pass |
 | `stop_when` | `null` | Arms this criterion for early stop (`pass`/`fail`/`decided`/`auto`); requires `run_limits.stop_early: true` and an observable criterion type (`skill_triggered`, `command_executed`). `auto` arms whichever polarity this instance can decide (for dataset-fanned criteria whose positive/distractor role flips per row). See [`stop_early`](#stop_early-opt-in-early-stop). |
+| `max_steps_to_decide` | `null` | **Only on live-observable criteria** (`skill_triggered`, `command_executed`) — requires `stop_when` to be set. Caps the tool-call steps this armed criterion may spend still undecided before the run gives up and force-fails. See [`stop_early`](#stop_early-opt-in-early-stop). |
 
 **Scoring types:**
 - **Binary** (1.0 or 0.0): `file_exists`, `run_command`, `file_matches_regex`, `classification_match`, `skill_triggered`

{coder_eval-0.9.2 → coder_eval-0.9.4}/litellm/README.md

@@ -55,7 +55,9 @@ bash litellm/start-litellm.sh          # foreground on :4000, Ctrl-C to stop
 ```
 
 Overridable via env: `LITELLM_PORT` (default 4000), `LITELLM_CONFIG`, `ENV_FILE`,
-`LITELLM_MASTER_KEY`.
+`LITELLM_MASTER_KEY`, and the proxy dep pins `LITELLM_SPEC` / `LITELLM_FASTAPI_SPEC`
+(full pip specifiers, e.g. `litellm[proxy]==1.95.0` / `fastapi==0.140.0` — set both
+together when bumping).
 
 ### Point coder_eval at it
 
@@ -190,4 +192,5 @@ subject to this. Bedrock models are single-provider and not affected.
 | `LiteLLM proxy not reachable at ...` (coder_eval startup) | Proxy not running — start it, or unset `LITELLM_BASE_URL`. |
 | `Invalid model name passed in model=...` | Model added to yaml but proxy not restarted — restart it. |
 | HTTP 401 / "Unable to locate credentials" | Missing `AWS_BEARER_TOKEN_BEDROCK` / `OPENROUTER_API_KEY` in `.env`, or key mismatch between `LITELLM_AUTH_TOKEN` (client) and the proxy's master key. |
+| `ModuleNotFoundError: No module named 'proxy_server'` (masked startup death) | fastapi drifted past 0.140.0 (`get_flat_dependant` removed). Use `start-litellm.sh` (it pins the deps), or run with `--with 'fastapi==0.140.0'`. If overriding `LITELLM_SPEC`, bump `LITELLM_FASTAPI_SPEC` to match. |
 | evalboard cost column blank for a model | Model missing from `evalboard/lib/pricing.ts`. |

{coder_eval-0.9.2 → coder_eval-0.9.4}/litellm/cost_logger.py

@@ -26,7 +26,7 @@ Register in ``litellm-config.yaml``::
 
 This module is intentionally **self-contained** (no ``coder_eval`` import): the
 proxy may run in its own ephemeral environment
-(``uvx --from 'litellm[proxy]' litellm``).
+(``uvx --from 'litellm[proxy]==1.95.0' --with 'fastapi==0.140.0' litellm``).
 """
 
 from __future__ import annotations

{coder_eval-0.9.2 → coder_eval-0.9.4}/litellm/litellm-config.yaml

@@ -18,8 +18,10 @@
 #   AWS_REGION=eu-north-1      — EU residency (Stockholm).
 #   LITELLM_MASTER_KEY         — the virtual key clients present as LITELLM_AUTH_TOKEN.
 #
-# Run manually:
-#   uvx --from 'litellm[proxy]' litellm --config litellm/litellm-config.yaml --port 4000
+# Run manually (pin fastapi==0.140.0 — a later 0.140.x patch removed a symbol
+# litellm's proxy imports; start-litellm.sh does this for you):
+#   uvx --from 'litellm[proxy]==1.95.0' --with 'fastapi==0.140.0' \
+#     litellm --config litellm/litellm-config.yaml --port 4000
 
 model_list:
   # DeepSeek V3.2 — cost lead ($0.74 / $2.22 per Mtok).

{coder_eval-0.9.2 → coder_eval-0.9.4}/litellm/start-litellm.sh

@@ -12,7 +12,9 @@
 # Usage:
 #   litellm/start-litellm.sh                 # foreground; Ctrl-C to stop
 # Overridable via env:
-#   LITELLM_PORT (default 4000), LITELLM_CONFIG, ENV_FILE, LITELLM_MASTER_KEY
+#   LITELLM_PORT (default 4000), LITELLM_CONFIG, ENV_FILE, LITELLM_MASTER_KEY,
+#   LITELLM_SPEC / LITELLM_FASTAPI_SPEC (proxy dep pins — FULL pip specifiers,
+#   e.g. 'litellm[proxy]==1.95.0' / 'fastapi==0.140.0'; see the pin comment below)
 set -euo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)"
@@ -69,11 +71,31 @@ if [ -z "$AWS_BEARER_TOKEN_BEDROCK" ]; then
   echo "       Set it in .env or 'export AWS_BEARER_TOKEN_BEDROCK=...' before running." >&2
   exit 1
 fi
+# Pin the proxy deps. `uvx --from 'litellm[proxy]'` unpinned drifts: litellm 1.95.0
+# declares `fastapi>=0.136.3,<1.0`, so uvx grabs the newest fastapi — but fastapi
+# dropped `get_flat_dependant` (which litellm's proxy still imports) in a 0.140.x
+# PATCH (0.140.0 has it, 0.140.13 doesn't), so a range cap isn't enough and startup
+# dies with a (masked) `ModuleNotFoundError: proxy_server`. Pin fastapi to an exact
+# verified-good version and pin litellm so the sidecar can't silently re-break.
+# Both are FULL pip specifiers; override for an upgrade (set BOTH together):
+#   LITELLM_SPEC='litellm[proxy]==<ver>' LITELLM_FASTAPI_SPEC='fastapi==<ver>'.
+LITELLM_SPEC="${LITELLM_SPEC:-litellm[proxy]==1.95.0}"
+LITELLM_FASTAPI_SPEC="${LITELLM_FASTAPI_SPEC:-fastapi==0.140.0}"
+# Fail loud on a bare version (e.g. '0.140.0'): `uvx --with 0.140.0` would die with
+# an opaque 'package not found' instead of a pin error.
+for _spec in "$LITELLM_SPEC" "$LITELLM_FASTAPI_SPEC"; do
+  case "$_spec" in
+    *[=\<\>~]*) ;;
+    *) echo "ERROR: '$_spec' is not a pip specifier (expected e.g. fastapi==0.140.0)." >&2; exit 1 ;;
+  esac
+done
+
 echo "config     : $CONFIG"
 echo "region     : $AWS_REGION"
 echo "bedrock tok: set (${#AWS_BEARER_TOKEN_BEDROCK} chars)"
 echo "master key : $LITELLM_MASTER_KEY"
 echo "cost log   : $LITELLM_COST_LOG"
+echo "proxy deps : $LITELLM_SPEC + $LITELLM_FASTAPI_SPEC"
 
 # --- stop any stale proxy on the port (the classic 'creds-less running proxy') ---
 existing=$(lsof -tiTCP:"$PORT" -sTCP:LISTEN 2>/dev/null || true)
@@ -96,4 +118,5 @@ Set these in coder_eval's .env (or shell) to use it:
 
 EOF
 
-exec uvx --from 'litellm[proxy]' litellm --config "$CONFIG" --host 127.0.0.1 --port "$PORT"
+# Launch with the pinned deps resolved above (rationale in the pin comment).
+exec uvx --from "$LITELLM_SPEC" --with "$LITELLM_FASTAPI_SPEC" litellm --config "$CONFIG" --host 127.0.0.1 --port "$PORT"

{coder_eval-0.9.2 → coder_eval-0.9.4}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "coder-eval"
-version = "0.9.2"
+version = "0.9.4"
 description = "Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity) with sandboxed, reproducible YAML task suites."
 readme = "README.md"
 license = "Apache-2.0"

{coder_eval-0.9.2 → coder_eval-0.9.4}/src/coder_eval/__init__.py

@@ -1,3 +1,3 @@
 """coder_eval - A framework for evaluating AI coding agents."""
 
-__version__ = "0.9.2"
+__version__ = "0.9.4"

{coder_eval-0.9.2 → coder_eval-0.9.4}/src/coder_eval/criteria/base.py

@@ -26,6 +26,19 @@ logger = logging.getLogger(__name__)
 
 # A criterion's verdict from a PARTIAL, mid-run trajectory (early-stop observability).
 # "undecided" means the outcome is not yet knowable from the events seen so far.
+#
+# CONTRACT every live_verdict override must satisfy (see BaseCriterion.live_verdict):
+#   - Deterministic: a pure function of the ``turn_records`` prefix passed in — no
+#     wall-clock, randomness, or other hidden state.
+#   - Monotonic: once it returns "pass"/"fail" for some trajectory prefix, it MUST
+#     return that SAME verdict for every longer prefix (i.e. every later call in the
+#     same run). "undecided" is the only verdict allowed to change on a later call.
+# EarlyStopWatcher's deferred fail-stop and pass/fail flip-attribution
+# (early_stop.py::_prev_verdicts) are correct only because both existing
+# implementations (skill_triggered, command_executed) honor this. A non-monotonic or
+# non-deterministic override compiles and passes CE025 (which only checks
+# LiveSuccessCriterion subclassing / live_verdict pairing, not this) but silently corrupts the stop
+# logic — there is currently no automated enforcement beyond this docstring.
 LiveVerdict = Literal["pass", "fail", "undecided"]
 
 
@@ -200,13 +213,6 @@ class BaseCriterion[C: BaseSuccessCriterion](ABC):
     # Subclasses MUST define this as a class variable
     criterion_type: ClassVar[str]
 
-    # Which polarities this criterion can decide from a PARTIAL, mid-run trajectory.
-    # Empty (base default) = not observable mid-run, so it can never arm early-stop.
-    # A subclass that reads only turn_records and can decide mid-run declares the
-    # polarities it supports (e.g. frozenset({"pass", "fail"})) AND overrides
-    # live_verdict; CE025 enforces that the two stay consistent.
-    live_stop_polarities: ClassVar[frozenset[str]] = frozenset()
-
     def __new__(cls, *args: Any, **kwargs: Any) -> "BaseCriterion[C]":
         """Block direct instantiation of ``BaseCriterion`` itself.
 
@@ -426,36 +432,17 @@ class BaseCriterion[C: BaseSuccessCriterion](ABC):
         from ``check()``/``_check_impl`` run on the frozen trajectory after the
         stop, so a live/final divergence can never corrupt scoring.
 
-        Base default: ``"undecided"`` (not observable mid-run). Subclasses that
-        override this MUST also declare a non-empty ``live_stop_polarities`` (and
-        vice versa) — enforced by lint rule CE025.
+        Base default: ``"undecided"`` (not observable mid-run). A checker
+        overrides this iff its criterion model is a ``LiveSuccessCriterion``
+        subclass (``models/criteria.py``) — that subclassing is the single
+        source of truth for "is this criterion type live-observable", checked
+        by ``validate_early_stop`` / ``EarlyStopWatcher`` and enforced by lint
+        rule CE025. An override MUST also satisfy the deterministic + monotonic
+        contract documented on the ``LiveVerdict`` type above (not enforced by
+        CE025 or any other automated check).
         """
         return "undecided"
 
-    @classmethod
-    def live_decidable_polarities(cls, criterion: C) -> frozenset[str]:
-        """Which polarities THIS criterion *instance* can actually decide mid-run.
-
-        ``live_stop_polarities`` is a class-level *capability* — the widest set
-        of polarities the checker's ``live_verdict`` could ever emit. But for
-        some criteria whether a given polarity can fire depends on the instance's
-        configuration, not just its type. ``command_executed`` is the canonical
-        case: it can live-``pass`` only with no upper bound, and live-``fail``
-        only with one, so a specific criterion may support strictly fewer
-        polarities than its class advertises (down to none — a "dead arm").
-
-        ``validate_early_stop`` gates the requested ``stop_when`` polarity on THIS
-        set, not the ClassVar, so an instance that can never decide its armed
-        polarity is rejected at resolution rather than silently degrading to a
-        full run (the "never a silent no-op" guarantee).
-
-        Default: the class-level ``live_stop_polarities`` — correct for every
-        criterion whose decidability is purely type-level (e.g. ``skill_triggered``).
-        Overrides MUST return a subset of ``live_stop_polarities`` (a criterion
-        cannot decide a polarity its ``live_verdict`` never emits).
-        """
-        return cls.live_stop_polarities
-
     def aggregate(
         self,
         criterion: C,

{coder_eval-0.9.2 → coder_eval-0.9.4}/src/coder_eval/criteria/command_executed.py

@@ -3,7 +3,7 @@
 import json
 import logging
 import re
-from typing import TYPE_CHECKING, ClassVar
+from typing import TYPE_CHECKING
 
 from coder_eval.criteria.base import BaseCriterion, CheckContext, LiveVerdict, register_criterion
 from coder_eval.models import CommandExecutedCriterion, CriterionResult
@@ -30,42 +30,6 @@ class CommandExecutedChecker(BaseCriterion[CommandExecutedCriterion]):
 
     criterion_type = "command_executed"
 
-    # Observable mid-run: command matches accumulate monotonically in the live
-    # stream, so a min_count pass (no upper bound) and a max_count exceedance
-    # (incl. the must-NOT-run 0/0 form) are both decidable before end-of-run.
-    live_stop_polarities: ClassVar[frozenset[str]] = frozenset({"pass", "fail"})
-
-    @classmethod
-    def live_decidable_polarities(cls, criterion: CommandExecutedCriterion) -> frozenset[str]:
-        """Narrow the class capability to what THIS instance can decide mid-run.
-
-        The class advertises ``{"pass", "fail"}``, but ``live_verdict`` can only:
-
-        - ``pass`` when there is no upper bound and a positive floor
-          (``max_count is None and min_count > 0``) — with an upper bound a pass
-          is not final until end-of-run, so it never fires live; and
-        - ``fail`` when there IS an upper bound (``max_count is not None``), the
-          moment the count exceeds it (this includes the ``min_count: 0,
-          max_count: 0`` "must NOT run" form).
-
-        So these instance shapes are dead arms the class-level check misses:
-
-        - ``stop_when: pass`` with ``max_count`` set → pass can never fire;
-        - ``stop_when: fail`` with ``max_count: None`` → fail can never fire;
-        - ``min_count: 0, max_count: None`` → neither can ever fire.
-
-        Reporting the true per-instance set here lets ``validate_early_stop``
-        reject such arming at resolution instead of silently degrading to a full
-        run. Stays a subset of ``live_stop_polarities`` by construction.
-        """
-        decidable: set[str] = set()
-        if criterion.max_count is None:
-            if criterion.min_count > 0:
-                decidable.add("pass")
-        else:
-            decidable.add("fail")
-        return frozenset(decidable)
-
     @staticmethod
     def _matching_commands(
         criterion: CommandExecutedCriterion,

{coder_eval-0.9.2 → coder_eval-0.9.4}/src/coder_eval/criteria/skill_triggered.py

@@ -10,7 +10,7 @@ from __future__ import annotations
 
 import logging
 import re
-from typing import TYPE_CHECKING, ClassVar
+from typing import TYPE_CHECKING
 
 from coder_eval.criteria._classification_aggregate import overlay_classification_metrics
 from coder_eval.criteria.base import BaseCriterion, LiveVerdict, register_criterion
@@ -109,14 +109,6 @@ class SkillTriggeredChecker(BaseCriterion[SkillTriggeredCriterion]):
 
     criterion_type = "skill_triggered"
 
-    # Observable mid-run: a Skill tool call (or a skill file read) is a positive
-    # event in the live stream. The TYPE can decide either polarity — a positive
-    # criterion live-passes when its expected skill is engaged, a
-    # distractor/negative one live-fails when its (wrong) skill is engaged — but
-    # any single INSTANCE decides only one of the two; see
-    # ``live_decidable_polarities``.
-    live_stop_polarities: ClassVar[frozenset[str]] = frozenset({"pass", "fail"})
-
     def _check_impl(
         self,
         criterion: SkillTriggeredCriterion,
@@ -179,37 +171,15 @@ class SkillTriggeredChecker(BaseCriterion[SkillTriggeredCriterion]):
         frozen trajectory by construction — whether or not the run stopped early.
         A positive criterion can therefore only ever live-``pass`` and a
         distractor/negative one only ever live-``fail``; their *absence* is never
-        decidable mid-run (see ``live_decidable_polarities``). This is the change
-        from first-engagement: a wrong skill engaged first no longer live-fails a
-        positive row — the run keeps going so the expected skill can still load.
+        decidable mid-run (see ``SkillTriggeredCriterion.live_decidable_polarities``
+        in models/criteria.py). This is the change from first-engagement: a wrong
+        skill engaged first no longer live-fails a positive row — the run keeps
+        going so the expected skill can still load.
         """
         if criterion.skill_name not in _all_engaged_skill_names(turn_records):
             return "undecided"
         return "pass" if criterion.expected_skill == criterion.skill_name else "fail"
 
-    @classmethod
-    def live_decidable_polarities(cls, criterion: SkillTriggeredCriterion) -> frozenset[str]:
-        """Per-instance narrowing under the any-engagement latch.
-
-        Unlike the type-level capability (``live_stop_polarities`` = both), a
-        single instance decides exactly one polarity:
-
-        - a **positive** criterion (``skill_name == expected_skill``) can only
-          live-``pass`` (the expected skill engaging is a decidable hit; its
-          absence is not knowable mid-run);
-        - a **distractor/negative** criterion (``skill_name != expected_skill``,
-          including the ``expected_skill == ""`` negatives) can only
-          live-``fail`` (a wrong skill engaging is a decidable miss; its absence
-          is not).
-
-        ``validate_early_stop`` gates the requested ``stop_when`` on this set, so
-        arming a positive with ``fail`` / a distractor with ``pass`` — or either
-        with ``decided`` (which needs both) — is rejected at resolution rather
-        than silently degrading to a full run.
-        """
-        expected_yes = criterion.expected_skill == criterion.skill_name
-        return frozenset({"pass"}) if expected_yes else frozenset({"fail"})
-
     def aggregate(
         self,
         criterion: SkillTriggeredCriterion,

{coder_eval-0.9.2 → coder_eval-0.9.4}/src/coder_eval/models/__init__.py

@@ -40,6 +40,8 @@ from coder_eval.models.criteria import (
     FileMatchesRegexCriterion,
     JMESPathAssertion,
     JsonCheckCriterion,
+    LivePolarity,
+    LiveSuccessCriterion,
     LLMJudgeCriterion,
     ReferenceComparisonCriterion,
     RegexPattern,
@@ -232,6 +234,8 @@ __all__ = [  # noqa: RUF022 - Keep grouped by category for readability
     "LLMJudgeCriterion",
     "AgentJudgeCriterion",
     "SkillTriggeredCriterion",
+    "LiveSuccessCriterion",
+    "LivePolarity",
     "SuccessCriterion",
     # Routing
     "ROUTE_NAMES",