coder-eval 0.9.2__tar.gz → 0.9.3__tar.gz

{coder_eval-0.9.2 → coder_eval-0.9.3}/CHANGELOG.md

@@ -2,6 +2,27 @@
 
 <!-- version list -->
 
+## v0.9.3 (2026-08-04)
+
+### Bug Fixes
+
+- **early-stop**: Address PR review — polarity-blind budget, pass_threshold displacement,
+  gate-semantic split ([#74](https://github.com/UiPath/coder_eval/pull/74),
+  [`800ac77`](https://github.com/UiPath/coder_eval/commit/800ac7730e53a0ce9e9128b42ea91f0030d01f89))
+
+### Chores
+
+- **deps**: Bump aiohttp 3.14.1→3.14.3, cryptography 49.0.0→50.0.0
+  ([#74](https://github.com/UiPath/coder_eval/pull/74),
+  [`800ac77`](https://github.com/UiPath/coder_eval/commit/800ac7730e53a0ce9e9128b42ea91f0030d01f89))
+
+### Features
+
+- **early-stop**: Weighted ceiling/floor bounds + decision-step budget
+  ([#74](https://github.com/UiPath/coder_eval/pull/74),
+  [`800ac77`](https://github.com/UiPath/coder_eval/commit/800ac7730e53a0ce9e9128b42ea91f0030d01f89))
+
+
 ## v0.9.2 (2026-07-31)
 
 ### Bug Fixes

{coder_eval-0.9.2 → coder_eval-0.9.3}/CLAUDE.md

@@ -141,8 +141,8 @@ action.yml                         # Published composite GitHub Action (coder-ev
 - **Sub-agent token accounting**: There is NO separate per-sub-agent field. Every sub-agent generation is captured as a `parent_tool_use_id`-tagged `AssistantMessage` in the turn transcript, so per-sub-agent usage is derived by grouping those messages on that id (the evalboard's `aggregateSubAgentUsage` does exactly this). Claude bubbles its sub-agent's intermediate generations into the parent stream natively, and the **terminal** generation (delivered as the Agent tool result, never streamed) is synthesized into one via `_synthesize_subagent_terminal_message` from `tool_use_result.usage`. Codex reconstructs all child generations from the child rollout (`_recover_subagent_tool_calls`). The turn total already includes sub-agent cost — Claude via the SDK's cumulative `model_usage`; Codex via `_fold_subagent_tokens`, which folds the child messages (their real per-generation tokens) into the parent total. `CommandTelemetry.result_summary` is stored **untruncated** (no 200-char cap) so sub-agent returns are preserved whole. Set `CODER_EVAL_RAW_SDK_LOG=1` to dump every raw SDK event to the task log for inspection.
 - **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The LiteLLM open-weight actual-cost join (`litellm_cost.apply_actual_cost`) deliberately writes cost at the TURN level only (`token_usage.total_cost_usd` = the real OpenRouter bill) plus the per-call `TurnRecord.provider_call_costs` audit record; it does NOT touch the message token buckets, so `EventCollector` stays the single writer and this invariant holds on every backend. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports.
 - **sandbox isolation**: Tasks that don't need MCP servers should set `setting_sources: []` in their `agent:` block to isolate the sandbox from the host project's CLAUDE.md and settings. Without this, the host project's CLAUDE.md (often 20 KB+) is injected into every API call, inflating cache-creation tokens and cost significantly.
-- **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block.
-- **Early stop on criterion (opt-in)**: `run_limits.stop_early` (default off) ends a single-shot Claude run early once the run's **armed** criteria are decided, so a raised `max_turns` isn't wasted on the smoke flavor. A criterion is armed by `stop_when: pass|fail|decided|auto`; only criteria that can decide from a partial trajectory may arm (non-empty `live_stop_polarities` ClassVar + `live_verdict` override — currently `skill_triggered`, `command_executed`; CE025 keeps the two consistent). `decided` arms **both** polarities; `auto` arms whichever polarities **this instance** can decide — the value for dataset-fanned criteria whose positive/distractor role flips per row. Stop rule: the pass-stop fires when every **pass-armed** criterion live-passes (fail-armed distractors are not required to pass; zero pass-armed ⇒ never pass-stops); the fail-stop fires on the first fail-armed live-fail but is **deferred while any pass-armed criterion is undecided** — a distractor misfire must not truncate a positive row's recall signal, so the latched misfire fires once the positives resolve (or the run continues to the cap). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a `stop_early: false` run. Driven by `orchestration/early_stop.py::EarlyStopWatcher` through the Claude agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all_async` on the frozen trajectory is authoritative. An early-stopped run gates on the **armed subset** (`EvaluationResult.armed_criteria_passed`); a completed run gates on the full set. Every unsupported use is a hard error at resolution (plan *and* run), and a runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo`, report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. Defaults off ⇒ behavior byte-for-behavior unchanged.
+- **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all *task-level* run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block. The one *per-criterion* cap, `max_steps_to_decide`, deliberately lives on `LiveSuccessCriterion` instead (see below) — the watcher must attribute a decision-step-budget breach to a specific criterion, which `RunLimits` (task-scoped, criterion-agnostic) cannot express.
+- **Early stop on criterion (opt-in)**: `run_limits.stop_early` (default off) ends a single-shot Claude run early once the run's **armed** criteria are decided, so a raised `max_turns` isn't wasted on the smoke flavor. A criterion is armed by `stop_when: pass|fail|decided|auto`; only criteria that can decide from a partial trajectory may arm — "is this criterion type live-observable" is `models.LiveSuccessCriterion` subclassing (currently `skill_triggered`, `command_executed`), the single source of truth `validate_early_stop`/`EarlyStopWatcher` check directly via `isinstance`; each subclass implements the abstract, checker-independent `live_decidable_polarities()` (a pure function of its own fields) alongside the checker's `live_verdict` override, and lint rule CE025 (`tests/test_custom_lint.py::TestCE025LiveVerdictConsistency`, a registry-based whole-tree check, not a per-file AST rule) keeps the two paired. `decided` arms **both** polarities; `auto` arms whichever polarities **this instance** can decide — the value for dataset-fanned criteria whose positive/distractor role flips per row. Stop rule is weighted, not strict-boolean: `run_limits.stop_early_gate_threshold` (default `1.0`, reproducing pre-weighting behavior byte-for-byte) is the minimum weighted score (`Σ weight·score / Σ weight` over the armed subset) required to pass; a fail-stop fires once the armed subset's **ceiling** (best case for everything still undecided) can no longer reach the threshold, a pass-stop once the pass-armed subset's **floor** (worst case) already meets it — both **deferred while any pass-armed criterion is undecided**, so a distractor misfire never truncates a positive row's recall signal before the positives resolve (or the run continues to the cap). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a `stop_early: false` run. A per-criterion `max_steps_to_decide` (on `LiveSuccessCriterion` only, requires `stop_when`) caps tool-call steps spent still undecided — cumulative across retry attempts of the same turn — before `EarlyStopReason.DECISION_BUDGET_EXCEEDED` force-fails the run outright, bypassing the weighted gate (nothing to weigh a criterion that never decided against). Driven by `orchestration/early_stop.py::EarlyStopWatcher` through the Claude agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all_async` on the frozen trajectory is authoritative. An early-stopped run gates on the **armed subset** via the weighted `EvaluationResult.armed_criteria_passed`; a run that completes naturally (even with `stop_early: true`) gates on the full set via the strict-AND `all_criteria_passed` — weight magnitude only forgives under the former, so the weighted gate is contingent on the watcher itself firing, not solely on the configured threshold. Every unsupported use is a hard error at resolution (plan *and* run), and a runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo` (incl. `gate_threshold` at stop time), report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. Defaults off ⇒ behavior byte-for-behavior unchanged.
 
 ## Success Criteria (14 types)
 

{coder_eval-0.9.2 → coder_eval-0.9.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: coder-eval
-Version: 0.9.2
+Version: 0.9.3
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval

{coder_eval-0.9.2 → coder_eval-0.9.3}/action.yml

@@ -33,7 +33,7 @@ inputs:
   version:
     description: coder-eval version to install from PyPI, or "local" to install from the action checkout
     required: false
-    default: "0.9.2"   # <-- kept in sync with releases by release.yml
+    default: "0.9.3"   # <-- kept in sync with releases by release.yml
   run-dir:
     description: Run directory (--run-dir)
     required: false

{coder_eval-0.9.2 → coder_eval-0.9.3}/docs/EXTENDING.md

@@ -210,8 +210,16 @@ Notes:
   `count/mean/median/std/min/max`, so your criterion is suite-thresholdable for free.
   Classification-style criteria return a `ClassificationCriterionResult` and layer
   accuracy / precision / recall / F1 / confusion on top.
-- For **early stop**, implement `live_verdict(...)` and declare
-  `live_stop_polarities` — a lint rule keeps the two consistent.
+- For **early stop**, make your criterion model subclass `LiveSuccessCriterion`
+  (`models/criteria.py`) instead of `BaseSuccessCriterion`, implement its
+  abstract `live_decidable_polarities()` (a pure function of the criterion's
+  own fields — no `turn_records`, no checker instance), and override the
+  checker's `live_verdict(...)`. `LiveSuccessCriterion` subclassing is the
+  single source of truth for "is this criterion type live-observable" —
+  `validate_early_stop`/`EarlyStopWatcher` check `isinstance(c,
+  LiveSuccessCriterion)` directly, no separate checker-side flag. A lint rule
+  (`tests/test_custom_lint.py::TestCE025LiveVerdictConsistency`) keeps the
+  model subclassing and the checker's `live_verdict` override paired.
 
 > A duplicate `criterion_type` **overwrites** the earlier checker with a warning (not
 > a hard error, unlike agents) — keep type strings unique.

{coder_eval-0.9.2 → coder_eval-0.9.3}/docs/REPORT_SCHEMA.md

@@ -188,10 +188,13 @@ backend), `num_turns`, `max_turns_exhausted`,
 ### EarlyStopInfo
 
 Present (non-`null`) iff the run stopped early — there is no separate boolean.
-Fields: `reason` (`criterion_passed` / `criterion_failed`),
+Fields: `reason` (`criterion_passed` / `criterion_failed` /
+`decision_budget_exceeded` — the last forces `FinalStatus.FAILURE` outright,
+bypassing the weighted gate),
 `deciding_criterion_type`, `deciding_criterion_description`, `armed_criteria`,
 `sdk_turn_index`, `tool_call_index` (1-based, includes the in-flight call),
-`elapsed_seconds`, `turns_remaining_at_stop`.
+`elapsed_seconds`, `turns_remaining_at_stop`, `gate_threshold` (the
+`run_limits.stop_early_gate_threshold` in effect for this stop; default `1.0`).
 
 ---
 

{coder_eval-0.9.2 → coder_eval-0.9.3}/docs/TASK_DEFINITION_GUIDE.md

@@ -260,6 +260,7 @@ run_limits:
 | `count_cached_input` | `false` | — | Count `cache_read_input_tokens` toward the input/total budgets. Off by default — cached reads are typically free. |
 | `count_cache_creation` | `false` | — | Count `cache_creation_input_tokens` toward the input/total budgets. Off by default. |
 | `stop_early` | `false` | — | Opt-in master switch for early-stop-on-criterion. See [`stop_early`](#stop_early-opt-in-early-stop). |
+| `stop_early_gate_threshold` | `1.0` | `[0.0, 1.0]` (but `> 0.0` is enforced at resolution when `stop_early: true`) | Minimum weighted score over the armed subset required to gate as a pass. See [`stop_early`](#stop_early-opt-in-early-stop). |
 
 The authoritative source is `src/coder_eval/models/limits.py`. A lint rule (CE030) fails the build if
 a field defined there goes undocumented in this guide, so the table can't quietly fall behind the
@@ -376,11 +377,14 @@ Semantics:
   (e.g. `stop_when: pass` alongside a `max_count`, or `auto` on an instance that
   can decide neither) is likewise a hard error at resolution, not a silent full
   run.
-- **Verdict.** An early-stopped run is gated on the **armed subset only**; the
-  non-armed criteria become **advisory** and are clearly marked (report badge +
-  per-criterion note + `stopped_early` row). A run that completes naturally is
-  gated on the **full** set, as always. This is what lets one file serve both a
-  `smoke` flavor (`stop_early: true`) and an `e2e` flavor (`stop_early: false`) —
+- **Verdict.** Any task armed for early-stop (`stop_early: true`) is gated on
+  the **armed subset only** — the non-armed criteria become **advisory** and
+  are clearly marked (report badge + per-criterion note + `stopped_early`
+  row when the watcher actually fired) — whether or not the watcher actually
+  cut the run short; one task config maps to one gate semantic. Only a task
+  that never armed `stop_early` at all is gated on the **full** set, as
+  always. This is what lets one file serve both a `smoke` flavor
+  (`stop_early: true`) and an `e2e` flavor (`stop_early: false`) —
   see [AB_EXPERIMENTS.md](AB_EXPERIMENTS.md). Verdict parity between the flavors
   is one-sided: a **fail-stop** is verdict-preserving (the deferral above
   guarantees every pass-armed signal was allowed to resolve first), but a
@@ -390,6 +394,57 @@ Semantics:
   authoritative precision/recall belongs on the `stop_early: false` run.
 - **Fail-safe.** A live-verdict bug **fails open** to a full run (logged loudly) —
   it can never silently disable a criterion or cause a false early stop.
+- **Weighting.** `run_limits.stop_early_gate_threshold` (default `1.0`) is the
+  minimum weighted score (`Σ weight·score / Σ weight`, over the armed subset)
+  required to gate as a pass — both for the post-hoc verdict and for the live
+  stop rule itself. A fail-stop fires once the armed subset's **ceiling** (best
+  case: every still-undecided or already-passed criterion ends up scoring 1.0,
+  every live-failed one scores 0) can no longer reach the threshold — the gate
+  is mathematically guaranteed to fail regardless of how the trajectory
+  continues. A pass-stop fires once the pass-armed subset's **floor** (worst
+  case: every still-undecided one scores 0) already meets it. At the default
+  `1.0` both bounds collapse to the pre-weighting rules above exactly (any
+  single armed criterion's live-fail already drops the ceiling below 1.0, and
+  the floor only reaches 1.0 once every pass-armed criterion has actually
+  passed) — lowering it lets a low-weight armed criterion's failure be absorbed
+  without truncating the run, at the cost of the gate becoming a genuine
+  weighted average rather than a strict AND. **The armed weighted gate applies
+  whenever `stop_early: true` is set — one task config, one gate semantic —
+  regardless of whether the watcher actually fired a stop.** A task armed for
+  early-stop that instead completes naturally (the agent finishes, or
+  `max_turns` is hit, before the bound ever trips) is gated on the *same*
+  weighted armed-subset formula as an actual early stop, not the full-run
+  `all_criteria_passed`; only a task that never armed `stop_early` at all uses
+  the strict full-set gate. Each armed criterion's own `pass_threshold` still
+  decides whether it individually passed (converted to a binary 1.0/0.0
+  before weighting) — only the combination rule (weighted average vs strict
+  AND) changes, which is what makes the `gate_threshold=1.0` default an exact
+  equivalence with the pre-weighting `all(...)` rule.
+- **Decision-step budget.** `max_steps_to_decide` (per armed criterion, only
+  on `skill_triggered` / `command_executed`, requires `stop_when`) caps how
+  many tool-call steps that criterion may spend still **undecided** before the
+  run gives up on it:
+
+  ```yaml
+  success_criteria:
+    - type: skill_triggered
+      description: "date-teller must activate within 5 steps"
+      skill_name: date-teller
+      expected_skill: date-teller
+      stop_when: pass
+      max_steps_to_decide: 5
+  ```
+
+  Once the cap is exceeded (checked AFTER the normal fail-/pass-stop checks
+  each round, so a criterion that decides on that very step is never
+  penalized), the watcher fires `reason: decision_budget_exceeded` and the run
+  is forced to `FinalStatus.FAILURE` outright — bypassing
+  `stop_early_gate_threshold`'s weighted gate entirely, since a criterion that
+  never reached a verdict has nothing meaningful to weigh against the others.
+  `None` (default) = no cap; the run relies solely on `run_limits.max_turns`.
+  The step count is **cumulative across every retry attempt** of the turn —
+  including an attempt that crashed or timed out before this criterion's own
+  investigation even began — so size the budget with that headroom in mind.
 
 Observability (every early-stopped run is flagged everywhere so analysis never
 compares a truncated run against a full one):
@@ -535,6 +590,7 @@ All criteria share these fields:
 | `weight` | 1.0 | Relative importance for weighted score. `0` = **informational**: excluded from both the score and the pass/fail gate |
 | `pass_threshold` | 0.9 | Minimum score (0.0–1.0) to pass |
 | `stop_when` | `null` | Arms this criterion for early stop (`pass`/`fail`/`decided`/`auto`); requires `run_limits.stop_early: true` and an observable criterion type (`skill_triggered`, `command_executed`). `auto` arms whichever polarity this instance can decide (for dataset-fanned criteria whose positive/distractor role flips per row). See [`stop_early`](#stop_early-opt-in-early-stop). |
+| `max_steps_to_decide` | `null` | **Only on live-observable criteria** (`skill_triggered`, `command_executed`) — requires `stop_when` to be set. Caps the tool-call steps this armed criterion may spend still undecided before the run gives up and force-fails. See [`stop_early`](#stop_early-opt-in-early-stop). |
 
 **Scoring types:**
 - **Binary** (1.0 or 0.0): `file_exists`, `run_command`, `file_matches_regex`, `classification_match`, `skill_triggered`

{coder_eval-0.9.2 → coder_eval-0.9.3}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "coder-eval"
-version = "0.9.2"
+version = "0.9.3"
 description = "Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity) with sandboxed, reproducible YAML task suites."
 readme = "README.md"
 license = "Apache-2.0"

{coder_eval-0.9.2 → coder_eval-0.9.3}/src/coder_eval/__init__.py

@@ -1,3 +1,3 @@
 """coder_eval - A framework for evaluating AI coding agents."""
 
-__version__ = "0.9.2"
+__version__ = "0.9.3"

{coder_eval-0.9.2 → coder_eval-0.9.3}/src/coder_eval/criteria/base.py

@@ -26,6 +26,19 @@ logger = logging.getLogger(__name__)
 
 # A criterion's verdict from a PARTIAL, mid-run trajectory (early-stop observability).
 # "undecided" means the outcome is not yet knowable from the events seen so far.
+#
+# CONTRACT every live_verdict override must satisfy (see BaseCriterion.live_verdict):
+#   - Deterministic: a pure function of the ``turn_records`` prefix passed in — no
+#     wall-clock, randomness, or other hidden state.
+#   - Monotonic: once it returns "pass"/"fail" for some trajectory prefix, it MUST
+#     return that SAME verdict for every longer prefix (i.e. every later call in the
+#     same run). "undecided" is the only verdict allowed to change on a later call.
+# EarlyStopWatcher's deferred fail-stop and pass/fail flip-attribution
+# (early_stop.py::_prev_verdicts) are correct only because both existing
+# implementations (skill_triggered, command_executed) honor this. A non-monotonic or
+# non-deterministic override compiles and passes CE025 (which only checks
+# LiveSuccessCriterion subclassing / live_verdict pairing, not this) but silently corrupts the stop
+# logic — there is currently no automated enforcement beyond this docstring.
 LiveVerdict = Literal["pass", "fail", "undecided"]
 
 
@@ -200,13 +213,6 @@ class BaseCriterion[C: BaseSuccessCriterion](ABC):
     # Subclasses MUST define this as a class variable
     criterion_type: ClassVar[str]
 
-    # Which polarities this criterion can decide from a PARTIAL, mid-run trajectory.
-    # Empty (base default) = not observable mid-run, so it can never arm early-stop.
-    # A subclass that reads only turn_records and can decide mid-run declares the
-    # polarities it supports (e.g. frozenset({"pass", "fail"})) AND overrides
-    # live_verdict; CE025 enforces that the two stay consistent.
-    live_stop_polarities: ClassVar[frozenset[str]] = frozenset()
-
     def __new__(cls, *args: Any, **kwargs: Any) -> "BaseCriterion[C]":
         """Block direct instantiation of ``BaseCriterion`` itself.
 
@@ -426,36 +432,17 @@ class BaseCriterion[C: BaseSuccessCriterion](ABC):
         from ``check()``/``_check_impl`` run on the frozen trajectory after the
         stop, so a live/final divergence can never corrupt scoring.
 
-        Base default: ``"undecided"`` (not observable mid-run). Subclasses that
-        override this MUST also declare a non-empty ``live_stop_polarities`` (and
-        vice versa) — enforced by lint rule CE025.
+        Base default: ``"undecided"`` (not observable mid-run). A checker
+        overrides this iff its criterion model is a ``LiveSuccessCriterion``
+        subclass (``models/criteria.py``) — that subclassing is the single
+        source of truth for "is this criterion type live-observable", checked
+        by ``validate_early_stop`` / ``EarlyStopWatcher`` and enforced by lint
+        rule CE025. An override MUST also satisfy the deterministic + monotonic
+        contract documented on the ``LiveVerdict`` type above (not enforced by
+        CE025 or any other automated check).
         """
         return "undecided"
 
-    @classmethod
-    def live_decidable_polarities(cls, criterion: C) -> frozenset[str]:
-        """Which polarities THIS criterion *instance* can actually decide mid-run.
-
-        ``live_stop_polarities`` is a class-level *capability* — the widest set
-        of polarities the checker's ``live_verdict`` could ever emit. But for
-        some criteria whether a given polarity can fire depends on the instance's
-        configuration, not just its type. ``command_executed`` is the canonical
-        case: it can live-``pass`` only with no upper bound, and live-``fail``
-        only with one, so a specific criterion may support strictly fewer
-        polarities than its class advertises (down to none — a "dead arm").
-
-        ``validate_early_stop`` gates the requested ``stop_when`` polarity on THIS
-        set, not the ClassVar, so an instance that can never decide its armed
-        polarity is rejected at resolution rather than silently degrading to a
-        full run (the "never a silent no-op" guarantee).
-
-        Default: the class-level ``live_stop_polarities`` — correct for every
-        criterion whose decidability is purely type-level (e.g. ``skill_triggered``).
-        Overrides MUST return a subset of ``live_stop_polarities`` (a criterion
-        cannot decide a polarity its ``live_verdict`` never emits).
-        """
-        return cls.live_stop_polarities
-
     def aggregate(
         self,
         criterion: C,

{coder_eval-0.9.2 → coder_eval-0.9.3}/src/coder_eval/criteria/command_executed.py

@@ -3,7 +3,7 @@
 import json
 import logging
 import re
-from typing import TYPE_CHECKING, ClassVar
+from typing import TYPE_CHECKING
 
 from coder_eval.criteria.base import BaseCriterion, CheckContext, LiveVerdict, register_criterion
 from coder_eval.models import CommandExecutedCriterion, CriterionResult
@@ -30,42 +30,6 @@ class CommandExecutedChecker(BaseCriterion[CommandExecutedCriterion]):
 
     criterion_type = "command_executed"
 
-    # Observable mid-run: command matches accumulate monotonically in the live
-    # stream, so a min_count pass (no upper bound) and a max_count exceedance
-    # (incl. the must-NOT-run 0/0 form) are both decidable before end-of-run.
-    live_stop_polarities: ClassVar[frozenset[str]] = frozenset({"pass", "fail"})
-
-    @classmethod
-    def live_decidable_polarities(cls, criterion: CommandExecutedCriterion) -> frozenset[str]:
-        """Narrow the class capability to what THIS instance can decide mid-run.
-
-        The class advertises ``{"pass", "fail"}``, but ``live_verdict`` can only:
-
-        - ``pass`` when there is no upper bound and a positive floor
-          (``max_count is None and min_count > 0``) — with an upper bound a pass
-          is not final until end-of-run, so it never fires live; and
-        - ``fail`` when there IS an upper bound (``max_count is not None``), the
-          moment the count exceeds it (this includes the ``min_count: 0,
-          max_count: 0`` "must NOT run" form).
-
-        So these instance shapes are dead arms the class-level check misses:
-
-        - ``stop_when: pass`` with ``max_count`` set → pass can never fire;
-        - ``stop_when: fail`` with ``max_count: None`` → fail can never fire;
-        - ``min_count: 0, max_count: None`` → neither can ever fire.
-
-        Reporting the true per-instance set here lets ``validate_early_stop``
-        reject such arming at resolution instead of silently degrading to a full
-        run. Stays a subset of ``live_stop_polarities`` by construction.
-        """
-        decidable: set[str] = set()
-        if criterion.max_count is None:
-            if criterion.min_count > 0:
-                decidable.add("pass")
-        else:
-            decidable.add("fail")
-        return frozenset(decidable)
-
     @staticmethod
     def _matching_commands(
         criterion: CommandExecutedCriterion,

{coder_eval-0.9.2 → coder_eval-0.9.3}/src/coder_eval/criteria/skill_triggered.py

@@ -10,7 +10,7 @@ from __future__ import annotations
 
 import logging
 import re
-from typing import TYPE_CHECKING, ClassVar
+from typing import TYPE_CHECKING
 
 from coder_eval.criteria._classification_aggregate import overlay_classification_metrics
 from coder_eval.criteria.base import BaseCriterion, LiveVerdict, register_criterion
@@ -109,14 +109,6 @@ class SkillTriggeredChecker(BaseCriterion[SkillTriggeredCriterion]):
 
     criterion_type = "skill_triggered"
 
-    # Observable mid-run: a Skill tool call (or a skill file read) is a positive
-    # event in the live stream. The TYPE can decide either polarity — a positive
-    # criterion live-passes when its expected skill is engaged, a
-    # distractor/negative one live-fails when its (wrong) skill is engaged — but
-    # any single INSTANCE decides only one of the two; see
-    # ``live_decidable_polarities``.
-    live_stop_polarities: ClassVar[frozenset[str]] = frozenset({"pass", "fail"})
-
     def _check_impl(
         self,
         criterion: SkillTriggeredCriterion,
@@ -179,37 +171,15 @@ class SkillTriggeredChecker(BaseCriterion[SkillTriggeredCriterion]):
         frozen trajectory by construction — whether or not the run stopped early.
         A positive criterion can therefore only ever live-``pass`` and a
         distractor/negative one only ever live-``fail``; their *absence* is never
-        decidable mid-run (see ``live_decidable_polarities``). This is the change
-        from first-engagement: a wrong skill engaged first no longer live-fails a
-        positive row — the run keeps going so the expected skill can still load.
+        decidable mid-run (see ``SkillTriggeredCriterion.live_decidable_polarities``
+        in models/criteria.py). This is the change from first-engagement: a wrong
+        skill engaged first no longer live-fails a positive row — the run keeps
+        going so the expected skill can still load.
         """
         if criterion.skill_name not in _all_engaged_skill_names(turn_records):
             return "undecided"
         return "pass" if criterion.expected_skill == criterion.skill_name else "fail"
 
-    @classmethod
-    def live_decidable_polarities(cls, criterion: SkillTriggeredCriterion) -> frozenset[str]:
-        """Per-instance narrowing under the any-engagement latch.
-
-        Unlike the type-level capability (``live_stop_polarities`` = both), a
-        single instance decides exactly one polarity:
-
-        - a **positive** criterion (``skill_name == expected_skill``) can only
-          live-``pass`` (the expected skill engaging is a decidable hit; its
-          absence is not knowable mid-run);
-        - a **distractor/negative** criterion (``skill_name != expected_skill``,
-          including the ``expected_skill == ""`` negatives) can only
-          live-``fail`` (a wrong skill engaging is a decidable miss; its absence
-          is not).
-
-        ``validate_early_stop`` gates the requested ``stop_when`` on this set, so
-        arming a positive with ``fail`` / a distractor with ``pass`` — or either
-        with ``decided`` (which needs both) — is rejected at resolution rather
-        than silently degrading to a full run.
-        """
-        expected_yes = criterion.expected_skill == criterion.skill_name
-        return frozenset({"pass"}) if expected_yes else frozenset({"fail"})
-
     def aggregate(
         self,
         criterion: SkillTriggeredCriterion,

{coder_eval-0.9.2 → coder_eval-0.9.3}/src/coder_eval/models/__init__.py

@@ -40,6 +40,8 @@ from coder_eval.models.criteria import (
     FileMatchesRegexCriterion,
     JMESPathAssertion,
     JsonCheckCriterion,
+    LivePolarity,
+    LiveSuccessCriterion,
     LLMJudgeCriterion,
     ReferenceComparisonCriterion,
     RegexPattern,
@@ -232,6 +234,8 @@ __all__ = [  # noqa: RUF022 - Keep grouped by category for readability
     "LLMJudgeCriterion",
     "AgentJudgeCriterion",
     "SkillTriggeredCriterion",
+    "LiveSuccessCriterion",
+    "LivePolarity",
     "SuccessCriterion",
     # Routing
     "ROUTE_NAMES",

{coder_eval-0.9.2 → coder_eval-0.9.3}/src/coder_eval/models/criteria.py

@@ -8,7 +8,7 @@
 
 from __future__ import annotations
 
-from abc import ABC
+from abc import ABC, abstractmethod
 from typing import Annotated, Any, ClassVar, Literal, Self
 
 from pydantic import BaseModel, ConfigDict, Field, model_validator
@@ -187,6 +187,86 @@ class BaseSuccessCriterion(BaseModel, ABC):
     # Business logic (check operations) moved to SuccessChecker in evaluator.py
 
 
+# The two polarities a live-observable criterion can decide mid-run — distinct
+# from the 3-value LiveVerdict ("pass"/"fail"/"undecided") the checker's
+# live_verdict returns: this is the narrower CAPABILITY type, "undecided" is
+# never a valid decidable polarity. Typed here (not a bare frozenset[str]) so
+# a live_decidable_polarities override returning a stray/typo'd string, or
+# "undecided" itself, is a pyright error rather than a runtime-only lint gap.
+LivePolarity = Literal["pass", "fail"]
+
+
+class LiveSuccessCriterion(BaseSuccessCriterion):
+    """Base for criteria observable from a PARTIAL, mid-run trajectory (early-stop).
+
+    ``live_decidable_polarities`` is a pure function of THIS instance's own
+    fields — no ``turn_records``, no sandbox, no checker instance needed (e.g.
+    ``command_executed`` can decide this purely from whether ``max_count`` is
+    set). That makes it genuinely computable on the data model rather than the
+    checker, unlike the checker's ``live_verdict`` (``criteria/base.py``),
+    which reads the actual trajectory and stays checker-side logic. Moving
+    decidability here also gives early-stop-only config (e.g.
+    ``max_steps_to_decide``) a home that doesn't pollute ``BaseSuccessCriterion``
+    with a field meaningless for every non-observable criterion type.
+
+    Only ``SkillTriggeredCriterion`` / ``CommandExecutedCriterion`` subclass
+    this today; a criterion type is "live-observable" iff it is a
+    ``LiveSuccessCriterion`` subclass — the single source of truth
+    ``validate_early_stop`` / ``EarlyStopWatcher`` consult (no separate
+    checker-side flag to keep in sync).
+    """
+
+    max_steps_to_decide: int | None = Field(
+        default=None,
+        ge=1,
+        description=(
+            "Cap on tool-call steps this ARMED criterion (stop_when must be set) "
+            "may spend still 'undecided' before the run gives up on it. Once "
+            "exceeded, EarlyStopWatcher fires an early stop with reason "
+            "'decision_budget_exceeded' and the run is forced to FinalStatus."
+            "FAILURE outright — regardless of what any other armed criterion's "
+            "weighted score would otherwise gate to (this criterion never "
+            "reached a verdict at all, so there is nothing to weigh). None "
+            "(default) = no cap; the run relies solely on run_limits.max_turns. "
+            "Requires run_limits.stop_early and this criterion's own stop_when. "
+            "The step count is CUMULATIVE across every retry attempt of the "
+            "turn (the same EarlyStopWatcher instance, and its counters, "
+            "persist across retries) — including attempts that ultimately "
+            "crashed or timed out before this criterion's own investigation "
+            "even began. Size the budget with that headroom in mind."
+        ),
+    )
+
+    @model_validator(mode="after")
+    def _check_max_steps_requires_armed(self) -> Self:
+        """Reject a decision-step cap on a criterion that isn't armed for early-stop.
+
+        ``max_steps_to_decide`` only means anything relative to a criterion
+        that ``EarlyStopWatcher`` is actually tracking (``stop_when`` set);
+        setting it without ``stop_when`` is a dead field that silently does
+        nothing, so reject it at load time rather than let it rot unnoticed.
+        """
+        if self.max_steps_to_decide is not None and self.stop_when is None:
+            raise ValueError(
+                f"criterion {self.type!r}: max_steps_to_decide requires stop_when to be set "
+                + "(the decision-step budget is meaningless for a criterion that isn't armed "
+                + "for early-stop)."
+            )
+        return self
+
+    @abstractmethod
+    def live_decidable_polarities(self) -> frozenset[LivePolarity]:
+        """Which polarities THIS instance can decide mid-run, from its own fields alone.
+
+        Must return a subset of the polarities the corresponding checker's
+        ``live_verdict`` can ever emit for this criterion type. Used by
+        ``validate_early_stop`` to reject arming a polarity this instance can
+        never reach, and by ``EarlyStopWatcher`` to resolve which polarities a
+        ``stop_when`` value actually arms for this instance (see
+        ``orchestration.early_stop._requested_polarities``).
+        """
+
+
 class FileExistsCriterion(BaseSuccessCriterion):
     """Check if a file exists at the specified path.
 
@@ -472,7 +552,7 @@ class CommandsEfficiencyCriterion(BaseSuccessCriterion):
     expected_commands: int = Field(ge=1, description="Expected number of tool commands to complete the task")
 
 
-class CommandExecutedCriterion(BaseSuccessCriterion):
+class CommandExecutedCriterion(LiveSuccessCriterion):
     """Check whether the agent executed specific commands/tools.
 
     Inspects CommandTelemetry records from TurnRecord.commands to verify
@@ -548,6 +628,32 @@ class CommandExecutedCriterion(BaseSuccessCriterion):
             raise ValueError(f"max_count ({self.max_count}) must be >= min_count ({self.min_count})")
         return self
 
+    def live_decidable_polarities(self) -> frozenset[LivePolarity]:
+        """Narrow to what THIS instance can decide mid-run.
+
+        The checker's ``live_verdict`` (``criteria/command_executed.py``) can
+        only:
+
+        - ``pass`` when there is no upper bound and a positive floor
+          (``max_count is None and min_count > 0``) — with an upper bound a
+          pass is not final until end-of-run, so it never fires live; and
+        - ``fail`` when there IS an upper bound (``max_count is not None``),
+          the moment the count exceeds it (this includes the
+          ``min_count: 0, max_count: 0`` "must-NOT-run" form).
+
+        So these instance shapes are dead arms the class-level check misses:
+        ``stop_when: pass`` with ``max_count`` set (pass can never fire);
+        ``stop_when: fail`` with ``max_count: None`` (fail can never fire);
+        ``min_count: 0, max_count: None`` (neither can ever fire).
+        """
+        decidable: set[LivePolarity] = set()
+        if self.max_count is None:
+            if self.min_count > 0:
+                decidable.add("pass")
+        else:
+            decidable.add("fail")
+        return frozenset(decidable)
+
 
 class UiPathEvalCriterion(BaseSuccessCriterion):
     """Check evaluation results against UiPath agent performance.
@@ -608,7 +714,7 @@ class ClassificationMatchCriterion(BaseSuccessCriterion):
     )
 
 
-class SkillTriggeredCriterion(BaseSuccessCriterion):
+class SkillTriggeredCriterion(LiveSuccessCriterion):
     """Binary classifier: did the agent engage the target skill during the run?
 
     Agent-agnostic. Observed label is ``"yes"`` when ``turn_records`` show the
@@ -644,6 +750,29 @@ class SkillTriggeredCriterion(BaseSuccessCriterion):
         description="Only count Skill invocations whose 'skill' parameter matches this name.",
     )
 
+    def live_decidable_polarities(self) -> frozenset[LivePolarity]:
+        """Per-instance narrowing under the checker's any-engagement latch.
+
+        The checker's ``live_verdict`` (``criteria/skill_triggered.py``) can
+        decide either polarity at the TYPE level, but a single INSTANCE only
+        ever resolves one of them:
+
+        - a **positive** criterion (``skill_name == expected_skill``) can only
+          live-``pass`` (the expected skill engaging is a decidable hit; its
+          absence is not knowable mid-run);
+        - a **distractor/negative** criterion (``skill_name != expected_skill``,
+          including the ``expected_skill == ""`` negatives) can only
+          live-``fail`` (a wrong skill engaging is a decidable miss; its
+          absence is not).
+
+        ``validate_early_stop`` gates the requested ``stop_when`` on this set,
+        so arming a positive with ``fail`` / a distractor with ``pass`` — or
+        either with ``decided`` (which needs both) — is rejected at resolution
+        rather than silently degrading to a full run.
+        """
+        expected_yes = self.expected_skill == self.skill_name
+        return frozenset({"pass"}) if expected_yes else frozenset({"fail"})
+
 
 class LLMJudgeCriterion(BaseSuccessCriterion):
     """Have an LLM grade the task's final state against an author-supplied prompt.