coder-eval 0.9.4__tar.gz → 0.9.5__tar.gz

{coder_eval-0.9.4 → coder_eval-0.9.5}/.claude/harness-candidates.md

@@ -107,3 +107,23 @@ Deferred lint/test guardrails surfaced during reviews. Promote to a `CExxx` rule
   caught them. The cleanup plan explicitly deferred this as YAGNI for the
   one-time purge, but any future doc rename/deletion re-opens the same blind
   spot — caught in the 2026-07-03 open-source-docs-cleanup implementation run.
+
+## From PR #77 (command-executed shell-normalize) — CE030-to-criteria deferred
+
+- [ ] **Extend CE030 doc/schema-parity to the `SuccessCriterion` union** so a new
+  criterion (or field) can't ship undocumented. Attempted in PR #77 and reverted:
+  CI installs `--extra uipath`, and in that environment `coder_eval.models.criteria`
+  gains a `CliCalledCriterion` (fields `log`/`positional`) that is NOT present in a
+  plain checkout (it did not reproduce on macOS, whose lockfile resolution omits the
+  contributing linux-only component). It defeated every discriminator tried — union
+  membership, a `__module__` string filter (it is spoofed to `coder_eval.models.criteria`),
+  a genuine-module-attribute scan (it is `setattr` onto the module), and even an AST
+  parse of the `SuccessCriterion` union literal in `criteria.py` source (CI's imported
+  criteria module resolves to a file whose union literal already contains it). No
+  runtime OR source signal available in the lint could separate the injected criterion
+  from an in-tree one. Revisit only with a way to identify the in-tree criterion set that
+  is provably immune to the uipath integration — e.g. a hardcoded name allowlist of the
+  in-tree criteria (losing auto-coverage of new ones), or first understanding exactly how
+  that environment injects the criterion. Until then CE030 stays scoped to the four
+  top-level models; the `command_pattern`/`exclude_pattern` contract this PR changed is
+  documented in the Field descriptions and TASK_DEFINITION_GUIDE regardless.

{coder_eval-0.9.4 → coder_eval-0.9.5}/CHANGELOG.md

@@ -2,6 +2,99 @@
 
 <!-- version list -->
 
+## v0.9.5 (2026-08-05)
+
+### Bug Fixes
+
+- **command-executed**: Keep whole argv-joined payload in shell unwrap
+  ([#77](https://github.com/UiPath/coder_eval/pull/77),
+  [`7abd080`](https://github.com/UiPath/coder_eval/commit/7abd08098146067890e16740c37239c2e0009a24))
+
+- **command-executed**: Match patterns against shell-normalized commands
+  ([#77](https://github.com/UiPath/coder_eval/pull/77),
+  [`7abd080`](https://github.com/UiPath/coder_eval/commit/7abd08098146067890e16740c37239c2e0009a24))
+
+- **command-executed**: Narrow command param to str before shell-normalizing
+  ([#77](https://github.com/UiPath/coder_eval/pull/77),
+  [`7abd080`](https://github.com/UiPath/coder_eval/commit/7abd08098146067890e16740c37239c2e0009a24))
+
+- **command-executed**: Recognize shell wrappers by predicate, not allowlist
+  ([#77](https://github.com/UiPath/coder_eval/pull/77),
+  [`7abd080`](https://github.com/UiPath/coder_eval/commit/7abd08098146067890e16740c37239c2e0009a24))
+
+- **criteria**: Add present predicate so asserting a switch cannot weaken a guard
+  ([#72](https://github.com/UiPath/coder_eval/pull/72),
+  [`8574ded`](https://github.com/UiPath/coder_eval/commit/8574dedb02d6e9ab883b042934f89297591e8abd))
+
+- **criteria**: Make cli_called guards fail loud instead of vacuously passing
+  ([#72](https://github.com/UiPath/coder_eval/pull/72),
+  [`8574ded`](https://github.com/UiPath/coder_eval/commit/8574dedb02d6e9ab883b042934f89297591e8abd))
+
+- **criteria**: Stop ignore_flags re-opening the guard false-PASS
+  ([#72](https://github.com/UiPath/coder_eval/pull/72),
+  [`8574ded`](https://github.com/UiPath/coder_eval/commit/8574dedb02d6e9ab883b042934f89297591e8abd))
+
+- **early-stop**: Address PR review — trajectory parity, reason determinism, doc restore
+  ([#78](https://github.com/UiPath/coder_eval/pull/78),
+  [`4cf8092`](https://github.com/UiPath/coder_eval/commit/4cf80920422e25fec663ef93e483902bdeffad24))
+
+- **lint**: Derive CE030 criteria from the source union literal, not runtime
+  ([#77](https://github.com/UiPath/coder_eval/pull/77),
+  [`7abd080`](https://github.com/UiPath/coder_eval/commit/7abd08098146067890e16740c37239c2e0009a24))
+
+- **lint**: Enumerate in-tree criteria by module attribute, not the union
+  ([#77](https://github.com/UiPath/coder_eval/pull/77),
+  [`7abd080`](https://github.com/UiPath/coder_eval/commit/7abd08098146067890e16740c37239c2e0009a24))
+
+- **lint**: Scope CE030 criterion parity to in-tree criteria only
+  ([#77](https://github.com/UiPath/coder_eval/pull/77),
+  [`7abd080`](https://github.com/UiPath/coder_eval/commit/7abd08098146067890e16740c37239c2e0009a24))
+
+- **reports**: Explicit return on every early_stop_gate_note path (CodeQL py/mixed-returns)
+  ([#78](https://github.com/UiPath/coder_eval/pull/78),
+  [`4cf8092`](https://github.com/UiPath/coder_eval/commit/4cf80920422e25fec663ef93e483902bdeffad24))
+
+- **reports**: Pre-initialize the gate note so CodeQL sees it bound on every path
+  ([#78](https://github.com/UiPath/coder_eval/pull/78),
+  [`4cf8092`](https://github.com/UiPath/coder_eval/commit/4cf80920422e25fec663ef93e483902bdeffad24))
+
+### Chores
+
+- **deps-dev**: Bump postcss from 8.5.18 to 8.5.23 in /evalboard
+  ([#75](https://github.com/UiPath/coder_eval/pull/75),
+  [`cdced15`](https://github.com/UiPath/coder_eval/commit/cdced152dea732096dcff939c45e7d7c927c2a5a))
+
+### Documentation
+
+- Surface the Marketplace listing and make the Action quickstarts self-sufficient
+  ([#80](https://github.com/UiPath/coder_eval/pull/80),
+  [`401245a`](https://github.com/UiPath/coder_eval/commit/401245ad14055c1da5d7b594e506686c59cedce7))
+
+- **command-executed**: Document shell-normalization contract + gate it (CE030)
+  ([#77](https://github.com/UiPath/coder_eval/pull/77),
+  [`7abd080`](https://github.com/UiPath/coder_eval/commit/7abd08098146067890e16740c37239c2e0009a24))
+
+### Features
+
+- **criteria**: Add cli_called for structured invocation matching
+  ([#72](https://github.com/UiPath/coder_eval/pull/72),
+  [`8574ded`](https://github.com/UiPath/coder_eval/commit/8574dedb02d6e9ab883b042934f89297591e8abd))
+
+- **criteria**: Match a flag across spellings with FlagMatch.aliases
+  ([#72](https://github.com/UiPath/coder_eval/pull/72),
+  [`8574ded`](https://github.com/UiPath/coder_eval/commit/8574dedb02d6e9ab883b042934f89297591e8abd))
+
+- **early-stop**: Per-criterion arming via stop_early blocks on live criteria
+  ([#78](https://github.com/UiPath/coder_eval/pull/78),
+  [`4cf8092`](https://github.com/UiPath/coder_eval/commit/4cf80920422e25fec663ef93e483902bdeffad24))
+
+### Refactoring
+
+- **command-executed**: Total _match_haystacks, shared window, memoized
+  ([#77](https://github.com/UiPath/coder_eval/pull/77),
+  [`7abd080`](https://github.com/UiPath/coder_eval/commit/7abd08098146067890e16740c37239c2e0009a24))
+
+
 ## v0.9.4 (2026-08-04)
 
 ### Bug Fixes

{coder_eval-0.9.4 → coder_eval-0.9.5}/CLAUDE.md

@@ -34,7 +34,7 @@ coder_eval/
 ├── models/                        # Pydantic data models (subpackage)
 │   ├── __init__.py                # Unified exports for all models
 │   ├── enums.py                   # AgentKind, AgentState, FinalStatus, ApiBackend
-│   ├── criteria.py                # 14 success criterion types + base + union
+│   ├── criteria.py                # 15 success criterion types + base + union
 │   ├── experiment.py              # ExperimentDefinition, ExperimentVariant, ResolvedTask, result models
 │   ├── judge_defaults.py          # DEFAULT_JUDGE_MODEL constant (cycle-free leaf)
 │   ├── mutations.py               # PromptMutation variants (prefix/suffix/replace/template)
@@ -141,10 +141,10 @@ action.yml                         # Published composite GitHub Action (coder-ev
 - **Sub-agent token accounting**: There is NO separate per-sub-agent field. Every sub-agent generation is captured as a `parent_tool_use_id`-tagged `AssistantMessage` in the turn transcript, so per-sub-agent usage is derived by grouping those messages on that id (the evalboard's `aggregateSubAgentUsage` does exactly this). Claude bubbles its sub-agent's intermediate generations into the parent stream natively, and the **terminal** generation (delivered as the Agent tool result, never streamed) is synthesized into one via `_synthesize_subagent_terminal_message` from `tool_use_result.usage`. Codex reconstructs all child generations from the child rollout (`_recover_subagent_tool_calls`). The turn total already includes sub-agent cost — Claude via the SDK's cumulative `model_usage`; Codex via `_fold_subagent_tokens`, which folds the child messages (their real per-generation tokens) into the parent total. `CommandTelemetry.result_summary` is stored **untruncated** (no 200-char cap) so sub-agent returns are preserved whole. Set `CODER_EVAL_RAW_SDK_LOG=1` to dump every raw SDK event to the task log for inspection.
 - **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The LiteLLM open-weight actual-cost join (`litellm_cost.apply_actual_cost`) deliberately writes cost at the TURN level only (`token_usage.total_cost_usd` = the real OpenRouter bill) plus the per-call `TurnRecord.provider_call_costs` audit record; it does NOT touch the message token buckets, so `EventCollector` stays the single writer and this invariant holds on every backend. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports.
 - **sandbox isolation**: Tasks that don't need MCP servers should set `setting_sources: []` in their `agent:` block to isolate the sandbox from the host project's CLAUDE.md and settings. Without this, the host project's CLAUDE.md (often 20 KB+) is injected into every API call, inflating cache-creation tokens and cost significantly.
-- **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all *task-level* run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block. The one *per-criterion* cap, `max_steps_to_decide`, deliberately lives on `LiveSuccessCriterion` instead (see below) — the watcher must attribute a decision-step-budget breach to a specific criterion, which `RunLimits` (task-scoped, criterion-agnostic) cannot express.
-- **Early stop on criterion (opt-in)**: `run_limits.stop_early` (default off) ends a single-shot Claude run early once the run's **armed** criteria are decided, so a raised `max_turns` isn't wasted on the smoke flavor. A criterion is armed by `stop_when: pass|fail|decided|auto`; only criteria that can decide from a partial trajectory may arm — "is this criterion type live-observable" is `models.LiveSuccessCriterion` subclassing (currently `skill_triggered`, `command_executed`), the single source of truth `validate_early_stop`/`EarlyStopWatcher` check directly via `isinstance`; each subclass implements the abstract, checker-independent `live_decidable_polarities()` (a pure function of its own fields) alongside the checker's `live_verdict` override, and lint rule CE025 (`tests/test_custom_lint.py::TestCE025LiveVerdictConsistency`, a registry-based whole-tree check, not a per-file AST rule) keeps the two paired. `decided` arms **both** polarities; `auto` arms whichever polarities **this instance** can decide — the value for dataset-fanned criteria whose positive/distractor role flips per row. Stop rule is weighted, not strict-boolean: `run_limits.stop_early_gate_threshold` (default `1.0`, reproducing pre-weighting behavior byte-for-byte) is the minimum weighted score (`Σ weight·score / Σ weight` over the armed subset) required to pass; a fail-stop fires once the armed subset's **ceiling** (best case for everything still undecided) can no longer reach the threshold, a pass-stop once the pass-armed subset's **floor** (worst case) already meets it — both **deferred while any pass-armed criterion is undecided**, so a distractor misfire never truncates a positive row's recall signal before the positives resolve (or the run continues to the cap). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a `stop_early: false` run. A per-criterion `max_steps_to_decide` (on `LiveSuccessCriterion` only, requires `stop_when`) caps tool-call steps spent still undecided — cumulative across retry attempts of the same turn — before `EarlyStopReason.DECISION_BUDGET_EXCEEDED` force-fails the run outright, bypassing the weighted gate (nothing to weigh a criterion that never decided against). Driven by `orchestration/early_stop.py::EarlyStopWatcher` through the Claude agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all_async` on the frozen trajectory is authoritative. An early-stopped run gates on the **armed subset** via the weighted `EvaluationResult.armed_criteria_passed`; a run that completes naturally (even with `stop_early: true`) gates on the full set via the strict-AND `all_criteria_passed` — weight magnitude only forgives under the former, so the weighted gate is contingent on the watcher itself firing, not solely on the configured threshold. Every unsupported use is a hard error at resolution (plan *and* run), and a runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo` (incl. `gate_threshold` at stop time), report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. Defaults off ⇒ behavior byte-for-behavior unchanged.
+- **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all *task-level* run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block. The one *per-criterion* cap, `stop_early.decide_within`, deliberately lives on `LiveSuccessCriterion` instead (see below) — the watcher must attribute a decision-step timeout to a specific criterion, which `RunLimits` (task-scoped, criterion-agnostic) cannot express.
+- **Early stop on criterion (opt-in, per-criterion arming)**: a `stop_early:` block (`StopEarlyPolicy`) on a criterion ends a single-shot run early once the run's **armed** criteria decide the outcome, so a raised `max_turns` isn't wasted on the smoke flavor. The block's PRESENCE is the arming and alone activates the watcher — there is **no run-level master switch**: `run_limits.stop_early: false` is the run-level KILL SWITCH that force-disarms every block (the one-line experiment-variant/`-D` override for an authoritative full run), and `run_limits.stop_early: true` (the removed master arm) is a hard `EarlyStopConfigError` at resolution. The block exists on `LiveSuccessCriterion` only (currently `skill_triggered`, `command_executed` — so arming an unobservable criterion is unrepresentable, a pydantic extra-forbid error). Arming carries one implicit trigger (a native live-fail may fail-stop the run); its keys refine it: `on_pass: stop` (pass-stop the moment the criterion live-passes; default `continue` just latches) and `decide_within: N` (still undecided after N tool-call steps latches an **effective fail**, fed through the same fail-stop rule, reported as `decision_budget_exceeded` — an ordinary weighted fail, NOT a gate-bypassing force-fail; cumulative across retry attempts of the same turn). A trigger whose polarity the instance can't decide (per the abstract, checker-independent `live_decidable_polarities()`, a pure function of the criterion's own fields, paired with the checker's `live_verdict` override by lint rule CE025, a registry-based whole-tree check) is **inert by design** — one dataset-fanned YAML line serves both positive rows (pass/timeout live) and distractor rows (fail live). Verdicts **latch**: once a criterion decides, its `live_verdict` is never polled again. Stop rule is weighted, not strict-boolean: `run_limits.stop_early_gate_threshold` (default `1.0`, reproducing strict-AND behavior exactly) is the minimum weighted score (`Σ weight·score / Σ weight` over the armed subset) required to pass; a fail-stop fires once the armed set's **ceiling** (best case for everything still undecided) can no longer reach the threshold — so a low-weight fail or timeout that can't doom the gate is absorbed and the run continues — and is **deferred while any pass-capable armed criterion is undecided** (a distractor misfire never truncates a positive row's recall signal); a pass-stop fires once the `on_pass: stop` subset's **floor** (worst case) already meets the threshold, and is symmetrically **deferred while any pass-capable armed criterion outside the `on_pass: stop` subset is undecided** (so an early pass never freezes a sibling `on_pass: continue` criterion's signal out of the trajectory). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a kill-switched (`stop_early: false`) run. Driven by `orchestration/early_stop.py::EarlyStopWatcher` (built when `early_stop_active(task)`: ≥1 armed criterion, kill switch not thrown) through the agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all_async` on the frozen trajectory is authoritative. Gating is **FIRED-ONLY**: a run the watcher actually cut gates on the **armed subset** via the weighted `EvaluationResult.armed_criteria_passed`; a run that completes naturally — armed or not — gates strict-AND via `all_criteria_passed`, so adding a block never changes the verdict of a run it didn't cut. Note the gate keys on the watcher having FIRED (`result.early_stop is not None`), not on confirmed truncation — an agent that ignores `should_stop`, or a stop firing on the final message, still gates armed-only. Every resolution-time guardrail violation is a hard error at resolution (plan *and* run); the one load-time case — a `stop_early:` block on a non-live criterion — is a pydantic schema error at task load, which the run surface reports as a skipped task like any other malformed task. A runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo` (incl. `gate_threshold` at stop time), report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. No blocks anywhere ⇒ behavior byte-for-byte unchanged.
 
-## Success Criteria (14 types)
+## Success Criteria (15 types)
 
 | Type | Scoring | Description |
 |------|---------|-------------|
@@ -156,6 +156,7 @@ action.yml                         # Published composite GitHub Action (coder-ev
 | `file_matches_regex` | Binary | Regex match on file |
 | `reference_comparison` | Continuous | AST/token/complexity similarity |
 | `command_executed` | Fractional | Agent tool usage verification |
+| `cli_called` | Binary | Structured match over a JSON Lines invocation log: verb / positional / per-flag predicates, with min_count/max_count bounds |
 | `commands_efficiency` | Continuous | Agent tool-call efficiency relative to expected budget |
 | `uipath_eval` | Fractional | UiPath agent evaluation results |
 | `classification_match` | Binary | File-based label match (observed vs expected) with `(none)`/`(other)` sentinels; emits `ClassificationCriterionResult` for suite-level P/R/F1 |
@@ -163,7 +164,7 @@ action.yml                         # Published composite GitHub Action (coder-ev
 | `llm_judge` | Continuous | LLM grades artifacts + optional trajectory + optional reference; routes through the run's backend (Bedrock / Anthropic) |
 | `agent_judge` | Continuous | Spawns a Claude Code SDK agent in an isolated sandbox copy; judge uses tools (Bash/Read/Grep/…) to investigate and returns a JSON verdict. Expensive; runs with evaluator credentials — see SECURITY note in the criterion docstring. |
 
-All criteria support `weight` (default 1.0) and `pass_threshold` (default 0.9), plus `stop_when` (`pass`/`fail`/`decided`/`auto`, default `null`) which arms the criterion for early stop when `run_limits.stop_early` is set (observable criteria only; `auto` arms the instance's own decidable polarities). On dataset-backed tasks, criteria may also set `suite_thresholds: {metric: min_value}` — the suite gate passes iff every listed metric (from the criterion's `aggregate()` output) meets its minimum.
+All criteria support `weight` (default 1.0) and `pass_threshold` (default 0.9), plus (on live criteria only) a `stop_early:` block (`on_pass`, `decide_within`) that arms the criterion for early stop by its presence. On dataset-backed tasks, criteria may also set `suite_thresholds: {metric: min_value}` — the suite gate passes iff every listed metric (from the criterion's `aggregate()` output) meets its minimum.
 
 ## Evaluation Flow
 
@@ -200,7 +201,7 @@ make lint        # custom architectural lint rules (CE001+)
 make verify      # All of the above + coverage check (CI equivalent)
 ```
 
-When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time — CE027–CE031 — are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name.)
+When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time — CE026–CE031 — are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name. CE026 keeps the GitHub Action's three onboarding surfaces honest: a page's *first* Action snippet must show the agent-runtime prerequisite steps (pinned to the `action-dogfood` job that proves them in CI), a zero-install absolute next to such a snippet must name the channel it means, and every `github.com/marketplace/actions/<slug>` link plus the shields badge label must match `action.yml`'s `name:`.)
 
 Adding a user-facing field to one of the models CE030 tracks (`TaskDefinition`, `RunLimits`, `Dataset`, `SimulationConfig` — see `tests/lint/doc_schema_parity.py`) means documenting it in its guide (mention the field name as inline code) or adding an `EXEMPT` entry with a reason it is not user-authored. `make lint` fails otherwise.
 

{coder_eval-0.9.4 → coder_eval-0.9.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: coder-eval
-Version: 0.9.4
+Version: 0.9.5
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval
@@ -64,8 +64,8 @@ Description-Content-Type: text/markdown
 # Coder Eval — evaluate & benchmark AI coding agents and Claude Code skills
 
 [![PyPI](https://img.shields.io/pypi/v/coder-eval.svg)](https://pypi.org/project/coder-eval/)
+[![GitHub Marketplace](https://img.shields.io/badge/marketplace-coder__eval-2ea44f.svg)](https://github.com/marketplace/actions/coder_eval)
 [![Website](https://img.shields.io/badge/website-coder--eval.com-1f6feb.svg)](https://coder-eval.com)
-[![Docs](https://img.shields.io/badge/docs-coder--eval.com%2Fdocs-1f6feb.svg)](https://coder-eval.com/docs)
 [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
 [![Python 3.13+](https://img.shields.io/badge/python-3.13%2B-blue.svg)](https://www.python.org/downloads/)
 [![CI](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml/badge.svg)](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml)
@@ -160,12 +160,18 @@ the full setup.
 
 ## Use as a GitHub Action
 
-A composite action at the repo root runs `coder-eval` as a CI gate — it installs
-the pinned CLI, runs your tasks, writes a JUnit XML report, appends `run.md` to
-the job summary, and fails the step on any task/gate failure:
+A composite action — on the Marketplace as
+[**coder_eval**](https://github.com/marketplace/actions/coder_eval) — runs
+`coder-eval` as a CI gate. It installs the pinned CLI, runs your tasks, writes a
+JUnit XML report, appends `run.md` to the job summary, and fails the step on any
+task/gate failure:
 
 ```yaml
-- uses: UiPath/coder_eval@v0    # becomes @v1 once 1.0.0 ships; @vX.Y.Z pins exactly
+- uses: actions/setup-node@v4      # the claude-code agent needs the Claude CLI…
+  with: { node-version: '20' }
+- run: npm install -g @anthropic-ai/claude-code
+
+- uses: UiPath/coder_eval@v0       # …then run the gate (@v1 once 1.0.0 ships; @vX.Y.Z pins exactly)
   with:
     tasks: tests/tasks/**/*.yaml
     model: claude-sonnet-5
@@ -217,9 +223,11 @@ code: the step fails if *either* coder-eval exits non-zero *or* any task's
 alone.
 
 > **Agent runtime is the caller's responsibility.** The action is agent-agnostic —
-> it installs `coder-eval` but no coding-agent runtime. Tasks using the default
-> `claude-code` agent need the `claude` CLI on `PATH` (`actions/setup-node` +
-> `npm install -g @anthropic-ai/claude-code`) in the job before the action runs.
+> it installs `coder-eval` but no coding-agent runtime, which is why the example
+> above starts with `actions/setup-node` +
+> `npm install -g @anthropic-ai/claude-code`: the default `claude-code` agent
+> needs the `claude` CLI on `PATH` before the action runs. Swap those steps for
+> your own agent's runtime as needed.
 
 > **Security.** Evaluated tasks execute agent-generated code. Do **not** run this
 > action under `pull_request_target` with secrets exposed to untrusted fork PRs —
@@ -251,7 +259,7 @@ alone.
 | [Bring Your Own Dataset](docs/DATASETS.md) | Fan a single task out over a dataset |
 | [Dialog Mode](docs/DIALOG_MODE.md) | Evaluate agents in multi-turn conversation via a simulated user |
 | [Docker Isolation](docs/DOCKER_ISOLATION.md) | The container sandbox driver, with custom images |
-| [CI Gate & GitHub Action](docs/CI_GATE.md) | Run Coder Eval as a CI gate — the packaged Action, JUnit output, score floor |
+| [CI Gate & GitHub Action](docs/CI_GATE.md) | Run Coder Eval as a CI gate — the Marketplace Action, JUnit output, score floor |
 | [Extending Coder Eval](docs/EXTENDING.md) | Author a custom agent, criterion, or model pricing via the plugin SPI |
 | [Report Schema](docs/REPORT_SCHEMA.md) | Field-level reference for run.json / variant.json / task.json |
 | [How It Compares](docs/comparison.md) | vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, hand-rolled scripts |

{coder_eval-0.9.4 → coder_eval-0.9.5}/README.md

@@ -1,8 +1,8 @@
 # Coder Eval — evaluate & benchmark AI coding agents and Claude Code skills
 
 [![PyPI](https://img.shields.io/pypi/v/coder-eval.svg)](https://pypi.org/project/coder-eval/)
+[![GitHub Marketplace](https://img.shields.io/badge/marketplace-coder__eval-2ea44f.svg)](https://github.com/marketplace/actions/coder_eval)
 [![Website](https://img.shields.io/badge/website-coder--eval.com-1f6feb.svg)](https://coder-eval.com)
-[![Docs](https://img.shields.io/badge/docs-coder--eval.com%2Fdocs-1f6feb.svg)](https://coder-eval.com/docs)
 [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
 [![Python 3.13+](https://img.shields.io/badge/python-3.13%2B-blue.svg)](https://www.python.org/downloads/)
 [![CI](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml/badge.svg)](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml)
@@ -97,12 +97,18 @@ the full setup.
 
 ## Use as a GitHub Action
 
-A composite action at the repo root runs `coder-eval` as a CI gate — it installs
-the pinned CLI, runs your tasks, writes a JUnit XML report, appends `run.md` to
-the job summary, and fails the step on any task/gate failure:
+A composite action — on the Marketplace as
+[**coder_eval**](https://github.com/marketplace/actions/coder_eval) — runs
+`coder-eval` as a CI gate. It installs the pinned CLI, runs your tasks, writes a
+JUnit XML report, appends `run.md` to the job summary, and fails the step on any
+task/gate failure:
 
 ```yaml
-- uses: UiPath/coder_eval@v0    # becomes @v1 once 1.0.0 ships; @vX.Y.Z pins exactly
+- uses: actions/setup-node@v4      # the claude-code agent needs the Claude CLI…
+  with: { node-version: '20' }
+- run: npm install -g @anthropic-ai/claude-code
+
+- uses: UiPath/coder_eval@v0       # …then run the gate (@v1 once 1.0.0 ships; @vX.Y.Z pins exactly)
   with:
     tasks: tests/tasks/**/*.yaml
     model: claude-sonnet-5
@@ -154,9 +160,11 @@ code: the step fails if *either* coder-eval exits non-zero *or* any task's
 alone.
 
 > **Agent runtime is the caller's responsibility.** The action is agent-agnostic —
-> it installs `coder-eval` but no coding-agent runtime. Tasks using the default
-> `claude-code` agent need the `claude` CLI on `PATH` (`actions/setup-node` +
-> `npm install -g @anthropic-ai/claude-code`) in the job before the action runs.
+> it installs `coder-eval` but no coding-agent runtime, which is why the example
+> above starts with `actions/setup-node` +
+> `npm install -g @anthropic-ai/claude-code`: the default `claude-code` agent
+> needs the `claude` CLI on `PATH` before the action runs. Swap those steps for
+> your own agent's runtime as needed.
 
 > **Security.** Evaluated tasks execute agent-generated code. Do **not** run this
 > action under `pull_request_target` with secrets exposed to untrusted fork PRs —
@@ -188,7 +196,7 @@ alone.
 | [Bring Your Own Dataset](docs/DATASETS.md) | Fan a single task out over a dataset |
 | [Dialog Mode](docs/DIALOG_MODE.md) | Evaluate agents in multi-turn conversation via a simulated user |
 | [Docker Isolation](docs/DOCKER_ISOLATION.md) | The container sandbox driver, with custom images |
-| [CI Gate & GitHub Action](docs/CI_GATE.md) | Run Coder Eval as a CI gate — the packaged Action, JUnit output, score floor |
+| [CI Gate & GitHub Action](docs/CI_GATE.md) | Run Coder Eval as a CI gate — the Marketplace Action, JUnit output, score floor |
 | [Extending Coder Eval](docs/EXTENDING.md) | Author a custom agent, criterion, or model pricing via the plugin SPI |
 | [Report Schema](docs/REPORT_SCHEMA.md) | Field-level reference for run.json / variant.json / task.json |
 | [How It Compares](docs/comparison.md) | vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, hand-rolled scripts |

{coder_eval-0.9.4 → coder_eval-0.9.5}/action.yml

@@ -41,7 +41,7 @@ inputs:
   version:
     description: coder-eval version to install from PyPI, or "local" to install from the action checkout
     required: false
-    default: "0.9.4"   # <-- kept in sync with releases by release.yml
+    default: "0.9.5"   # <-- kept in sync with releases by release.yml
   run-dir:
     description: Run directory (--run-dir)
     required: false

{coder_eval-0.9.4 → coder_eval-0.9.5}/docs/AB_EXPERIMENTS.md

@@ -268,11 +268,12 @@ variant's, on the already-mutated prompt.
 ## Recipe: Smoke vs. e2e Flavors (Early Stop)
 
 Run the **same** task file as both a fast `smoke` flavor and a full `e2e` flavor
-by flipping one boolean per variant — `run_limits.stop_early`. Arm the criteria
-that define "the interesting thing happened" with `stop_when` in the task file;
+with a one-line kill switch on the reference variant. Arm the criteria
+that define "the interesting thing happened" with `stop_early:` blocks in the
+task file;
 the `smoke` variant cuts off as soon as they're decided, while `e2e` runs to
 completion. Because the field merge is per-key, the variant sets only
-`stop_early` without disturbing the task's `max_turns`.
+`stop_early` (the run-level kill switch) without disturbing the task's `max_turns`.
 
 ```yaml
 experiment_id: early-stop-ab
@@ -281,13 +282,12 @@ description: "Smoke vs. e2e from one file via opt-in early stop"
 variants:
   - variant_id: e2e
     run_limits:
-      stop_early: false # full run to completion (the reference flavor)
+      stop_early: false # kill switch: force-disarm the blocks (the reference flavor)
   - variant_id: smoke
-    run_limits:
-      stop_early: true # cut off once the armed criteria are decided
+    # no override needed: the task's stop_early: blocks arm the watcher
 ```
 
-The task file supplies the arming (`stop_when` on the criteria that gate the
+The task file supplies the arming (`stop_early:` blocks on the criteria that gate the
 flavor) and a `max_turns` generous enough for `e2e`; see
 [`stop_early`](TASK_DEFINITION_GUIDE.md#stop_early-opt-in-early-stop). This recipe
 ships as `experiments/early-stop-ab.yaml`.

{coder_eval-0.9.4 → coder_eval-0.9.5}/docs/CI_GATE.md

@@ -1,25 +1,34 @@
 ---
 description: >-
-  Run Coder Eval as a CI gate — the packaged composite GitHub Action, JUnit XML
-  output for test-report ingestion, and an optional per-task score floor.
+  Run Coder Eval as a CI gate — the coder_eval GitHub Action from the Actions
+  Marketplace, JUnit XML output for test-report ingestion, and an optional
+  per-task score floor.
 ---
 
 # CI Gate: GitHub Action & JUnit reports
 
-Coder Eval ships a **packaged CI gate**: a composite GitHub Action that installs
-the CLI, runs your tasks, emits a JUnit XML report, appends the run summary to the
-job summary, and fails the build on any task/gate failure. This page is the
-reference for the Action and the JUnit output. For a step-by-step walkthrough
+Coder Eval ships a **packaged CI gate**: a composite GitHub Action — on the
+Actions Marketplace as
+[**coder_eval**](https://github.com/marketplace/actions/coder_eval) — that
+installs the CLI, runs your tasks, emits a JUnit XML report, appends the run
+summary to the job summary, and fails the build on any task/gate failure. This
+page is the reference for the Action and the JUnit output. For a walkthrough
 (including a hand-rolled workflow), see
 [Tutorial 02 — Running Coder Eval in CI](tutorials/02-ci-pipeline.md).
 
 ## The GitHub Action
 
-A composite action lives at the repo root (`action.yml`), so you can reference it
-directly:
+The action is published on the GitHub Actions Marketplace as
+[**coder_eval**](https://github.com/marketplace/actions/coder_eval). It is a
+composite action living at the repo root (`action.yml`), so you reference it by
+repo path — there is no Marketplace install step:
 
 ```yaml
-- uses: UiPath/coder_eval@v0    # becomes @v1 once 1.0.0 ships; @vX.Y.Z pins exactly
+- uses: actions/setup-node@v4      # the claude-code agent needs the Claude CLI…
+  with: { node-version: '20' }
+- run: npm install -g @anthropic-ai/claude-code
+
+- uses: UiPath/coder_eval@v0       # …then run the gate (@v1 once 1.0.0 ships; @vX.Y.Z pins exactly)
   with:
     tasks: tests/tasks/**/*.yaml
     model: claude-sonnet-5
@@ -27,10 +36,11 @@ directly:
       ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}
 ```
 
-The action is **agent-agnostic** — it installs `coder-eval` but *not* any
-coding-agent runtime. Tasks using the default `claude-code` agent need the
-`claude` CLI on `PATH` (Node + `@anthropic-ai/claude-code`), provided by your job
-*before* this step runs.
+The first two steps are there because the action is **agent-agnostic** — it
+installs `coder-eval` but *not* any coding-agent runtime. Tasks using the default
+`claude-code` agent need the `claude` CLI on `PATH` (Node +
+`@anthropic-ai/claude-code`), provided by your job *before* the action runs; swap
+those steps for your own agent's runtime as needed.
 
 ### Inputs
 

{coder_eval-0.9.4 → coder_eval-0.9.5}/docs/DIALOG_MODE.md

@@ -69,9 +69,10 @@ calls the agent's `communicate()`, so Codex and plugin agents work. The *simulat
 always a Claude Code agent. A dialog run therefore needs the `claude` CLI on `PATH` and working
 Anthropic or Bedrock credentials **even when the agent under test is not Claude**.
 
-**Early stop is not available here.** [`run_limits.stop_early`](TASK_DEFINITION_GUIDE.md#stop_early-opt-in-early-stop)
-is single-shot-only; arming it alongside `simulation.enabled` is a hard error at resolution time, not
-a silent no-op. Use `stop_on_criteria_pass` (below) for the dialog equivalent.
+**Early stop is not available here.** [Criterion-level `stop_early:` arming](TASK_DEFINITION_GUIDE.md#stop_early-opt-in-early-stop)
+is single-shot-only; an armed criterion alongside `simulation.enabled` is a hard error at resolution
+time, not a silent no-op (disarm with the `run_limits.stop_early: false` kill switch to run anyway).
+Use `stop_on_criteria_pass` (below) for the dialog equivalent.
 
 Every field's default and constraint lives in one place — the
 [Task Definition Guide's simulation section](TASK_DEFINITION_GUIDE.md#simulation-multi-turn-user-dialog).

{coder_eval-0.9.4 → coder_eval-0.9.5}/docs/EXTENDING.md

@@ -109,7 +109,7 @@ turn, and `ToolStart`/`ToolEnd` per tool call (close orphaned tools with
 returned `TurnRecord`, the single agent-agnostic capture path.
 
 Set `supports_cooperative_stop: ClassVar[bool] = True` only if your `communicate()`
-actually honors `should_stop` (needed for `run_limits.stop_early`). Leaving it
+actually honors `should_stop` (needed for criterion-level `stop_early:` arming). Leaving it
 `False` means early stop is rejected at resolution for your agent — which is correct
 if you can't stop cooperatively.
 
@@ -146,7 +146,7 @@ class MyCriterion(BaseSuccessCriterion):
 Union membership is required — a run validates that every union member's `type` has a
 registered checker, and rejects unknown `type` tags in YAML. `BaseSuccessCriterion`
 gives you `description`, `weight` (default 1.0; `0` = informational/non-gating),
-`pass_threshold` (default 0.9), `stop_when`, and `suite_thresholds` for free, with
+`pass_threshold` (default 0.9) and `suite_thresholds` for free, with
 `extra="forbid"` so YAML typos are caught.
 
 ### Step 2 — the checker

{coder_eval-0.9.4 → coder_eval-0.9.5}/docs/REPORT_SCHEMA.md

@@ -189,8 +189,9 @@ backend), `num_turns`, `max_turns_exhausted`,
 
 Present (non-`null`) iff the run stopped early — there is no separate boolean.
 Fields: `reason` (`criterion_passed` / `criterion_failed` /
-`decision_budget_exceeded` — the last forces `FinalStatus.FAILURE` outright,
-bypassing the weighted gate),
+`decision_budget_exceeded` — the last marks a fail-stop whose deciding
+criterion timed out undecided past its `stop_early.decide_within`; it gates through
+the same weighted armed gate as a native fail),
 `deciding_criterion_type`, `deciding_criterion_description`, `armed_criteria`,
 `sdk_turn_index`, `tool_call_index` (1-based, includes the in-flight call),
 `elapsed_seconds`, `turns_remaining_at_stop`, `gate_threshold` (the