coder-eval 0.8.3__tar.gz → 0.8.4__tar.gz

{coder_eval-0.8.3 → coder_eval-0.8.4}/.claude/harness-candidates.md

@@ -5,21 +5,27 @@ Deferred lint/test guardrails surfaced during reviews. Promote to a `CExxx` rule
 
 ## From code review 260701-1954 (fix-review-top5 run) — deferred to a dedicated guardrail plan
 
-- **CE024** — workflow-YAML rule: forbid any `uses:` step pinned to a floating ref
+> Numbering note: these are *proposed* ids. `CE024` (discriminated-unions) and
+> `CE025` (live-verdict consistency) have since been **implemented** for other
+> rules, so the candidates below were renumbered to the next free ids. Always
+> claim the next unused number in `tests/lint/rules/` — the id-uniqueness assert
+> in `tests/lint/runner.py` is the source of truth.
+
+- **CE026** — workflow-YAML rule: forbid any `uses:` step pinned to a floating ref
   (`@v3`, `@main`) rather than a 40-hex commit SHA. Would have caught
   `mxschmitt/action-tmate@v3` (fixed manually in this run). >30 min: needs a
   non-Python file-walk branch in `tests/lint/runner.py`.
-- **CE025** — retired-token grep gate: fail when a removed-subsystem token
+- **CE027** — retired-token grep gate: fail when a removed-subsystem token
   (`LLMGW_`, `API_BACKEND=proxy`, `uipath_llmgw_client`) reappears outside an
   allowlist across docs/config/src. Would have caught the LLM-Gateway residue
   swept in this run. >30 min: needs an allowlist + repo-wide text scan.
-- **CE026** — assert the Makefile `lint:` help does not hardcode a stale `CE0NN`
+- **CE028** — assert the Makefile `lint:` help does not hardcode a stale `CE0NN`
   upper bound (use `CE001+`). Would have caught the `CE001–CE005` drift fixed here.
 - **docs-vs-harness smoke test** — execute the CI tutorial's `coder-eval run`
   command against a NoOp task and assert the produced tree matches the documented
   globs. Would have caught the `--run-dir runs` layout bug. Not statically
   reachable (needs a live run).
-- [ ] CE-rule: `type: Literal[...]` fields on models in `coder_eval/models/` must declare their tag default (`type: Literal["x"] = "x"`) — a member without the default degrades `validate_registry` diagnostics (PydanticUndefined in expected_types) and breaks direct construction. Nothing guards it today; needs a rule-design call (second violation class inside CE024 vs. a new CE025), and the failure is already double-caught by the MINIMAL_PAYLOADS parity test + direct-construction tests — caught in the 2026-07-03 top5-review-fixes run (Phase 1 quality review).
+- [ ] CE-rule: `type: Literal[...]` fields on models in `coder_eval/models/` must declare their tag default (`type: Literal["x"] = "x"`) — a member without the default degrades `validate_registry` diagnostics (PydanticUndefined in expected_types) and breaks direct construction. Nothing guards it today; needs a rule-design call (second violation class inside CE024 vs. a new CExxx at the next free id), and the failure is already double-caught by the MINIMAL_PAYLOADS parity test + direct-construction tests — caught in the 2026-07-03 top5-review-fixes run (Phase 1 quality review).
 
 ## From 2026-07-03 open-source docs cleanup
 

{coder_eval-0.8.3 → coder_eval-0.8.4}/CHANGELOG.md

@@ -2,6 +2,65 @@
 
 <!-- version list -->
 
+## v0.8.4 (2026-07-13)
+
+### Bug Fixes
+
+- **deps**: Upgrade click to 8.4.2 to resolve PYSEC-2026-2132
+  ([#19](https://github.com/UiPath/coder_eval/pull/19),
+  [`a240d4e`](https://github.com/UiPath/coder_eval/commit/a240d4e2f8b970665743d0dc3d212c6c8e8a30ac))
+
+- **deps**: Upgrade click to 8.4.2 to resolve PYSEC-2026-2132
+  ([#20](https://github.com/UiPath/coder_eval/pull/20),
+  [`bba8645`](https://github.com/UiPath/coder_eval/commit/bba8645fc3f6c7b6e9784af496641ac0eaf8bede))
+
+- **evalboard**: Address search-box code review comments
+  ([#13](https://github.com/UiPath/coder_eval/pull/13),
+  [`382d80d`](https://github.com/UiPath/coder_eval/commit/382d80d4a84da39327b0196c97c01a367f094c0e))
+
+- **evalboard**: Fix search bar clear issue ([#13](https://github.com/UiPath/coder_eval/pull/13),
+  [`382d80d`](https://github.com/UiPath/coder_eval/commit/382d80d4a84da39327b0196c97c01a367f094c0e))
+
+- **evalboard**: Prevent search bar from resetting mid-type
+  ([#13](https://github.com/UiPath/coder_eval/pull/13),
+  [`382d80d`](https://github.com/UiPath/coder_eval/commit/382d80d4a84da39327b0196c97c01a367f094c0e))
+
+- **sandbox**: Exclude home-dir dotfiles from capture_to artifacts
+  ([#19](https://github.com/UiPath/coder_eval/pull/19),
+  [`a240d4e`](https://github.com/UiPath/coder_eval/commit/a240d4e2f8b970665743d0dc3d212c6c8e8a30ac))
+
+- **sandbox**: Extend capture_to denylist with credential stores
+  ([#19](https://github.com/UiPath/coder_eval/pull/19),
+  [`a240d4e`](https://github.com/UiPath/coder_eval/commit/a240d4e2f8b970665743d0dc3d212c6c8e8a30ac))
+
+### Code Style
+
+- Fix ruff formatting in _WORKSPACE_CAPTURE_IGNORE
+  ([#19](https://github.com/UiPath/coder_eval/pull/19),
+  [`a240d4e`](https://github.com/UiPath/coder_eval/commit/a240d4e2f8b970665743d0dc3d212c6c8e8a30ac))
+
+### Documentation
+
+- **readme**: Clarify framing before hero gif ([#16](https://github.com/UiPath/coder_eval/pull/16),
+  [`b7dee1c`](https://github.com/UiPath/coder_eval/commit/b7dee1c0616d9a98dc98432ca066c1a8c9264ef4))
+
+- **readme**: Reframe title toward agents & their skills
+  ([#16](https://github.com/UiPath/coder_eval/pull/16),
+  [`b7dee1c`](https://github.com/UiPath/coder_eval/commit/b7dee1c0616d9a98dc98432ca066c1a8c9264ef4))
+
+### Features
+
+- **early-stop**: Opt-in early stop once armed criteria are decided
+  ([#14](https://github.com/UiPath/coder_eval/pull/14),
+  [`b0c1ade`](https://github.com/UiPath/coder_eval/commit/b0c1ade364119573f172e4d64ee3fff0a387db32))
+
+### Testing
+
+- **sandbox**: Cover home-dir dotfile exclusion in capture_to
+  ([#19](https://github.com/UiPath/coder_eval/pull/19),
+  [`a240d4e`](https://github.com/UiPath/coder_eval/commit/a240d4e2f8b970665743d0dc3d212c6c8e8a30ac))
+
+
 ## v0.8.3 (2026-07-09)
 
 ### Bug Fixes

{coder_eval-0.8.3 → coder_eval-0.8.4}/CLAUDE.md

@@ -36,7 +36,7 @@ coder_eval/
 │   ├── experiment.py              # ExperimentDefinition, ExperimentVariant, ResolvedTask, result models
 │   ├── judge_defaults.py          # DEFAULT_JUDGE_MODEL constant (cycle-free leaf)
 │   ├── mutations.py               # PromptMutation variants (prefix/suffix/replace/template/rephrase)
-│   ├── results.py                 # CriterionResult (+ ClassificationCriterionResult), TurnRecord, EvaluationResult, CriterionAggregate, ThresholdCheck, SuiteRollup
+│   ├── results.py                 # CriterionResult (+ ClassificationCriterionResult), TurnRecord, EvaluationResult, EarlyStopInfo/EarlyStopReason, CriterionAggregate, ThresholdCheck, SuiteRollup
 │   ├── routing.py                 # ApiRoute (DirectRoute/BedrockRoute)
 │   ├── sandbox.py                 # SandboxConfig, ResourceLimits
 │   ├── tasks.py                   # TaskDefinition, AgentConfig, Dataset (dataset fan-out + sample)
@@ -79,6 +79,7 @@ coder_eval/
 ├── orchestration/                 # Batch execution utilities
 │   ├── batch.py                   # Parallel task execution (run_batch + run_batch_resolved)
 │   ├── config.py                  # Batch run configuration
+│   ├── early_stop.py              # validate_early_stop guardrails + EarlyStopWatcher (armed live-verdict observer)
 │   ├── evaluation.py              # Evaluation helpers
 │   ├── experiment.py              # ExperimentRunner, resolve_task_for_variant, load_experiment
 │   └── task_loader.py             # YAML task loading
@@ -138,6 +139,7 @@ templates/                         # Sandbox template directories
 - **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports.
 - **sandbox isolation**: Tasks that don't need MCP servers should set `setting_sources: []` in their `agent:` block to isolate the sandbox from the host project's CLAUDE.md and settings. Without this, the host project's CLAUDE.md (often 20 KB+) is injected into every API call, inflating cache-creation tokens and cost significantly.
 - **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block.
+- **Early stop on criterion (opt-in)**: `run_limits.stop_early` (default off) ends a single-shot Claude run early once the run's **armed** criteria are decided, so a raised `max_turns` isn't wasted on the smoke flavor. A criterion is armed by `stop_when: pass|fail|decided`; only criteria that can decide from a partial trajectory may arm (they declare a non-empty `live_stop_polarities` ClassVar and override `live_verdict` on `BaseCriterion` — currently `skill_triggered`, `command_executed`; CE025 enforces the two stay consistent). It uses a cooperative `should_stop` seam on the Claude agent's between-messages guard (tool-call granularity, no SIGKILL) driven by `orchestration/early_stop.py::EarlyStopWatcher` (own `EventCollector` + stop rule). Live verdicts only *trigger* the stop; the standard `check_all` on the frozen trajectory is authoritative. An early-stopped run gates on the **armed subset** (`EvaluationResult.armed_criteria_passed`); a completed run gates on the full set. Every unsupported use (non-observable criterion, non-Claude agent, wrong polarity, no armed criterion, simulation mode) is an error at resolution (plan *and* run), and a runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo` (reason + deciding criterion + when), report notes/badges, `stopped_early` run.json rows, and `EarlyStopped`/`EarlyStopReason` telemetry dims. Defaults off ⇒ behavior byte-for-behavior unchanged.
 
 ## Success Criteria (14 types)
 
@@ -158,7 +160,7 @@ templates/                         # Sandbox template directories
 | `llm_judge` | Continuous | LLM grades artifacts + optional trajectory + optional reference; routes through the run's backend (Bedrock / Anthropic) |
 | `agent_judge` | Continuous | Spawns a Claude Code SDK agent in an isolated sandbox copy; judge uses tools (Bash/Read/Grep/…) to investigate and returns a JSON verdict. Expensive; runs with evaluator credentials — see SECURITY note in the criterion docstring. |
 
-All criteria support `weight` (default 1.0) and `pass_threshold` (default 0.9). On dataset-backed tasks, criteria may also set `suite_thresholds: {metric: min_value}` — the suite gate passes iff every listed metric (from the criterion's `aggregate()` output) meets its minimum.
+All criteria support `weight` (default 1.0) and `pass_threshold` (default 0.9), plus `stop_when` (`pass`/`fail`/`decided`, default `null`) which arms the criterion for early stop when `run_limits.stop_early` is set (observable criteria only). On dataset-backed tasks, criteria may also set `suite_thresholds: {metric: min_value}` — the suite gate passes iff every listed metric (from the criterion's `aggregate()` output) meets its minimum.
 
 ## Evaluation Flow
 
@@ -191,11 +193,11 @@ make format      # ruff format
 make check       # ruff check (lint)
 make typecheck   # pyright
 make test        # pytest
-make lint        # custom architectural lint rules (CE001–CE013)
+make lint        # custom architectural lint rules (CE001–CE025)
 make verify      # All of the above + coverage check (CI equivalent)
 ```
 
-When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001–CE013 pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested.
+When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001–CE025 pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested.
 
 ## Configuration
 

{coder_eval-0.8.3 → coder_eval-0.8.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: coder-eval
-Version: 0.8.3
+Version: 0.8.4
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://github.com/UiPath/coder_eval
 Project-URL: Repository, https://github.com/UiPath/coder_eval
@@ -27,6 +27,7 @@ Requires-Dist: anthropic>=0.86.0
 Requires-Dist: anyio>=4.13.0
 Requires-Dist: azure-monitor-opentelemetry-exporter<1.1.0,>=1.0.0b30
 Requires-Dist: claude-agent-sdk>=0.2.82
+Requires-Dist: click>=8.3.3
 Requires-Dist: jmespath>=1.1.0
 Requires-Dist: jsonschema>=4.26.0
 Requires-Dist: opentelemetry-sdk<2.0.0,>=1.30.0
@@ -59,7 +60,7 @@ Provides-Extra: uipath
 Requires-Dist: uipath>=2.10.31; extra == 'uipath'
 Description-Content-Type: text/markdown
 
-# coder_eval — evaluate & benchmark AI coding agents
+# coder_eval — evaluate AI coding agents & their skills
 
 [![PyPI](https://img.shields.io/pypi/v/coder-eval.svg)](https://pypi.org/project/coder-eval/)
 [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
@@ -67,6 +68,11 @@ Description-Content-Type: text/markdown
 [![CI](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml/badge.svg)](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml)
 [![Code style: Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
 
+A framework for evaluating AI coding agents **and their skills** — built for CLI
+and skill builders — with sandboxing, reproducibility, and data-driven analysis.
+Not an "agentic coding" benchmark: it measures how effective your CLI and skills
+are when used by coding agents.
+
 <p align="center">
   <img src="docs/assets/hero.gif" alt="coder_eval running the hello_date task: a sandboxed agent writes and runs a script from a YAML task, then the scored result is browsed in evalboard" width="100%">
 </p>

{coder_eval-0.8.3 → coder_eval-0.8.4}/README.md

@@ -1,4 +1,4 @@
-# coder_eval — evaluate & benchmark AI coding agents
+# coder_eval — evaluate AI coding agents & their skills
 
 [![PyPI](https://img.shields.io/pypi/v/coder-eval.svg)](https://pypi.org/project/coder-eval/)
 [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
@@ -6,6 +6,11 @@
 [![CI](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml/badge.svg)](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml)
 [![Code style: Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
 
+A framework for evaluating AI coding agents **and their skills** — built for CLI
+and skill builders — with sandboxing, reproducibility, and data-driven analysis.
+Not an "agentic coding" benchmark: it measures how effective your CLI and skills
+are when used by coding agents.
+
 <p align="center">
   <img src="docs/assets/hero.gif" alt="coder_eval running the hello_date task: a sandboxed agent writes and runs a script from a YAML task, then the scored result is browsed in evalboard" width="100%">
 </p>

{coder_eval-0.8.3 → coder_eval-0.8.4}/docs/AB_EXPERIMENTS.md

@@ -17,6 +17,7 @@ to the orchestrator are involved.
 - [Recipe: A/B a Skill](#recipe-ab-a-skill)
 - [Recipe: A/B a Model](#recipe-ab-a-model)
 - [Recipe: A/B a Prompt](#recipe-ab-a-prompt)
+- [Recipe: Smoke vs. e2e Flavors (Early Stop)](#recipe-smoke-vs-e2e-flavors-early-stop)
 - [Replicates (Statistical Power)](#replicates-statistical-power)
 - [Measuring the Difference](#measuring-the-difference)
 - [CLI Reference](#cli-reference)
@@ -222,6 +223,39 @@ variants:
 The full mutation catalog (prefix / suffix / replace / template / rephrase) is
 defined in `coder_eval/models/mutations.py`.
 
+## Recipe: Smoke vs. e2e Flavors (Early Stop)
+
+Run the **same** task file as both a fast `smoke` flavor and a full `e2e` flavor
+by flipping one boolean per variant — `run_limits.stop_early`. Arm the criteria
+that define "the interesting thing happened" with `stop_when` in the task file;
+the `smoke` variant cuts off as soon as they're decided, while `e2e` runs to
+completion. Because the field merge is per-key, the variant sets only
+`stop_early` without disturbing the task's `max_turns`.
+
+```yaml
+experiment_id: early-stop-ab
+description: "Smoke vs. e2e from one file via opt-in early stop"
+
+variants:
+  - variant_id: e2e
+    run_limits:
+      stop_early: false # full run to completion (the reference flavor)
+  - variant_id: smoke
+    run_limits:
+      stop_early: true # cut off once the armed criteria are decided
+```
+
+The task file supplies the arming (`stop_when` on the criteria that gate the
+flavor) and a `max_turns` generous enough for `e2e`; see
+[`stop_early`](TASK_DEFINITION_GUIDE.md#stop_early-opt-in-early-stop). This recipe
+ships as `experiments/early-stop-ab.yaml`.
+
+Expect **identical pass/fail verdicts** between the two variants — an
+early-stopped run is gated on the armed subset only, and the non-armed criteria
+become advisory (clearly marked in the report), so the `smoke` flavor can't
+"pass for free" — with the `smoke` variant significantly lower on turns,
+duration, and tokens.
+
 ## Replicates (Statistical Power)
 
 Agents are stochastic — a single run per arm is noise, not signal. Set `repeats`

{coder_eval-0.8.3 → coder_eval-0.8.4}/docs/TASK_DEFINITION_GUIDE.md

@@ -192,6 +192,61 @@ budgets consistently across a suite — the headline % is only comparable when
 tasks are measured against realistic, like-for-like targets. Omit it (the
 default) to exclude a task from the metric entirely.
 
+### `stop_early` (opt-in early stop)
+
+`run_limits.stop_early` (default `false`) ends a single-shot run **early** once
+the run's **armed** criteria are decided — so you can raise `max_turns` for the
+full-run flavor without paying for turns the smoke flavor doesn't need. A
+criterion is *armed* by giving it a `stop_when` (see the criterion-fields table);
+`stop_early` is the master switch that turns arming on for the run.
+
+```yaml
+run_limits:
+  max_turns: 30
+  stop_early: true            # opt in; default false leaves behavior unchanged
+success_criteria:
+  - type: skill_triggered
+    skill_name: date-teller
+    expected_skill: date-teller
+    stop_when: decided        # arm on pass OR definitive fail
+  - type: file_exists         # not armed → advisory on an early-stopped run
+    path: report.md
+```
+
+Semantics:
+
+- **Opt-in, per run.** With `stop_early: false` (the default) the run behaves
+  exactly as before — `stop_when` is inert and every criterion gates normally.
+- **Polarity.** `stop_when: pass` stops the moment all armed criteria are decided
+  in the pass direction; `stop_when: fail` stops on a definitive wrong-signal
+  fail; `stop_when: decided` stops on either. Only criteria that can decide from a
+  partial trajectory (currently `skill_triggered`, `command_executed`) may be
+  armed — arming any other criterion is a hard error at resolution (plan *and*
+  run), never a silent no-op. Decidability can also depend on a criterion's own
+  fields: `command_executed` can live-**pass** only with `max_count` unset and
+  `min_count > 0`, and live-**fail** only with `max_count` set (which includes
+  the `min_count: 0, max_count: 0` "must-NOT-run" form). Arming a polarity the
+  configured criterion can never reach (e.g. `stop_when: pass` alongside a
+  `max_count`) is likewise a hard error at resolution, not a silent full run.
+- **Verdict.** An early-stopped run is gated on the **armed subset only**; the
+  non-armed criteria become **advisory** and are clearly marked (report badge +
+  per-criterion note + `stopped_early` row). A run that completes naturally is
+  gated on the **full** set, as always. This is what lets one file serve both a
+  `smoke` flavor (`stop_early: true`) and an `e2e` flavor (`stop_early: false`)
+  with identical verdicts — see [AB_EXPERIMENTS.md](AB_EXPERIMENTS.md).
+- **Fail-safe.** A live-verdict bug **fails open** to a full run (logged loudly) —
+  it can never silently disable a criterion or cause a false early stop.
+
+Observability (every early-stopped run is flagged everywhere so analysis never
+compares a truncated run against a full one):
+
+| Surface | Field / marker |
+|---------|----------------|
+| `run.json` row | `stopped_early`, `early_stop_reason`, `turns_remaining_at_stop` |
+| `run.md` | `> **NOTE:** […] stopped early (<reason>); <= N turn(s) avoided …` |
+| `task.html` | header badge `stopped early (<reason>)` + `advisory — not gated` markers |
+| Telemetry | `EarlyStopped` / `EarlyStopReason` dimensions on `CoderEval.Task.End` |
+
 ## Sandbox Configuration
 
 The `sandbox` block is optional. When omitted, it defaults to `driver: "tempdir"` with standard Python environment.
@@ -319,6 +374,7 @@ All criteria share these fields:
 | `description` | — | Human-readable description (required) |
 | `weight` | 1.0 | Relative importance for weighted score |
 | `pass_threshold` | 0.9 | Minimum score (0.0–1.0) to pass |
+| `stop_when` | `null` | Arms this criterion for early stop (`pass`/`fail`/`decided`); requires `run_limits.stop_early: true` and an observable criterion type (`skill_triggered`, `command_executed`). See [`stop_early`](#stop_early-opt-in-early-stop). |
 
 **Scoring types:**
 - **Binary** (1.0 or 0.0): `file_exists`, `run_command`, `file_matches_regex`

{coder_eval-0.8.3 → coder_eval-0.8.4}/docs/tutorials/04-writing-a-task.md

@@ -95,6 +95,7 @@ created are preserved under `runs/latest/<variant>/<task>/<NN>/` (`task.json`,
 ## Where to go deeper
 
 - **All 14 criterion types, weights, thresholds** → [Task Definition Guide](../TASK_DEFINITION_GUIDE.md)
+- **Stop a run early once the key criteria are decided** (opt-in `run_limits.stop_early` + `stop_when` on a criterion) → [Task Definition Guide → `stop_early`](../TASK_DEFINITION_GUIDE.md#stop_early-opt-in-early-stop)
 - **Fan one task out over a dataset of rows** → [Bring Your Own Data](../BYOD.md)
 - **Full CLI & config reference** → [User Guide](../USER_GUIDE.md)
 - **Compare two configurations on this task** → [Tutorial 05](05-comparing-models.md)

coder_eval-0.8.4/evalboard/app/_components/__tests__/search-box.test.tsx

@@ -0,0 +1,120 @@
+import { describe, expect, test, vi, beforeEach, afterEach } from "vitest";
+import { render, screen, act, fireEvent } from "@testing-library/react";
+
+// vi.hoisted ensures these are initialized before vi.mock hoists its factory.
+const { mockReplace, navState } = vi.hoisted(() => ({
+    mockReplace: vi.fn(),
+    navState: { q: "" as string },
+}));
+
+vi.mock("next/navigation", () => ({
+    useRouter: () => ({ replace: mockReplace }),
+    usePathname: () => "/",
+    useSearchParams: () => new URLSearchParams(navState.q ? `q=${navState.q}` : ""),
+}));
+
+const { SearchBox } = await import("../search-box");
+
+describe("SearchBox — typing-ahead race condition", () => {
+    beforeEach(() => {
+        vi.useFakeTimers();
+        mockReplace.mockClear();
+        navState.q = "";
+    });
+    afterEach(() => {
+        vi.useRealTimers();
+    });
+
+    test("preserves in-progress input when a navigation resolves mid-typing", () => {
+        // Reproduces the race: user types "foo" → debounce fires → user types
+        // more → navigation for "foo" resolves → input must NOT reset to "foo".
+        const { rerender } = render(<SearchBox />);
+        const input = screen.getByRole("textbox");
+
+        fireEvent.change(input, { target: { value: "foo" } });
+
+        // Debounce fires; typingAhead becomes false.
+        act(() => { vi.advanceTimersByTime(300); });
+        expect(mockReplace).toHaveBeenCalledOnce();
+
+        // User types more before the navigation resolves.
+        fireEvent.change(input, { target: { value: "foobar" } });
+
+        // Navigation for "foo" resolves — URL now reports "foo".
+        navState.q = "foo";
+        rerender(<SearchBox />);
+
+        // typingAhead is true, so the sync effect must NOT overwrite the input.
+        expect(input).toHaveValue("foobar");
+    });
+
+    test("syncs from URL when the user is not typing (external navigation)", () => {
+        // Back/forward nav or a tag click should still update the input when the
+        // user hasn't typed anything since the last URL write.
+        const { rerender } = render(<SearchBox />);
+        const input = screen.getByRole("textbox");
+
+        navState.q = "tag:alpha";
+        rerender(<SearchBox />);
+
+        expect(input).toHaveValue("tag:alpha");
+    });
+
+    test("clears the input when the URL is cleared externally", () => {
+        navState.q = "foo";
+        const { rerender } = render(<SearchBox />);
+        const input = screen.getByRole("textbox");
+
+        expect(input).toHaveValue("foo");
+
+        navState.q = "";
+        rerender(<SearchBox />);
+
+        expect(input).toHaveValue("");
+    });
+
+    test("clear button does not get repopulated when a stale navigation resolves", () => {
+        // The PR's named fix: user clicks × to clear a settled search, then a
+        // navigation that had already resolved with the old value re-renders the
+        // component — the input must stay empty, not snap back to the old value.
+        navState.q = "foo";
+        const { rerender } = render(<SearchBox />);
+        const input = screen.getByRole("textbox");
+        expect(input).toHaveValue("foo");
+
+        // User clicks ×.
+        fireEvent.click(screen.getByRole("button", { name: /clear search/i }));
+        expect(input).toHaveValue("");
+
+        // A stale navigation reports q="foo" — must NOT repopulate the input.
+        navState.q = "foo";
+        rerender(<SearchBox />);
+
+        expect(input).toHaveValue("");
+    });
+
+    test("typingAhead latch releases after the debounce fires so later external nav syncs", () => {
+        // Guards against a stuck-true latch: after the debounce settles (user
+        // stops typing, timer fires, typingAhead → false), a subsequent genuine
+        // external navigation must still update the input.
+        const { rerender } = render(<SearchBox />);
+        const input = screen.getByRole("textbox");
+
+        fireEvent.change(input, { target: { value: "foo" } });
+
+        // Let the debounce fire — typingAhead resets to false.
+        act(() => { vi.advanceTimersByTime(300); });
+
+        // Simulate the navigation resolving (URL catches up).
+        navState.q = "foo";
+        rerender(<SearchBox />);
+        expect(input).toHaveValue("foo");
+
+        // Now a genuine external navigation changes q (e.g. browser Back).
+        navState.q = "bar";
+        rerender(<SearchBox />);
+
+        // typingAhead is false, so the sync must apply.
+        expect(input).toHaveValue("bar");
+    });
+});

{coder_eval-0.8.3 → coder_eval-0.8.4}/evalboard/app/_components/search-box.tsx

@@ -1,7 +1,7 @@
 "use client";
 
 import { usePathname, useRouter, useSearchParams } from "next/navigation";
-import { useEffect, useState } from "react";
+import { useEffect, useRef, useState } from "react";
 
 const Q_DEBOUNCE_MS = 300;
 
@@ -18,18 +18,31 @@ export function SearchBox({
 
     const urlQ = searchParams.get("q") ?? "";
     const [q, setQ] = useState(urlQ);
+    // Invariant: true iff local input is ahead of the last URL write.
+    // Written in three places by the debounce effect (cleared on catch-up at
+    // the early-return, set when a new timer arms, cleared when the timer
+    // fires). Read by the sync effect to decide whether to accept a URL change.
+    const typingAhead = useRef(false);
 
     // Sync local state when the URL changes externally (back/forward, link
-    // clicks). The debounced write below early-returns when state and URL
-    // agree, so this can't loop.
+    // clicks). Skipped while the user is ahead of the URL to avoid clobbering
+    // in-progress input with a stale value from a just-resolved navigation.
+    // Note: external q changes that arrive during an active debounce window are
+    // intentionally deferred — the user's in-progress typing takes priority.
     useEffect(() => {
+        if (typingAhead.current) return;
         setQ((prev) => (prev.trim() === urlQ ? prev : urlQ));
     }, [urlQ]);
 
     useEffect(() => {
         const trimmed = q.trim();
-        if (trimmed === urlQ) return;
+        if (trimmed === urlQ) {
+            typingAhead.current = false;
+            return;
+        }
+        typingAhead.current = true;
         const timer = setTimeout(() => {
+            typingAhead.current = false;
             // Read the live URL at fire time so a concurrent write (e.g. a
             // tag click that landed during the debounce) isn't clobbered.
             const params = new URLSearchParams(window.location.search);
@@ -40,6 +53,9 @@ export function SearchBox({
                 scroll: false,
             });
         }, Q_DEBOUNCE_MS);
+        // Don't reset typingAhead in cleanup — cleanup fires on any dep change
+        // (q, urlQ, pathname, router). The effect body re-run re-establishes
+        // the correct value: false on catch-up, true when a new timer arms.
         return () => clearTimeout(timer);
     }, [q, urlQ, pathname, router]);
 

coder_eval-0.8.4/experiments/early-stop-ab.yaml

@@ -0,0 +1,23 @@
+experiment_id: early-stop-ab
+description: |
+  Smoke vs. e2e flavors from ONE task file via opt-in early-stop. Both variants
+  share the same tasks, criteria, and max_turns; they differ only in the
+  run_limits.stop_early boolean (field-merged, so neither replaces the task's
+  run_limits block). The task's armed criteria (those carrying `stop_when`)
+  decide when the `smoke` variant cuts off — as soon as the designated criteria
+  are decided — while `e2e` runs every task to completion.
+
+  Expect identical pass/fail verdicts between the two variants (the armed subset
+  gates an early-stopped run; other criteria are advisory and clearly marked),
+  with the `smoke` variant significantly lower on turns, duration, and tokens.
+
+variants:
+  - variant_id: e2e
+    description: "Full run to completion — no early stop (the reference flavor)."
+    run_limits:
+      stop_early: false
+
+  - variant_id: smoke
+    description: "Opt-in early stop — cut off once the armed criteria are decided."
+    run_limits:
+      stop_early: true

{coder_eval-0.8.3 → coder_eval-0.8.4}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "coder-eval"
-version = "0.8.3"
+version = "0.8.4"
 description = "Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity) with sandboxed, reproducible YAML task suites."
 readme = "README.md"
 license = "Apache-2.0"
@@ -30,6 +30,7 @@ dependencies = [
     "pydantic-settings>=2.14.2",
     "pyyaml>=6.0.3",
     "typer>=0.24.1",
+    "click>=8.3.3",
     "rich>=14.3.3",
     "python-dotenv>=1.2.2",
     "anthropic>=0.86.0",

{coder_eval-0.8.3 → coder_eval-0.8.4}/src/coder_eval/__init__.py

@@ -1,3 +1,3 @@
 """coder_eval - A framework for evaluating AI coding agents."""
 
-__version__ = "0.8.3"
+__version__ = "0.8.4"

{coder_eval-0.8.3 → coder_eval-0.8.4}/src/coder_eval/agent.py

@@ -5,7 +5,8 @@
 
 import logging
 from abc import ABC, abstractmethod
-from typing import Any, NoReturn, Protocol
+from collections.abc import Callable
+from typing import Any, ClassVar, NoReturn, Protocol
 
 from .errors import AgentCrashError, TurnTimeoutError
 from .errors.agent import format_timeout_reason, truncate_crash_message
@@ -71,6 +72,13 @@ class Agent[ConfigT: BaseAgentConfig](ABC):
     _iteration: int = 0
     _iteration_was_incremented: bool = False
 
+    # Capability flag: whether this agent honors the cooperative ``should_stop``
+    # interrupt threaded through ``communicate()`` (early-stop-on-criterion).
+    # Default False — arming early-stop on an agent that does not set this True
+    # is rejected at resolution time. Concrete agents that check ``should_stop``
+    # between messages override it to True.
+    supports_cooperative_stop: ClassVar[bool] = False
+
     def _begin_turn(self) -> None:
         """Mark the start of a ``communicate()`` turn: reset the pending slot and
         bump the iteration counter so a mid-turn failure can be rolled back.
@@ -183,6 +191,7 @@ class Agent[ConfigT: BaseAgentConfig](ABC):
         stream_callback: StreamCallback | None = None,
         timeout: float | None = None,
         max_turns: int | None = None,
+        should_stop: Callable[[], bool] | None = None,
     ) -> TurnRecord:
         """Send a message to the agent and receive its response.
 
@@ -198,6 +207,15 @@ class Agent[ConfigT: BaseAgentConfig](ABC):
                 ``communicate()`` call. When the agent would exceed it, the
                 returned ``TurnRecord`` has ``max_turns_exhausted=True``.
                 None defers to the underlying SDK default.
+            should_stop: Cooperative early-stop poll for early-stop-on-criterion.
+                When provided, an implementation that supports cooperative
+                stopping (``supports_cooperative_stop=True``) should call it at
+                each safe message boundary and, when it returns True, stop
+                pulling further work and finalize the turn cleanly
+                (``crashed=False``, no raise). ``None`` (default) preserves the
+                pre-existing behavior exactly. Agents that do not support it
+                accept the argument and ignore it (the orchestrator only passes
+                it to a capable agent).
 
         Returns:
             TurnRecord containing the complete interaction

{coder_eval-0.8.3 → coder_eval-0.8.4}/src/coder_eval/agents/antigravity_agent.py

@@ -23,7 +23,7 @@ import contextlib
 import logging
 import os
 import time
-from collections.abc import AsyncIterator
+from collections.abc import AsyncIterator, Callable
 from contextlib import AsyncExitStack
 from datetime import datetime
 from pathlib import Path
@@ -409,9 +409,15 @@ class AntigravityAgent(Agent[AntigravityAgentConfig]):
         stream_callback: StreamCallback | None = None,
         timeout: float | None = None,
         max_turns: int | None = None,
+        should_stop: Callable[[], bool] | None = None,
     ) -> TurnRecord:
         """Send a message to the Antigravity agent and receive its response.
 
+        ``should_stop`` is accepted for ``Agent.communicate`` override
+        compatibility and ignored — Antigravity does not support cooperative
+        early-stop (``supports_cooperative_stop`` is False), so the orchestrator
+        never passes it.
+
         Drives one logical turn: ``conversation.send(prompt)`` then iterate
         ``receive_steps()`` until the turn goes idle, mapping the Gemini step
         stream onto the standardized event protocol.