coder-eval 0.8.3__tar.gz → 0.8.5__tar.gz

{coder_eval-0.8.3 → coder_eval-0.8.5}/.claude/harness-candidates.md

@@ -5,21 +5,27 @@ Deferred lint/test guardrails surfaced during reviews. Promote to a `CExxx` rule
 
 ## From code review 260701-1954 (fix-review-top5 run) — deferred to a dedicated guardrail plan
 
-- **CE024** — workflow-YAML rule: forbid any `uses:` step pinned to a floating ref
+> Numbering note: these are *proposed* ids. `CE024` (discriminated-unions) and
+> `CE025` (live-verdict consistency) have since been **implemented** for other
+> rules, so the candidates below were renumbered to the next free ids. Always
+> claim the next unused number in `tests/lint/rules/` — the id-uniqueness assert
+> in `tests/lint/runner.py` is the source of truth.
+
+- **CE026** — workflow-YAML rule: forbid any `uses:` step pinned to a floating ref
   (`@v3`, `@main`) rather than a 40-hex commit SHA. Would have caught
   `mxschmitt/action-tmate@v3` (fixed manually in this run). >30 min: needs a
   non-Python file-walk branch in `tests/lint/runner.py`.
-- **CE025** — retired-token grep gate: fail when a removed-subsystem token
+- **CE027** — retired-token grep gate: fail when a removed-subsystem token
   (`LLMGW_`, `API_BACKEND=proxy`, `uipath_llmgw_client`) reappears outside an
   allowlist across docs/config/src. Would have caught the LLM-Gateway residue
   swept in this run. >30 min: needs an allowlist + repo-wide text scan.
-- **CE026** — assert the Makefile `lint:` help does not hardcode a stale `CE0NN`
+- **CE028** — assert the Makefile `lint:` help does not hardcode a stale `CE0NN`
   upper bound (use `CE001+`). Would have caught the `CE001–CE005` drift fixed here.
 - **docs-vs-harness smoke test** — execute the CI tutorial's `coder-eval run`
   command against a NoOp task and assert the produced tree matches the documented
   globs. Would have caught the `--run-dir runs` layout bug. Not statically
   reachable (needs a live run).
-- [ ] CE-rule: `type: Literal[...]` fields on models in `coder_eval/models/` must declare their tag default (`type: Literal["x"] = "x"`) — a member without the default degrades `validate_registry` diagnostics (PydanticUndefined in expected_types) and breaks direct construction. Nothing guards it today; needs a rule-design call (second violation class inside CE024 vs. a new CE025), and the failure is already double-caught by the MINIMAL_PAYLOADS parity test + direct-construction tests — caught in the 2026-07-03 top5-review-fixes run (Phase 1 quality review).
+- [ ] CE-rule: `type: Literal[...]` fields on models in `coder_eval/models/` must declare their tag default (`type: Literal["x"] = "x"`) — a member without the default degrades `validate_registry` diagnostics (PydanticUndefined in expected_types) and breaks direct construction. Nothing guards it today; needs a rule-design call (second violation class inside CE024 vs. a new CExxx at the next free id), and the failure is already double-caught by the MINIMAL_PAYLOADS parity test + direct-construction tests — caught in the 2026-07-03 top5-review-fixes run (Phase 1 quality review).
 
 ## From 2026-07-03 open-source docs cleanup
 

{coder_eval-0.8.3 → coder_eval-0.8.5}/.github/dependabot.yml

@@ -26,7 +26,7 @@ updates:
     directory: "/"
     schedule:
       interval: "monthly"
-    open-pull-requests-limit: 3  # Raised from 1: actions are SHA-pinned, so SHA bumps come as patch/minor PRs.
+    open-pull-requests-limit: 3  # Grouped below into one PR; headroom in case a major is held out of the group.
     labels:
       - "dependencies"
       - "github-actions"
@@ -36,3 +36,29 @@ updates:
       prefix: "chore"
     # No ignore block — keep SHA pins current for patch and minor releases too,
     # otherwise SHA pinning fossilizes the actions in place.
+    # One batched PR instead of one per action, matching the npm block below.
+    groups:
+      actions-all:
+        patterns: ["*"]
+
+  # evalboard frontend (npm). Without this block nothing routinely refreshes the
+  # Next.js app, so it drifts until a CVE fires and alerts pile up. Monthly,
+  # grouped bumps keep next/React/Azure ahead of most advisories.
+  - package-ecosystem: "npm"
+    directory: "/evalboard"
+    schedule:
+      interval: "monthly"
+    open-pull-requests-limit: 3
+    labels:
+      - "dependencies"
+      - "javascript"
+    # `chore` keeps the commit + PR title within the Conventional Commits
+    # allowed-type set enforced by conventional-commits.yml (see pip block).
+    commit-message:
+      prefix: "chore"
+      include: "scope"
+    # One batched PR instead of N noisy ones, matching the low-noise style of
+    # the pip/actions blocks above.
+    groups:
+      npm-all:
+        patterns: ["*"]

{coder_eval-0.8.3 → coder_eval-0.8.5}/CHANGELOG.md

@@ -2,6 +2,121 @@
 
 <!-- version list -->
 
+## v0.8.5 (2026-07-20)
+
+### Bug Fixes
+
+- **deps**: Bump mcp to >=1.28.1 for CVE-2026-52869/52870/59950
+  ([#25](https://github.com/UiPath/coder_eval/pull/25),
+  [`fb1ad4c`](https://github.com/UiPath/coder_eval/commit/fb1ad4cba2973414a6a883468f62c908deee532b))
+
+- **orchestration**: Isolate per-task config-resolution failures from the suite
+  ([#25](https://github.com/UiPath/coder_eval/pull/25),
+  [`fb1ad4c`](https://github.com/UiPath/coder_eval/commit/fb1ad4cba2973414a6a883468f62c908deee532b))
+
+- **orchestration**: Normalize the all-fail config-resolution abort to ValueError
+  ([#25](https://github.com/UiPath/coder_eval/pull/25),
+  [`fb1ad4c`](https://github.com/UiPath/coder_eval/commit/fb1ad4cba2973414a6a883468f62c908deee532b))
+
+- **orchestration**: Re-raise ValueError verbatim in the all-fail abort
+  ([#25](https://github.com/UiPath/coder_eval/pull/25),
+  [`fb1ad4c`](https://github.com/UiPath/coder_eval/commit/fb1ad4cba2973414a6a883468f62c908deee532b))
+
+- **orchestrator**: Interrupt-proof teardown so a timeout can't drop task.json
+  ([#29](https://github.com/UiPath/coder_eval/pull/29),
+  [`89ec0d0`](https://github.com/UiPath/coder_eval/commit/89ec0d0eda74616d21a87302680e3099473119bc))
+
+- **sandbox**: Prune capture-ignored entries on every preservation path; interrupt-proof teardown
+  ([#29](https://github.com/UiPath/coder_eval/pull/29),
+  [`89ec0d0`](https://github.com/UiPath/coder_eval/commit/89ec0d0eda74616d21a87302680e3099473119bc))
+
+### Chores
+
+- **evalboard**: Address npm Dependabot alerts ([#18](https://github.com/UiPath/coder_eval/pull/18),
+  [`3d5e7b7`](https://github.com/UiPath/coder_eval/commit/3d5e7b7d2358e10d95d59487c54596df7da1bef2))
+
+- **evalboard**: Batch github-actions bumps into one grouped PR
+  ([#18](https://github.com/UiPath/coder_eval/pull/18),
+  [`3d5e7b7`](https://github.com/UiPath/coder_eval/commit/3d5e7b7d2358e10d95d59487c54596df7da1bef2))
+
+### Documentation
+
+- **orchestrator**: Correct teardown comment to scope of the fix
+  ([#29](https://github.com/UiPath/coder_eval/pull/29),
+  [`89ec0d0`](https://github.com/UiPath/coder_eval/commit/89ec0d0eda74616d21a87302680e3099473119bc))
+
+### Features
+
+- **evalboard**: Show conversation transcript for simulation tasks
+  ([#23](https://github.com/UiPath/coder_eval/pull/23),
+  [`252722a`](https://github.com/UiPath/coder_eval/commit/252722ab17edf3dbc1b68a7ca3a357b2dd3cb0d3))
+
+### Refactoring
+
+- **orchestrator**: Trim to interrupt-proof teardown; drop preservation-prune
+  ([#29](https://github.com/UiPath/coder_eval/pull/29),
+  [`89ec0d0`](https://github.com/UiPath/coder_eval/commit/89ec0d0eda74616d21a87302680e3099473119bc))
+
+
+## v0.8.4 (2026-07-13)
+
+### Bug Fixes
+
+- **deps**: Upgrade click to 8.4.2 to resolve PYSEC-2026-2132
+  ([#19](https://github.com/UiPath/coder_eval/pull/19),
+  [`a240d4e`](https://github.com/UiPath/coder_eval/commit/a240d4e2f8b970665743d0dc3d212c6c8e8a30ac))
+
+- **deps**: Upgrade click to 8.4.2 to resolve PYSEC-2026-2132
+  ([#20](https://github.com/UiPath/coder_eval/pull/20),
+  [`bba8645`](https://github.com/UiPath/coder_eval/commit/bba8645fc3f6c7b6e9784af496641ac0eaf8bede))
+
+- **evalboard**: Address search-box code review comments
+  ([#13](https://github.com/UiPath/coder_eval/pull/13),
+  [`382d80d`](https://github.com/UiPath/coder_eval/commit/382d80d4a84da39327b0196c97c01a367f094c0e))
+
+- **evalboard**: Fix search bar clear issue ([#13](https://github.com/UiPath/coder_eval/pull/13),
+  [`382d80d`](https://github.com/UiPath/coder_eval/commit/382d80d4a84da39327b0196c97c01a367f094c0e))
+
+- **evalboard**: Prevent search bar from resetting mid-type
+  ([#13](https://github.com/UiPath/coder_eval/pull/13),
+  [`382d80d`](https://github.com/UiPath/coder_eval/commit/382d80d4a84da39327b0196c97c01a367f094c0e))
+
+- **sandbox**: Exclude home-dir dotfiles from capture_to artifacts
+  ([#19](https://github.com/UiPath/coder_eval/pull/19),
+  [`a240d4e`](https://github.com/UiPath/coder_eval/commit/a240d4e2f8b970665743d0dc3d212c6c8e8a30ac))
+
+- **sandbox**: Extend capture_to denylist with credential stores
+  ([#19](https://github.com/UiPath/coder_eval/pull/19),
+  [`a240d4e`](https://github.com/UiPath/coder_eval/commit/a240d4e2f8b970665743d0dc3d212c6c8e8a30ac))
+
+### Code Style
+
+- Fix ruff formatting in _WORKSPACE_CAPTURE_IGNORE
+  ([#19](https://github.com/UiPath/coder_eval/pull/19),
+  [`a240d4e`](https://github.com/UiPath/coder_eval/commit/a240d4e2f8b970665743d0dc3d212c6c8e8a30ac))
+
+### Documentation
+
+- **readme**: Clarify framing before hero gif ([#16](https://github.com/UiPath/coder_eval/pull/16),
+  [`b7dee1c`](https://github.com/UiPath/coder_eval/commit/b7dee1c0616d9a98dc98432ca066c1a8c9264ef4))
+
+- **readme**: Reframe title toward agents & their skills
+  ([#16](https://github.com/UiPath/coder_eval/pull/16),
+  [`b7dee1c`](https://github.com/UiPath/coder_eval/commit/b7dee1c0616d9a98dc98432ca066c1a8c9264ef4))
+
+### Features
+
+- **early-stop**: Opt-in early stop once armed criteria are decided
+  ([#14](https://github.com/UiPath/coder_eval/pull/14),
+  [`b0c1ade`](https://github.com/UiPath/coder_eval/commit/b0c1ade364119573f172e4d64ee3fff0a387db32))
+
+### Testing
+
+- **sandbox**: Cover home-dir dotfile exclusion in capture_to
+  ([#19](https://github.com/UiPath/coder_eval/pull/19),
+  [`a240d4e`](https://github.com/UiPath/coder_eval/commit/a240d4e2f8b970665743d0dc3d212c6c8e8a30ac))
+
+
 ## v0.8.3 (2026-07-09)
 
 ### Bug Fixes

{coder_eval-0.8.3 → coder_eval-0.8.5}/CLAUDE.md

@@ -36,7 +36,7 @@ coder_eval/
 │   ├── experiment.py              # ExperimentDefinition, ExperimentVariant, ResolvedTask, result models
 │   ├── judge_defaults.py          # DEFAULT_JUDGE_MODEL constant (cycle-free leaf)
 │   ├── mutations.py               # PromptMutation variants (prefix/suffix/replace/template/rephrase)
-│   ├── results.py                 # CriterionResult (+ ClassificationCriterionResult), TurnRecord, EvaluationResult, CriterionAggregate, ThresholdCheck, SuiteRollup
+│   ├── results.py                 # CriterionResult (+ ClassificationCriterionResult), TurnRecord, EvaluationResult, EarlyStopInfo/EarlyStopReason, CriterionAggregate, ThresholdCheck, SuiteRollup
 │   ├── routing.py                 # ApiRoute (DirectRoute/BedrockRoute)
 │   ├── sandbox.py                 # SandboxConfig, ResourceLimits
 │   ├── tasks.py                   # TaskDefinition, AgentConfig, Dataset (dataset fan-out + sample)
@@ -79,6 +79,7 @@ coder_eval/
 ├── orchestration/                 # Batch execution utilities
 │   ├── batch.py                   # Parallel task execution (run_batch + run_batch_resolved)
 │   ├── config.py                  # Batch run configuration
+│   ├── early_stop.py              # validate_early_stop guardrails + EarlyStopWatcher (armed live-verdict observer)
 │   ├── evaluation.py              # Evaluation helpers
 │   ├── experiment.py              # ExperimentRunner, resolve_task_for_variant, load_experiment
 │   └── task_loader.py             # YAML task loading
@@ -138,6 +139,7 @@ templates/                         # Sandbox template directories
 - **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports.
 - **sandbox isolation**: Tasks that don't need MCP servers should set `setting_sources: []` in their `agent:` block to isolate the sandbox from the host project's CLAUDE.md and settings. Without this, the host project's CLAUDE.md (often 20 KB+) is injected into every API call, inflating cache-creation tokens and cost significantly.
 - **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block.
+- **Early stop on criterion (opt-in)**: `run_limits.stop_early` (default off) ends a single-shot Claude run early once the run's **armed** criteria are decided, so a raised `max_turns` isn't wasted on the smoke flavor. A criterion is armed by `stop_when: pass|fail|decided`; only criteria that can decide from a partial trajectory may arm (they declare a non-empty `live_stop_polarities` ClassVar and override `live_verdict` on `BaseCriterion` — currently `skill_triggered`, `command_executed`; CE025 enforces the two stay consistent). It uses a cooperative `should_stop` seam on the Claude agent's between-messages guard (tool-call granularity, no SIGKILL) driven by `orchestration/early_stop.py::EarlyStopWatcher` (own `EventCollector` + stop rule). Live verdicts only *trigger* the stop; the standard `check_all` on the frozen trajectory is authoritative. An early-stopped run gates on the **armed subset** (`EvaluationResult.armed_criteria_passed`); a completed run gates on the full set. Every unsupported use (non-observable criterion, non-Claude agent, wrong polarity, no armed criterion, simulation mode) is an error at resolution (plan *and* run), and a runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo` (reason + deciding criterion + when), report notes/badges, `stopped_early` run.json rows, and `EarlyStopped`/`EarlyStopReason` telemetry dims. Defaults off ⇒ behavior byte-for-behavior unchanged.
 
 ## Success Criteria (14 types)
 
@@ -158,7 +160,7 @@ templates/                         # Sandbox template directories
 | `llm_judge` | Continuous | LLM grades artifacts + optional trajectory + optional reference; routes through the run's backend (Bedrock / Anthropic) |
 | `agent_judge` | Continuous | Spawns a Claude Code SDK agent in an isolated sandbox copy; judge uses tools (Bash/Read/Grep/…) to investigate and returns a JSON verdict. Expensive; runs with evaluator credentials — see SECURITY note in the criterion docstring. |
 
-All criteria support `weight` (default 1.0) and `pass_threshold` (default 0.9). On dataset-backed tasks, criteria may also set `suite_thresholds: {metric: min_value}` — the suite gate passes iff every listed metric (from the criterion's `aggregate()` output) meets its minimum.
+All criteria support `weight` (default 1.0) and `pass_threshold` (default 0.9), plus `stop_when` (`pass`/`fail`/`decided`, default `null`) which arms the criterion for early stop when `run_limits.stop_early` is set (observable criteria only). On dataset-backed tasks, criteria may also set `suite_thresholds: {metric: min_value}` — the suite gate passes iff every listed metric (from the criterion's `aggregate()` output) meets its minimum.
 
 ## Evaluation Flow
 
@@ -191,11 +193,11 @@ make format      # ruff format
 make check       # ruff check (lint)
 make typecheck   # pyright
 make test        # pytest
-make lint        # custom architectural lint rules (CE001–CE013)
+make lint        # custom architectural lint rules (CE001–CE025)
 make verify      # All of the above + coverage check (CI equivalent)
 ```
 
-When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001–CE013 pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested.
+When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001–CE025 pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested.
 
 ## Configuration
 

{coder_eval-0.8.3 → coder_eval-0.8.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: coder-eval
-Version: 0.8.3
+Version: 0.8.5
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://github.com/UiPath/coder_eval
 Project-URL: Repository, https://github.com/UiPath/coder_eval
@@ -27,6 +27,7 @@ Requires-Dist: anthropic>=0.86.0
 Requires-Dist: anyio>=4.13.0
 Requires-Dist: azure-monitor-opentelemetry-exporter<1.1.0,>=1.0.0b30
 Requires-Dist: claude-agent-sdk>=0.2.82
+Requires-Dist: click>=8.3.3
 Requires-Dist: jmespath>=1.1.0
 Requires-Dist: jsonschema>=4.26.0
 Requires-Dist: opentelemetry-sdk<2.0.0,>=1.30.0
@@ -45,7 +46,7 @@ Provides-Extra: codex
 Requires-Dist: openai-codex>=0.1.0b3; extra == 'codex'
 Provides-Extra: dev
 Requires-Dist: bandit[toml]>=1.9.4; extra == 'dev'
-Requires-Dist: mcp>=1.26.0; extra == 'dev'
+Requires-Dist: mcp>=1.28.1; extra == 'dev'
 Requires-Dist: pip-audit>=2.10.0; extra == 'dev'
 Requires-Dist: pre-commit>=4.5.1; extra == 'dev'
 Requires-Dist: pyright>=1.1.408; extra == 'dev'
@@ -59,7 +60,7 @@ Provides-Extra: uipath
 Requires-Dist: uipath>=2.10.31; extra == 'uipath'
 Description-Content-Type: text/markdown
 
-# coder_eval — evaluate & benchmark AI coding agents
+# coder_eval — evaluate AI coding agents & their skills
 
 [![PyPI](https://img.shields.io/pypi/v/coder-eval.svg)](https://pypi.org/project/coder-eval/)
 [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
@@ -67,6 +68,11 @@ Description-Content-Type: text/markdown
 [![CI](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml/badge.svg)](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml)
 [![Code style: Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
 
+A framework for evaluating AI coding agents **and their skills** — built for CLI
+and skill builders — with sandboxing, reproducibility, and data-driven analysis.
+Not an "agentic coding" benchmark: it measures how effective your CLI and skills
+are when used by coding agents.
+
 <p align="center">
   <img src="docs/assets/hero.gif" alt="coder_eval running the hello_date task: a sandboxed agent writes and runs a script from a YAML task, then the scored result is browsed in evalboard" width="100%">
 </p>

{coder_eval-0.8.3 → coder_eval-0.8.5}/README.md

@@ -1,4 +1,4 @@
-# coder_eval — evaluate & benchmark AI coding agents
+# coder_eval — evaluate AI coding agents & their skills
 
 [![PyPI](https://img.shields.io/pypi/v/coder-eval.svg)](https://pypi.org/project/coder-eval/)
 [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
@@ -6,6 +6,11 @@
 [![CI](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml/badge.svg)](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml)
 [![Code style: Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
 
+A framework for evaluating AI coding agents **and their skills** — built for CLI
+and skill builders — with sandboxing, reproducibility, and data-driven analysis.
+Not an "agentic coding" benchmark: it measures how effective your CLI and skills
+are when used by coding agents.
+
 <p align="center">
   <img src="docs/assets/hero.gif" alt="coder_eval running the hello_date task: a sandboxed agent writes and runs a script from a YAML task, then the scored result is browsed in evalboard" width="100%">
 </p>

{coder_eval-0.8.3 → coder_eval-0.8.5}/docs/AB_EXPERIMENTS.md

@@ -17,6 +17,7 @@ to the orchestrator are involved.
 - [Recipe: A/B a Skill](#recipe-ab-a-skill)
 - [Recipe: A/B a Model](#recipe-ab-a-model)
 - [Recipe: A/B a Prompt](#recipe-ab-a-prompt)
+- [Recipe: Smoke vs. e2e Flavors (Early Stop)](#recipe-smoke-vs-e2e-flavors-early-stop)
 - [Replicates (Statistical Power)](#replicates-statistical-power)
 - [Measuring the Difference](#measuring-the-difference)
 - [CLI Reference](#cli-reference)
@@ -222,6 +223,39 @@ variants:
 The full mutation catalog (prefix / suffix / replace / template / rephrase) is
 defined in `coder_eval/models/mutations.py`.
 
+## Recipe: Smoke vs. e2e Flavors (Early Stop)
+
+Run the **same** task file as both a fast `smoke` flavor and a full `e2e` flavor
+by flipping one boolean per variant — `run_limits.stop_early`. Arm the criteria
+that define "the interesting thing happened" with `stop_when` in the task file;
+the `smoke` variant cuts off as soon as they're decided, while `e2e` runs to
+completion. Because the field merge is per-key, the variant sets only
+`stop_early` without disturbing the task's `max_turns`.
+
+```yaml
+experiment_id: early-stop-ab
+description: "Smoke vs. e2e from one file via opt-in early stop"
+
+variants:
+  - variant_id: e2e
+    run_limits:
+      stop_early: false # full run to completion (the reference flavor)
+  - variant_id: smoke
+    run_limits:
+      stop_early: true # cut off once the armed criteria are decided
+```
+
+The task file supplies the arming (`stop_when` on the criteria that gate the
+flavor) and a `max_turns` generous enough for `e2e`; see
+[`stop_early`](TASK_DEFINITION_GUIDE.md#stop_early-opt-in-early-stop). This recipe
+ships as `experiments/early-stop-ab.yaml`.
+
+Expect **identical pass/fail verdicts** between the two variants — an
+early-stopped run is gated on the armed subset only, and the non-armed criteria
+become advisory (clearly marked in the report), so the `smoke` flavor can't
+"pass for free" — with the `smoke` variant significantly lower on turns,
+duration, and tokens.
+
 ## Replicates (Statistical Power)
 
 Agents are stochastic — a single run per arm is noise, not signal. Set `repeats`

{coder_eval-0.8.3 → coder_eval-0.8.5}/docs/TASK_DEFINITION_GUIDE.md

@@ -192,6 +192,61 @@ budgets consistently across a suite — the headline % is only comparable when
 tasks are measured against realistic, like-for-like targets. Omit it (the
 default) to exclude a task from the metric entirely.
 
+### `stop_early` (opt-in early stop)
+
+`run_limits.stop_early` (default `false`) ends a single-shot run **early** once
+the run's **armed** criteria are decided — so you can raise `max_turns` for the
+full-run flavor without paying for turns the smoke flavor doesn't need. A
+criterion is *armed* by giving it a `stop_when` (see the criterion-fields table);
+`stop_early` is the master switch that turns arming on for the run.
+
+```yaml
+run_limits:
+  max_turns: 30
+  stop_early: true            # opt in; default false leaves behavior unchanged
+success_criteria:
+  - type: skill_triggered
+    skill_name: date-teller
+    expected_skill: date-teller
+    stop_when: decided        # arm on pass OR definitive fail
+  - type: file_exists         # not armed → advisory on an early-stopped run
+    path: report.md
+```
+
+Semantics:
+
+- **Opt-in, per run.** With `stop_early: false` (the default) the run behaves
+  exactly as before — `stop_when` is inert and every criterion gates normally.
+- **Polarity.** `stop_when: pass` stops the moment all armed criteria are decided
+  in the pass direction; `stop_when: fail` stops on a definitive wrong-signal
+  fail; `stop_when: decided` stops on either. Only criteria that can decide from a
+  partial trajectory (currently `skill_triggered`, `command_executed`) may be
+  armed — arming any other criterion is a hard error at resolution (plan *and*
+  run), never a silent no-op. Decidability can also depend on a criterion's own
+  fields: `command_executed` can live-**pass** only with `max_count` unset and
+  `min_count > 0`, and live-**fail** only with `max_count` set (which includes
+  the `min_count: 0, max_count: 0` "must-NOT-run" form). Arming a polarity the
+  configured criterion can never reach (e.g. `stop_when: pass` alongside a
+  `max_count`) is likewise a hard error at resolution, not a silent full run.
+- **Verdict.** An early-stopped run is gated on the **armed subset only**; the
+  non-armed criteria become **advisory** and are clearly marked (report badge +
+  per-criterion note + `stopped_early` row). A run that completes naturally is
+  gated on the **full** set, as always. This is what lets one file serve both a
+  `smoke` flavor (`stop_early: true`) and an `e2e` flavor (`stop_early: false`)
+  with identical verdicts — see [AB_EXPERIMENTS.md](AB_EXPERIMENTS.md).
+- **Fail-safe.** A live-verdict bug **fails open** to a full run (logged loudly) —
+  it can never silently disable a criterion or cause a false early stop.
+
+Observability (every early-stopped run is flagged everywhere so analysis never
+compares a truncated run against a full one):
+
+| Surface | Field / marker |
+|---------|----------------|
+| `run.json` row | `stopped_early`, `early_stop_reason`, `turns_remaining_at_stop` |
+| `run.md` | `> **NOTE:** […] stopped early (<reason>); <= N turn(s) avoided …` |
+| `task.html` | header badge `stopped early (<reason>)` + `advisory — not gated` markers |
+| Telemetry | `EarlyStopped` / `EarlyStopReason` dimensions on `CoderEval.Task.End` |
+
 ## Sandbox Configuration
 
 The `sandbox` block is optional. When omitted, it defaults to `driver: "tempdir"` with standard Python environment.
@@ -319,6 +374,7 @@ All criteria share these fields:
 | `description` | — | Human-readable description (required) |
 | `weight` | 1.0 | Relative importance for weighted score |
 | `pass_threshold` | 0.9 | Minimum score (0.0–1.0) to pass |
+| `stop_when` | `null` | Arms this criterion for early stop (`pass`/`fail`/`decided`); requires `run_limits.stop_early: true` and an observable criterion type (`skill_triggered`, `command_executed`). See [`stop_early`](#stop_early-opt-in-early-stop). |
 
 **Scoring types:**
 - **Binary** (1.0 or 0.0): `file_exists`, `run_command`, `file_matches_regex`

{coder_eval-0.8.3 → coder_eval-0.8.5}/docs/tutorials/04-writing-a-task.md

@@ -95,6 +95,7 @@ created are preserved under `runs/latest/<variant>/<task>/<NN>/` (`task.json`,
 ## Where to go deeper
 
 - **All 14 criterion types, weights, thresholds** → [Task Definition Guide](../TASK_DEFINITION_GUIDE.md)
+- **Stop a run early once the key criteria are decided** (opt-in `run_limits.stop_early` + `stop_when` on a criterion) → [Task Definition Guide → `stop_early`](../TASK_DEFINITION_GUIDE.md#stop_early-opt-in-early-stop)
 - **Fan one task out over a dataset of rows** → [Bring Your Own Data](../BYOD.md)
 - **Full CLI & config reference** → [User Guide](../USER_GUIDE.md)
 - **Compare two configurations on this task** → [Tutorial 05](05-comparing-models.md)

coder_eval-0.8.5/evalboard/app/_components/__tests__/search-box.test.tsx

@@ -0,0 +1,120 @@
+import { describe, expect, test, vi, beforeEach, afterEach } from "vitest";
+import { render, screen, act, fireEvent } from "@testing-library/react";
+
+// vi.hoisted ensures these are initialized before vi.mock hoists its factory.
+const { mockReplace, navState } = vi.hoisted(() => ({
+    mockReplace: vi.fn(),
+    navState: { q: "" as string },
+}));
+
+vi.mock("next/navigation", () => ({
+    useRouter: () => ({ replace: mockReplace }),
+    usePathname: () => "/",
+    useSearchParams: () => new URLSearchParams(navState.q ? `q=${navState.q}` : ""),
+}));
+
+const { SearchBox } = await import("../search-box");
+
+describe("SearchBox — typing-ahead race condition", () => {
+    beforeEach(() => {
+        vi.useFakeTimers();
+        mockReplace.mockClear();
+        navState.q = "";
+    });
+    afterEach(() => {
+        vi.useRealTimers();
+    });
+
+    test("preserves in-progress input when a navigation resolves mid-typing", () => {
+        // Reproduces the race: user types "foo" → debounce fires → user types
+        // more → navigation for "foo" resolves → input must NOT reset to "foo".
+        const { rerender } = render(<SearchBox />);
+        const input = screen.getByRole("textbox");
+
+        fireEvent.change(input, { target: { value: "foo" } });
+
+        // Debounce fires; typingAhead becomes false.
+        act(() => { vi.advanceTimersByTime(300); });
+        expect(mockReplace).toHaveBeenCalledOnce();
+
+        // User types more before the navigation resolves.
+        fireEvent.change(input, { target: { value: "foobar" } });
+
+        // Navigation for "foo" resolves — URL now reports "foo".
+        navState.q = "foo";
+        rerender(<SearchBox />);
+
+        // typingAhead is true, so the sync effect must NOT overwrite the input.
+        expect(input).toHaveValue("foobar");
+    });
+
+    test("syncs from URL when the user is not typing (external navigation)", () => {
+        // Back/forward nav or a tag click should still update the input when the
+        // user hasn't typed anything since the last URL write.
+        const { rerender } = render(<SearchBox />);
+        const input = screen.getByRole("textbox");
+
+        navState.q = "tag:alpha";
+        rerender(<SearchBox />);
+
+        expect(input).toHaveValue("tag:alpha");
+    });
+
+    test("clears the input when the URL is cleared externally", () => {
+        navState.q = "foo";
+        const { rerender } = render(<SearchBox />);
+        const input = screen.getByRole("textbox");
+
+        expect(input).toHaveValue("foo");
+
+        navState.q = "";
+        rerender(<SearchBox />);
+
+        expect(input).toHaveValue("");
+    });
+
+    test("clear button does not get repopulated when a stale navigation resolves", () => {
+        // The PR's named fix: user clicks × to clear a settled search, then a
+        // navigation that had already resolved with the old value re-renders the
+        // component — the input must stay empty, not snap back to the old value.
+        navState.q = "foo";
+        const { rerender } = render(<SearchBox />);
+        const input = screen.getByRole("textbox");
+        expect(input).toHaveValue("foo");
+
+        // User clicks ×.
+        fireEvent.click(screen.getByRole("button", { name: /clear search/i }));
+        expect(input).toHaveValue("");
+
+        // A stale navigation reports q="foo" — must NOT repopulate the input.
+        navState.q = "foo";
+        rerender(<SearchBox />);
+
+        expect(input).toHaveValue("");
+    });
+
+    test("typingAhead latch releases after the debounce fires so later external nav syncs", () => {
+        // Guards against a stuck-true latch: after the debounce settles (user
+        // stops typing, timer fires, typingAhead → false), a subsequent genuine
+        // external navigation must still update the input.
+        const { rerender } = render(<SearchBox />);
+        const input = screen.getByRole("textbox");
+
+        fireEvent.change(input, { target: { value: "foo" } });
+
+        // Let the debounce fire — typingAhead resets to false.
+        act(() => { vi.advanceTimersByTime(300); });
+
+        // Simulate the navigation resolving (URL catches up).
+        navState.q = "foo";
+        rerender(<SearchBox />);
+        expect(input).toHaveValue("foo");
+
+        // Now a genuine external navigation changes q (e.g. browser Back).
+        navState.q = "bar";
+        rerender(<SearchBox />);
+
+        // typingAhead is false, so the sync must apply.
+        expect(input).toHaveValue("bar");
+    });
+});

{coder_eval-0.8.3 → coder_eval-0.8.5}/evalboard/app/_components/search-box.tsx

@@ -1,7 +1,7 @@
 "use client";
 
 import { usePathname, useRouter, useSearchParams } from "next/navigation";
-import { useEffect, useState } from "react";
+import { useEffect, useRef, useState } from "react";
 
 const Q_DEBOUNCE_MS = 300;
 
@@ -18,18 +18,31 @@ export function SearchBox({
 
     const urlQ = searchParams.get("q") ?? "";
     const [q, setQ] = useState(urlQ);
+    // Invariant: true iff local input is ahead of the last URL write.
+    // Written in three places by the debounce effect (cleared on catch-up at
+    // the early-return, set when a new timer arms, cleared when the timer
+    // fires). Read by the sync effect to decide whether to accept a URL change.
+    const typingAhead = useRef(false);
 
     // Sync local state when the URL changes externally (back/forward, link
-    // clicks). The debounced write below early-returns when state and URL
-    // agree, so this can't loop.
+    // clicks). Skipped while the user is ahead of the URL to avoid clobbering
+    // in-progress input with a stale value from a just-resolved navigation.
+    // Note: external q changes that arrive during an active debounce window are
+    // intentionally deferred — the user's in-progress typing takes priority.
     useEffect(() => {
+        if (typingAhead.current) return;
         setQ((prev) => (prev.trim() === urlQ ? prev : urlQ));
     }, [urlQ]);
 
     useEffect(() => {
         const trimmed = q.trim();
-        if (trimmed === urlQ) return;
+        if (trimmed === urlQ) {
+            typingAhead.current = false;
+            return;
+        }
+        typingAhead.current = true;
         const timer = setTimeout(() => {
+            typingAhead.current = false;
             // Read the live URL at fire time so a concurrent write (e.g. a
             // tag click that landed during the debounce) isn't clobbered.
             const params = new URLSearchParams(window.location.search);
@@ -40,6 +53,9 @@ export function SearchBox({
                 scroll: false,
             });
         }, Q_DEBOUNCE_MS);
+        // Don't reset typingAhead in cleanup — cleanup fires on any dep change
+        // (q, urlQ, pathname, router). The effect body re-run re-establishes
+        // the correct value: false on catch-up, true when a new timer arms.
         return () => clearTimeout(timer);
     }, [q, urlQ, pathname, router]);
 

{coder_eval-0.8.3 → coder_eval-0.8.5}/evalboard/package.json

@@ -12,15 +12,17 @@
         "verify": "tsc --noEmit && vitest run && next build"
     },
     "dependencies": {
-        "@azure/identity": "^4.13.1",
-        "@azure/storage-blob": "^12.31.0",
-        "next": "^15.1.0",
+        "next": "^15.5.20",
         "react": "^19.0.0",
         "react-dom": "^19.0.0",
         "react-markdown": "^10.1.0",
         "recharts": "^2.15.0",
         "remark-gfm": "^4.0.1"
     },
+    "optionalDependencies": {
+        "@azure/identity": "^4.13.1",
+        "@azure/storage-blob": ">=12.31.0 <12.33.0"
+    },
     "devDependencies": {
         "@tailwindcss/typography": "^0.5.19",
         "@testing-library/jest-dom": "^6.0.0",
@@ -36,5 +38,13 @@
         "tailwindcss": "^3.4.17",
         "typescript": "^5.6.3",
         "vitest": "^2.1.0"
+    },
+    "pnpm": {
+        "overrides": {
+            "postcss": "^8.5.10",
+            "fast-xml-builder": "^1.1.7",
+            "@babel/core": "^7.29.6",
+            "uuid": "^11.1.1"
+        }
     }
 }