coder-eval 0.8.8__tar.gz → 0.8.10__tar.gz

{coder_eval-0.8.8 → coder_eval-0.8.10}/.claude/commands/coder-eval-code-review.md

@@ -12,7 +12,7 @@ description: Run a multi-model code review on uncommitted changes or a described
 
 Run a thorough code review using multiple AI models in parallel, then fix all high-confidence medium-severity and above findings.
 
-**Cost note**: This command runs `make verify` + a multi-model review (`gemini-3` + `codex` via MCP, plus an Opus sub-agent) + automatic fixes. Expensive in time and tokens. Use this for changes you're about to ship; for broad codebase audits use `/coder-eval-code-review-full`; for quick local checks, run targeted tools (`ruff`, `pyright`, `pytest`) directly.
+**Cost note**: This command runs `make verify` + a multi-model review (`gemini-3` + `gpt-5` via MCP, plus an Opus sub-agent) + automatic fixes. Expensive in time and tokens. Use this for changes you're about to ship; for broad codebase audits use `/coder-eval-code-review-full`; for quick local checks, run targeted tools (`ruff`, `pyright`, `pytest`) directly.
 
 Input: $ARGUMENTS
 
@@ -65,7 +65,7 @@ If it fails, report what's failing and stop — code review on broken code is pr
 
 Launch two reviews **in parallel**:
 
-**Review A — Multi-model (via `mcp__multi__codereview`)**: run it per the shared procedure in `.claude/shared/multi-model-review.md` — `models: ["gemini-3", "codex"]`, `relevant_files` = absolute paths of all changed files, `content` = a review request citing the Severity Standard, Review Principles, and Checklist. **Heed the multi-step protocol described there** (the step-1 response is usually an `in_progress` checklist, not findings — you must make the `step_number: 2` follow-up call with the same `thread_id`). Falls back to two Opus sub-agents if the tool is unavailable.
+**Review A — Multi-model (via `mcp__multi__codereview`)**: run it per the shared procedure in `.claude/shared/multi-model-review.md` — `models: ["gemini-3", "gpt-5"]`, `relevant_files` = absolute paths of all changed files, `content` = a review request citing the Severity Standard, Review Principles, and Checklist. **Heed the multi-step protocol described there** (the step-1 response is usually an `in_progress` checklist, not findings — you must make the `step_number: 2` follow-up call with the same `thread_id`). Falls back to two Opus sub-agents if the tool is unavailable.
 
 **Review B — Opus sub-agent**: Use the `Agent` tool with `model: "opus"`. Give it the list of changed files and the following specialized tasks to run in parallel internally:
 
@@ -120,7 +120,7 @@ across sessions and provides forensic context if a fix later proves wrong:
 - Git SHA: <`git rev-parse HEAD`>
 - Branch: <`git rev-parse --abbrev-ref HEAD`>
 - Scope: <uncommitted | described "<input>" | last commit>
-- Reviewers: <e.g. gemini-3, codex, opus-fallback-1, opus-fallback-2 — list actual reviewers used>
+- Reviewers: <e.g. gemini-3, gpt-5, opus-fallback-1, opus-fallback-2 — list actual reviewers used>
 
 ### Scope
 - Files reviewed: (list)

{coder_eval-0.8.8 → coder_eval-0.8.10}/.claude/harness-candidates.md

@@ -27,6 +27,22 @@ Deferred lint/test guardrails surfaced during reviews. Promote to a `CExxx` rule
   reachable (needs a live run).
 - [ ] CE-rule: `type: Literal[...]` fields on models in `coder_eval/models/` must declare their tag default (`type: Literal["x"] = "x"`) — a member without the default degrades `validate_registry` diagnostics (PydanticUndefined in expected_types) and breaks direct construction. Nothing guards it today; needs a rule-design call (second violation class inside CE024 vs. a new CExxx at the next free id), and the failure is already double-caught by the MINIMAL_PAYLOADS parity test + direct-construction tests — caught in the 2026-07-03 top5-review-fixes run (Phase 1 quality review).
 
+## From 2026-07-23 stop_when:auto early-stop review
+
+- [ ] CE-rule: the early-stop watcher stop rule must decide polarity via the
+  resolved `_armed_polarities`, never a raw `criterion.stop_when` comparison —
+  forbid `.stop_when` attribute reads inside `EarlyStopWatcher._evaluate` /
+  `_resolve_armed_polarities`'s callers in `orchestration/early_stop.py`. This
+  diff *was* the fix for exactly that class of bug (the old rule compared
+  `stop_when in ("pass","decided")` and so vetoed every mixed `auto` pass-stop).
+  Deferred, not cheap: existing CE rules scope by file/module, not by a specific
+  method, so a method-scoped attribute-ban needs a new AST-walk shape (and risks
+  false positives on the legitimate `is not None` membership reads elsewhere in
+  the file). Claim the next free id in `tests/lint/rules/`. Caught in the
+  2026-07-23 stop_when:auto review; the behavior itself is guarded by
+  `test_auto_mixed_pass_stops_ignoring_undecided_distractors` +
+  `test_mixed_static_arming_pass_stops_ignoring_fail_armed`.
+
 ## From 2026-07-03 open-source docs cleanup
 
 - [ ] **Dead-relative-link checker for `docs/**/*.md`** — resolve every relative

{coder_eval-0.8.8 → coder_eval-0.8.10}/.claude/shared/multi-model-review.md

@@ -14,7 +14,7 @@ If `mcp__multi__codereview` is available (check the deferred-tools list; if list
 but not loaded, load it with `ToolSearch` → `select:mcp__multi__codereview`), call
 it with:
 
-- `models`: the strongest available (e.g. `["gemini-3", "codex"]`)
+- `models`: the strongest available (e.g. `["gemini-3", "gpt-5"]` — as of July 2026 these resolve to `gemini-3.1-pro-preview` and `gpt-5.6-sol`; check `mcp__multi__models` if unsure)
 - `relevant_files`: absolute paths of every changed / in-scope file
 - `content`: the review request, citing the consumer's rubric (the shared **Review Criteria**, the **Severity Standard**, etc.)
 - `base_path`: project root

coder_eval-0.8.10/.github/ISSUE_TEMPLATE/adopter.yml

@@ -0,0 +1,84 @@
+name: Adopter — we're using coder_eval
+description: Tell us your team uses coder_eval so we can add you to ADOPTERS.md
+title: "[adopter] <organization>"
+labels: ["adopter"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for using coder_eval! Adopter entries help other teams see the project is
+        real and in production, and they tell us which use cases to keep supporting.
+
+        Everything below is optional except the organization and how you use it. We only
+        publish what you explicitly allow here.
+  - type: input
+    id: organization
+    attributes:
+      label: Organization
+      description: The name you'd like listed.
+      placeholder: Acme Corp
+    validations:
+      required: true
+  - type: input
+    id: website
+    attributes:
+      label: Website
+      placeholder: https://example.com
+    validations:
+      required: false
+  - type: textarea
+    id: usage
+    attributes:
+      label: How do you use coder_eval?
+      description: |
+        A sentence or two. What do you evaluate — Claude Code skills, agent A/B tests,
+        CI quality gates, your own repos — and roughly at what scale?
+      placeholder: |
+        We gate CI on a suite of ~40 YAML tasks that check our internal Claude Code
+        skills still trigger, and A/B Claude Code against Codex each release.
+    validations:
+      required: true
+  - type: dropdown
+    id: listing
+    attributes:
+      label: How may we list you?
+      description: |
+        Pick the most you're comfortable with. **Logo use needs someone who can approve
+        trademark use for your organization** — if that isn't you, choose a name-only or
+        private option and we'll follow up.
+      options:
+        - Name only, in ADOPTERS.md
+        - Name in ADOPTERS.md and on the project website
+        - Name and logo in ADOPTERS.md and on the project website
+        - Please don't list us publicly — this is just for the maintainers
+      default: 0
+    validations:
+      required: true
+  - type: checkboxes
+    id: logo-permission
+    attributes:
+      label: Trademark permission
+      description: Only required if you selected a logo option above.
+      options:
+        - label: >-
+            I am authorized to grant this permission on behalf of my organization, and I
+            grant the coder_eval maintainers a non-exclusive, revocable license to display
+            our name and logo as an adopter in ADOPTERS.md and on the project website. We
+            can withdraw it at any time by commenting on this issue.
+          required: false
+  - type: input
+    id: contact
+    attributes:
+      label: Contact
+      description: GitHub handle or email, so we can reach you before publishing anything.
+    validations:
+      required: false
+  - type: checkboxes
+    id: case-study
+    attributes:
+      label: Anything else?
+      options:
+        - label: We'd be open to a short case study or a talk about our setup.
+          required: false
+        - label: We'd like to hear about breaking changes before they ship.
+          required: false

{coder_eval-0.8.8 → coder_eval-0.8.10}/.github/workflows/docs.yml

@@ -10,12 +10,17 @@ name: Docs
 # Source: "Deploy from a branch" → branch `gh-pages` / `/ (root)`. That's the same
 # (legacy, branch-based) configuration uipath-python already runs on.
 on:
-  push:
-    branches: [main]
-    paths:
-      - "docs/**"
-      - "mkdocs.yml"
-      - ".github/workflows/docs.yml"
+  # Auto-publish on push is DISABLED: GitHub Pages is not yet enabled for this
+  # repo on uipath.github.io (org owner must switch it on once — Settings → Pages
+  # → Deploy from a branch → gh-pages / root). The published docs currently serve
+  # from coder-eval.com/docs (synced separately by the website), so the gh-pages
+  # deploy is not needed yet. Re-enable by uncommenting the `push:` trigger below.
+  # push:
+  #   branches: [main]
+  #   paths:
+  #     - "docs/**"
+  #     - "mkdocs.yml"
+  #     - ".github/workflows/docs.yml"
   workflow_dispatch:
 
 permissions:

{coder_eval-0.8.8 → coder_eval-0.8.10}/.github/workflows/pr-checks.yml

@@ -831,3 +831,61 @@ jobs:
           name: byoa-live-output
           path: tmp/
           retention-days: 7
+
+  action-dogfood:
+    name: Action Dogfood (composite action, real API)
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    # Skip on fork PRs where secrets aren't available (matches e2e-smoke).
+    if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
+
+      # The composite action is agent-agnostic and does NOT install a coding-agent
+      # runtime. The dogfood task uses the default claude-code agent, so provide
+      # Node + the Claude CLI here (as e2e-smoke does), before invoking the action.
+      - name: Set up Node.js 20
+        uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020  # v4.4.0
+        with:
+          node-version: "20"
+      - name: Install Claude CLI
+        run: npm install -g @anthropic-ai/claude-code
+
+      - name: Run coder-eval via local action
+        id: dogfood
+        uses: ./
+        with:
+          version: local
+          tasks: tasks/hello_date.yaml
+          model: claude-haiku-4-5-20251001
+          run-dir: runs/ci-action-dogfood
+          junit-path: runs/ci-action-dogfood/junit.xml
+          # Credentials go through the generic env passthrough (the only channel);
+          # ANTHROPIC_API_KEY reaching the run is proven by the API-backed task
+          # succeeding. A floor of 0.0 passes for any produced score (exercises
+          # the gate path green in CI without flakiness); the second line
+          # exercises multi-line env parsing.
+          minimum-task-score: "0.0"
+          env: |
+            ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}
+            CE_DOGFOOD_MARKER=1
+
+      - name: Verify outputs and JUnit file
+        env:
+          JUNIT: ${{ steps.dogfood.outputs.junit-path }}
+          RUNDIR: ${{ steps.dogfood.outputs.run-dir }}
+        run: |
+          set -euo pipefail
+          test -n "$JUNIT" && test -f "$JUNIT" || { echo "junit output missing"; exit 1; }
+          # Well-formedness check on a file this job just generated (trusted input;
+          # our writer emits no DTDs/entities) — stdlib ET is fine here.
+          python3 -c "import sys, xml.etree.ElementTree as ET; ET.parse(sys.argv[1])" "$JUNIT"
+          test -f "$RUNDIR/run.json" || { echo "run.json missing"; exit 1; }
+
+      - name: Upload dogfood run on failure
+        if: failure()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a  # v7.0.1
+        with:
+          name: action-dogfood-runs
+          path: runs/ci-action-dogfood/
+          retention-days: 7

{coder_eval-0.8.8 → coder_eval-0.8.10}/.github/workflows/release.yml

@@ -182,23 +182,50 @@ jobs:
           echo "version=$V" >> "$GITHUB_OUTPUT"
           echo "Publishing version: $V"
 
-      - name: Regenerate uv.lock and amend release commit
+      - name: Regenerate uv.lock, bump action.yml pin, and amend release commit
         if: steps.mode.outputs.prerelease != 'true' && steps.release.outputs.version != ''
+        env:
+          # Passed via env (not interpolated into the script) per GitHub's
+          # injection guidance.
+          VERSION: ${{ steps.release.outputs.version }}
         run: |
+          set -euo pipefail
+          git config user.email "github-actions[bot]@users.noreply.github.com"
+          git config user.name "github-actions[bot]"
+          # Bump the composite action's default `version:` pin to the just-released
+          # version so `UiPath/coder_eval@vX.Y.Z` installs `coder-eval==X.Y.Z`. The
+          # anchor is indentation-tolerant and keyed on the unique trailing
+          # "# <-- kept in sync" comment; the grep guard fails the release loudly
+          # if a reformat ever detaches it (rather than shipping a stale pin).
+          sed -i -E 's/^([[:space:]]*default: ")[0-9]+\.[0-9]+\.[0-9]+("   # <-- kept in sync)/\1'"${VERSION}"'\2/' action.yml
+          grep -q "default: \"${VERSION}\"" action.yml || { echo "action.yml version bump failed"; exit 1; }
+          git add action.yml
+          # Regenerate the lock too; stage it (a no-op if unchanged).
           uv lock
-          if ! git diff --quiet uv.lock; then
-            git config user.email "github-actions[bot]@users.noreply.github.com"
-            git config user.name "github-actions[bot]"
-            git add uv.lock
+          git add uv.lock
+          # Amend only if action.yml/uv.lock actually changed the tree.
+          if ! git diff --cached --quiet; then
             git commit --amend --no-edit
             # Amend replaced the commit the tag points at; re-point it before pushing.
-            git tag -f "v${{ steps.release.outputs.version }}"
+            git tag -f "v${VERSION}"
           fi
 
       - name: Push release commit and tags
         if: steps.mode.outputs.prerelease != 'true' && steps.release.outputs.version != ''
         run: git push origin main "v${{ steps.release.outputs.version }}"
 
+      - name: Move major action tag (vN -> this release)
+        if: steps.release.outputs.version != ''
+        env:
+          VERSION: ${{ steps.release.outputs.version }}
+        run: |
+          set -euo pipefail
+          # Consumers pin `UiPath/coder_eval@v0` (becomes `@v1` at 1.0.0). Force-move
+          # the moving major tag to this release. Force on a missing tag creates it.
+          MAJOR="v${VERSION%%.*}"
+          git tag -f "$MAJOR" "v${VERSION}"
+          git push -f origin "$MAJOR"
+
       - name: Build wheel + sdist
         if: steps.ver.outputs.version != ''
         run: uv build

coder_eval-0.8.10/ADOPTERS.md

@@ -0,0 +1,13 @@
+# Adopters
+
+Teams and organizations using `coder_eval` in their development or CI workflows.
+
+Using it? [Open an adopter issue](https://github.com/UiPath/coder_eval/issues/new?template=adopter.yml)
+and we'll add you here. Name-only listings are welcome — a logo is never required.
+
+Every entry below is published with the organization's permission, and any of them can be
+removed on request by opening an issue.
+
+| Organization | How they use it |
+|---|---|
+| [UiPath](https://www.uipath.com) | Nightly evaluation of internal Claude Code skills and agent A/B experiments; maintains `coder_eval`. |

{coder_eval-0.8.8 → coder_eval-0.8.10}/CHANGELOG.md

@@ -2,6 +2,218 @@
 
 <!-- version list -->
 
+## v0.8.10 (2026-07-24)
+
+### Bug Fixes
+
+- Remove dead SimulationConfig.parallel_trials; add CE031 to guard the class
+  ([`947acd3`](https://github.com/UiPath/coder_eval/commit/947acd34a4f0451c69cb7bf0eb863127b9995f76))
+
+- **early-stop**: Add stop_when 'auto' — per-instance arming + pass-armed-subset stop rule
+  ([#51](https://github.com/UiPath/coder_eval/pull/51),
+  [`08e21e6`](https://github.com/UiPath/coder_eval/commit/08e21e67b67fdab84d26b8a0e5551e04289d92a8))
+
+- **early-stop**: Defer fail-stop while a pass-armed criterion is undecided
+  ([#51](https://github.com/UiPath/coder_eval/pull/51),
+  [`08e21e6`](https://github.com/UiPath/coder_eval/commit/08e21e67b67fdab84d26b8a0e5551e04289d92a8))
+
+- **early-stop**: Return assert_never explicitly to satisfy CodeQL
+  ([#51](https://github.com/UiPath/coder_eval/pull/51),
+  [`08e21e6`](https://github.com/UiPath/coder_eval/commit/08e21e67b67fdab84d26b8a0e5551e04289d92a8))
+
+- **reports**: Code review fixes for the JUnit CI gate
+  ([#37](https://github.com/UiPath/coder_eval/pull/37),
+  [`74db6fa`](https://github.com/UiPath/coder_eval/commit/74db6facfea6f898af4db6709e563d90af0d7b30))
+
+- **reports**: JUnit CI-gate review fixes + CE027 env-var lint
+  ([#37](https://github.com/UiPath/coder_eval/pull/37),
+  [`74db6fa`](https://github.com/UiPath/coder_eval/commit/74db6facfea6f898af4db6709e563d90af0d7b30))
+
+- **reports**: Make skipped-task JUnit names platform-independent
+  ([#37](https://github.com/UiPath/coder_eval/pull/37),
+  [`74db6fa`](https://github.com/UiPath/coder_eval/commit/74db6facfea6f898af4db6709e563d90af0d7b30))
+
+### Chores
+
+- Re-trigger CI (GitHub dropped the force-push event)
+  ([#37](https://github.com/UiPath/coder_eval/pull/37),
+  [`74db6fa`](https://github.com/UiPath/coder_eval/commit/74db6facfea6f898af4db6709e563d90af0d7b30))
+
+- **deps**: Lock defusedxml (dev-only, test-side XML parsing)
+  ([#37](https://github.com/UiPath/coder_eval/pull/37),
+  [`74db6fa`](https://github.com/UiPath/coder_eval/commit/74db6facfea6f898af4db6709e563d90af0d7b30))
+
+### Continuous Integration
+
+- Disable Docs gh-pages auto-publish on push (Pages not enabled yet)
+  ([`c289d46`](https://github.com/UiPath/coder_eval/commit/c289d46c5cec269f205224e4dfce27affab5557e))
+
+### Documentation
+
+- 1/8 — add DATASETS.md and a task-schema dataset: section
+  ([`e3d37ac`](https://github.com/UiPath/coder_eval/commit/e3d37ac793e6f12c020b71a4ed19df2279b4b2c1))
+
+- 2/8 — retire BYOD.md into DOCKER_ISOLATION.md
+  ([`1a4a2a5`](https://github.com/UiPath/coder_eval/commit/1a4a2a5893639d14832dbd1614163aa7c18bf70a))
+
+- 3/8 — one complete run_limits reference; document skip
+  ([`bcb7e24`](https://github.com/UiPath/coder_eval/commit/bcb7e24132c229114efe9a38281a37e736d83312))
+
+- 4/8 — add DIALOG_MODE.md and correct four stale simulation claims
+  ([`18dcbe9`](https://github.com/UiPath/coder_eval/commit/18dcbe95d156856149538406ededc8c75ec594f8))
+
+- 5/8 — fix prompt_mutations example; add CE029
+  ([`b524009`](https://github.com/UiPath/coder_eval/commit/b5240099c58f5b50c7b5fe531a12f1f8b98aec29))
+
+- 7/8 — generate flat indexes from the mkdocs nav; add CE028
+  ([`1514bcb`](https://github.com/UiPath/coder_eval/commit/1514bcb8f075ed48f2857a84269513c60878ea81))
+
+- Add CI Gate reference (GitHub Action + JUnit) and wire into indexes
+  ([`b8c6301`](https://github.com/UiPath/coder_eval/commit/b8c63018bd93ba91751511d95f4af96a444d9873))
+
+- Agent guides, extending & report-schema references, and fixes
+  ([`ce74824`](https://github.com/UiPath/coder_eval/commit/ce74824bde2b5322dc745bace045e1806f77324b))
+
+- Fold nav long tail into one Advanced group; align index ordering
+  ([`e2ff053`](https://github.com/UiPath/coder_eval/commit/e2ff0539af8c9e3e6deb85c0a0124c0232224033))
+
+- Point docs links to coder-eval.com/docs; drop Ruff badge
+  ([`60430e5`](https://github.com/UiPath/coder_eval/commit/60430e5e857f0af6fbf08cde00f0621de4610562))
+
+- Point pyproject Documentation URL to coder-eval.com/docs
+  ([`be39df2`](https://github.com/UiPath/coder_eval/commit/be39df23c8ba1a0f6f945dc99e3827eb5d6f2772))
+
+- Reword CODER_EVAL_RAW_SDK_LOG to prose form (satisfy CE027)
+  ([`d7d5b59`](https://github.com/UiPath/coder_eval/commit/d7d5b59b990127810fcfe33272f3ff733a9498cd))
+
+- Use the brand name "Coder Eval" in prose and titles
+  ([`821f11b`](https://github.com/UiPath/coder_eval/commit/821f11bf94762f6578913090568f5b092e237596))
+
+### Features
+
+- Packaged CI gate — JUnit XML output + composite GitHub Action
+  ([#37](https://github.com/UiPath/coder_eval/pull/37),
+  [`74db6fa`](https://github.com/UiPath/coder_eval/commit/74db6facfea6f898af4db6709e563d90af0d7b30))
+
+- **action**: Generic env passthrough + minimum-task-score gate
+  ([#37](https://github.com/UiPath/coder_eval/pull/37),
+  [`74db6fa`](https://github.com/UiPath/coder_eval/commit/74db6facfea6f898af4db6709e563d90af0d7b30))
+
+- **ci**: 3/3 — publish composite action, release automation, PR dogfood
+  ([#37](https://github.com/UiPath/coder_eval/pull/37),
+  [`74db6fa`](https://github.com/UiPath/coder_eval/commit/74db6facfea6f898af4db6709e563d90af0d7b30))
+
+- **cli**: 2/3 — wire run --junit-xml and report -f junit
+  ([#37](https://github.com/UiPath/coder_eval/pull/37),
+  [`74db6fa`](https://github.com/UiPath/coder_eval/commit/74db6facfea6f898af4db6709e563d90af0d7b30))
+
+- **reports**: 1/3 — add reports_junit.py disk-driven JUnit XML writer
+  ([#37](https://github.com/UiPath/coder_eval/pull/37),
+  [`74db6fa`](https://github.com/UiPath/coder_eval/commit/74db6facfea6f898af4db6709e563d90af0d7b30))
+
+### Testing
+
+- 6/8 — CE030 documents-or-exempts model fields
+  ([`c9a3b16`](https://github.com/UiPath/coder_eval/commit/c9a3b160deeec385386a7009812025bc26021363))
+
+
+## v0.8.9 (2026-07-23)
+
+### Bug Fixes
+
+- Code review fixes for welch-t-test-exact ([#38](https://github.com/UiPath/coder_eval/pull/38),
+  [`6df6e9b`](https://github.com/UiPath/coder_eval/commit/6df6e9bb1f3bd009592185ec3473e8f2e67d3816))
+
+- Render weight:0 criteria as informational on every display surface
+  ([#34](https://github.com/UiPath/coder_eval/pull/34),
+  [`9a34e90`](https://github.com/UiPath/coder_eval/commit/9a34e90bba2854316149cad61613a75dd3bd91e4))
+
+- Weight:0 un-gates criteria (informational criteria)
+  ([#34](https://github.com/UiPath/coder_eval/pull/34),
+  [`9a34e90`](https://github.com/UiPath/coder_eval/commit/9a34e90bba2854316149cad61613a75dd3bd91e4))
+
+- Weight:0 un-gates criteria and renders as informational
+  ([#34](https://github.com/UiPath/coder_eval/pull/34),
+  [`9a34e90`](https://github.com/UiPath/coder_eval/commit/9a34e90bba2854316149cad61613a75dd3bd91e4))
+
+- **early-stop**: Decide skill activation on the tool call, not its result
+  ([#43](https://github.com/UiPath/coder_eval/pull/43),
+  [`d34aa97`](https://github.com/UiPath/coder_eval/commit/d34aa97416ae3af09d09cdf78cb6607b4e2e6f7c))
+
+- **early-stop**: Latch skill activation on any engagement, not first
+  ([#43](https://github.com/UiPath/coder_eval/pull/43),
+  [`d34aa97`](https://github.com/UiPath/coder_eval/commit/d34aa97416ae3af09d09cdf78cb6607b4e2e6f7c))
+
+- **evalboard**: Match watchlist skeleton header to avoid layout shift
+  ([#45](https://github.com/UiPath/coder_eval/pull/45),
+  [`acc1c86`](https://github.com/UiPath/coder_eval/commit/acc1c86bc97b80383ce1ef402356853447088e4a))
+
+- **reports**: 1/2 — exact Student-t p-values in welch_t_test
+  ([#38](https://github.com/UiPath/coder_eval/pull/38),
+  [`6df6e9b`](https://github.com/UiPath/coder_eval/commit/6df6e9bb1f3bd009592185ec3473e8f2e67d3816))
+
+- **reports**: Exact Student-t p-values and a paired comparison section
+  ([#38](https://github.com/UiPath/coder_eval/pull/38),
+  [`6df6e9b`](https://github.com/UiPath/coder_eval/commit/6df6e9bb1f3bd009592185ec3473e8f2e67d3816))
+
+- **reports**: Fail loud on t* overflow; surface excluded paired tasks
+  ([#38](https://github.com/UiPath/coder_eval/pull/38),
+  [`6df6e9b`](https://github.com/UiPath/coder_eval/commit/6df6e9bb1f3bd009592185ec3473e8f2e67d3816))
+
+- **reports**: One source of truth for variant series and paired stats
+  ([#38](https://github.com/UiPath/coder_eval/pull/38),
+  [`6df6e9b`](https://github.com/UiPath/coder_eval/commit/6df6e9bb1f3bd009592185ec3473e8f2e67d3816))
+
+- **reports**: Validate confidence and n_resamples in bootstrap_mean_ci
+  ([#38](https://github.com/UiPath/coder_eval/pull/38),
+  [`6df6e9b`](https://github.com/UiPath/coder_eval/commit/6df6e9bb1f3bd009592185ec3473e8f2e67d3816))
+
+### Chores
+
+- **harness**: Defer two guards from the welch-t-test-exact run
+  ([#38](https://github.com/UiPath/coder_eval/pull/38),
+  [`6df6e9b`](https://github.com/UiPath/coder_eval/commit/6df6e9bb1f3bd009592185ec3473e8f2e67d3816))
+
+### Documentation
+
+- Add adopter issue template and ADOPTERS.md ([#40](https://github.com/UiPath/coder_eval/pull/40),
+  [`bfbed4e`](https://github.com/UiPath/coder_eval/commit/bfbed4e4a2ca857167fac2cb15462b7159e97684))
+
+- Switch multi-model review from codex to gpt-5 alias
+  ([#36](https://github.com/UiPath/coder_eval/pull/36),
+  [`0a3f2a7`](https://github.com/UiPath/coder_eval/commit/0a3f2a71a0a1e1e56f42a668fff57bac06eb99ec))
+
+### Features
+
+- **evalboard**: Make all pages harness-aware and stream tables
+  ([#45](https://github.com/UiPath/coder_eval/pull/45),
+  [`acc1c86`](https://github.com/UiPath/coder_eval/commit/acc1c86bc97b80383ce1ef402356853447088e4a))
+
+- **evalboard**: Make analytics surfaces harness-aware and stream tables
+  ([#45](https://github.com/UiPath/coder_eval/pull/45),
+  [`acc1c86`](https://github.com/UiPath/coder_eval/commit/acc1c86bc97b80383ce1ef402356853447088e4a))
+
+- **evalboard**: Scope task trends to one harness
+  ([#45](https://github.com/UiPath/coder_eval/pull/45),
+  [`acc1c86`](https://github.com/UiPath/coder_eval/commit/acc1c86bc97b80383ce1ef402356853447088e4a))
+
+- **reports**: 2/2 — add a Paired Comparison section to experiment reports
+  ([#38](https://github.com/UiPath/coder_eval/pull/38),
+  [`6df6e9b`](https://github.com/UiPath/coder_eval/commit/6df6e9bb1f3bd009592185ec3473e8f2e67d3816))
+
+### Refactoring
+
+- **evalboard**: Address review nits on harness plumbing
+  ([#45](https://github.com/UiPath/coder_eval/pull/45),
+  [`acc1c86`](https://github.com/UiPath/coder_eval/commit/acc1c86bc97b80383ce1ef402356853447088e4a))
+
+### Testing
+
+- **early-stop**: Cover second-review items (two-AgentStart, golden corpus, parity)
+  ([#43](https://github.com/UiPath/coder_eval/pull/43),
+  [`d34aa97`](https://github.com/UiPath/coder_eval/commit/d34aa97416ae3af09d09cdf78cb6607b4e2e6f7c))
+
+
 ## v0.8.8 (2026-07-22)
 
 ### Bug Fixes

{coder_eval-0.8.8 → coder_eval-0.8.10}/CLAUDE.md

@@ -20,6 +20,7 @@ coder_eval/
 ├── orchestrator.py                # Main evaluation loop
 ├── reports.py                     # Markdown/JSON report generation (run-level + per-suite rollup via write_suite_rollups)
 ├── reports_experiment.py          # Experiment/cross-variant report generation
+├── reports_junit.py               # JUnit XML report from a finalized run dir (run.json spine; for CI test-report ingestion)
 ├── analysis.py                    # Command statistics aggregation
 ├── logging_config.py              # Structured logging setup
 ├── path_utils.py                  # Run ID generation, path utilities
@@ -35,7 +36,7 @@ coder_eval/
 │   ├── criteria.py                # 14 success criterion types + base + union
 │   ├── experiment.py              # ExperimentDefinition, ExperimentVariant, ResolvedTask, result models
 │   ├── judge_defaults.py          # DEFAULT_JUDGE_MODEL constant (cycle-free leaf)
-│   ├── mutations.py               # PromptMutation variants (prefix/suffix/replace/template/rephrase)
+│   ├── mutations.py               # PromptMutation variants (prefix/suffix/replace/template)
 │   ├── results.py                 # CriterionResult (+ ClassificationCriterionResult), TurnRecord, EvaluationResult, EarlyStopInfo/EarlyStopReason, CriterionAggregate, ThresholdCheck, SuiteRollup
 │   ├── routing.py                 # ApiRoute (DirectRoute/BedrockRoute)
 │   ├── sandbox.py                 # SandboxConfig, ResourceLimits
@@ -120,6 +121,7 @@ tasks/                             # Task definition YAML files
 tests/                             # Test suite
 docs/                              # Documentation
 templates/                         # Sandbox template directories
+action.yml                         # Published composite GitHub Action (coder-eval as a CI gate). release.yml maintains its `version:` default + the moving `v<major>` tag.
 ```
 
 ## Key Architectural Patterns
@@ -139,7 +141,7 @@ templates/                         # Sandbox template directories
 - **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports.
 - **sandbox isolation**: Tasks that don't need MCP servers should set `setting_sources: []` in their `agent:` block to isolate the sandbox from the host project's CLAUDE.md and settings. Without this, the host project's CLAUDE.md (often 20 KB+) is injected into every API call, inflating cache-creation tokens and cost significantly.
 - **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block.
-- **Early stop on criterion (opt-in)**: `run_limits.stop_early` (default off) ends a single-shot Claude run early once the run's **armed** criteria are decided, so a raised `max_turns` isn't wasted on the smoke flavor. A criterion is armed by `stop_when: pass|fail|decided`; only criteria that can decide from a partial trajectory may arm (they declare a non-empty `live_stop_polarities` ClassVar and override `live_verdict` on `BaseCriterion` — currently `skill_triggered`, `command_executed`; CE025 enforces the two stay consistent). It uses a cooperative `should_stop` seam on the Claude agent's between-messages guard (tool-call granularity, no SIGKILL) driven by `orchestration/early_stop.py::EarlyStopWatcher` (own `EventCollector` + stop rule). Live verdicts only *trigger* the stop; the standard `check_all` on the frozen trajectory is authoritative. An early-stopped run gates on the **armed subset** (`EvaluationResult.armed_criteria_passed`); a completed run gates on the full set. Every unsupported use (non-observable criterion, non-Claude agent, wrong polarity, no armed criterion, simulation mode) is an error at resolution (plan *and* run), and a runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo` (reason + deciding criterion + when), report notes/badges, `stopped_early` run.json rows, and `EarlyStopped`/`EarlyStopReason` telemetry dims. Defaults off ⇒ behavior byte-for-behavior unchanged.
+- **Early stop on criterion (opt-in)**: `run_limits.stop_early` (default off) ends a single-shot Claude run early once the run's **armed** criteria are decided, so a raised `max_turns` isn't wasted on the smoke flavor. A criterion is armed by `stop_when: pass|fail|decided|auto`; only criteria that can decide from a partial trajectory may arm (non-empty `live_stop_polarities` ClassVar + `live_verdict` override — currently `skill_triggered`, `command_executed`; CE025 keeps the two consistent). `decided` arms **both** polarities; `auto` arms whichever polarities **this instance** can decide — the value for dataset-fanned criteria whose positive/distractor role flips per row. Stop rule: the pass-stop fires when every **pass-armed** criterion live-passes (fail-armed distractors are not required to pass; zero pass-armed ⇒ never pass-stops); the fail-stop fires on the first fail-armed live-fail but is **deferred while any pass-armed criterion is undecided** — a distractor misfire must not truncate a positive row's recall signal, so the latched misfire fires once the positives resolve (or the run continues to the cap). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a `stop_early: false` run. Driven by `orchestration/early_stop.py::EarlyStopWatcher` through the Claude agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all` on the frozen trajectory is authoritative. An early-stopped run gates on the **armed subset** (`EvaluationResult.armed_criteria_passed`); a completed run gates on the full set. Every unsupported use is a hard error at resolution (plan *and* run), and a runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo`, report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. Defaults off ⇒ behavior byte-for-behavior unchanged.
 
 ## Success Criteria (14 types)
 
@@ -160,7 +162,7 @@ templates/                         # Sandbox template directories
 | `llm_judge` | Continuous | LLM grades artifacts + optional trajectory + optional reference; routes through the run's backend (Bedrock / Anthropic) |
 | `agent_judge` | Continuous | Spawns a Claude Code SDK agent in an isolated sandbox copy; judge uses tools (Bash/Read/Grep/…) to investigate and returns a JSON verdict. Expensive; runs with evaluator credentials — see SECURITY note in the criterion docstring. |
 
-All criteria support `weight` (default 1.0) and `pass_threshold` (default 0.9), plus `stop_when` (`pass`/`fail`/`decided`, default `null`) which arms the criterion for early stop when `run_limits.stop_early` is set (observable criteria only). On dataset-backed tasks, criteria may also set `suite_thresholds: {metric: min_value}` — the suite gate passes iff every listed metric (from the criterion's `aggregate()` output) meets its minimum.
+All criteria support `weight` (default 1.0) and `pass_threshold` (default 0.9), plus `stop_when` (`pass`/`fail`/`decided`/`auto`, default `null`) which arms the criterion for early stop when `run_limits.stop_early` is set (observable criteria only; `auto` arms the instance's own decidable polarities). On dataset-backed tasks, criteria may also set `suite_thresholds: {metric: min_value}` — the suite gate passes iff every listed metric (from the criterion's `aggregate()` output) meets its minimum.
 
 ## Evaluation Flow
 
@@ -193,11 +195,17 @@ make format      # ruff format
 make check       # ruff check (lint)
 make typecheck   # pyright
 make test        # pytest
-make lint        # custom architectural lint rules (CE001–CE025)
+make lint        # custom architectural lint rules (CE001+)
 make verify      # All of the above + coverage check (CI equivalent)
 ```
 
-When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001–CE025 pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested.
+When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time — CE027–CE031 — are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name.)
+
+Adding a user-facing field to one of the models CE030 tracks (`TaskDefinition`, `RunLimits`, `Dataset`, `SimulationConfig` — see `tests/lint/doc_schema_parity.py`) means documenting it in its guide (mention the field name as inline code) or adding an `EXEMPT` entry with a reason it is not user-authored. `make lint` fails otherwise.
+
+**Docs index SSOT.** `nav:` plus `extra.docs_index` (blurbs) in `mkdocs.yml` are the single source of truth for the flat index surfaces — `README.md`'s Documentation table, `docs/index.md`'s "Where to go next" table, and the `## Docs` / `## Tutorials` sections of `docs/llms.txt`. Regenerate all three with `make docs-indexes`; **CE028** fails the build if any drifts, if a nav page lacks a blurb (or vice-versa), or if a `docs/*.md` page is missing from the nav. The website sidebar derives from the same `nav:`. When adding or renaming a docs page, edit `nav:` + `extra.docs_index` and run `make docs-indexes` — never hand-edit the generated tables (they sit between `<!-- docs-index:start -->` / `<!-- docs-index:end -->` markers).
+
+**Anchor slugger convention.** The docs are rendered by three sluggers (GitHub, Starlight/github-slugger on coder-eval.com, and python-markdown/mkdocs), which disagree on headings containing `&` or punctuation (`api-routing--benchmarking` vs `api-routing-benchmarking`). Prefer punctuation-free headings so all three agree; if a heading needs `&`, add a GitHub-form `<a id="…"></a>` shim above it and link that form. Verify a new intra-doc anchor link resolves in the built HTML (`mkdocs build`), not by eye.
 
 ## Configuration
 
@@ -294,7 +302,7 @@ Tasks are YAML files. See [docs/TASK_DEFINITION_GUIDE.md](docs/TASK_DEFINITION_G
 
 **Runtime (always)**: pydantic, pydantic-settings, pyyaml, typer, rich, python-dotenv, anthropic, claude-agent-sdk, anyio, radon, tqdm, jmespath, jsonschema
 
-**Runtime (optional, `[uipath]` extra)**: uipath — the in-host `uipath` SDK (handy for local sandbox parity with tasks that invoke `uv run uipath eval ...`). Base installs without this extra still run end-to-end; UiPath-dependent paths fail at dispatch with a clear `pip install 'coder-eval[uipath]'` hint. The LLM judge and the `rephrase` mutation no longer use the LLM Gateway client — they route through the run's backend (Bedrock / Anthropic), so `uipath-llmgw-client` is no longer a dependency.
+**Runtime (optional, `[uipath]` extra)**: uipath — the in-host `uipath` SDK (handy for local sandbox parity with tasks that invoke `uv run uipath eval ...`). Base installs without this extra still run end-to-end; UiPath-dependent paths fail at dispatch with a clear `pip install 'coder-eval[uipath]'` hint. The LLM judge no longer uses the LLM Gateway client — it routes through the run's backend (Bedrock / Anthropic), so `uipath-llmgw-client` is no longer a dependency.
 
 **Dev**: pytest, pytest-asyncio, pytest-mock, pytest-cov, ruff, pyright, pip-audit, bandit, pre-commit, mcp
 

{coder_eval-0.8.8 → coder_eval-0.8.10}/Makefile

@@ -1,4 +1,4 @@
-.PHONY: help install format check typecheck test test-live test-smoke verify verify-noextra clean run lint docker-image docker-image-full coder-eval-runtime docker-images
+.PHONY: help install format check typecheck test test-live test-smoke verify verify-noextra clean run lint docs-indexes docker-image docker-image-full coder-eval-runtime docker-images
 
 # Single source of the installed coder-eval version (used to tag the docker
 # images). Referenced lazily inside the docker recipes, so it doesn't run on
@@ -24,6 +24,9 @@ check:  ## Run linting checks
 lint:  ## Run custom architectural lint rules (CE001+)
 	uv run pytest tests/test_custom_lint.py -v --tb=short --no-header -p no:warnings
 
+docs-indexes:  ## Regenerate README/docs indexes from the mkdocs nav (SSOT)
+	uv run python -m tests.lint.doc_indexes
+
 typecheck:  ## Run type checking with pyright
 	uv run pyright