coder-eval 0.8.9__tar.gz → 0.8.10__tar.gz

{coder_eval-0.8.9 → coder_eval-0.8.10}/.claude/harness-candidates.md

@@ -27,6 +27,22 @@ Deferred lint/test guardrails surfaced during reviews. Promote to a `CExxx` rule
   reachable (needs a live run).
 - [ ] CE-rule: `type: Literal[...]` fields on models in `coder_eval/models/` must declare their tag default (`type: Literal["x"] = "x"`) — a member without the default degrades `validate_registry` diagnostics (PydanticUndefined in expected_types) and breaks direct construction. Nothing guards it today; needs a rule-design call (second violation class inside CE024 vs. a new CExxx at the next free id), and the failure is already double-caught by the MINIMAL_PAYLOADS parity test + direct-construction tests — caught in the 2026-07-03 top5-review-fixes run (Phase 1 quality review).
 
+## From 2026-07-23 stop_when:auto early-stop review
+
+- [ ] CE-rule: the early-stop watcher stop rule must decide polarity via the
+  resolved `_armed_polarities`, never a raw `criterion.stop_when` comparison —
+  forbid `.stop_when` attribute reads inside `EarlyStopWatcher._evaluate` /
+  `_resolve_armed_polarities`'s callers in `orchestration/early_stop.py`. This
+  diff *was* the fix for exactly that class of bug (the old rule compared
+  `stop_when in ("pass","decided")` and so vetoed every mixed `auto` pass-stop).
+  Deferred, not cheap: existing CE rules scope by file/module, not by a specific
+  method, so a method-scoped attribute-ban needs a new AST-walk shape (and risks
+  false positives on the legitimate `is not None` membership reads elsewhere in
+  the file). Claim the next free id in `tests/lint/rules/`. Caught in the
+  2026-07-23 stop_when:auto review; the behavior itself is guarded by
+  `test_auto_mixed_pass_stops_ignoring_undecided_distractors` +
+  `test_mixed_static_arming_pass_stops_ignoring_fail_armed`.
+
 ## From 2026-07-03 open-source docs cleanup
 
 - [ ] **Dead-relative-link checker for `docs/**/*.md`** — resolve every relative

{coder_eval-0.8.9 → coder_eval-0.8.10}/.github/workflows/docs.yml

@@ -10,12 +10,17 @@ name: Docs
 # Source: "Deploy from a branch" → branch `gh-pages` / `/ (root)`. That's the same
 # (legacy, branch-based) configuration uipath-python already runs on.
 on:
-  push:
-    branches: [main]
-    paths:
-      - "docs/**"
-      - "mkdocs.yml"
-      - ".github/workflows/docs.yml"
+  # Auto-publish on push is DISABLED: GitHub Pages is not yet enabled for this
+  # repo on uipath.github.io (org owner must switch it on once — Settings → Pages
+  # → Deploy from a branch → gh-pages / root). The published docs currently serve
+  # from coder-eval.com/docs (synced separately by the website), so the gh-pages
+  # deploy is not needed yet. Re-enable by uncommenting the `push:` trigger below.
+  # push:
+  #   branches: [main]
+  #   paths:
+  #     - "docs/**"
+  #     - "mkdocs.yml"
+  #     - ".github/workflows/docs.yml"
   workflow_dispatch:
 
 permissions:

{coder_eval-0.8.9 → coder_eval-0.8.10}/.github/workflows/pr-checks.yml

@@ -831,3 +831,61 @@ jobs:
           name: byoa-live-output
           path: tmp/
           retention-days: 7
+
+  action-dogfood:
+    name: Action Dogfood (composite action, real API)
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    # Skip on fork PRs where secrets aren't available (matches e2e-smoke).
+    if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
+
+      # The composite action is agent-agnostic and does NOT install a coding-agent
+      # runtime. The dogfood task uses the default claude-code agent, so provide
+      # Node + the Claude CLI here (as e2e-smoke does), before invoking the action.
+      - name: Set up Node.js 20
+        uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020  # v4.4.0
+        with:
+          node-version: "20"
+      - name: Install Claude CLI
+        run: npm install -g @anthropic-ai/claude-code
+
+      - name: Run coder-eval via local action
+        id: dogfood
+        uses: ./
+        with:
+          version: local
+          tasks: tasks/hello_date.yaml
+          model: claude-haiku-4-5-20251001
+          run-dir: runs/ci-action-dogfood
+          junit-path: runs/ci-action-dogfood/junit.xml
+          # Credentials go through the generic env passthrough (the only channel);
+          # ANTHROPIC_API_KEY reaching the run is proven by the API-backed task
+          # succeeding. A floor of 0.0 passes for any produced score (exercises
+          # the gate path green in CI without flakiness); the second line
+          # exercises multi-line env parsing.
+          minimum-task-score: "0.0"
+          env: |
+            ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}
+            CE_DOGFOOD_MARKER=1
+
+      - name: Verify outputs and JUnit file
+        env:
+          JUNIT: ${{ steps.dogfood.outputs.junit-path }}
+          RUNDIR: ${{ steps.dogfood.outputs.run-dir }}
+        run: |
+          set -euo pipefail
+          test -n "$JUNIT" && test -f "$JUNIT" || { echo "junit output missing"; exit 1; }
+          # Well-formedness check on a file this job just generated (trusted input;
+          # our writer emits no DTDs/entities) — stdlib ET is fine here.
+          python3 -c "import sys, xml.etree.ElementTree as ET; ET.parse(sys.argv[1])" "$JUNIT"
+          test -f "$RUNDIR/run.json" || { echo "run.json missing"; exit 1; }
+
+      - name: Upload dogfood run on failure
+        if: failure()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a  # v7.0.1
+        with:
+          name: action-dogfood-runs
+          path: runs/ci-action-dogfood/
+          retention-days: 7

{coder_eval-0.8.9 → coder_eval-0.8.10}/.github/workflows/release.yml

@@ -182,23 +182,50 @@ jobs:
           echo "version=$V" >> "$GITHUB_OUTPUT"
           echo "Publishing version: $V"
 
-      - name: Regenerate uv.lock and amend release commit
+      - name: Regenerate uv.lock, bump action.yml pin, and amend release commit
         if: steps.mode.outputs.prerelease != 'true' && steps.release.outputs.version != ''
+        env:
+          # Passed via env (not interpolated into the script) per GitHub's
+          # injection guidance.
+          VERSION: ${{ steps.release.outputs.version }}
         run: |
+          set -euo pipefail
+          git config user.email "github-actions[bot]@users.noreply.github.com"
+          git config user.name "github-actions[bot]"
+          # Bump the composite action's default `version:` pin to the just-released
+          # version so `UiPath/coder_eval@vX.Y.Z` installs `coder-eval==X.Y.Z`. The
+          # anchor is indentation-tolerant and keyed on the unique trailing
+          # "# <-- kept in sync" comment; the grep guard fails the release loudly
+          # if a reformat ever detaches it (rather than shipping a stale pin).
+          sed -i -E 's/^([[:space:]]*default: ")[0-9]+\.[0-9]+\.[0-9]+("   # <-- kept in sync)/\1'"${VERSION}"'\2/' action.yml
+          grep -q "default: \"${VERSION}\"" action.yml || { echo "action.yml version bump failed"; exit 1; }
+          git add action.yml
+          # Regenerate the lock too; stage it (a no-op if unchanged).
           uv lock
-          if ! git diff --quiet uv.lock; then
-            git config user.email "github-actions[bot]@users.noreply.github.com"
-            git config user.name "github-actions[bot]"
-            git add uv.lock
+          git add uv.lock
+          # Amend only if action.yml/uv.lock actually changed the tree.
+          if ! git diff --cached --quiet; then
             git commit --amend --no-edit
             # Amend replaced the commit the tag points at; re-point it before pushing.
-            git tag -f "v${{ steps.release.outputs.version }}"
+            git tag -f "v${VERSION}"
           fi
 
       - name: Push release commit and tags
         if: steps.mode.outputs.prerelease != 'true' && steps.release.outputs.version != ''
         run: git push origin main "v${{ steps.release.outputs.version }}"
 
+      - name: Move major action tag (vN -> this release)
+        if: steps.release.outputs.version != ''
+        env:
+          VERSION: ${{ steps.release.outputs.version }}
+        run: |
+          set -euo pipefail
+          # Consumers pin `UiPath/coder_eval@v0` (becomes `@v1` at 1.0.0). Force-move
+          # the moving major tag to this release. Force on a missing tag creates it.
+          MAJOR="v${VERSION%%.*}"
+          git tag -f "$MAJOR" "v${VERSION}"
+          git push -f origin "$MAJOR"
+
       - name: Build wheel + sdist
         if: steps.ver.outputs.version != ''
         run: uv build

{coder_eval-0.8.9 → coder_eval-0.8.10}/CHANGELOG.md

@@ -2,6 +2,121 @@
 
 <!-- version list -->
 
+## v0.8.10 (2026-07-24)
+
+### Bug Fixes
+
+- Remove dead SimulationConfig.parallel_trials; add CE031 to guard the class
+  ([`947acd3`](https://github.com/UiPath/coder_eval/commit/947acd34a4f0451c69cb7bf0eb863127b9995f76))
+
+- **early-stop**: Add stop_when 'auto' — per-instance arming + pass-armed-subset stop rule
+  ([#51](https://github.com/UiPath/coder_eval/pull/51),
+  [`08e21e6`](https://github.com/UiPath/coder_eval/commit/08e21e67b67fdab84d26b8a0e5551e04289d92a8))
+
+- **early-stop**: Defer fail-stop while a pass-armed criterion is undecided
+  ([#51](https://github.com/UiPath/coder_eval/pull/51),
+  [`08e21e6`](https://github.com/UiPath/coder_eval/commit/08e21e67b67fdab84d26b8a0e5551e04289d92a8))
+
+- **early-stop**: Return assert_never explicitly to satisfy CodeQL
+  ([#51](https://github.com/UiPath/coder_eval/pull/51),
+  [`08e21e6`](https://github.com/UiPath/coder_eval/commit/08e21e67b67fdab84d26b8a0e5551e04289d92a8))
+
+- **reports**: Code review fixes for the JUnit CI gate
+  ([#37](https://github.com/UiPath/coder_eval/pull/37),
+  [`74db6fa`](https://github.com/UiPath/coder_eval/commit/74db6facfea6f898af4db6709e563d90af0d7b30))
+
+- **reports**: JUnit CI-gate review fixes + CE027 env-var lint
+  ([#37](https://github.com/UiPath/coder_eval/pull/37),
+  [`74db6fa`](https://github.com/UiPath/coder_eval/commit/74db6facfea6f898af4db6709e563d90af0d7b30))
+
+- **reports**: Make skipped-task JUnit names platform-independent
+  ([#37](https://github.com/UiPath/coder_eval/pull/37),
+  [`74db6fa`](https://github.com/UiPath/coder_eval/commit/74db6facfea6f898af4db6709e563d90af0d7b30))
+
+### Chores
+
+- Re-trigger CI (GitHub dropped the force-push event)
+  ([#37](https://github.com/UiPath/coder_eval/pull/37),
+  [`74db6fa`](https://github.com/UiPath/coder_eval/commit/74db6facfea6f898af4db6709e563d90af0d7b30))
+
+- **deps**: Lock defusedxml (dev-only, test-side XML parsing)
+  ([#37](https://github.com/UiPath/coder_eval/pull/37),
+  [`74db6fa`](https://github.com/UiPath/coder_eval/commit/74db6facfea6f898af4db6709e563d90af0d7b30))
+
+### Continuous Integration
+
+- Disable Docs gh-pages auto-publish on push (Pages not enabled yet)
+  ([`c289d46`](https://github.com/UiPath/coder_eval/commit/c289d46c5cec269f205224e4dfce27affab5557e))
+
+### Documentation
+
+- 1/8 — add DATASETS.md and a task-schema dataset: section
+  ([`e3d37ac`](https://github.com/UiPath/coder_eval/commit/e3d37ac793e6f12c020b71a4ed19df2279b4b2c1))
+
+- 2/8 — retire BYOD.md into DOCKER_ISOLATION.md
+  ([`1a4a2a5`](https://github.com/UiPath/coder_eval/commit/1a4a2a5893639d14832dbd1614163aa7c18bf70a))
+
+- 3/8 — one complete run_limits reference; document skip
+  ([`bcb7e24`](https://github.com/UiPath/coder_eval/commit/bcb7e24132c229114efe9a38281a37e736d83312))
+
+- 4/8 — add DIALOG_MODE.md and correct four stale simulation claims
+  ([`18dcbe9`](https://github.com/UiPath/coder_eval/commit/18dcbe95d156856149538406ededc8c75ec594f8))
+
+- 5/8 — fix prompt_mutations example; add CE029
+  ([`b524009`](https://github.com/UiPath/coder_eval/commit/b5240099c58f5b50c7b5fe531a12f1f8b98aec29))
+
+- 7/8 — generate flat indexes from the mkdocs nav; add CE028
+  ([`1514bcb`](https://github.com/UiPath/coder_eval/commit/1514bcb8f075ed48f2857a84269513c60878ea81))
+
+- Add CI Gate reference (GitHub Action + JUnit) and wire into indexes
+  ([`b8c6301`](https://github.com/UiPath/coder_eval/commit/b8c63018bd93ba91751511d95f4af96a444d9873))
+
+- Agent guides, extending & report-schema references, and fixes
+  ([`ce74824`](https://github.com/UiPath/coder_eval/commit/ce74824bde2b5322dc745bace045e1806f77324b))
+
+- Fold nav long tail into one Advanced group; align index ordering
+  ([`e2ff053`](https://github.com/UiPath/coder_eval/commit/e2ff0539af8c9e3e6deb85c0a0124c0232224033))
+
+- Point docs links to coder-eval.com/docs; drop Ruff badge
+  ([`60430e5`](https://github.com/UiPath/coder_eval/commit/60430e5e857f0af6fbf08cde00f0621de4610562))
+
+- Point pyproject Documentation URL to coder-eval.com/docs
+  ([`be39df2`](https://github.com/UiPath/coder_eval/commit/be39df23c8ba1a0f6f945dc99e3827eb5d6f2772))
+
+- Reword CODER_EVAL_RAW_SDK_LOG to prose form (satisfy CE027)
+  ([`d7d5b59`](https://github.com/UiPath/coder_eval/commit/d7d5b59b990127810fcfe33272f3ff733a9498cd))
+
+- Use the brand name "Coder Eval" in prose and titles
+  ([`821f11b`](https://github.com/UiPath/coder_eval/commit/821f11bf94762f6578913090568f5b092e237596))
+
+### Features
+
+- Packaged CI gate — JUnit XML output + composite GitHub Action
+  ([#37](https://github.com/UiPath/coder_eval/pull/37),
+  [`74db6fa`](https://github.com/UiPath/coder_eval/commit/74db6facfea6f898af4db6709e563d90af0d7b30))
+
+- **action**: Generic env passthrough + minimum-task-score gate
+  ([#37](https://github.com/UiPath/coder_eval/pull/37),
+  [`74db6fa`](https://github.com/UiPath/coder_eval/commit/74db6facfea6f898af4db6709e563d90af0d7b30))
+
+- **ci**: 3/3 — publish composite action, release automation, PR dogfood
+  ([#37](https://github.com/UiPath/coder_eval/pull/37),
+  [`74db6fa`](https://github.com/UiPath/coder_eval/commit/74db6facfea6f898af4db6709e563d90af0d7b30))
+
+- **cli**: 2/3 — wire run --junit-xml and report -f junit
+  ([#37](https://github.com/UiPath/coder_eval/pull/37),
+  [`74db6fa`](https://github.com/UiPath/coder_eval/commit/74db6facfea6f898af4db6709e563d90af0d7b30))
+
+- **reports**: 1/3 — add reports_junit.py disk-driven JUnit XML writer
+  ([#37](https://github.com/UiPath/coder_eval/pull/37),
+  [`74db6fa`](https://github.com/UiPath/coder_eval/commit/74db6facfea6f898af4db6709e563d90af0d7b30))
+
+### Testing
+
+- 6/8 — CE030 documents-or-exempts model fields
+  ([`c9a3b16`](https://github.com/UiPath/coder_eval/commit/c9a3b160deeec385386a7009812025bc26021363))
+
+
 ## v0.8.9 (2026-07-23)
 
 ### Bug Fixes

{coder_eval-0.8.9 → coder_eval-0.8.10}/CLAUDE.md

@@ -20,6 +20,7 @@ coder_eval/
 ├── orchestrator.py                # Main evaluation loop
 ├── reports.py                     # Markdown/JSON report generation (run-level + per-suite rollup via write_suite_rollups)
 ├── reports_experiment.py          # Experiment/cross-variant report generation
+├── reports_junit.py               # JUnit XML report from a finalized run dir (run.json spine; for CI test-report ingestion)
 ├── analysis.py                    # Command statistics aggregation
 ├── logging_config.py              # Structured logging setup
 ├── path_utils.py                  # Run ID generation, path utilities
@@ -120,6 +121,7 @@ tasks/                             # Task definition YAML files
 tests/                             # Test suite
 docs/                              # Documentation
 templates/                         # Sandbox template directories
+action.yml                         # Published composite GitHub Action (coder-eval as a CI gate). release.yml maintains its `version:` default + the moving `v<major>` tag.
 ```
 
 ## Key Architectural Patterns
@@ -139,7 +141,7 @@ templates/                         # Sandbox template directories
 - **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports.
 - **sandbox isolation**: Tasks that don't need MCP servers should set `setting_sources: []` in their `agent:` block to isolate the sandbox from the host project's CLAUDE.md and settings. Without this, the host project's CLAUDE.md (often 20 KB+) is injected into every API call, inflating cache-creation tokens and cost significantly.
 - **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block.
-- **Early stop on criterion (opt-in)**: `run_limits.stop_early` (default off) ends a single-shot Claude run early once the run's **armed** criteria are decided, so a raised `max_turns` isn't wasted on the smoke flavor. A criterion is armed by `stop_when: pass|fail|decided`; only criteria that can decide from a partial trajectory may arm (they declare a non-empty `live_stop_polarities` ClassVar and override `live_verdict` on `BaseCriterion` — currently `skill_triggered`, `command_executed`; CE025 enforces the two stay consistent). It uses a cooperative `should_stop` seam on the Claude agent's between-messages guard (tool-call granularity, no SIGKILL) driven by `orchestration/early_stop.py::EarlyStopWatcher` (own `EventCollector` + stop rule). Live verdicts only *trigger* the stop; the standard `check_all` on the frozen trajectory is authoritative. An early-stopped run gates on the **armed subset** (`EvaluationResult.armed_criteria_passed`); a completed run gates on the full set. Every unsupported use (non-observable criterion, non-Claude agent, wrong polarity, no armed criterion, simulation mode) is an error at resolution (plan *and* run), and a runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo` (reason + deciding criterion + when), report notes/badges, `stopped_early` run.json rows, and `EarlyStopped`/`EarlyStopReason` telemetry dims. Defaults off ⇒ behavior byte-for-behavior unchanged.
+- **Early stop on criterion (opt-in)**: `run_limits.stop_early` (default off) ends a single-shot Claude run early once the run's **armed** criteria are decided, so a raised `max_turns` isn't wasted on the smoke flavor. A criterion is armed by `stop_when: pass|fail|decided|auto`; only criteria that can decide from a partial trajectory may arm (non-empty `live_stop_polarities` ClassVar + `live_verdict` override — currently `skill_triggered`, `command_executed`; CE025 keeps the two consistent). `decided` arms **both** polarities; `auto` arms whichever polarities **this instance** can decide — the value for dataset-fanned criteria whose positive/distractor role flips per row. Stop rule: the pass-stop fires when every **pass-armed** criterion live-passes (fail-armed distractors are not required to pass; zero pass-armed ⇒ never pass-stops); the fail-stop fires on the first fail-armed live-fail but is **deferred while any pass-armed criterion is undecided** — a distractor misfire must not truncate a positive row's recall signal, so the latched misfire fires once the positives resolve (or the run continues to the cap). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a `stop_early: false` run. Driven by `orchestration/early_stop.py::EarlyStopWatcher` through the Claude agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all` on the frozen trajectory is authoritative. An early-stopped run gates on the **armed subset** (`EvaluationResult.armed_criteria_passed`); a completed run gates on the full set. Every unsupported use is a hard error at resolution (plan *and* run), and a runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo`, report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. Defaults off ⇒ behavior byte-for-behavior unchanged.
 
 ## Success Criteria (14 types)
 
@@ -160,7 +162,7 @@ templates/                         # Sandbox template directories
 | `llm_judge` | Continuous | LLM grades artifacts + optional trajectory + optional reference; routes through the run's backend (Bedrock / Anthropic) |
 | `agent_judge` | Continuous | Spawns a Claude Code SDK agent in an isolated sandbox copy; judge uses tools (Bash/Read/Grep/…) to investigate and returns a JSON verdict. Expensive; runs with evaluator credentials — see SECURITY note in the criterion docstring. |
 
-All criteria support `weight` (default 1.0) and `pass_threshold` (default 0.9), plus `stop_when` (`pass`/`fail`/`decided`, default `null`) which arms the criterion for early stop when `run_limits.stop_early` is set (observable criteria only). On dataset-backed tasks, criteria may also set `suite_thresholds: {metric: min_value}` — the suite gate passes iff every listed metric (from the criterion's `aggregate()` output) meets its minimum.
+All criteria support `weight` (default 1.0) and `pass_threshold` (default 0.9), plus `stop_when` (`pass`/`fail`/`decided`/`auto`, default `null`) which arms the criterion for early stop when `run_limits.stop_early` is set (observable criteria only; `auto` arms the instance's own decidable polarities). On dataset-backed tasks, criteria may also set `suite_thresholds: {metric: min_value}` — the suite gate passes iff every listed metric (from the criterion's `aggregate()` output) meets its minimum.
 
 ## Evaluation Flow
 
@@ -193,11 +195,17 @@ make format      # ruff format
 make check       # ruff check (lint)
 make typecheck   # pyright
 make test        # pytest
-make lint        # custom architectural lint rules (CE001–CE025)
+make lint        # custom architectural lint rules (CE001+)
 make verify      # All of the above + coverage check (CI equivalent)
 ```
 
-When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001–CE025 pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested.
+When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time — CE027–CE031 — are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name.)
+
+Adding a user-facing field to one of the models CE030 tracks (`TaskDefinition`, `RunLimits`, `Dataset`, `SimulationConfig` — see `tests/lint/doc_schema_parity.py`) means documenting it in its guide (mention the field name as inline code) or adding an `EXEMPT` entry with a reason it is not user-authored. `make lint` fails otherwise.
+
+**Docs index SSOT.** `nav:` plus `extra.docs_index` (blurbs) in `mkdocs.yml` are the single source of truth for the flat index surfaces — `README.md`'s Documentation table, `docs/index.md`'s "Where to go next" table, and the `## Docs` / `## Tutorials` sections of `docs/llms.txt`. Regenerate all three with `make docs-indexes`; **CE028** fails the build if any drifts, if a nav page lacks a blurb (or vice-versa), or if a `docs/*.md` page is missing from the nav. The website sidebar derives from the same `nav:`. When adding or renaming a docs page, edit `nav:` + `extra.docs_index` and run `make docs-indexes` — never hand-edit the generated tables (they sit between `<!-- docs-index:start -->` / `<!-- docs-index:end -->` markers).
+
+**Anchor slugger convention.** The docs are rendered by three sluggers (GitHub, Starlight/github-slugger on coder-eval.com, and python-markdown/mkdocs), which disagree on headings containing `&` or punctuation (`api-routing--benchmarking` vs `api-routing-benchmarking`). Prefer punctuation-free headings so all three agree; if a heading needs `&`, add a GitHub-form `<a id="…"></a>` shim above it and link that form. Verify a new intra-doc anchor link resolves in the built HTML (`mkdocs build`), not by eye.
 
 ## Configuration
 

{coder_eval-0.8.9 → coder_eval-0.8.10}/Makefile

@@ -1,4 +1,4 @@
-.PHONY: help install format check typecheck test test-live test-smoke verify verify-noextra clean run lint docker-image docker-image-full coder-eval-runtime docker-images
+.PHONY: help install format check typecheck test test-live test-smoke verify verify-noextra clean run lint docs-indexes docker-image docker-image-full coder-eval-runtime docker-images
 
 # Single source of the installed coder-eval version (used to tag the docker
 # images). Referenced lazily inside the docker recipes, so it doesn't run on
@@ -24,6 +24,9 @@ check:  ## Run linting checks
 lint:  ## Run custom architectural lint rules (CE001+)
 	uv run pytest tests/test_custom_lint.py -v --tb=short --no-header -p no:warnings
 
+docs-indexes:  ## Regenerate README/docs indexes from the mkdocs nav (SSOT)
+	uv run python -m tests.lint.doc_indexes
+
 typecheck:  ## Run type checking with pyright
 	uv run pyright
 

{coder_eval-0.8.9 → coder_eval-0.8.10}/PKG-INFO

@@ -1,10 +1,10 @@
 Metadata-Version: 2.4
 Name: coder-eval
-Version: 0.8.9
+Version: 0.8.10
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval
-Project-URL: Documentation, https://github.com/UiPath/coder_eval/tree/main/docs
+Project-URL: Documentation, https://coder-eval.com/docs
 Project-URL: Issues, https://github.com/UiPath/coder_eval/issues
 Project-URL: Changelog, https://github.com/UiPath/coder_eval/blob/main/CHANGELOG.md
 Author-email: UiPath <coder-eval@uipath.com>
@@ -46,6 +46,7 @@ Provides-Extra: codex
 Requires-Dist: openai-codex>=0.144.4; extra == 'codex'
 Provides-Extra: dev
 Requires-Dist: bandit[toml]>=1.9.4; extra == 'dev'
+Requires-Dist: defusedxml>=0.7.1; extra == 'dev'
 Requires-Dist: mcp>=1.28.1; extra == 'dev'
 Requires-Dist: pip-audit>=2.10.0; extra == 'dev'
 Requires-Dist: pre-commit>=4.5.1; extra == 'dev'
@@ -64,11 +65,10 @@ Description-Content-Type: text/markdown
 
 [![PyPI](https://img.shields.io/pypi/v/coder-eval.svg)](https://pypi.org/project/coder-eval/)
 [![Website](https://img.shields.io/badge/website-coder--eval.com-1f6feb.svg)](https://coder-eval.com)
-[![Docs](https://img.shields.io/badge/docs-uipath.github.io%2Fcoder__eval-1f6feb.svg)](https://uipath.github.io/coder_eval/)
+[![Docs](https://img.shields.io/badge/docs-coder--eval.com%2Fdocs-1f6feb.svg)](https://coder-eval.com/docs)
 [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
 [![Python 3.13+](https://img.shields.io/badge/python-3.13%2B-blue.svg)](https://www.python.org/downloads/)
 [![CI](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml/badge.svg)](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml)
-[![Code style: Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
 
 **Coder Eval** (`pip install coder-eval` / `uv tool install coder-eval`) is an open-source framework for
 **evaluating and benchmarking AI coding agents and their skills** — built for CLI
@@ -84,8 +84,8 @@ prompt), or **gate CI on coding-agent quality**. Unlike fixed datasets (SWE-benc
 SkillsBench) that rank models on a shared leaderboard, Coder Eval evaluates the
 tasks, skills, and workflows *you* ship — with weighted 0.0–1.0 criteria, a
 `skill_triggered` activation check, an A/B experiment layer, and per-tool cost
-telemetry. See [How it compares](https://uipath.github.io/coder_eval/comparison/).
-📚 **Full docs:** **[uipath.github.io/coder_eval](https://uipath.github.io/coder_eval/)**.
+telemetry. See [How it compares](https://coder-eval.com/docs/comparison).
+📚 **Full docs:** **[coder-eval.com/docs](https://coder-eval.com/docs)**.
 
 <p align="center">
   <img src="docs/assets/hero.gif" alt="Coder Eval running the hello_date task: a sandboxed agent writes and runs a script from a YAML task, then the scored result is browsed in evalboard" width="100%">
@@ -111,7 +111,7 @@ telemetry. See [How it compares](https://uipath.github.io/coder_eval/comparison/
 > **Keeping skills fresh?** Run Coder Eval as a scheduled GitHub Actions job so your
 > skills are continuously re-evaluated against the latest model — a skill that quietly
 > stops triggering surfaces as a failing criterion before your users hit it. See
-> **[Tutorial 02 — Running coder_eval in CI](docs/tutorials/02-ci-pipeline.md)**.
+> **[Tutorial 02 — Running Coder Eval in CI](docs/tutorials/02-ci-pipeline.md)**.
 
 ## Quick Start
 
@@ -155,9 +155,77 @@ To add it as a project dependency instead: `uv add coder-eval` or
 `pip install coder-eval`. In a real CI gate, pin to a specific released version
 so a harness upgrade can't silently move your results. (The example `tasks/`
 live in this repo — clone it or point the CLI at your own task files.) See
-[Tutorial 02 — Running coder_eval in CI](docs/tutorials/02-ci-pipeline.md) for
+[Tutorial 02 — Running Coder Eval in CI](docs/tutorials/02-ci-pipeline.md) for
 the full setup.
 
+## Use as a GitHub Action
+
+A composite action at the repo root runs `coder-eval` as a CI gate — it installs
+the pinned CLI, runs your tasks, writes a JUnit XML report, appends `run.md` to
+the job summary, and fails the step on any task/gate failure:
+
+```yaml
+- uses: UiPath/coder_eval@v0    # becomes @v1 once 1.0.0 ships; @vX.Y.Z pins exactly
+  with:
+    tasks: tests/tasks/**/*.yaml
+    model: claude-sonnet-5
+    env: |
+      ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}
+```
+
+| Input | Default | Purpose |
+| --- | --- | --- |
+| `tasks` | *(all `tasks/`)* | Task YAML path(s)/glob |
+| `tags` | — | `--tags` filter |
+| `model` | — | `--model` override |
+| `extra-args` | — | Verbatim extra args (`--experiment`, `-D …`, …) |
+| `version` | pinned release | PyPI version, or `local` to install from the checkout |
+| `run-dir` | `runs/ci` | Run directory |
+| `junit-path` | `coder-eval-junit.xml` | Where to write the JUnit report |
+| `step-summary` | `true` | Append `run.md` to the job summary |
+| `env` | — | Credentials/backend passthrough: newline-separated `NAME=VALUE` pairs, exported for the run step only |
+| `minimum-task-score` | *(off)* | Strict floor (0.0–1.0): fail the step if any task's `weighted_score` is below it |
+
+Outputs: `run-dir` and `junit-path`. Feed the JUnit file to your platform's
+test-report renderer — e.g. on GitHub Actions with
+[`mikepenz/action-junit-report`](https://github.com/mikepenz/action-junit-report):
+
+```yaml
+- uses: mikepenz/action-junit-report@v5
+  if: always()
+  with:
+    report_paths: coder-eval-junit.xml
+```
+
+**Credentials and backend config** are the sole responsibility of `env` — a
+passthrough exported for the run step only (never written to `$GITHUB_ENV`, so
+it can't leak into later steps). Set whatever the run needs, Anthropic or not:
+
+```yaml
+- uses: UiPath/coder_eval@v0
+  with:
+    tasks: tests/tasks/**/*.yaml
+    minimum-task-score: "0.8"   # fail the build if any task scores below 0.8
+    env: |
+      API_BACKEND=bedrock
+      AWS_BEARER_TOKEN_BEDROCK=${{ secrets.BEDROCK_TOKEN }}
+```
+
+`minimum-task-score` is a strict floor **on top of** coder-eval's own exit
+code: the step fails if *either* coder-eval exits non-zero *or* any task's
+`weighted_score` falls below the floor. Leave it unset to gate on the exit code
+alone.
+
+> **Agent runtime is the caller's responsibility.** The action is agent-agnostic —
+> it installs `coder-eval` but no coding-agent runtime. Tasks using the default
+> `claude-code` agent need the `claude` CLI on `PATH` (`actions/setup-node` +
+> `npm install -g @anthropic-ai/claude-code`) in the job before the action runs.
+
+> **Security.** Evaluated tasks execute agent-generated code. Do **not** run this
+> action under `pull_request_target` with secrets exposed to untrusted fork PRs —
+> use `pull_request` and gate on the same-repo condition, as this repo's own
+> dogfood job does.
+
 ## Telemetry
 
 > 📊 **Usage telemetry is on by default.** `coder-eval` sends **anonymous** usage
@@ -170,15 +238,27 @@ the full setup.
 
 ## Documentation
 
+<!-- docs-index:start -->
 | Guide | What's in it |
 | --- | --- |
 | [Tutorials](docs/tutorials/README.md) | Step-by-step walkthroughs — start here |
 | [User Guide](docs/USER_GUIDE.md) | Full CLI, configuration, output, and environment-variable reference |
 | [Task Definition Guide](docs/TASK_DEFINITION_GUIDE.md) | The task-file schema — all criterion types, scoring, templates |
+| [Claude Code](docs/agents/CLAUDE_CODE.md) | Configuring and running the default Claude Code agent |
+| [Codex](docs/agents/CODEX.md) | Running the OpenAI Codex agent |
+| [Antigravity (Gemini)](docs/agents/ANTIGRAVITY.md) | Running the Google Antigravity / Gemini agent |
 | [A/B Experiments](docs/AB_EXPERIMENTS.md) | Compare models / tools / prompts across the same tasks |
-| [Bring Your Own Dataset](docs/BYOD.md) | Fan a single task out over a dataset |
-| [Codex Agent Guide](docs/CODEX_AGENT_GUIDE.md) | Running the Codex agent |
-| [Docker Isolation](docs/DOCKER_ISOLATION.md) | The container sandbox driver |
+| [Bring Your Own Dataset](docs/DATASETS.md) | Fan a single task out over a dataset |
+| [Dialog Mode](docs/DIALOG_MODE.md) | Evaluate agents in multi-turn conversation via a simulated user |
+| [Docker Isolation](docs/DOCKER_ISOLATION.md) | The container sandbox driver, with custom images |
+| [CI Gate & GitHub Action](docs/CI_GATE.md) | Run Coder Eval as a CI gate — the packaged Action, JUnit output, score floor |
+| [Extending Coder Eval](docs/EXTENDING.md) | Author a custom agent, criterion, or model pricing via the plugin SPI |
+| [Report Schema](docs/REPORT_SCHEMA.md) | Field-level reference for run.json / variant.json / task.json |
+| [How It Compares](docs/comparison.md) | vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, hand-rolled scripts |
+<!-- docs-index:end -->
+
+| Repo doc | What's in it |
+| --- | --- |
 | [CLAUDE.md](CLAUDE.md) | Architecture, key patterns, and extension points |
 | [CONTRIBUTING.md](CONTRIBUTING.md) | Dev setup, quality bar, and how to contribute |
 
@@ -186,7 +266,7 @@ the full setup.
 
 - **vs. fixed benchmarks (SWE-bench, SkillsBench)** — they score a canonical dataset;
   Coder Eval scores *your* tasks with continuous 0.0–1.0 weighted criteria (and can
-  still wrap a fixed dataset via [Bring Your Own Dataset](docs/BYOD.md)).
+  still wrap a fixed dataset via [Bring Your Own Dataset](docs/DATASETS.md)).
 - **vs. large-scale / RL harnesses (Harbor)** — Harbor targets scale and RL rollouts;
   Coder Eval targets weighted, skill-aware suites gated in CI.
 - **vs. model-output eval tools (OpenAI Evals)** — they grade model text; Coder Eval
@@ -194,7 +274,7 @@ the full setup.
 - **vs. hand-rolled scripts** — reproducible sandboxes, weighted criteria,
   cost/token telemetry, A/B experiments, and CI-ready pass/fail gates out of the box.
 
-See the full [comparison — with sources](https://uipath.github.io/coder_eval/comparison/).
+See the full [comparison — with sources](https://coder-eval.com/docs/comparison).
 
 ## Task Definition