coder-eval 0.9.5__tar.gz → 0.9.6__tar.gz

{coder_eval-0.9.5 → coder_eval-0.9.6}/.claude/harness-candidates.md

@@ -127,3 +127,82 @@ Deferred lint/test guardrails surfaced during reviews. Promote to a `CExxx` rule
   that environment injects the criterion. Until then CE030 stays scoped to the four
   top-level models; the `command_pattern`/`exclude_pattern` contract this PR changed is
   documented in the Field descriptions and TASK_DEFINITION_GUIDE regardless.
+
+## From the evalboard Path-to-GA de-tag / mature-passes fix (4e5bbc4…dd5f7e9) — TS-side guards deferred
+
+Context: the CExxx harness is a **Python** AST runner over `src/coder_eval/`, so none
+of the invariants below are mechanizable in it. Each would need a TypeScript lint
+harness (eslint config + custom rules) that `evalboard/` does not have today —
+standing one up for three call sites fails the KISS/YAGNI gate. Deferring rather
+than dropping; promote if a fourth TS-side invariant appears, and stand up the
+harness once for all of them.
+
+> **Update (PR #94 review round 2).** The *execution* half of this gap is closed:
+> `evalboard/` is now gated by the `evalboard` job in `.github/workflows/pr-checks.yml`
+> and reachable locally via `make evalboard-verify`, so the vitest suite (including
+> the pricing drift guard) is enforcement rather than documentation. What remains
+> deferred below is the *static-analysis* half — eslint has still not been stood up.
+> The review that prompted this round names four more candidate TS rules (raw
+> `status === "SUCCESS"` outside `lib/status.ts`; DOM-global shadowing in props;
+> inline copies of the tag predicate; per-run tooltip copy reused on aggregate
+> surfaces), which meets the "fourth invariant" promotion bar stated above —
+> **stand up eslint next time `evalboard/` is touched substantively.**
+
+- [ ] **"Every consumer of `RunOverviewTask.matureSkipped` must decide explicitly
+  whether a carry-forward row counts."** Four consumers now, and they deliberately
+  DISAGREE: `lib/trends.ts` and `app/runs/[id]/run-view.tsx` count a mature skip as
+  a pass; `lib/overview.ts::buildTagTaskRows` excludes it from both terms
+  (`/path-to-ga` is a GA-readiness page). A new consumer silently inheriting either
+  convention is a real hazard. Guard shape: flag a file that reads `.matureSkipped`
+  without a nearby comment naming its convention — weak, hence the deferral. Closed
+  for now by unit tests that assert the exclusion from BOTH numerator and denominator
+  (`lib/__tests__/overview.test.ts` → `describe("buildTagTaskRows")`).
+
+- [x] ~~**`taskCarriesRepoTag` is the single repo-provenance tag predicate — but one
+  duplicate survives.**~~ **RESOLVED in PR #94 review round 2.** The predicate moved to
+  a dependency-free `lib/tags.ts` (structurally typed on `{skill, tags}` so
+  `RunOverviewTask`, `TaskResultSummary` and `TaskTrend` all satisfy it), re-exported
+  from `lib/overview.ts` for existing callers. Both inline copies now import it:
+  `app/runs/[id]/run-view.tsx` (the `"use client"` one that could not before) and
+  `lib/trends.ts::trendMatchesTag` (a third copy the original deferral missed).
+  Still worth a lint rule ("no inline `tags.includes(x) || skill === x`") to catch
+  future copies — folded into the eslint promotion noted above.
+
+- [ ] **The de-tag rule fails CLOSED on a newest run that loads fine but stamps no
+  `tags`** (`lib/overview.ts::buildTagTaskRows`): every tagged task would read as
+  de-tagged and the table would empty, rendering an empty state indistinguishable from
+  a genuine full de-tagging. Its sibling failure mode (`overview == null`, a transient
+  blob read failure) IS guarded, with exactly this rationale. Currently unreachable —
+  0 of ~116k date-shaped non-ad-hoc task rows in `runs-remote/` lack `tags`, and the
+  six zero-tag runs found are all ad-hoc (filtered upstream by id shape + `meta.adhoc`)
+  — so the barrier is two upstream filters rather than a check at the seam. Left
+  unguarded on purpose: a `if (taggedInRun.size === 0) skip the de-tag signal` guard
+  would also mask a real, total de-tagging. Revisit if the pipeline ever stops
+  stamping tags, or if a non-ad-hoc run legitimately carries zero tagged rows.
+
+- [ ] **Discriminating-test discipline for predicate narrowings.** Two tests in this
+  change passed for the wrong reason — a downstream rule (the de-tag drop) masked the
+  mutation they claimed to catch — and the plan leaned on a `grep` acceptance criterion
+  that CI never runs. Both were found by mutation-testing the suite and fixed. No
+  mechanizable guard; the durable lesson is: when a test names a narrowing, construct
+  the fixture so the row SURVIVES every other rule, or the assertion proves nothing.
+
+- [ ] **CE034 — runner-label registry + dogfood runner parity** over
+  `.github/workflows/*.yml`. Two clauses: (a) every label a job can land on must appear
+  in `.github/actionlint.yaml`'s `self-hosted-runner.labels` or a stock GitHub-hosted
+  allowlist — including *both* branches of an expression-valued `runs-on:`, which
+  actionlint treats as opaque; (b) `action-dogfood`'s label must equal the one the
+  consumer snippet in `docs/tutorials/02-ci-pipeline.md` advertises. Nothing guards
+  either today: actionlint is not wired into `make verify` or pre-commit (grep: the
+  config file is its only mention), and CE026 parses that job's prerequisite *steps*
+  but never its `runs-on:`. Why it matters: an undeclared label is not a runtime error,
+  the job queues until GitHub cancels it hours later — indistinguishable from a pool
+  outage; and a repo-wide `runs-on:` migration has twice swept up `action-dogfood`
+  (#306, then 027121e in this PR), which exists precisely to prove the published Action
+  works on the image external integrators use. Implemented and verified once (both
+  clauses caught their regression class on the real tree) but reverted as out of
+  proportion to a 16-line runner migration — ~240 lines including tests. Note when
+  writing it: discriminate labels from expression operands structurally, on the
+  preceding `&&`/`||`, NOT on the string's shape — a "contains 'ubuntu'" heuristic
+  silently fails on `uipath-ubunut-latest`, the exact transposition typo the rule is
+  for. Caught in the multi-model review of PR #86.

coder_eval-0.9.6/.github/actionlint.yaml

@@ -0,0 +1,11 @@
+# Declares this repo's custom runner labels so actionlint stops reporting them as
+# unknown. Advisory only: nothing in `make verify` runs actionlint, so this file helps
+# a local run or an editor integration and is not a gate.
+#
+# These are UiPath's centralized managed GitHub pool labels. Jobs that deliberately
+# stay on stock `ubuntu-latest` say why at their own `runs-on:`.
+self-hosted-runner:
+  labels:
+    - uipath-ubuntu-latest
+    - uipath-ubuntu-24.04
+    - uipath-windows-latest

{coder_eval-0.9.5 → coder_eval-0.9.6}/.github/workflows/claude-pr-review.yml

@@ -22,7 +22,7 @@ permissions: {}
 jobs:
   claude-review:
     name: Claude Code Review
-    runs-on: ubuntu-latest
+    runs-on: uipath-ubuntu-latest
     timeout-minutes: 15
 
     # Trusted triggers only — this repo is public.

{coder_eval-0.9.5 → coder_eval-0.9.6}/.github/workflows/codeql.yml

@@ -16,7 +16,7 @@ permissions:
 jobs:
   analyze:
     name: Analyze Python code
-    runs-on: ubuntu-latest
+    runs-on: uipath-ubuntu-latest
     timeout-minutes: 15
     permissions:
       security-events: write

{coder_eval-0.9.5 → coder_eval-0.9.6}/.github/workflows/conventional-commits.yml

@@ -16,7 +16,7 @@ permissions:
 jobs:
   check-conventional-commits:
     name: Check PR Title & Commit Messages
-    runs-on: ubuntu-latest
+    runs-on: uipath-ubuntu-latest
     timeout-minutes: 5
 
     steps:

{coder_eval-0.9.5 → coder_eval-0.9.6}/.github/workflows/docker-publish.yml

@@ -41,13 +41,13 @@ env:
 jobs:
   publish:
     name: Build and push to GHCR
-    runs-on: ubuntu-latest
+    runs-on: uipath-ubuntu-latest
     # Skip semantic-release's own commit so we don't double-build on the
     # version-bump push.
     if: "!contains(github.event.head_commit.message, 'chore(release):')"
     timeout-minutes: 30
-    env:
-      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: "openai-codex-cli-bin,openai-codex"
+    # No SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS on purpose: nothing installs on the
+    # host here, and the Docker build doesn't inherit job env (it uses its own ARG).
 
     steps:
       - name: Checkout code
@@ -65,17 +65,17 @@ jobs:
           echo "owner_lc=${OWNER_LC}"   >> "$GITHUB_OUTPUT"
 
       - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v3
+        uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f  # v3.12.0
 
       - name: Log in to GHCR
-        uses: docker/login-action@v3
+        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9  # v3.7.0
         with:
           registry: ${{ env.REGISTRY }}
           username: ${{ github.actor }}
           password: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Build and push image
-        uses: docker/build-push-action@v6
+        uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8  # v6.19.2
         with:
           context: .
           file: docker/Dockerfile

{coder_eval-0.9.5 → coder_eval-0.9.6}/.github/workflows/docs.yml

@@ -35,9 +35,12 @@ concurrency:
 
 jobs:
   publish:
-    runs-on: ubuntu-latest
+    runs-on: uipath-ubuntu-latest
+    # Without a budget, a hang in the unattended `git push --force` below would hold a
+    # shared-pool runner slot for GitHub's 6h default.
+    timeout-minutes: 10
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
 
       - name: Assemble the published tree
         run: |

{coder_eval-0.9.5 → coder_eval-0.9.6}/.github/workflows/pr-checks.yml

@@ -26,17 +26,26 @@ permissions:
 # here — TELEMETRY_ENABLED is the single canonical disable gate.
 env:
   TELEMETRY_ENABLED: "false"
+  # The `uipath-*` pool enforces a minimum package-age safe-chain check on installs.
+  # Workflow-level so every installing job inherits it; per-job copies are how some
+  # jobs previously ended up with no exclusions at all. The literal is the operative
+  # value — no secret of that name exists at repo or org level, so the bare `secrets.`
+  # reference this replaced resolved to an empty list. (Image builds carry their own
+  # list in docker/Dockerfile; deliberately not the same set.)
+  SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }}
 
 jobs:
   quality-gate:
     name: Quality Gate (Format, Lint, Type, Test, Security)
-    runs-on: ubuntu-latest
+    # Fork PRs go to stock GitHub-hosted runners: this job runs the PR's own
+    # `uv.lock` build hooks and test files, and the repo is public, so untrusted code
+    # must not land on the shared pool image. Everything else uses the pool.
+    runs-on: ${{ (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository) && 'ubuntu-latest' || 'uipath-ubuntu-latest' }}
     timeout-minutes: 10
 
-    # Shared env for all steps: safe-chain min-age exclusions and a dummy
-    # Anthropic key so CI tests that construct a client don't fail on a missing key.
+    # A dummy Anthropic key so CI tests that construct a client don't fail on a
+    # missing key.
     env:
-      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS }}
       ANTHROPIC_API_KEY: "sk-ant-test-dummy-key-for-ci-tests-only"
 
     steps:
@@ -157,20 +166,62 @@ jobs:
           echo "✅ Quality gate complete!"
           echo "📊 All checks passed: formatting, linting, types, security, tests"
 
+  evalboard:
+    # The dashboard's own gate. `evalboard/` ships ~460 vitest assertions,
+    # including the pricing drift guard that asserts lib/pricing.ts still agrees
+    # with src/coder_eval/pricing.py — and until this job existed NOTHING ran
+    # them: not a workflow, not a Makefile target, not a pre-commit hook. The
+    # guard was consequently red on `main` for weeks while a 3x-wrong Opus rate
+    # and five unpriced in-use models shipped to the board. An unrun assertion is
+    # documentation, not enforcement.
+    #
+    # Deliberately NOT path-filtered. `paths:` is workflow-scoped in GitHub
+    # Actions, and the parity guard's whole point is that a reprice in
+    # src/coder_eval/pricing.py — a pure-Python diff touching no evalboard file —
+    # must trip it. A `evalboard/**`-only filter would skip exactly the change
+    # class this job exists to catch.
+    name: Evalboard (Types, Tests, Build)
+    # Fork-PR carve-out — see `quality-gate`. `pnpm install --frozen-lockfile` runs
+    # the PR's own lockfile install scripts, same untrusted-code class.
+    runs-on: ${{ (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository) && 'ubuntu-latest' || 'uipath-ubuntu-latest' }}
+    timeout-minutes: 15
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+
+      - name: Set up Node.js 20
+        uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
+        with:
+          node-version: "20"
+
+      - name: Enable pnpm
+        # The version comes from evalboard/package.json's `packageManager` field,
+        # so corepack pins it without a second place to keep in sync.
+        run: corepack enable
+
+      - name: Install dependencies (lockfile-pinned)
+        working-directory: evalboard
+        run: pnpm install --frozen-lockfile
+
+      - name: Verify (tsc --noEmit && vitest run && next build)
+        working-directory: evalboard
+        run: pnpm verify
+
   no-uipath-extra:
     # Proves that `pip install coder-eval` (without the optional `[uipath]`
     # extra) yields a working framework: imports succeed, the criterion
     # registry validates, and the uipath-specific code paths fail with a
     # clear hint instead of an import error.
     name: No-Extra Install (uipath optional)
-    runs-on: ubuntu-latest
+    # Fork-PR carve-out — see the comment on `quality-gate` above.
+    runs-on: ${{ (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository) && 'ubuntu-latest' || 'uipath-ubuntu-latest' }}
     timeout-minutes: 5
     steps:
       - name: Checkout code
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
 
       - name: Set up Python 3.13
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405  # v6.2.0
         with:
           python-version: "3.13"
 
@@ -208,7 +259,7 @@ jobs:
 
   windows-smoke:
     name: Windows Smoke Test
-    runs-on: windows-latest
+    runs-on: uipath-windows-latest
     # 15min headroom: the smoke task itself completes in ~7min, but the
     # actions/cache post-step on Windows is slow when ``.venv`` is large.
     # We also exclude ``.venv`` from the cached paths (uv re-creates it
@@ -222,7 +273,6 @@ jobs:
         shell: bash
 
     env:
-      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS }}
       # Job-level dummy key keeps the unit-test step deterministic and isolated
       # from real-API leakage. The e2e step below overrides to Bedrock at step scope.
       ANTHROPIC_API_KEY: "sk-ant-test-dummy-key-for-ci-tests-only"
@@ -326,7 +376,7 @@ jobs:
 
   e2e-smoke:
     name: E2E Smoke Tests (Real API)
-    runs-on: ubuntu-latest
+    runs-on: uipath-ubuntu-latest
     timeout-minutes: 10
     # Skip on fork PRs where secrets aren't available
     if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
@@ -341,7 +391,6 @@ jobs:
       AWS_BEARER_TOKEN_BEDROCK: ${{ secrets.AWS_BEARER_TOKEN_BEDROCK }}
       AWS_REGION: ${{ secrets.AWS_REGION }}
       BEDROCK_MODEL: ${{ secrets.BEDROCK_MODEL }}
-      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS }}
       # tasks_run for --tags smoke-pass. 6 task files (hello_date, dataset_example,
       # smoke_llm_judge, smoke_agent_judge, byod_smoke_test, agentless_smoke_test);
       # dataset_example fans out to 2 inline rows, so 7 sub-tasks. If you add/remove a
@@ -509,7 +558,7 @@ jobs:
 
   live-tests:
     name: Live Integration Tests (Settings Enforcement + Cost Budget)
-    runs-on: ubuntu-24.04
+    runs-on: uipath-ubuntu-24.04
     timeout-minutes: 15
     # Skip on fork PRs where secrets aren't available
     if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
@@ -519,7 +568,6 @@ jobs:
       # settings-enforcement and cost-budget steps use this. The Bedrock
       # settings-enforcement step adds API_BACKEND=bedrock at step scope only.
       ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
-      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS }}
       # Bedrock backend for the third settings-enforcement run. Job-level so
       # the preflight secrets check can validate them; the BedrockRoute is
       # only engaged via API_BACKEND=bedrock at step scope.
@@ -582,10 +630,12 @@ jobs:
         # `-ra` surfaces skipped tests in the summary so CI logs show what ran vs skipped.
         # `--strict-markers` rejects unregistered @pytest.mark.* (cheap typo insurance).
         # JUnit XML feeds the post-run "tests actually passed" assertion below.
+        # `-n 4` overrides pyproject's `-n auto`: these hit the real API, so
+        # concurrency must not vary with the runner's vCPU count. 4 = the old shape.
         run: |
           mkdir -p tmp
           .venv/bin/pytest tests/test_claude_settings_enforcement_live.py \
-            -m live -v --tb=short --strict-markers -ra -n auto \
+            -m live -v --tb=short --strict-markers -ra -n 4 \
             --junit-xml=tmp/junit-settings.xml
 
       - name: Run claude-settings enforcement live tests (BedrockRoute)
@@ -597,7 +647,7 @@ jobs:
           API_BACKEND: "bedrock"
         run: |
           .venv/bin/pytest tests/test_claude_settings_enforcement_live.py \
-            -m live -v --tb=short --strict-markers -ra -n auto \
+            -m live -v --tb=short --strict-markers -ra -n 4 \
             --junit-xml=tmp/junit-settings-bedrock.xml
 
       - name: Assert live tests actually ran (not silently skipped)
@@ -680,7 +730,7 @@ jobs:
 
   codex-live-tests:
     name: Live Integration Tests (Codex)
-    runs-on: ubuntu-24.04
+    runs-on: uipath-ubuntu-24.04
     timeout-minutes: 15
     # Skip on fork PRs where secrets aren't available.
     if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
@@ -692,7 +742,6 @@ jobs:
       CODEX_API_KEY: ${{ secrets.CODEX_API_KEY }}
       CODEX_BASE_URL: ${{ secrets.CODEX_BASE_URL }}
       CODEX_MODEL: ${{ secrets.CODEX_MODEL }}
-      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS }}
 
     steps:
       - name: Checkout code
@@ -756,7 +805,7 @@ jobs:
 
   byoa-live-tests:
     name: Live Integration Tests (BYOA Plugin)
-    runs-on: ubuntu-24.04
+    runs-on: uipath-ubuntu-24.04
     timeout-minutes: 15
     # Skip on fork PRs where secrets aren't available.
     if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
@@ -764,7 +813,6 @@ jobs:
     env:
       # DirectRoute: a plugin agent that subclasses ClaudeCodeAgent uses ANTHROPIC_API_KEY.
       ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
-      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS }}
 
     steps:
       - name: Checkout code
@@ -834,6 +882,11 @@ jobs:
 
   action-dogfood:
     name: Action Dogfood (composite action, real API)
+    # Deliberately NOT on the `uipath-*` pool, and keep it that way: this job is the
+    # executable proof of the published Action, and docs/tutorials/02-ci-pipeline.md
+    # (the one consumer snippet naming a runner) says `ubuntu-latest`. Nothing else in
+    # CI exercises the image integrators actually use. A bulk `runs-on:` migration has
+    # swept this up twice — check it by hand.
     runs-on: ubuntu-latest
     timeout-minutes: 15
     # Skip on fork PRs where secrets aren't available (matches e2e-smoke).

{coder_eval-0.9.5 → coder_eval-0.9.6}/.github/workflows/publish-testpypi.yml

@@ -34,8 +34,11 @@ permissions:
 jobs:
   publish-testpypi:
     name: Build and publish to TestPyPI
-    runs-on: ubuntu-latest
+    runs-on: uipath-ubuntu-latest
     timeout-minutes: 10
+    env:
+      # `uv build` resolves build deps under the pool's safe-chain gate.
+      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }}
     environment:
       name: testpypi
       url: https://test.pypi.org/project/coder-eval/

{coder_eval-0.9.5 → coder_eval-0.9.6}/.github/workflows/release.yml

@@ -49,21 +49,22 @@ permissions:
 jobs:
   release:
     name: Bump version and publish
-    # GitHub-hosted so cutting a release does not depend on the self-hosted
-    # `uipath-ubuntu-latest` pool. semantic-release, uv, twine, and the docker
-    # buildx -> GHCR push all run fine here.
-    runs-on: ubuntu-latest
+    # KNOWINGLY reverses PR #6, which moved this job to a GitHub-hosted runner because
+    # "cutting a release was blocked whenever that pool was unavailable". The pool's
+    # availability is what changed; the accepted trade-off is that the release path has
+    # no GitHub-hosted fallback again. Both jobs here are `workflow_dispatch`-only, so
+    # no PR check exercises them -- dry-run publish-testpypi.yml after editing this file.
+    runs-on: uipath-ubuntu-latest
     timeout-minutes: 15
     outputs:
       # Exposed so the downstream publish-pypi job gates on a version having been
       # produced (real release on main, or a stamped prerelease on a branch).
       version: ${{ steps.ver.outputs.version }}
     env:
-      # The self-hosted `uipath-ubuntu-latest` runners enforce a minimum
-      # package-age safe-chain check on uv installs; on GitHub-hosted runners
-      # this is a no-op. Kept (matching pr-checks.yml) so parity is preserved
-      # if the job ever moves back to the self-hosted pool.
-      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: "openai-codex-cli-bin,openai-codex"
+      # Load-bearing on the release path: the pool enforces a package-age safe-chain
+      # check on uv installs. Same expression as pr-checks.yml (see the comment there),
+      # so a package can't pass PR CI and then fail the release install.
+      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }}
 
     steps:
       # Only a real release (main) needs the app token: semantic-release pushes the
@@ -409,14 +410,14 @@ jobs:
           cache-from: type=registry,ref=ghcr.io/${{ steps.img.outputs.owner_lc }}/coder-eval-agent:buildcache
 
   # Publish the wheel+sdist to public PyPI. This runs as its own job so OIDC
-  # Trusted Publishing is scoped to a dedicated, environment-gated context on
-  # GitHub-hosted runners -- no PyPI token/secret is stored. Gated on the
-  # release job having actually cut a version.
+  # Trusted Publishing is scoped to a dedicated, environment-gated context --
+  # no PyPI token/secret is stored. Gated on the release job having actually
+  # cut a version.
   publish-pypi:
     name: Publish to PyPI
     needs: release
     if: needs.release.outputs.version != ''
-    runs-on: ubuntu-latest
+    runs-on: uipath-ubuntu-latest
     timeout-minutes: 10
     environment:
       name: pypi

{coder_eval-0.9.5 → coder_eval-0.9.6}/CHANGELOG.md

@@ -2,6 +2,80 @@
 
 <!-- version list -->
 
+## v0.9.6 (2026-08-11)
+
+### Bug Fixes
+
+- Code review fixes for evalboard path-to-ga stale tags and mature passes
+  ([#94](https://github.com/UiPath/coder_eval/pull/94),
+  [`ce006c1`](https://github.com/UiPath/coder_eval/commit/ce006c19b594701e90aec1d63cae15b337c0e06d))
+
+- Review round 2 — gate evalboard in CI, correct GPT-5.6 rates
+  ([#94](https://github.com/UiPath/coder_eval/pull/94),
+  [`ce006c1`](https://github.com/UiPath/coder_eval/commit/ce006c19b594701e90aec1d63cae15b337c0e06d))
+
+- **ci**: Correct defects in the uipath runner migration
+  ([#86](https://github.com/UiPath/coder_eval/pull/86),
+  [`57556af`](https://github.com/UiPath/coder_eval/commit/57556af6e47bffcd5fbbb708b9f8b399cceea4c7))
+
+- **criteria**: Make glob path resolution literal-first and ignore-filtered
+  ([#65](https://github.com/UiPath/coder_eval/pull/65),
+  [`b3bba2b`](https://github.com/UiPath/coder_eval/commit/b3bba2b6c00e3772351b28bf63d5cde9e97b8e7e))
+
+- **criteria**: Split clustered short flags and keep negative numbers positional
+  ([#73](https://github.com/UiPath/coder_eval/pull/73),
+  [`a7ec3ea`](https://github.com/UiPath/coder_eval/commit/a7ec3eababc907a0220a598d974540eb579a538c))
+
+- **evalboard**: 1/3 — drop de-tagged tasks and score only executed runs
+  ([#94](https://github.com/UiPath/coder_eval/pull/94),
+  [`ce006c1`](https://github.com/UiPath/coder_eval/commit/ce006c19b594701e90aec1d63cae15b337c0e06d))
+
+- **evalboard**: Path-to-GA shows only still-tagged tasks, scored on runs that executed
+  ([#94](https://github.com/UiPath/coder_eval/pull/94),
+  [`ce006c1`](https://github.com/UiPath/coder_eval/commit/ce006c19b594701e90aec1d63cae15b337c0e06d))
+
+- **evalboard**: Resync lib/pricing.ts with the authoritative pricing.py table
+  ([#94](https://github.com/UiPath/coder_eval/pull/94),
+  [`ce006c1`](https://github.com/UiPath/coder_eval/commit/ce006c19b594701e90aec1d63cae15b337c0e06d))
+
+- **sandbox**: Close the remaining record_cli findings from #73
+  ([#73](https://github.com/UiPath/coder_eval/pull/73),
+  [`a7ec3ea`](https://github.com/UiPath/coder_eval/commit/a7ec3eababc907a0220a598d974540eb579a538c))
+
+- **sandbox**: Repair record_cli defects found reviewing #73
+  ([#73](https://github.com/UiPath/coder_eval/pull/73),
+  [`a7ec3ea`](https://github.com/UiPath/coder_eval/commit/a7ec3eababc907a0220a598d974540eb579a538c))
+
+- **sandbox**: Stop the recorder dir defeating the PLUGIN_TOOLS_DIR pin
+  ([#73](https://github.com/UiPath/coder_eval/pull/73),
+  [`a7ec3ea`](https://github.com/UiPath/coder_eval/commit/a7ec3eababc907a0220a598d974540eb579a538c))
+
+### Chores
+
+- Change to centralized managed GitHub pool ([#86](https://github.com/UiPath/coder_eval/pull/86),
+  [`57556af`](https://github.com/UiPath/coder_eval/commit/57556af6e47bffcd5fbbb708b9f8b399cceea4c7))
+
+### Documentation
+
+- **harness**: Defer four TS-side evalboard invariants from the path-to-ga fix
+  ([#94](https://github.com/UiPath/coder_eval/pull/94),
+  [`ce006c1`](https://github.com/UiPath/coder_eval/commit/ce006c19b594701e90aec1d63cae15b337c0e06d))
+
+### Features
+
+- **criteria**: Accept glob patterns in criterion path fields
+  ([#65](https://github.com/UiPath/coder_eval/pull/65),
+  [`b3bba2b`](https://github.com/UiPath/coder_eval/commit/b3bba2b6c00e3772351b28bf63d5cde9e97b8e7e))
+
+- **evalboard**: 2/3 — surface last-seen and maturity on the Path-to-GA table
+  ([#94](https://github.com/UiPath/coder_eval/pull/94),
+  [`ce006c1`](https://github.com/UiPath/coder_eval/commit/ce006c19b594701e90aec1d63cae15b337c0e06d))
+
+- **sandbox**: Generate CLI recording shims via record_cli
+  ([#73](https://github.com/UiPath/coder_eval/pull/73),
+  [`a7ec3ea`](https://github.com/UiPath/coder_eval/commit/a7ec3eababc907a0220a598d974540eb579a538c))
+
+
 ## v0.9.5 (2026-08-05)
 
 ### Bug Fixes

{coder_eval-0.9.5 → coder_eval-0.9.6}/CLAUDE.md

@@ -199,8 +199,14 @@ make typecheck   # pyright
 make test        # pytest
 make lint        # custom architectural lint rules (CE001+)
 make verify      # All of the above + coverage check (CI equivalent)
+
+# The JS half (evalboard/). Separate because it needs a Node/pnpm toolchain,
+# but gated in CI by the `evalboard` job just like the Python side.
+make evalboard-verify   # tsc --noEmit + vitest + next build
 ```
 
+Editing `src/coder_eval/pricing.py` means editing `evalboard/lib/pricing.ts` too — it is a hand-copied mirror, and `evalboard/lib/__tests__/pricing-parity.test.ts` fails the build on drift in either direction.
+
 When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time — CE026–CE031 — are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name. CE026 keeps the GitHub Action's three onboarding surfaces honest: a page's *first* Action snippet must show the agent-runtime prerequisite steps (pinned to the `action-dogfood` job that proves them in CI), a zero-install absolute next to such a snippet must name the channel it means, and every `github.com/marketplace/actions/<slug>` link plus the shields badge label must match `action.yml`'s `name:`.)
 
 Adding a user-facing field to one of the models CE030 tracks (`TaskDefinition`, `RunLimits`, `Dataset`, `SimulationConfig` — see `tests/lint/doc_schema_parity.py`) means documenting it in its guide (mention the field name as inline code) or adding an `EXEMPT` entry with a reason it is not user-authored. `make lint` fails otherwise.

{coder_eval-0.9.5 → coder_eval-0.9.6}/CONTRIBUTING.md

@@ -96,6 +96,25 @@ after the type/scope (or a `BREAKING CHANGE:` footer) marks a breaking change.
 Keep PRs small and single-purpose where possible — it makes review faster and
 bisection easier.
 
+### CI runners
+
+Workflows run on UiPath's centralized managed GitHub pool, whose labels are listed in
+[`.github/actionlint.yaml`](.github/actionlint.yaml). Nothing enforces that list, so
+check a new label against it by hand — an unknown label is not a build error, the job
+just queues until GitHub cancels it.
+
+That pool enforces a minimum package-age safe-chain check on installs, so jobs that
+install dependencies set `SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS`. The literal in
+that expression is the operative value — no secret of that name exists at repo or org
+level. `pr-checks.yml` sets it once at the **workflow** level; don't add per-job copies.
+
+Some jobs deliberately use stock `ubuntu-latest`, each explained at its `runs-on:`:
+jobs that execute PR-supplied code (`quality-gate`, `no-uipath-extra`, `evalboard`)
+fall back to it **for fork PRs only**, since this repo is public and untrusted code
+should not run on the shared pool image — any new job running PR-supplied code needs
+the same carve-out. `action-dogfood` always uses it, because it is the executable proof
+behind the published Action and must exercise the image integrators actually use.
+
 ## Adding Tasks or Criteria
 
 - Task YAMLs live in `tasks/`; see

{coder_eval-0.9.5 → coder_eval-0.9.6}/Makefile

@@ -1,4 +1,4 @@
-.PHONY: help install format check typecheck test test-live test-smoke verify verify-noextra clean run lint docs-indexes docker-image docker-image-full coder-eval-runtime docker-images
+.PHONY: help install format check typecheck test test-live test-smoke verify verify-noextra evalboard-verify clean run lint docs-indexes docker-image docker-image-full coder-eval-runtime docker-images
 
 # Single source of the installed coder-eval version (used to tag the docker
 # images). Referenced lazily inside the docker recipes, so it doesn't run on
@@ -56,6 +56,14 @@ verify:  ## Run all verification steps (CI equivalent)
 	# uv run bandit -r src/ -ll --format json -o bandit-report.json
 	uv run pytest tests/ -n auto -m "not live and not lint" --cov=coder_eval --cov-report=term-missing --cov-report=xml --cov-fail-under=80
 
+evalboard-verify:  ## Run the evalboard (Next.js dashboard) checks: tsc + vitest + next build
+	# The JS half of the repo. Not folded into `make verify` because it needs a
+	# Node/pnpm toolchain a Python-only contributor may not have — but it IS
+	# gated in CI by the `evalboard` job, so a red run here is a red PR.
+	# Includes the pricing drift guard: a reprice in src/coder_eval/pricing.py
+	# without the matching edit to evalboard/lib/pricing.ts fails right here.
+	cd evalboard && pnpm install --frozen-lockfile && pnpm verify
+
 verify-noextra:  ## Verify the framework works without the optional [uipath] extra
 	# Build a throwaway venv that has ONLY the [dev] extra (no [uipath]); confirms
 	# `pip install coder-eval` (without the extra) is functional. Mirrors the

{coder_eval-0.9.5 → coder_eval-0.9.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: coder-eval
-Version: 0.9.5
+Version: 0.9.6
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval

{coder_eval-0.9.5 → coder_eval-0.9.6}/action.yml

@@ -41,7 +41,7 @@ inputs:
   version:
     description: coder-eval version to install from PyPI, or "local" to install from the action checkout
     required: false
-    default: "0.9.5"   # <-- kept in sync with releases by release.yml
+    default: "0.9.6"   # <-- kept in sync with releases by release.yml
   run-dir:
     description: Run directory (--run-dir)
     required: false