coder-eval 0.9.4__tar.gz → 0.9.6__tar.gz

{coder_eval-0.9.4 → coder_eval-0.9.6}/.claude/harness-candidates.md

@@ -107,3 +107,102 @@ Deferred lint/test guardrails surfaced during reviews. Promote to a `CExxx` rule
   caught them. The cleanup plan explicitly deferred this as YAGNI for the
   one-time purge, but any future doc rename/deletion re-opens the same blind
   spot — caught in the 2026-07-03 open-source-docs-cleanup implementation run.
+
+## From PR #77 (command-executed shell-normalize) — CE030-to-criteria deferred
+
+- [ ] **Extend CE030 doc/schema-parity to the `SuccessCriterion` union** so a new
+  criterion (or field) can't ship undocumented. Attempted in PR #77 and reverted:
+  CI installs `--extra uipath`, and in that environment `coder_eval.models.criteria`
+  gains a `CliCalledCriterion` (fields `log`/`positional`) that is NOT present in a
+  plain checkout (it did not reproduce on macOS, whose lockfile resolution omits the
+  contributing linux-only component). It defeated every discriminator tried — union
+  membership, a `__module__` string filter (it is spoofed to `coder_eval.models.criteria`),
+  a genuine-module-attribute scan (it is `setattr` onto the module), and even an AST
+  parse of the `SuccessCriterion` union literal in `criteria.py` source (CI's imported
+  criteria module resolves to a file whose union literal already contains it). No
+  runtime OR source signal available in the lint could separate the injected criterion
+  from an in-tree one. Revisit only with a way to identify the in-tree criterion set that
+  is provably immune to the uipath integration — e.g. a hardcoded name allowlist of the
+  in-tree criteria (losing auto-coverage of new ones), or first understanding exactly how
+  that environment injects the criterion. Until then CE030 stays scoped to the four
+  top-level models; the `command_pattern`/`exclude_pattern` contract this PR changed is
+  documented in the Field descriptions and TASK_DEFINITION_GUIDE regardless.
+
+## From the evalboard Path-to-GA de-tag / mature-passes fix (4e5bbc4…dd5f7e9) — TS-side guards deferred
+
+Context: the CExxx harness is a **Python** AST runner over `src/coder_eval/`, so none
+of the invariants below are mechanizable in it. Each would need a TypeScript lint
+harness (eslint config + custom rules) that `evalboard/` does not have today —
+standing one up for three call sites fails the KISS/YAGNI gate. Deferring rather
+than dropping; promote if a fourth TS-side invariant appears, and stand up the
+harness once for all of them.
+
+> **Update (PR #94 review round 2).** The *execution* half of this gap is closed:
+> `evalboard/` is now gated by the `evalboard` job in `.github/workflows/pr-checks.yml`
+> and reachable locally via `make evalboard-verify`, so the vitest suite (including
+> the pricing drift guard) is enforcement rather than documentation. What remains
+> deferred below is the *static-analysis* half — eslint has still not been stood up.
+> The review that prompted this round names four more candidate TS rules (raw
+> `status === "SUCCESS"` outside `lib/status.ts`; DOM-global shadowing in props;
+> inline copies of the tag predicate; per-run tooltip copy reused on aggregate
+> surfaces), which meets the "fourth invariant" promotion bar stated above —
+> **stand up eslint next time `evalboard/` is touched substantively.**
+
+- [ ] **"Every consumer of `RunOverviewTask.matureSkipped` must decide explicitly
+  whether a carry-forward row counts."** Four consumers now, and they deliberately
+  DISAGREE: `lib/trends.ts` and `app/runs/[id]/run-view.tsx` count a mature skip as
+  a pass; `lib/overview.ts::buildTagTaskRows` excludes it from both terms
+  (`/path-to-ga` is a GA-readiness page). A new consumer silently inheriting either
+  convention is a real hazard. Guard shape: flag a file that reads `.matureSkipped`
+  without a nearby comment naming its convention — weak, hence the deferral. Closed
+  for now by unit tests that assert the exclusion from BOTH numerator and denominator
+  (`lib/__tests__/overview.test.ts` → `describe("buildTagTaskRows")`).
+
+- [x] ~~**`taskCarriesRepoTag` is the single repo-provenance tag predicate — but one
+  duplicate survives.**~~ **RESOLVED in PR #94 review round 2.** The predicate moved to
+  a dependency-free `lib/tags.ts` (structurally typed on `{skill, tags}` so
+  `RunOverviewTask`, `TaskResultSummary` and `TaskTrend` all satisfy it), re-exported
+  from `lib/overview.ts` for existing callers. Both inline copies now import it:
+  `app/runs/[id]/run-view.tsx` (the `"use client"` one that could not before) and
+  `lib/trends.ts::trendMatchesTag` (a third copy the original deferral missed).
+  Still worth a lint rule ("no inline `tags.includes(x) || skill === x`") to catch
+  future copies — folded into the eslint promotion noted above.
+
+- [ ] **The de-tag rule fails CLOSED on a newest run that loads fine but stamps no
+  `tags`** (`lib/overview.ts::buildTagTaskRows`): every tagged task would read as
+  de-tagged and the table would empty, rendering an empty state indistinguishable from
+  a genuine full de-tagging. Its sibling failure mode (`overview == null`, a transient
+  blob read failure) IS guarded, with exactly this rationale. Currently unreachable —
+  0 of ~116k date-shaped non-ad-hoc task rows in `runs-remote/` lack `tags`, and the
+  six zero-tag runs found are all ad-hoc (filtered upstream by id shape + `meta.adhoc`)
+  — so the barrier is two upstream filters rather than a check at the seam. Left
+  unguarded on purpose: a `if (taggedInRun.size === 0) skip the de-tag signal` guard
+  would also mask a real, total de-tagging. Revisit if the pipeline ever stops
+  stamping tags, or if a non-ad-hoc run legitimately carries zero tagged rows.
+
+- [ ] **Discriminating-test discipline for predicate narrowings.** Two tests in this
+  change passed for the wrong reason — a downstream rule (the de-tag drop) masked the
+  mutation they claimed to catch — and the plan leaned on a `grep` acceptance criterion
+  that CI never runs. Both were found by mutation-testing the suite and fixed. No
+  mechanizable guard; the durable lesson is: when a test names a narrowing, construct
+  the fixture so the row SURVIVES every other rule, or the assertion proves nothing.
+
+- [ ] **CE034 — runner-label registry + dogfood runner parity** over
+  `.github/workflows/*.yml`. Two clauses: (a) every label a job can land on must appear
+  in `.github/actionlint.yaml`'s `self-hosted-runner.labels` or a stock GitHub-hosted
+  allowlist — including *both* branches of an expression-valued `runs-on:`, which
+  actionlint treats as opaque; (b) `action-dogfood`'s label must equal the one the
+  consumer snippet in `docs/tutorials/02-ci-pipeline.md` advertises. Nothing guards
+  either today: actionlint is not wired into `make verify` or pre-commit (grep: the
+  config file is its only mention), and CE026 parses that job's prerequisite *steps*
+  but never its `runs-on:`. Why it matters: an undeclared label is not a runtime error,
+  the job queues until GitHub cancels it hours later — indistinguishable from a pool
+  outage; and a repo-wide `runs-on:` migration has twice swept up `action-dogfood`
+  (#306, then 027121e in this PR), which exists precisely to prove the published Action
+  works on the image external integrators use. Implemented and verified once (both
+  clauses caught their regression class on the real tree) but reverted as out of
+  proportion to a 16-line runner migration — ~240 lines including tests. Note when
+  writing it: discriminate labels from expression operands structurally, on the
+  preceding `&&`/`||`, NOT on the string's shape — a "contains 'ubuntu'" heuristic
+  silently fails on `uipath-ubunut-latest`, the exact transposition typo the rule is
+  for. Caught in the multi-model review of PR #86.

coder_eval-0.9.6/.github/actionlint.yaml

@@ -0,0 +1,11 @@
+# Declares this repo's custom runner labels so actionlint stops reporting them as
+# unknown. Advisory only: nothing in `make verify` runs actionlint, so this file helps
+# a local run or an editor integration and is not a gate.
+#
+# These are UiPath's centralized managed GitHub pool labels. Jobs that deliberately
+# stay on stock `ubuntu-latest` say why at their own `runs-on:`.
+self-hosted-runner:
+  labels:
+    - uipath-ubuntu-latest
+    - uipath-ubuntu-24.04
+    - uipath-windows-latest

{coder_eval-0.9.4 → coder_eval-0.9.6}/.github/workflows/claude-pr-review.yml

@@ -22,7 +22,7 @@ permissions: {}
 jobs:
   claude-review:
     name: Claude Code Review
-    runs-on: ubuntu-latest
+    runs-on: uipath-ubuntu-latest
     timeout-minutes: 15
 
     # Trusted triggers only — this repo is public.

{coder_eval-0.9.4 → coder_eval-0.9.6}/.github/workflows/codeql.yml

@@ -16,7 +16,7 @@ permissions:
 jobs:
   analyze:
     name: Analyze Python code
-    runs-on: ubuntu-latest
+    runs-on: uipath-ubuntu-latest
     timeout-minutes: 15
     permissions:
       security-events: write

{coder_eval-0.9.4 → coder_eval-0.9.6}/.github/workflows/conventional-commits.yml

@@ -16,7 +16,7 @@ permissions:
 jobs:
   check-conventional-commits:
     name: Check PR Title & Commit Messages
-    runs-on: ubuntu-latest
+    runs-on: uipath-ubuntu-latest
     timeout-minutes: 5
 
     steps:

{coder_eval-0.9.4 → coder_eval-0.9.6}/.github/workflows/docker-publish.yml

@@ -41,13 +41,13 @@ env:
 jobs:
   publish:
     name: Build and push to GHCR
-    runs-on: ubuntu-latest
+    runs-on: uipath-ubuntu-latest
     # Skip semantic-release's own commit so we don't double-build on the
     # version-bump push.
     if: "!contains(github.event.head_commit.message, 'chore(release):')"
     timeout-minutes: 30
-    env:
-      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: "openai-codex-cli-bin,openai-codex"
+    # No SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS on purpose: nothing installs on the
+    # host here, and the Docker build doesn't inherit job env (it uses its own ARG).
 
     steps:
       - name: Checkout code
@@ -65,17 +65,17 @@ jobs:
           echo "owner_lc=${OWNER_LC}"   >> "$GITHUB_OUTPUT"
 
       - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v3
+        uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f  # v3.12.0
 
       - name: Log in to GHCR
-        uses: docker/login-action@v3
+        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9  # v3.7.0
         with:
           registry: ${{ env.REGISTRY }}
           username: ${{ github.actor }}
           password: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Build and push image
-        uses: docker/build-push-action@v6
+        uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8  # v6.19.2
         with:
           context: .
           file: docker/Dockerfile

{coder_eval-0.9.4 → coder_eval-0.9.6}/.github/workflows/docs.yml

@@ -35,9 +35,12 @@ concurrency:
 
 jobs:
   publish:
-    runs-on: ubuntu-latest
+    runs-on: uipath-ubuntu-latest
+    # Without a budget, a hang in the unattended `git push --force` below would hold a
+    # shared-pool runner slot for GitHub's 6h default.
+    timeout-minutes: 10
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
 
       - name: Assemble the published tree
         run: |

{coder_eval-0.9.4 → coder_eval-0.9.6}/.github/workflows/pr-checks.yml

@@ -26,17 +26,26 @@ permissions:
 # here — TELEMETRY_ENABLED is the single canonical disable gate.
 env:
   TELEMETRY_ENABLED: "false"
+  # The `uipath-*` pool enforces a minimum package-age safe-chain check on installs.
+  # Workflow-level so every installing job inherits it; per-job copies are how some
+  # jobs previously ended up with no exclusions at all. The literal is the operative
+  # value — no secret of that name exists at repo or org level, so the bare `secrets.`
+  # reference this replaced resolved to an empty list. (Image builds carry their own
+  # list in docker/Dockerfile; deliberately not the same set.)
+  SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }}
 
 jobs:
   quality-gate:
     name: Quality Gate (Format, Lint, Type, Test, Security)
-    runs-on: ubuntu-latest
+    # Fork PRs go to stock GitHub-hosted runners: this job runs the PR's own
+    # `uv.lock` build hooks and test files, and the repo is public, so untrusted code
+    # must not land on the shared pool image. Everything else uses the pool.
+    runs-on: ${{ (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository) && 'ubuntu-latest' || 'uipath-ubuntu-latest' }}
     timeout-minutes: 10
 
-    # Shared env for all steps: safe-chain min-age exclusions and a dummy
-    # Anthropic key so CI tests that construct a client don't fail on a missing key.
+    # A dummy Anthropic key so CI tests that construct a client don't fail on a
+    # missing key.
     env:
-      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS }}
       ANTHROPIC_API_KEY: "sk-ant-test-dummy-key-for-ci-tests-only"
 
     steps:
@@ -157,20 +166,62 @@ jobs:
           echo "✅ Quality gate complete!"
           echo "📊 All checks passed: formatting, linting, types, security, tests"
 
+  evalboard:
+    # The dashboard's own gate. `evalboard/` ships ~460 vitest assertions,
+    # including the pricing drift guard that asserts lib/pricing.ts still agrees
+    # with src/coder_eval/pricing.py — and until this job existed NOTHING ran
+    # them: not a workflow, not a Makefile target, not a pre-commit hook. The
+    # guard was consequently red on `main` for weeks while a 3x-wrong Opus rate
+    # and five unpriced in-use models shipped to the board. An unrun assertion is
+    # documentation, not enforcement.
+    #
+    # Deliberately NOT path-filtered. `paths:` is workflow-scoped in GitHub
+    # Actions, and the parity guard's whole point is that a reprice in
+    # src/coder_eval/pricing.py — a pure-Python diff touching no evalboard file —
+    # must trip it. A `evalboard/**`-only filter would skip exactly the change
+    # class this job exists to catch.
+    name: Evalboard (Types, Tests, Build)
+    # Fork-PR carve-out — see `quality-gate`. `pnpm install --frozen-lockfile` runs
+    # the PR's own lockfile install scripts, same untrusted-code class.
+    runs-on: ${{ (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository) && 'ubuntu-latest' || 'uipath-ubuntu-latest' }}
+    timeout-minutes: 15
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+
+      - name: Set up Node.js 20
+        uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
+        with:
+          node-version: "20"
+
+      - name: Enable pnpm
+        # The version comes from evalboard/package.json's `packageManager` field,
+        # so corepack pins it without a second place to keep in sync.
+        run: corepack enable
+
+      - name: Install dependencies (lockfile-pinned)
+        working-directory: evalboard
+        run: pnpm install --frozen-lockfile
+
+      - name: Verify (tsc --noEmit && vitest run && next build)
+        working-directory: evalboard
+        run: pnpm verify
+
   no-uipath-extra:
     # Proves that `pip install coder-eval` (without the optional `[uipath]`
     # extra) yields a working framework: imports succeed, the criterion
     # registry validates, and the uipath-specific code paths fail with a
     # clear hint instead of an import error.
     name: No-Extra Install (uipath optional)
-    runs-on: ubuntu-latest
+    # Fork-PR carve-out — see the comment on `quality-gate` above.
+    runs-on: ${{ (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository) && 'ubuntu-latest' || 'uipath-ubuntu-latest' }}
     timeout-minutes: 5
     steps:
       - name: Checkout code
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
 
       - name: Set up Python 3.13
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405  # v6.2.0
         with:
           python-version: "3.13"
 
@@ -208,7 +259,7 @@ jobs:
 
   windows-smoke:
     name: Windows Smoke Test
-    runs-on: windows-latest
+    runs-on: uipath-windows-latest
     # 15min headroom: the smoke task itself completes in ~7min, but the
     # actions/cache post-step on Windows is slow when ``.venv`` is large.
     # We also exclude ``.venv`` from the cached paths (uv re-creates it
@@ -222,7 +273,6 @@ jobs:
         shell: bash
 
     env:
-      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS }}
       # Job-level dummy key keeps the unit-test step deterministic and isolated
       # from real-API leakage. The e2e step below overrides to Bedrock at step scope.
       ANTHROPIC_API_KEY: "sk-ant-test-dummy-key-for-ci-tests-only"
@@ -326,7 +376,7 @@ jobs:
 
   e2e-smoke:
     name: E2E Smoke Tests (Real API)
-    runs-on: ubuntu-latest
+    runs-on: uipath-ubuntu-latest
     timeout-minutes: 10
     # Skip on fork PRs where secrets aren't available
     if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
@@ -341,7 +391,6 @@ jobs:
       AWS_BEARER_TOKEN_BEDROCK: ${{ secrets.AWS_BEARER_TOKEN_BEDROCK }}
       AWS_REGION: ${{ secrets.AWS_REGION }}
       BEDROCK_MODEL: ${{ secrets.BEDROCK_MODEL }}
-      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS }}
       # tasks_run for --tags smoke-pass. 6 task files (hello_date, dataset_example,
       # smoke_llm_judge, smoke_agent_judge, byod_smoke_test, agentless_smoke_test);
       # dataset_example fans out to 2 inline rows, so 7 sub-tasks. If you add/remove a
@@ -509,7 +558,7 @@ jobs:
 
   live-tests:
     name: Live Integration Tests (Settings Enforcement + Cost Budget)
-    runs-on: ubuntu-24.04
+    runs-on: uipath-ubuntu-24.04
     timeout-minutes: 15
     # Skip on fork PRs where secrets aren't available
     if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
@@ -519,7 +568,6 @@ jobs:
       # settings-enforcement and cost-budget steps use this. The Bedrock
       # settings-enforcement step adds API_BACKEND=bedrock at step scope only.
       ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
-      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS }}
       # Bedrock backend for the third settings-enforcement run. Job-level so
       # the preflight secrets check can validate them; the BedrockRoute is
       # only engaged via API_BACKEND=bedrock at step scope.
@@ -582,10 +630,12 @@ jobs:
         # `-ra` surfaces skipped tests in the summary so CI logs show what ran vs skipped.
         # `--strict-markers` rejects unregistered @pytest.mark.* (cheap typo insurance).
         # JUnit XML feeds the post-run "tests actually passed" assertion below.
+        # `-n 4` overrides pyproject's `-n auto`: these hit the real API, so
+        # concurrency must not vary with the runner's vCPU count. 4 = the old shape.
         run: |
           mkdir -p tmp
           .venv/bin/pytest tests/test_claude_settings_enforcement_live.py \
-            -m live -v --tb=short --strict-markers -ra -n auto \
+            -m live -v --tb=short --strict-markers -ra -n 4 \
             --junit-xml=tmp/junit-settings.xml
 
       - name: Run claude-settings enforcement live tests (BedrockRoute)
@@ -597,7 +647,7 @@ jobs:
           API_BACKEND: "bedrock"
         run: |
           .venv/bin/pytest tests/test_claude_settings_enforcement_live.py \
-            -m live -v --tb=short --strict-markers -ra -n auto \
+            -m live -v --tb=short --strict-markers -ra -n 4 \
             --junit-xml=tmp/junit-settings-bedrock.xml
 
       - name: Assert live tests actually ran (not silently skipped)
@@ -680,7 +730,7 @@ jobs:
 
   codex-live-tests:
     name: Live Integration Tests (Codex)
-    runs-on: ubuntu-24.04
+    runs-on: uipath-ubuntu-24.04
     timeout-minutes: 15
     # Skip on fork PRs where secrets aren't available.
     if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
@@ -692,7 +742,6 @@ jobs:
       CODEX_API_KEY: ${{ secrets.CODEX_API_KEY }}
       CODEX_BASE_URL: ${{ secrets.CODEX_BASE_URL }}
       CODEX_MODEL: ${{ secrets.CODEX_MODEL }}
-      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS }}
 
     steps:
       - name: Checkout code
@@ -756,7 +805,7 @@ jobs:
 
   byoa-live-tests:
     name: Live Integration Tests (BYOA Plugin)
-    runs-on: ubuntu-24.04
+    runs-on: uipath-ubuntu-24.04
     timeout-minutes: 15
     # Skip on fork PRs where secrets aren't available.
     if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
@@ -764,7 +813,6 @@ jobs:
     env:
       # DirectRoute: a plugin agent that subclasses ClaudeCodeAgent uses ANTHROPIC_API_KEY.
       ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
-      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS }}
 
     steps:
       - name: Checkout code
@@ -834,6 +882,11 @@ jobs:
 
   action-dogfood:
     name: Action Dogfood (composite action, real API)
+    # Deliberately NOT on the `uipath-*` pool, and keep it that way: this job is the
+    # executable proof of the published Action, and docs/tutorials/02-ci-pipeline.md
+    # (the one consumer snippet naming a runner) says `ubuntu-latest`. Nothing else in
+    # CI exercises the image integrators actually use. A bulk `runs-on:` migration has
+    # swept this up twice — check it by hand.
     runs-on: ubuntu-latest
     timeout-minutes: 15
     # Skip on fork PRs where secrets aren't available (matches e2e-smoke).

{coder_eval-0.9.4 → coder_eval-0.9.6}/.github/workflows/publish-testpypi.yml

@@ -34,8 +34,11 @@ permissions:
 jobs:
   publish-testpypi:
     name: Build and publish to TestPyPI
-    runs-on: ubuntu-latest
+    runs-on: uipath-ubuntu-latest
     timeout-minutes: 10
+    env:
+      # `uv build` resolves build deps under the pool's safe-chain gate.
+      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }}
     environment:
       name: testpypi
       url: https://test.pypi.org/project/coder-eval/

{coder_eval-0.9.4 → coder_eval-0.9.6}/.github/workflows/release.yml

@@ -49,21 +49,22 @@ permissions:
 jobs:
   release:
     name: Bump version and publish
-    # GitHub-hosted so cutting a release does not depend on the self-hosted
-    # `uipath-ubuntu-latest` pool. semantic-release, uv, twine, and the docker
-    # buildx -> GHCR push all run fine here.
-    runs-on: ubuntu-latest
+    # KNOWINGLY reverses PR #6, which moved this job to a GitHub-hosted runner because
+    # "cutting a release was blocked whenever that pool was unavailable". The pool's
+    # availability is what changed; the accepted trade-off is that the release path has
+    # no GitHub-hosted fallback again. Both jobs here are `workflow_dispatch`-only, so
+    # no PR check exercises them -- dry-run publish-testpypi.yml after editing this file.
+    runs-on: uipath-ubuntu-latest
     timeout-minutes: 15
     outputs:
       # Exposed so the downstream publish-pypi job gates on a version having been
       # produced (real release on main, or a stamped prerelease on a branch).
       version: ${{ steps.ver.outputs.version }}
     env:
-      # The self-hosted `uipath-ubuntu-latest` runners enforce a minimum
-      # package-age safe-chain check on uv installs; on GitHub-hosted runners
-      # this is a no-op. Kept (matching pr-checks.yml) so parity is preserved
-      # if the job ever moves back to the self-hosted pool.
-      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: "openai-codex-cli-bin,openai-codex"
+      # Load-bearing on the release path: the pool enforces a package-age safe-chain
+      # check on uv installs. Same expression as pr-checks.yml (see the comment there),
+      # so a package can't pass PR CI and then fail the release install.
+      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }}
 
     steps:
       # Only a real release (main) needs the app token: semantic-release pushes the
@@ -409,14 +410,14 @@ jobs:
           cache-from: type=registry,ref=ghcr.io/${{ steps.img.outputs.owner_lc }}/coder-eval-agent:buildcache
 
   # Publish the wheel+sdist to public PyPI. This runs as its own job so OIDC
-  # Trusted Publishing is scoped to a dedicated, environment-gated context on
-  # GitHub-hosted runners -- no PyPI token/secret is stored. Gated on the
-  # release job having actually cut a version.
+  # Trusted Publishing is scoped to a dedicated, environment-gated context --
+  # no PyPI token/secret is stored. Gated on the release job having actually
+  # cut a version.
   publish-pypi:
     name: Publish to PyPI
     needs: release
     if: needs.release.outputs.version != ''
-    runs-on: ubuntu-latest
+    runs-on: uipath-ubuntu-latest
     timeout-minutes: 10
     environment:
       name: pypi

{coder_eval-0.9.4 → coder_eval-0.9.6}/CHANGELOG.md

@@ -2,6 +2,173 @@
 
 <!-- version list -->
 
+## v0.9.6 (2026-08-11)
+
+### Bug Fixes
+
+- Code review fixes for evalboard path-to-ga stale tags and mature passes
+  ([#94](https://github.com/UiPath/coder_eval/pull/94),
+  [`ce006c1`](https://github.com/UiPath/coder_eval/commit/ce006c19b594701e90aec1d63cae15b337c0e06d))
+
+- Review round 2 — gate evalboard in CI, correct GPT-5.6 rates
+  ([#94](https://github.com/UiPath/coder_eval/pull/94),
+  [`ce006c1`](https://github.com/UiPath/coder_eval/commit/ce006c19b594701e90aec1d63cae15b337c0e06d))
+
+- **ci**: Correct defects in the uipath runner migration
+  ([#86](https://github.com/UiPath/coder_eval/pull/86),
+  [`57556af`](https://github.com/UiPath/coder_eval/commit/57556af6e47bffcd5fbbb708b9f8b399cceea4c7))
+
+- **criteria**: Make glob path resolution literal-first and ignore-filtered
+  ([#65](https://github.com/UiPath/coder_eval/pull/65),
+  [`b3bba2b`](https://github.com/UiPath/coder_eval/commit/b3bba2b6c00e3772351b28bf63d5cde9e97b8e7e))
+
+- **criteria**: Split clustered short flags and keep negative numbers positional
+  ([#73](https://github.com/UiPath/coder_eval/pull/73),
+  [`a7ec3ea`](https://github.com/UiPath/coder_eval/commit/a7ec3eababc907a0220a598d974540eb579a538c))
+
+- **evalboard**: 1/3 — drop de-tagged tasks and score only executed runs
+  ([#94](https://github.com/UiPath/coder_eval/pull/94),
+  [`ce006c1`](https://github.com/UiPath/coder_eval/commit/ce006c19b594701e90aec1d63cae15b337c0e06d))
+
+- **evalboard**: Path-to-GA shows only still-tagged tasks, scored on runs that executed
+  ([#94](https://github.com/UiPath/coder_eval/pull/94),
+  [`ce006c1`](https://github.com/UiPath/coder_eval/commit/ce006c19b594701e90aec1d63cae15b337c0e06d))
+
+- **evalboard**: Resync lib/pricing.ts with the authoritative pricing.py table
+  ([#94](https://github.com/UiPath/coder_eval/pull/94),
+  [`ce006c1`](https://github.com/UiPath/coder_eval/commit/ce006c19b594701e90aec1d63cae15b337c0e06d))
+
+- **sandbox**: Close the remaining record_cli findings from #73
+  ([#73](https://github.com/UiPath/coder_eval/pull/73),
+  [`a7ec3ea`](https://github.com/UiPath/coder_eval/commit/a7ec3eababc907a0220a598d974540eb579a538c))
+
+- **sandbox**: Repair record_cli defects found reviewing #73
+  ([#73](https://github.com/UiPath/coder_eval/pull/73),
+  [`a7ec3ea`](https://github.com/UiPath/coder_eval/commit/a7ec3eababc907a0220a598d974540eb579a538c))
+
+- **sandbox**: Stop the recorder dir defeating the PLUGIN_TOOLS_DIR pin
+  ([#73](https://github.com/UiPath/coder_eval/pull/73),
+  [`a7ec3ea`](https://github.com/UiPath/coder_eval/commit/a7ec3eababc907a0220a598d974540eb579a538c))
+
+### Chores
+
+- Change to centralized managed GitHub pool ([#86](https://github.com/UiPath/coder_eval/pull/86),
+  [`57556af`](https://github.com/UiPath/coder_eval/commit/57556af6e47bffcd5fbbb708b9f8b399cceea4c7))
+
+### Documentation
+
+- **harness**: Defer four TS-side evalboard invariants from the path-to-ga fix
+  ([#94](https://github.com/UiPath/coder_eval/pull/94),
+  [`ce006c1`](https://github.com/UiPath/coder_eval/commit/ce006c19b594701e90aec1d63cae15b337c0e06d))
+
+### Features
+
+- **criteria**: Accept glob patterns in criterion path fields
+  ([#65](https://github.com/UiPath/coder_eval/pull/65),
+  [`b3bba2b`](https://github.com/UiPath/coder_eval/commit/b3bba2b6c00e3772351b28bf63d5cde9e97b8e7e))
+
+- **evalboard**: 2/3 — surface last-seen and maturity on the Path-to-GA table
+  ([#94](https://github.com/UiPath/coder_eval/pull/94),
+  [`ce006c1`](https://github.com/UiPath/coder_eval/commit/ce006c19b594701e90aec1d63cae15b337c0e06d))
+
+- **sandbox**: Generate CLI recording shims via record_cli
+  ([#73](https://github.com/UiPath/coder_eval/pull/73),
+  [`a7ec3ea`](https://github.com/UiPath/coder_eval/commit/a7ec3eababc907a0220a598d974540eb579a538c))
+
+
+## v0.9.5 (2026-08-05)
+
+### Bug Fixes
+
+- **command-executed**: Keep whole argv-joined payload in shell unwrap
+  ([#77](https://github.com/UiPath/coder_eval/pull/77),
+  [`7abd080`](https://github.com/UiPath/coder_eval/commit/7abd08098146067890e16740c37239c2e0009a24))
+
+- **command-executed**: Match patterns against shell-normalized commands
+  ([#77](https://github.com/UiPath/coder_eval/pull/77),
+  [`7abd080`](https://github.com/UiPath/coder_eval/commit/7abd08098146067890e16740c37239c2e0009a24))
+
+- **command-executed**: Narrow command param to str before shell-normalizing
+  ([#77](https://github.com/UiPath/coder_eval/pull/77),
+  [`7abd080`](https://github.com/UiPath/coder_eval/commit/7abd08098146067890e16740c37239c2e0009a24))
+
+- **command-executed**: Recognize shell wrappers by predicate, not allowlist
+  ([#77](https://github.com/UiPath/coder_eval/pull/77),
+  [`7abd080`](https://github.com/UiPath/coder_eval/commit/7abd08098146067890e16740c37239c2e0009a24))
+
+- **criteria**: Add present predicate so asserting a switch cannot weaken a guard
+  ([#72](https://github.com/UiPath/coder_eval/pull/72),
+  [`8574ded`](https://github.com/UiPath/coder_eval/commit/8574dedb02d6e9ab883b042934f89297591e8abd))
+
+- **criteria**: Make cli_called guards fail loud instead of vacuously passing
+  ([#72](https://github.com/UiPath/coder_eval/pull/72),
+  [`8574ded`](https://github.com/UiPath/coder_eval/commit/8574dedb02d6e9ab883b042934f89297591e8abd))
+
+- **criteria**: Stop ignore_flags re-opening the guard false-PASS
+  ([#72](https://github.com/UiPath/coder_eval/pull/72),
+  [`8574ded`](https://github.com/UiPath/coder_eval/commit/8574dedb02d6e9ab883b042934f89297591e8abd))
+
+- **early-stop**: Address PR review — trajectory parity, reason determinism, doc restore
+  ([#78](https://github.com/UiPath/coder_eval/pull/78),
+  [`4cf8092`](https://github.com/UiPath/coder_eval/commit/4cf80920422e25fec663ef93e483902bdeffad24))
+
+- **lint**: Derive CE030 criteria from the source union literal, not runtime
+  ([#77](https://github.com/UiPath/coder_eval/pull/77),
+  [`7abd080`](https://github.com/UiPath/coder_eval/commit/7abd08098146067890e16740c37239c2e0009a24))
+
+- **lint**: Enumerate in-tree criteria by module attribute, not the union
+  ([#77](https://github.com/UiPath/coder_eval/pull/77),
+  [`7abd080`](https://github.com/UiPath/coder_eval/commit/7abd08098146067890e16740c37239c2e0009a24))
+
+- **lint**: Scope CE030 criterion parity to in-tree criteria only
+  ([#77](https://github.com/UiPath/coder_eval/pull/77),
+  [`7abd080`](https://github.com/UiPath/coder_eval/commit/7abd08098146067890e16740c37239c2e0009a24))
+
+- **reports**: Explicit return on every early_stop_gate_note path (CodeQL py/mixed-returns)
+  ([#78](https://github.com/UiPath/coder_eval/pull/78),
+  [`4cf8092`](https://github.com/UiPath/coder_eval/commit/4cf80920422e25fec663ef93e483902bdeffad24))
+
+- **reports**: Pre-initialize the gate note so CodeQL sees it bound on every path
+  ([#78](https://github.com/UiPath/coder_eval/pull/78),
+  [`4cf8092`](https://github.com/UiPath/coder_eval/commit/4cf80920422e25fec663ef93e483902bdeffad24))
+
+### Chores
+
+- **deps-dev**: Bump postcss from 8.5.18 to 8.5.23 in /evalboard
+  ([#75](https://github.com/UiPath/coder_eval/pull/75),
+  [`cdced15`](https://github.com/UiPath/coder_eval/commit/cdced152dea732096dcff939c45e7d7c927c2a5a))
+
+### Documentation
+
+- Surface the Marketplace listing and make the Action quickstarts self-sufficient
+  ([#80](https://github.com/UiPath/coder_eval/pull/80),
+  [`401245a`](https://github.com/UiPath/coder_eval/commit/401245ad14055c1da5d7b594e506686c59cedce7))
+
+- **command-executed**: Document shell-normalization contract + gate it (CE030)
+  ([#77](https://github.com/UiPath/coder_eval/pull/77),
+  [`7abd080`](https://github.com/UiPath/coder_eval/commit/7abd08098146067890e16740c37239c2e0009a24))
+
+### Features
+
+- **criteria**: Add cli_called for structured invocation matching
+  ([#72](https://github.com/UiPath/coder_eval/pull/72),
+  [`8574ded`](https://github.com/UiPath/coder_eval/commit/8574dedb02d6e9ab883b042934f89297591e8abd))
+
+- **criteria**: Match a flag across spellings with FlagMatch.aliases
+  ([#72](https://github.com/UiPath/coder_eval/pull/72),
+  [`8574ded`](https://github.com/UiPath/coder_eval/commit/8574dedb02d6e9ab883b042934f89297591e8abd))
+
+- **early-stop**: Per-criterion arming via stop_early blocks on live criteria
+  ([#78](https://github.com/UiPath/coder_eval/pull/78),
+  [`4cf8092`](https://github.com/UiPath/coder_eval/commit/4cf80920422e25fec663ef93e483902bdeffad24))
+
+### Refactoring
+
+- **command-executed**: Total _match_haystacks, shared window, memoized
+  ([#77](https://github.com/UiPath/coder_eval/pull/77),
+  [`7abd080`](https://github.com/UiPath/coder_eval/commit/7abd08098146067890e16740c37239c2e0009a24))
+
+
 ## v0.9.4 (2026-08-04)
 
 ### Bug Fixes