coder-eval 0.8.7rc11__tar.gz → 0.8.9__tar.gz

{coder_eval-0.8.7rc11 → coder_eval-0.8.9}/.claude/commands/coder-eval-code-review.md

@@ -12,7 +12,7 @@ description: Run a multi-model code review on uncommitted changes or a described
 
 Run a thorough code review using multiple AI models in parallel, then fix all high-confidence medium-severity and above findings.
 
-**Cost note**: This command runs `make verify` + a multi-model review (`gemini-3` + `codex` via MCP, plus an Opus sub-agent) + automatic fixes. Expensive in time and tokens. Use this for changes you're about to ship; for broad codebase audits use `/coder-eval-code-review-full`; for quick local checks, run targeted tools (`ruff`, `pyright`, `pytest`) directly.
+**Cost note**: This command runs `make verify` + a multi-model review (`gemini-3` + `gpt-5` via MCP, plus an Opus sub-agent) + automatic fixes. Expensive in time and tokens. Use this for changes you're about to ship; for broad codebase audits use `/coder-eval-code-review-full`; for quick local checks, run targeted tools (`ruff`, `pyright`, `pytest`) directly.
 
 Input: $ARGUMENTS
 
@@ -65,7 +65,7 @@ If it fails, report what's failing and stop — code review on broken code is pr
 
 Launch two reviews **in parallel**:
 
-**Review A — Multi-model (via `mcp__multi__codereview`)**: run it per the shared procedure in `.claude/shared/multi-model-review.md` — `models: ["gemini-3", "codex"]`, `relevant_files` = absolute paths of all changed files, `content` = a review request citing the Severity Standard, Review Principles, and Checklist. **Heed the multi-step protocol described there** (the step-1 response is usually an `in_progress` checklist, not findings — you must make the `step_number: 2` follow-up call with the same `thread_id`). Falls back to two Opus sub-agents if the tool is unavailable.
+**Review A — Multi-model (via `mcp__multi__codereview`)**: run it per the shared procedure in `.claude/shared/multi-model-review.md` — `models: ["gemini-3", "gpt-5"]`, `relevant_files` = absolute paths of all changed files, `content` = a review request citing the Severity Standard, Review Principles, and Checklist. **Heed the multi-step protocol described there** (the step-1 response is usually an `in_progress` checklist, not findings — you must make the `step_number: 2` follow-up call with the same `thread_id`). Falls back to two Opus sub-agents if the tool is unavailable.
 
 **Review B — Opus sub-agent**: Use the `Agent` tool with `model: "opus"`. Give it the list of changed files and the following specialized tasks to run in parallel internally:
 
@@ -120,7 +120,7 @@ across sessions and provides forensic context if a fix later proves wrong:
 - Git SHA: <`git rev-parse HEAD`>
 - Branch: <`git rev-parse --abbrev-ref HEAD`>
 - Scope: <uncommitted | described "<input>" | last commit>
-- Reviewers: <e.g. gemini-3, codex, opus-fallback-1, opus-fallback-2 — list actual reviewers used>
+- Reviewers: <e.g. gemini-3, gpt-5, opus-fallback-1, opus-fallback-2 — list actual reviewers used>
 
 ### Scope
 - Files reviewed: (list)

{coder_eval-0.8.7rc11 → coder_eval-0.8.9}/.claude/shared/multi-model-review.md

@@ -14,7 +14,7 @@ If `mcp__multi__codereview` is available (check the deferred-tools list; if list
 but not loaded, load it with `ToolSearch` → `select:mcp__multi__codereview`), call
 it with:
 
-- `models`: the strongest available (e.g. `["gemini-3", "codex"]`)
+- `models`: the strongest available (e.g. `["gemini-3", "gpt-5"]` — as of July 2026 these resolve to `gemini-3.1-pro-preview` and `gpt-5.6-sol`; check `mcp__multi__models` if unsure)
 - `relevant_files`: absolute paths of every changed / in-scope file
 - `content`: the review request, citing the consumer's rubric (the shared **Review Criteria**, the **Severity Standard**, etc.)
 - `base_path`: project root

coder_eval-0.8.9/.github/ISSUE_TEMPLATE/adopter.yml

@@ -0,0 +1,84 @@
+name: Adopter — we're using coder_eval
+description: Tell us your team uses coder_eval so we can add you to ADOPTERS.md
+title: "[adopter] <organization>"
+labels: ["adopter"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for using coder_eval! Adopter entries help other teams see the project is
+        real and in production, and they tell us which use cases to keep supporting.
+
+        Everything below is optional except the organization and how you use it. We only
+        publish what you explicitly allow here.
+  - type: input
+    id: organization
+    attributes:
+      label: Organization
+      description: The name you'd like listed.
+      placeholder: Acme Corp
+    validations:
+      required: true
+  - type: input
+    id: website
+    attributes:
+      label: Website
+      placeholder: https://example.com
+    validations:
+      required: false
+  - type: textarea
+    id: usage
+    attributes:
+      label: How do you use coder_eval?
+      description: |
+        A sentence or two. What do you evaluate — Claude Code skills, agent A/B tests,
+        CI quality gates, your own repos — and roughly at what scale?
+      placeholder: |
+        We gate CI on a suite of ~40 YAML tasks that check our internal Claude Code
+        skills still trigger, and A/B Claude Code against Codex each release.
+    validations:
+      required: true
+  - type: dropdown
+    id: listing
+    attributes:
+      label: How may we list you?
+      description: |
+        Pick the most you're comfortable with. **Logo use needs someone who can approve
+        trademark use for your organization** — if that isn't you, choose a name-only or
+        private option and we'll follow up.
+      options:
+        - Name only, in ADOPTERS.md
+        - Name in ADOPTERS.md and on the project website
+        - Name and logo in ADOPTERS.md and on the project website
+        - Please don't list us publicly — this is just for the maintainers
+      default: 0
+    validations:
+      required: true
+  - type: checkboxes
+    id: logo-permission
+    attributes:
+      label: Trademark permission
+      description: Only required if you selected a logo option above.
+      options:
+        - label: >-
+            I am authorized to grant this permission on behalf of my organization, and I
+            grant the coder_eval maintainers a non-exclusive, revocable license to display
+            our name and logo as an adopter in ADOPTERS.md and on the project website. We
+            can withdraw it at any time by commenting on this issue.
+          required: false
+  - type: input
+    id: contact
+    attributes:
+      label: Contact
+      description: GitHub handle or email, so we can reach you before publishing anything.
+    validations:
+      required: false
+  - type: checkboxes
+    id: case-study
+    attributes:
+      label: Anything else?
+      options:
+        - label: We'd be open to a short case study or a talk about our setup.
+          required: false
+        - label: We'd like to hear about breaking changes before they ship.
+          required: false

coder_eval-0.8.9/.github/workflows/docs.yml

@@ -0,0 +1,48 @@
+name: Docs
+
+# Build the MkDocs Material site and publish it to the `gh-pages` branch via
+# `mkdocs gh-deploy` — the same mechanism UiPath/uipath-python uses. This workflow
+# only pushes a branch (needs `contents: write`); it never calls the Pages API, so
+# it succeeds even before Pages is switched on.
+#
+# One-time setup (org owner): the UiPath org blocks Pages *creation*, so enable
+# Pages for this repo once — Settings → Pages → Build and deployment →
+# Source: "Deploy from a branch" → branch `gh-pages` / `/ (root)`. That's the same
+# (legacy, branch-based) configuration uipath-python already runs on.
+on:
+  push:
+    branches: [main]
+    paths:
+      - "docs/**"
+      - "mkdocs.yml"
+      - ".github/workflows/docs.yml"
+  workflow_dispatch:
+
+permissions:
+  contents: write
+
+concurrency:
+  group: docs-gh-pages
+  cancel-in-progress: true
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+      - name: Install social-card system libraries
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y --no-install-recommends \
+            libcairo2 libfreetype6 libjpeg-turbo8 libpng16-16 pngquant
+      - name: Install MkDocs Material
+        run: pip install "mkdocs-material[imaging]>=9.5,<10"
+      - name: Configure git identity for gh-pages commits
+        run: |
+          git config user.name "coder-eval"
+          git config user.email "coder-eval@uipath.com"
+      - name: Build strictly and publish to gh-pages
+        run: mkdocs gh-deploy --force --strict

{coder_eval-0.8.7rc11 → coder_eval-0.8.9}/.github/workflows/release.yml

@@ -25,7 +25,7 @@ on:
   workflow_dispatch:
     inputs:
       bump:
-        description: 'Version bump to release'
+        description: 'Version bump to release (ignored on non-main / prerelease dispatch, which always stamps a next-patch rc)'
         type: choice
         default: patch
         options:

{coder_eval-0.8.7rc11 → coder_eval-0.8.9}/.gitignore

@@ -1,3 +1,7 @@
+# MkDocs build output
+site/
+.cache/
+
 # Python-generated files
 __pycache__/
 *.py[oc]

coder_eval-0.8.9/ADOPTERS.md

@@ -0,0 +1,13 @@
+# Adopters
+
+Teams and organizations using `coder_eval` in their development or CI workflows.
+
+Using it? [Open an adopter issue](https://github.com/UiPath/coder_eval/issues/new?template=adopter.yml)
+and we'll add you here. Name-only listings are welcome — a logo is never required.
+
+Every entry below is published with the organization's permission, and any of them can be
+removed on request by opening an issue.
+
+| Organization | How they use it |
+|---|---|
+| [UiPath](https://www.uipath.com) | Nightly evaluation of internal Claude Code skills and agent A/B experiments; maintains `coder_eval`. |

{coder_eval-0.8.7rc11 → coder_eval-0.8.9}/CHANGELOG.md

@@ -2,6 +2,212 @@
 
 <!-- version list -->
 
+## v0.8.9 (2026-07-23)
+
+### Bug Fixes
+
+- Code review fixes for welch-t-test-exact ([#38](https://github.com/UiPath/coder_eval/pull/38),
+  [`6df6e9b`](https://github.com/UiPath/coder_eval/commit/6df6e9bb1f3bd009592185ec3473e8f2e67d3816))
+
+- Render weight:0 criteria as informational on every display surface
+  ([#34](https://github.com/UiPath/coder_eval/pull/34),
+  [`9a34e90`](https://github.com/UiPath/coder_eval/commit/9a34e90bba2854316149cad61613a75dd3bd91e4))
+
+- Weight:0 un-gates criteria (informational criteria)
+  ([#34](https://github.com/UiPath/coder_eval/pull/34),
+  [`9a34e90`](https://github.com/UiPath/coder_eval/commit/9a34e90bba2854316149cad61613a75dd3bd91e4))
+
+- Weight:0 un-gates criteria and renders as informational
+  ([#34](https://github.com/UiPath/coder_eval/pull/34),
+  [`9a34e90`](https://github.com/UiPath/coder_eval/commit/9a34e90bba2854316149cad61613a75dd3bd91e4))
+
+- **early-stop**: Decide skill activation on the tool call, not its result
+  ([#43](https://github.com/UiPath/coder_eval/pull/43),
+  [`d34aa97`](https://github.com/UiPath/coder_eval/commit/d34aa97416ae3af09d09cdf78cb6607b4e2e6f7c))
+
+- **early-stop**: Latch skill activation on any engagement, not first
+  ([#43](https://github.com/UiPath/coder_eval/pull/43),
+  [`d34aa97`](https://github.com/UiPath/coder_eval/commit/d34aa97416ae3af09d09cdf78cb6607b4e2e6f7c))
+
+- **evalboard**: Match watchlist skeleton header to avoid layout shift
+  ([#45](https://github.com/UiPath/coder_eval/pull/45),
+  [`acc1c86`](https://github.com/UiPath/coder_eval/commit/acc1c86bc97b80383ce1ef402356853447088e4a))
+
+- **reports**: 1/2 — exact Student-t p-values in welch_t_test
+  ([#38](https://github.com/UiPath/coder_eval/pull/38),
+  [`6df6e9b`](https://github.com/UiPath/coder_eval/commit/6df6e9bb1f3bd009592185ec3473e8f2e67d3816))
+
+- **reports**: Exact Student-t p-values and a paired comparison section
+  ([#38](https://github.com/UiPath/coder_eval/pull/38),
+  [`6df6e9b`](https://github.com/UiPath/coder_eval/commit/6df6e9bb1f3bd009592185ec3473e8f2e67d3816))
+
+- **reports**: Fail loud on t* overflow; surface excluded paired tasks
+  ([#38](https://github.com/UiPath/coder_eval/pull/38),
+  [`6df6e9b`](https://github.com/UiPath/coder_eval/commit/6df6e9bb1f3bd009592185ec3473e8f2e67d3816))
+
+- **reports**: One source of truth for variant series and paired stats
+  ([#38](https://github.com/UiPath/coder_eval/pull/38),
+  [`6df6e9b`](https://github.com/UiPath/coder_eval/commit/6df6e9bb1f3bd009592185ec3473e8f2e67d3816))
+
+- **reports**: Validate confidence and n_resamples in bootstrap_mean_ci
+  ([#38](https://github.com/UiPath/coder_eval/pull/38),
+  [`6df6e9b`](https://github.com/UiPath/coder_eval/commit/6df6e9bb1f3bd009592185ec3473e8f2e67d3816))
+
+### Chores
+
+- **harness**: Defer two guards from the welch-t-test-exact run
+  ([#38](https://github.com/UiPath/coder_eval/pull/38),
+  [`6df6e9b`](https://github.com/UiPath/coder_eval/commit/6df6e9bb1f3bd009592185ec3473e8f2e67d3816))
+
+### Documentation
+
+- Add adopter issue template and ADOPTERS.md ([#40](https://github.com/UiPath/coder_eval/pull/40),
+  [`bfbed4e`](https://github.com/UiPath/coder_eval/commit/bfbed4e4a2ca857167fac2cb15462b7159e97684))
+
+- Switch multi-model review from codex to gpt-5 alias
+  ([#36](https://github.com/UiPath/coder_eval/pull/36),
+  [`0a3f2a7`](https://github.com/UiPath/coder_eval/commit/0a3f2a71a0a1e1e56f42a668fff57bac06eb99ec))
+
+### Features
+
+- **evalboard**: Make all pages harness-aware and stream tables
+  ([#45](https://github.com/UiPath/coder_eval/pull/45),
+  [`acc1c86`](https://github.com/UiPath/coder_eval/commit/acc1c86bc97b80383ce1ef402356853447088e4a))
+
+- **evalboard**: Make analytics surfaces harness-aware and stream tables
+  ([#45](https://github.com/UiPath/coder_eval/pull/45),
+  [`acc1c86`](https://github.com/UiPath/coder_eval/commit/acc1c86bc97b80383ce1ef402356853447088e4a))
+
+- **evalboard**: Scope task trends to one harness
+  ([#45](https://github.com/UiPath/coder_eval/pull/45),
+  [`acc1c86`](https://github.com/UiPath/coder_eval/commit/acc1c86bc97b80383ce1ef402356853447088e4a))
+
+- **reports**: 2/2 — add a Paired Comparison section to experiment reports
+  ([#38](https://github.com/UiPath/coder_eval/pull/38),
+  [`6df6e9b`](https://github.com/UiPath/coder_eval/commit/6df6e9bb1f3bd009592185ec3473e8f2e67d3816))
+
+### Refactoring
+
+- **evalboard**: Address review nits on harness plumbing
+  ([#45](https://github.com/UiPath/coder_eval/pull/45),
+  [`acc1c86`](https://github.com/UiPath/coder_eval/commit/acc1c86bc97b80383ce1ef402356853447088e4a))
+
+### Testing
+
+- **early-stop**: Cover second-review items (two-AgentStart, golden corpus, parity)
+  ([#43](https://github.com/UiPath/coder_eval/pull/43),
+  [`d34aa97`](https://github.com/UiPath/coder_eval/commit/d34aa97416ae3af09d09cdf78cb6607b4e2e6f7c))
+
+
+## v0.8.8 (2026-07-22)
+
+### Bug Fixes
+
+- **codex**: Create CODEX_HOME before pinning it in the app-server env
+  ([#39](https://github.com/UiPath/coder_eval/pull/39),
+  [`8d98b91`](https://github.com/UiPath/coder_eval/commit/8d98b912ae4767904c7d1c395913c3d0aaec6c66))
+
+### Documentation
+
+- Add Website badge and point PyPI Homepage at coder-eval.com
+  ([#41](https://github.com/UiPath/coder_eval/pull/41),
+  [`0551534`](https://github.com/UiPath/coder_eval/commit/055153409d86b8d537bba73ef7ed58dfd17fee26))
+
+
+## v0.8.7 (2026-07-22)
+
+### Bug Fixes
+
+- Detect Windows skill paths in telemetry ([#24](https://github.com/UiPath/coder_eval/pull/24),
+  [`c57a6b0`](https://github.com/UiPath/coder_eval/commit/c57a6b0408ef116177b15bc9b09a768096aaa33f))
+
+- **agents**: Run codex + antigravity harnesses on the tempdir/host path
+  ([#33](https://github.com/UiPath/coder_eval/pull/33),
+  [`04498a9`](https://github.com/UiPath/coder_eval/commit/04498a96f6b9ac0cc5c73f36e73a361f3f3fa0ae))
+
+- **antigravity**: Pin google-antigravity 0.1.7 to load on glibc 2.35
+  ([#33](https://github.com/UiPath/coder_eval/pull/33),
+  [`04498a9`](https://github.com/UiPath/coder_eval/commit/04498a96f6b9ac0cc5c73f36e73a361f3f3fa0ae))
+
+- **codex**: Always run full-access; drop the in-process OS sandbox
+  ([#33](https://github.com/UiPath/coder_eval/pull/33),
+  [`04498a9`](https://github.com/UiPath/coder_eval/commit/04498a96f6b9ac0cc5c73f36e73a361f3f3fa0ae))
+
+- **codex**: Cover zsh login shells (macOS default) in the mock-PATH home shim
+  ([#26](https://github.com/UiPath/coder_eval/pull/26),
+  [`73f0db2`](https://github.com/UiPath/coder_eval/commit/73f0db24df0b67f82d46e3bf15c85a21837237ef))
+
+- **codex**: Keep mock CLIs shadowed in bash login shells
+  ([#26](https://github.com/UiPath/coder_eval/pull/26),
+  [`73f0db2`](https://github.com/UiPath/coder_eval/commit/73f0db24df0b67f82d46e3bf15c85a21837237ef))
+
+- **codex**: Make login-shell temp-home lifecycle exception-safe incl. kill_sync
+  ([#26](https://github.com/UiPath/coder_eval/pull/26),
+  [`73f0db2`](https://github.com/UiPath/coder_eval/commit/73f0db24df0b67f82d46e3bf15c85a21837237ef))
+
+- **codex**: Restore mock-CLI PATH prepend in login shells via per-task HOME profile
+  ([#26](https://github.com/UiPath/coder_eval/pull/26),
+  [`73f0db2`](https://github.com/UiPath/coder_eval/commit/73f0db24df0b67f82d46e3bf15c85a21837237ef))
+
+- **codex**: Restore original HOME inside generated login-shell profile
+  ([#26](https://github.com/UiPath/coder_eval/pull/26),
+  [`73f0db2`](https://github.com/UiPath/coder_eval/commit/73f0db24df0b67f82d46e3bf15c85a21837237ef))
+
+- **codex**: Use full-access sandbox on coder_eval-managed tempdir path
+  ([#33](https://github.com/UiPath/coder_eval/pull/33),
+  [`04498a9`](https://github.com/UiPath/coder_eval/commit/04498a96f6b9ac0cc5c73f36e73a361f3f3fa0ae))
+
+- **deps**: Bump pyasn1 to 0.6.4 for GHSA-8ppf-4f7h-5ppj / GHSA-hm4w-wwcw-mr6r
+  ([#33](https://github.com/UiPath/coder_eval/pull/33),
+  [`04498a9`](https://github.com/UiPath/coder_eval/commit/04498a96f6b9ac0cc5c73f36e73a361f3f3fa0ae))
+
+### Chores
+
+- **deps**: Bump pyasn1 0.6.3 -> 0.6.4 (GHSA-8ppf-4f7h-5ppj, GHSA-hm4w-wwcw-mr6r)
+  ([#32](https://github.com/UiPath/coder_eval/pull/32),
+  [`f1f4f9f`](https://github.com/UiPath/coder_eval/commit/f1f4f9fba2bd034a271e50fc1a95bcc74c2ac19b))
+
+- **deps**: Upgrade agent SDKs to latest (claude 0.2.124, codex 0.144.4)
+  ([#33](https://github.com/UiPath/coder_eval/pull/33),
+  [`04498a9`](https://github.com/UiPath/coder_eval/commit/04498a96f6b9ac0cc5c73f36e73a361f3f3fa0ae))
+
+### Continuous Integration
+
+- **release**: Publish a prerelease from a non-main branch
+  ([#33](https://github.com/UiPath/coder_eval/pull/33),
+  [`04498a9`](https://github.com/UiPath/coder_eval/commit/04498a96f6b9ac0cc5c73f36e73a361f3f3fa0ae))
+
+### Documentation
+
+- Add MkDocs docs site, comparison page, and SEO metadata
+  ([#32](https://github.com/UiPath/coder_eval/pull/32),
+  [`f1f4f9f`](https://github.com/UiPath/coder_eval/commit/f1f4f9fba2bd034a271e50fc1a95bcc74c2ac19b))
+
+- Address PR review — Coder Eval naming, cleaner table, gh-deploy workflow
+  ([#32](https://github.com/UiPath/coder_eval/pull/32),
+  [`f1f4f9f`](https://github.com/UiPath/coder_eval/commit/f1f4f9fba2bd034a271e50fc1a95bcc74c2ac19b))
+
+- Address review — git identity for gh-pages, single strict deploy, table glyphs, uv tool install in
+  Tutorial 01 ([#32](https://github.com/UiPath/coder_eval/pull/32),
+  [`f1f4f9f`](https://github.com/UiPath/coder_eval/commit/f1f4f9fba2bd034a271e50fc1a95bcc74c2ac19b))
+
+### Refactoring
+
+- **codex**: Drop dead sandbox branch + honest full-access messaging
+  ([#33](https://github.com/UiPath/coder_eval/pull/33),
+  [`04498a9`](https://github.com/UiPath/coder_eval/commit/04498a96f6b9ac0cc5c73f36e73a361f3f3fa0ae))
+
+### Testing
+
+- Accept sandbox_managed kwarg in agent test doubles
+  ([#33](https://github.com/UiPath/coder_eval/pull/33),
+  [`04498a9`](https://github.com/UiPath/coder_eval/commit/04498a96f6b9ac0cc5c73f36e73a361f3f3fa0ae))
+
+- **codex**: Pin start-to-CodexConfig env composition and tidy test imports
+  ([#26](https://github.com/UiPath/coder_eval/pull/26),
+  [`73f0db2`](https://github.com/UiPath/coder_eval/commit/73f0db24df0b67f82d46e3bf15c85a21837237ef))
+
+
 ## v0.8.6 (2026-07-20)
 
 ### Features

{coder_eval-0.8.7rc11 → coder_eval-0.8.9}/CLAUDE.md

@@ -35,7 +35,7 @@ coder_eval/
 │   ├── criteria.py                # 14 success criterion types + base + union
 │   ├── experiment.py              # ExperimentDefinition, ExperimentVariant, ResolvedTask, result models
 │   ├── judge_defaults.py          # DEFAULT_JUDGE_MODEL constant (cycle-free leaf)
-│   ├── mutations.py               # PromptMutation variants (prefix/suffix/replace/template/rephrase)
+│   ├── mutations.py               # PromptMutation variants (prefix/suffix/replace/template)
 │   ├── results.py                 # CriterionResult (+ ClassificationCriterionResult), TurnRecord, EvaluationResult, EarlyStopInfo/EarlyStopReason, CriterionAggregate, ThresholdCheck, SuiteRollup
 │   ├── routing.py                 # ApiRoute (DirectRoute/BedrockRoute)
 │   ├── sandbox.py                 # SandboxConfig, ResourceLimits
@@ -294,7 +294,7 @@ Tasks are YAML files. See [docs/TASK_DEFINITION_GUIDE.md](docs/TASK_DEFINITION_G
 
 **Runtime (always)**: pydantic, pydantic-settings, pyyaml, typer, rich, python-dotenv, anthropic, claude-agent-sdk, anyio, radon, tqdm, jmespath, jsonschema
 
-**Runtime (optional, `[uipath]` extra)**: uipath — the in-host `uipath` SDK (handy for local sandbox parity with tasks that invoke `uv run uipath eval ...`). Base installs without this extra still run end-to-end; UiPath-dependent paths fail at dispatch with a clear `pip install 'coder-eval[uipath]'` hint. The LLM judge and the `rephrase` mutation no longer use the LLM Gateway client — they route through the run's backend (Bedrock / Anthropic), so `uipath-llmgw-client` is no longer a dependency.
+**Runtime (optional, `[uipath]` extra)**: uipath — the in-host `uipath` SDK (handy for local sandbox parity with tasks that invoke `uv run uipath eval ...`). Base installs without this extra still run end-to-end; UiPath-dependent paths fail at dispatch with a clear `pip install 'coder-eval[uipath]'` hint. The LLM judge no longer uses the LLM Gateway client — it routes through the run's backend (Bedrock / Anthropic), so `uipath-llmgw-client` is no longer a dependency.
 
 **Dev**: pytest, pytest-asyncio, pytest-mock, pytest-cov, ruff, pyright, pip-audit, bandit, pre-commit, mcp
 

{coder_eval-0.8.7rc11 → coder_eval-0.8.9}/PKG-INFO

@@ -1,8 +1,8 @@
 Metadata-Version: 2.4
 Name: coder-eval
-Version: 0.8.7rc11
+Version: 0.8.9
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity) with sandboxed, reproducible YAML task suites.
-Project-URL: Homepage, https://github.com/UiPath/coder_eval
+Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval
 Project-URL: Documentation, https://github.com/UiPath/coder_eval/tree/main/docs
 Project-URL: Issues, https://github.com/UiPath/coder_eval/issues
@@ -11,7 +11,7 @@ Author-email: UiPath <coder-eval@uipath.com>
 License-Expression: Apache-2.0
 License-File: LICENSE
 License-File: NOTICE
-Keywords: agent,agent-evaluation,agent-skills,ai,anthropic,antigravity,benchmark,claude,claude-code,claude-skills,code-generation,codex,coding-agent,eval,evals,evaluation,gemini,llm,llm-evaluation,sandbox,skills-evaluation,skillsbench,swe-bench
+Keywords: agent,agent-evaluation,agent-skills,agent-testing,ai,ai-evaluation,anthropic,antigravity,benchmark,claude,claude-code,claude-code-skills,claude-skills,code-generation,codex,coding-agent,eval,evals,evaluation,gemini,llm,llm-eval,llm-evaluation,llmops,sandbox,skills-evaluation,skillsbench,swe-bench
 Classifier: Development Status :: 4 - Beta
 Classifier: Environment :: Console
 Classifier: Intended Audience :: Developers
@@ -60,28 +60,37 @@ Provides-Extra: uipath
 Requires-Dist: uipath>=2.10.31; extra == 'uipath'
 Description-Content-Type: text/markdown
 
-# coder_eval — evaluate AI coding agents & their skills
+# Coder Eval — evaluate & benchmark AI coding agents and Claude Code skills
 
 [![PyPI](https://img.shields.io/pypi/v/coder-eval.svg)](https://pypi.org/project/coder-eval/)
+[![Website](https://img.shields.io/badge/website-coder--eval.com-1f6feb.svg)](https://coder-eval.com)
+[![Docs](https://img.shields.io/badge/docs-uipath.github.io%2Fcoder__eval-1f6feb.svg)](https://uipath.github.io/coder_eval/)
 [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
 [![Python 3.13+](https://img.shields.io/badge/python-3.13%2B-blue.svg)](https://www.python.org/downloads/)
 [![CI](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml/badge.svg)](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml)
 [![Code style: Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
 
-A framework for evaluating AI coding agents **and their skills** — built for CLI
+**Coder Eval** (`pip install coder-eval` / `uv tool install coder-eval`) is an open-source framework for
+**evaluating and benchmarking AI coding agents and their skills** — built for CLI
 and skill builders — with sandboxing, reproducibility, and data-driven analysis.
-Not an "agentic coding" benchmark: it measures how effective your CLI and skills
-are when used by coding agents.
+It runs a real agent (**Claude Code**, **Codex**, or **Google Antigravity /
+Gemini**) in a sandbox against declarative YAML tasks, then scores the files and
+commands it actually produced. Not an "agentic coding" benchmark: it measures how
+effective your CLI and skills are when used by coding agents.
+
+Reach for it when you want to **test whether a Claude Code skill triggers**,
+**A/B-test Claude Code vs. Codex vs. Gemini** (or model vs. model, prompt vs.
+prompt), or **gate CI on coding-agent quality**. Unlike fixed datasets (SWE-bench,
+SkillsBench) that rank models on a shared leaderboard, Coder Eval evaluates the
+tasks, skills, and workflows *you* ship — with weighted 0.0–1.0 criteria, a
+`skill_triggered` activation check, an A/B experiment layer, and per-tool cost
+telemetry. See [How it compares](https://uipath.github.io/coder_eval/comparison/).
+📚 **Full docs:** **[uipath.github.io/coder_eval](https://uipath.github.io/coder_eval/)**.
 
 <p align="center">
-  <img src="docs/assets/hero.gif" alt="coder_eval running the hello_date task: a sandboxed agent writes and runs a script from a YAML task, then the scored result is browsed in evalboard" width="100%">
+  <img src="docs/assets/hero.gif" alt="Coder Eval running the hello_date task: a sandboxed agent writes and runs a script from a YAML task, then the scored result is browsed in evalboard" width="100%">
 </p>
 
-> **The Coding Agents Gym.** A sandboxed, reproducible framework to evaluate,
-> benchmark, and A/B-test AI coding agents — Claude Code, Codex, and Google
-> Antigravity (Gemini) today, any agent via a plugin SPI — with declarative
-> YAML tasks and weighted scoring.
-
 - **Declarative YAML tasks** with pinned dependencies and clear success criteria
 - **Sandboxed execution** in isolated environments with resource limits
 - **Weighted, continuous scoring** (0.0–1.0) with fractional credit and thresholds
@@ -92,14 +101,14 @@ are when used by coding agents.
 
 ## What you can do with it
 
-- **Benchmark coding agents** — score an agent across a suite of tasks with weighted, pass/fail thresholds
+- **Benchmark coding agents** — score an agent across a suite of tasks with weighted scoring and pass/fail thresholds
 - **Compare models & configs** — A/B-test Claude vs. Codex vs. Gemini, model vs. model, tool-on vs. tool-off, prompt vs. prompt
 - **Evaluate skills** — verify an agent actually engages a target skill (`skill_triggered`) and score skill-driven suites (SkillsBench-style)
 - **Keep skills up to date in CI** — re-validate your skills on every change or on a schedule; catch silent regressions when models, prompts, or the skills themselves drift
 - **Gate CI on agent quality** — run the suite in GitHub Actions and fail the build on regressions
 - **Bring your own dataset** — fan one task out over many rows for larger benchmark suites
 
-> **Keeping skills fresh?** Run coder_eval as a scheduled GitHub Actions job so your
+> **Keeping skills fresh?** Run Coder Eval as a scheduled GitHub Actions job so your
 > skills are continuously re-evaluated against the latest model — a skill that quietly
 > stops triggering surfaces as a failing criterion before your users hit it. See
 > **[Tutorial 02 — Running coder_eval in CI](docs/tutorials/02-ci-pipeline.md)**.
@@ -115,7 +124,9 @@ git clone https://github.com/UiPath/coder_eval.git
 cd coder_eval
 
 uv sync --extra dev          # install core + dev tools
-cp .env.example .env         # then set ANTHROPIC_API_KEY
+cp .env.example .env         # then set ANTHROPIC_API_KEY — or skip that: an
+                             # existing Claude Code login (`claude login`) is
+                             # picked up automatically
 
 uv run coder-eval plan tasks/hello_date.yaml   # validate (no tokens spent)
 uv run coder-eval run  tasks/hello_date.yaml   # run your first evaluation
@@ -129,11 +140,23 @@ The optional `[uipath]` extra (`uv sync --extra dev --extra uipath`) adds the in
 required). Without it the framework runs end-to-end; uipath-dependent features fail
 at dispatch with a clear hint.
 
-> **Using coder_eval in CI or another project?** Install the published package:
-> `pip install coder-eval` (or `uv add coder-eval`; extras install the same way —
-> `pip install "coder-eval[codex,antigravity]"`). In a real CI gate, pin to a
-> specific released version so a harness upgrade can't silently move your results.
-> See [Tutorial 02 — Running coder_eval in CI](docs/tutorials/02-ci-pipeline.md) for the full setup.
+**Using Coder Eval in CI or another project?** Install the published package
+instead of cloning:
+
+```bash
+uv tool install coder-eval    # puts the `coder-eval` CLI on your PATH,
+                              # in its own isolated environment
+
+uv tool install "coder-eval[codex,antigravity]"   # same, with agent extras
+coder-eval --version                              # verify the install
+```
+
+To add it as a project dependency instead: `uv add coder-eval` or
+`pip install coder-eval`. In a real CI gate, pin to a specific released version
+so a harness upgrade can't silently move your results. (The example `tasks/`
+live in this repo — clone it or point the CLI at your own task files.) See
+[Tutorial 02 — Running coder_eval in CI](docs/tutorials/02-ci-pipeline.md) for
+the full setup.
 
 ## Telemetry
 
@@ -161,17 +184,18 @@ at dispatch with a clear hint.
 
 ## How it compares
 
-- **vs. SWE-bench and fixed benchmarks** — SWE-bench is a fixed dataset of GitHub
-  issues; coder_eval is a *framework* for authoring your own tasks in declarative
-  YAML, so you evaluate the skills and workflows you care about (and can still wrap
-  a fixed dataset via [Bring Your Own Dataset](docs/BYOD.md)).
-- **vs. LLM-output eval harnesses (e.g. OpenAI Evals)** — those grade a model's text;
-  coder_eval runs a full **agent** in a **sandbox** with real tool use and multi-turn
-  dialog, then scores the files and commands it actually produced (continuous
-  0.0–1.0) — not just a judge over a string.
+- **vs. fixed benchmarks (SWE-bench, SkillsBench)** — they score a canonical dataset;
+  Coder Eval scores *your* tasks with continuous 0.0–1.0 weighted criteria (and can
+  still wrap a fixed dataset via [Bring Your Own Dataset](docs/BYOD.md)).
+- **vs. large-scale / RL harnesses (Harbor)** — Harbor targets scale and RL rollouts;
+  Coder Eval targets weighted, skill-aware suites gated in CI.
+- **vs. model-output eval tools (OpenAI Evals)** — they grade model text; Coder Eval
+  runs a full agent in a sandbox and scores the files and commands it produced.
 - **vs. hand-rolled scripts** — reproducible sandboxes, weighted criteria,
   cost/token telemetry, A/B experiments, and CI-ready pass/fail gates out of the box.
 
+See the full [comparison — with sources](https://uipath.github.io/coder_eval/comparison/).
+
 ## Task Definition
 
 A task is a YAML file: a prompt, the agent config, a sandbox, and success criteria.
@@ -221,12 +245,12 @@ extension points (new criteria, new agents).
 
 ## Known limits & non-goals
 
-- **Not a fixed benchmark or leaderboard** — coder_eval scores *your* tasks and ships
+- **Not a fixed benchmark or leaderboard** — Coder Eval scores *your* tasks and ships
   example tasks, not a canonical scored dataset.
 - **Tasks execute real code** — run untrusted tasks only under the container driver
   (see [Docker Isolation](docs/DOCKER_ISOLATION.md)); the `tempdir` driver is not a
   security boundary.
-- **Bring your own model credentials** — Anthropic, Bedrock, or Gemini keys; coder_eval
+- **Bring your own model credentials** — Anthropic, Bedrock, or Gemini keys; Coder Eval
   does not proxy or supply model access.
 - **Python 3.13+ only.**