coder-eval 0.8.7rc10__tar.gz → 0.8.8__tar.gz

coder_eval-0.8.8/.github/workflows/docs.yml

@@ -0,0 +1,48 @@
+name: Docs
+
+# Build the MkDocs Material site and publish it to the `gh-pages` branch via
+# `mkdocs gh-deploy` — the same mechanism UiPath/uipath-python uses. This workflow
+# only pushes a branch (needs `contents: write`); it never calls the Pages API, so
+# it succeeds even before Pages is switched on.
+#
+# One-time setup (org owner): the UiPath org blocks Pages *creation*, so enable
+# Pages for this repo once — Settings → Pages → Build and deployment →
+# Source: "Deploy from a branch" → branch `gh-pages` / `/ (root)`. That's the same
+# (legacy, branch-based) configuration uipath-python already runs on.
+on:
+  push:
+    branches: [main]
+    paths:
+      - "docs/**"
+      - "mkdocs.yml"
+      - ".github/workflows/docs.yml"
+  workflow_dispatch:
+
+permissions:
+  contents: write
+
+concurrency:
+  group: docs-gh-pages
+  cancel-in-progress: true
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+      - name: Install social-card system libraries
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y --no-install-recommends \
+            libcairo2 libfreetype6 libjpeg-turbo8 libpng16-16 pngquant
+      - name: Install MkDocs Material
+        run: pip install "mkdocs-material[imaging]>=9.5,<10"
+      - name: Configure git identity for gh-pages commits
+        run: |
+          git config user.name "coder-eval"
+          git config user.email "coder-eval@uipath.com"
+      - name: Build strictly and publish to gh-pages
+        run: mkdocs gh-deploy --force --strict

{coder_eval-0.8.7rc10 → coder_eval-0.8.8}/.github/workflows/release.yml

@@ -25,7 +25,7 @@ on:
   workflow_dispatch:
     inputs:
       bump:
-        description: 'Version bump to release'
+        description: 'Version bump to release (ignored on non-main / prerelease dispatch, which always stamps a next-patch rc)'
         type: choice
         default: patch
         options:

{coder_eval-0.8.7rc10 → coder_eval-0.8.8}/.gitignore

@@ -1,3 +1,7 @@
+# MkDocs build output
+site/
+.cache/
+
 # Python-generated files
 __pycache__/
 *.py[oc]

{coder_eval-0.8.7rc10 → coder_eval-0.8.8}/CHANGELOG.md

@@ -2,6 +2,115 @@
 
 <!-- version list -->
 
+## v0.8.8 (2026-07-22)
+
+### Bug Fixes
+
+- **codex**: Create CODEX_HOME before pinning it in the app-server env
+  ([#39](https://github.com/UiPath/coder_eval/pull/39),
+  [`8d98b91`](https://github.com/UiPath/coder_eval/commit/8d98b912ae4767904c7d1c395913c3d0aaec6c66))
+
+### Documentation
+
+- Add Website badge and point PyPI Homepage at coder-eval.com
+  ([#41](https://github.com/UiPath/coder_eval/pull/41),
+  [`0551534`](https://github.com/UiPath/coder_eval/commit/055153409d86b8d537bba73ef7ed58dfd17fee26))
+
+
+## v0.8.7 (2026-07-22)
+
+### Bug Fixes
+
+- Detect Windows skill paths in telemetry ([#24](https://github.com/UiPath/coder_eval/pull/24),
+  [`c57a6b0`](https://github.com/UiPath/coder_eval/commit/c57a6b0408ef116177b15bc9b09a768096aaa33f))
+
+- **agents**: Run codex + antigravity harnesses on the tempdir/host path
+  ([#33](https://github.com/UiPath/coder_eval/pull/33),
+  [`04498a9`](https://github.com/UiPath/coder_eval/commit/04498a96f6b9ac0cc5c73f36e73a361f3f3fa0ae))
+
+- **antigravity**: Pin google-antigravity 0.1.7 to load on glibc 2.35
+  ([#33](https://github.com/UiPath/coder_eval/pull/33),
+  [`04498a9`](https://github.com/UiPath/coder_eval/commit/04498a96f6b9ac0cc5c73f36e73a361f3f3fa0ae))
+
+- **codex**: Always run full-access; drop the in-process OS sandbox
+  ([#33](https://github.com/UiPath/coder_eval/pull/33),
+  [`04498a9`](https://github.com/UiPath/coder_eval/commit/04498a96f6b9ac0cc5c73f36e73a361f3f3fa0ae))
+
+- **codex**: Cover zsh login shells (macOS default) in the mock-PATH home shim
+  ([#26](https://github.com/UiPath/coder_eval/pull/26),
+  [`73f0db2`](https://github.com/UiPath/coder_eval/commit/73f0db24df0b67f82d46e3bf15c85a21837237ef))
+
+- **codex**: Keep mock CLIs shadowed in bash login shells
+  ([#26](https://github.com/UiPath/coder_eval/pull/26),
+  [`73f0db2`](https://github.com/UiPath/coder_eval/commit/73f0db24df0b67f82d46e3bf15c85a21837237ef))
+
+- **codex**: Make login-shell temp-home lifecycle exception-safe incl. kill_sync
+  ([#26](https://github.com/UiPath/coder_eval/pull/26),
+  [`73f0db2`](https://github.com/UiPath/coder_eval/commit/73f0db24df0b67f82d46e3bf15c85a21837237ef))
+
+- **codex**: Restore mock-CLI PATH prepend in login shells via per-task HOME profile
+  ([#26](https://github.com/UiPath/coder_eval/pull/26),
+  [`73f0db2`](https://github.com/UiPath/coder_eval/commit/73f0db24df0b67f82d46e3bf15c85a21837237ef))
+
+- **codex**: Restore original HOME inside generated login-shell profile
+  ([#26](https://github.com/UiPath/coder_eval/pull/26),
+  [`73f0db2`](https://github.com/UiPath/coder_eval/commit/73f0db24df0b67f82d46e3bf15c85a21837237ef))
+
+- **codex**: Use full-access sandbox on coder_eval-managed tempdir path
+  ([#33](https://github.com/UiPath/coder_eval/pull/33),
+  [`04498a9`](https://github.com/UiPath/coder_eval/commit/04498a96f6b9ac0cc5c73f36e73a361f3f3fa0ae))
+
+- **deps**: Bump pyasn1 to 0.6.4 for GHSA-8ppf-4f7h-5ppj / GHSA-hm4w-wwcw-mr6r
+  ([#33](https://github.com/UiPath/coder_eval/pull/33),
+  [`04498a9`](https://github.com/UiPath/coder_eval/commit/04498a96f6b9ac0cc5c73f36e73a361f3f3fa0ae))
+
+### Chores
+
+- **deps**: Bump pyasn1 0.6.3 -> 0.6.4 (GHSA-8ppf-4f7h-5ppj, GHSA-hm4w-wwcw-mr6r)
+  ([#32](https://github.com/UiPath/coder_eval/pull/32),
+  [`f1f4f9f`](https://github.com/UiPath/coder_eval/commit/f1f4f9fba2bd034a271e50fc1a95bcc74c2ac19b))
+
+- **deps**: Upgrade agent SDKs to latest (claude 0.2.124, codex 0.144.4)
+  ([#33](https://github.com/UiPath/coder_eval/pull/33),
+  [`04498a9`](https://github.com/UiPath/coder_eval/commit/04498a96f6b9ac0cc5c73f36e73a361f3f3fa0ae))
+
+### Continuous Integration
+
+- **release**: Publish a prerelease from a non-main branch
+  ([#33](https://github.com/UiPath/coder_eval/pull/33),
+  [`04498a9`](https://github.com/UiPath/coder_eval/commit/04498a96f6b9ac0cc5c73f36e73a361f3f3fa0ae))
+
+### Documentation
+
+- Add MkDocs docs site, comparison page, and SEO metadata
+  ([#32](https://github.com/UiPath/coder_eval/pull/32),
+  [`f1f4f9f`](https://github.com/UiPath/coder_eval/commit/f1f4f9fba2bd034a271e50fc1a95bcc74c2ac19b))
+
+- Address PR review — Coder Eval naming, cleaner table, gh-deploy workflow
+  ([#32](https://github.com/UiPath/coder_eval/pull/32),
+  [`f1f4f9f`](https://github.com/UiPath/coder_eval/commit/f1f4f9fba2bd034a271e50fc1a95bcc74c2ac19b))
+
+- Address review — git identity for gh-pages, single strict deploy, table glyphs, uv tool install in
+  Tutorial 01 ([#32](https://github.com/UiPath/coder_eval/pull/32),
+  [`f1f4f9f`](https://github.com/UiPath/coder_eval/commit/f1f4f9fba2bd034a271e50fc1a95bcc74c2ac19b))
+
+### Refactoring
+
+- **codex**: Drop dead sandbox branch + honest full-access messaging
+  ([#33](https://github.com/UiPath/coder_eval/pull/33),
+  [`04498a9`](https://github.com/UiPath/coder_eval/commit/04498a96f6b9ac0cc5c73f36e73a361f3f3fa0ae))
+
+### Testing
+
+- Accept sandbox_managed kwarg in agent test doubles
+  ([#33](https://github.com/UiPath/coder_eval/pull/33),
+  [`04498a9`](https://github.com/UiPath/coder_eval/commit/04498a96f6b9ac0cc5c73f36e73a361f3f3fa0ae))
+
+- **codex**: Pin start-to-CodexConfig env composition and tidy test imports
+  ([#26](https://github.com/UiPath/coder_eval/pull/26),
+  [`73f0db2`](https://github.com/UiPath/coder_eval/commit/73f0db24df0b67f82d46e3bf15c85a21837237ef))
+
+
 ## v0.8.6 (2026-07-20)
 
 ### Features

{coder_eval-0.8.7rc10 → coder_eval-0.8.8}/PKG-INFO

@@ -1,8 +1,8 @@
 Metadata-Version: 2.4
 Name: coder-eval
-Version: 0.8.7rc10
+Version: 0.8.8
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity) with sandboxed, reproducible YAML task suites.
-Project-URL: Homepage, https://github.com/UiPath/coder_eval
+Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval
 Project-URL: Documentation, https://github.com/UiPath/coder_eval/tree/main/docs
 Project-URL: Issues, https://github.com/UiPath/coder_eval/issues
@@ -11,7 +11,7 @@ Author-email: UiPath <coder-eval@uipath.com>
 License-Expression: Apache-2.0
 License-File: LICENSE
 License-File: NOTICE
-Keywords: agent,agent-evaluation,agent-skills,ai,anthropic,antigravity,benchmark,claude,claude-code,claude-skills,code-generation,codex,coding-agent,eval,evals,evaluation,gemini,llm,llm-evaluation,sandbox,skills-evaluation,skillsbench,swe-bench
+Keywords: agent,agent-evaluation,agent-skills,agent-testing,ai,ai-evaluation,anthropic,antigravity,benchmark,claude,claude-code,claude-code-skills,claude-skills,code-generation,codex,coding-agent,eval,evals,evaluation,gemini,llm,llm-eval,llm-evaluation,llmops,sandbox,skills-evaluation,skillsbench,swe-bench
 Classifier: Development Status :: 4 - Beta
 Classifier: Environment :: Console
 Classifier: Intended Audience :: Developers
@@ -60,28 +60,37 @@ Provides-Extra: uipath
 Requires-Dist: uipath>=2.10.31; extra == 'uipath'
 Description-Content-Type: text/markdown
 
-# coder_eval — evaluate AI coding agents & their skills
+# Coder Eval — evaluate & benchmark AI coding agents and Claude Code skills
 
 [![PyPI](https://img.shields.io/pypi/v/coder-eval.svg)](https://pypi.org/project/coder-eval/)
+[![Website](https://img.shields.io/badge/website-coder--eval.com-1f6feb.svg)](https://coder-eval.com)
+[![Docs](https://img.shields.io/badge/docs-uipath.github.io%2Fcoder__eval-1f6feb.svg)](https://uipath.github.io/coder_eval/)
 [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
 [![Python 3.13+](https://img.shields.io/badge/python-3.13%2B-blue.svg)](https://www.python.org/downloads/)
 [![CI](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml/badge.svg)](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml)
 [![Code style: Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
 
-A framework for evaluating AI coding agents **and their skills** — built for CLI
+**Coder Eval** (`pip install coder-eval` / `uv tool install coder-eval`) is an open-source framework for
+**evaluating and benchmarking AI coding agents and their skills** — built for CLI
 and skill builders — with sandboxing, reproducibility, and data-driven analysis.
-Not an "agentic coding" benchmark: it measures how effective your CLI and skills
-are when used by coding agents.
+It runs a real agent (**Claude Code**, **Codex**, or **Google Antigravity /
+Gemini**) in a sandbox against declarative YAML tasks, then scores the files and
+commands it actually produced. Not an "agentic coding" benchmark: it measures how
+effective your CLI and skills are when used by coding agents.
+
+Reach for it when you want to **test whether a Claude Code skill triggers**,
+**A/B-test Claude Code vs. Codex vs. Gemini** (or model vs. model, prompt vs.
+prompt), or **gate CI on coding-agent quality**. Unlike fixed datasets (SWE-bench,
+SkillsBench) that rank models on a shared leaderboard, Coder Eval evaluates the
+tasks, skills, and workflows *you* ship — with weighted 0.0–1.0 criteria, a
+`skill_triggered` activation check, an A/B experiment layer, and per-tool cost
+telemetry. See [How it compares](https://uipath.github.io/coder_eval/comparison/).
+📚 **Full docs:** **[uipath.github.io/coder_eval](https://uipath.github.io/coder_eval/)**.
 
 <p align="center">
-  <img src="docs/assets/hero.gif" alt="coder_eval running the hello_date task: a sandboxed agent writes and runs a script from a YAML task, then the scored result is browsed in evalboard" width="100%">
+  <img src="docs/assets/hero.gif" alt="Coder Eval running the hello_date task: a sandboxed agent writes and runs a script from a YAML task, then the scored result is browsed in evalboard" width="100%">
 </p>
 
-> **The Coding Agents Gym.** A sandboxed, reproducible framework to evaluate,
-> benchmark, and A/B-test AI coding agents — Claude Code, Codex, and Google
-> Antigravity (Gemini) today, any agent via a plugin SPI — with declarative
-> YAML tasks and weighted scoring.
-
 - **Declarative YAML tasks** with pinned dependencies and clear success criteria
 - **Sandboxed execution** in isolated environments with resource limits
 - **Weighted, continuous scoring** (0.0–1.0) with fractional credit and thresholds
@@ -92,14 +101,14 @@ are when used by coding agents.
 
 ## What you can do with it
 
-- **Benchmark coding agents** — score an agent across a suite of tasks with weighted, pass/fail thresholds
+- **Benchmark coding agents** — score an agent across a suite of tasks with weighted scoring and pass/fail thresholds
 - **Compare models & configs** — A/B-test Claude vs. Codex vs. Gemini, model vs. model, tool-on vs. tool-off, prompt vs. prompt
 - **Evaluate skills** — verify an agent actually engages a target skill (`skill_triggered`) and score skill-driven suites (SkillsBench-style)
 - **Keep skills up to date in CI** — re-validate your skills on every change or on a schedule; catch silent regressions when models, prompts, or the skills themselves drift
 - **Gate CI on agent quality** — run the suite in GitHub Actions and fail the build on regressions
 - **Bring your own dataset** — fan one task out over many rows for larger benchmark suites
 
-> **Keeping skills fresh?** Run coder_eval as a scheduled GitHub Actions job so your
+> **Keeping skills fresh?** Run Coder Eval as a scheduled GitHub Actions job so your
 > skills are continuously re-evaluated against the latest model — a skill that quietly
 > stops triggering surfaces as a failing criterion before your users hit it. See
 > **[Tutorial 02 — Running coder_eval in CI](docs/tutorials/02-ci-pipeline.md)**.
@@ -115,7 +124,9 @@ git clone https://github.com/UiPath/coder_eval.git
 cd coder_eval
 
 uv sync --extra dev          # install core + dev tools
-cp .env.example .env         # then set ANTHROPIC_API_KEY
+cp .env.example .env         # then set ANTHROPIC_API_KEY — or skip that: an
+                             # existing Claude Code login (`claude login`) is
+                             # picked up automatically
 
 uv run coder-eval plan tasks/hello_date.yaml   # validate (no tokens spent)
 uv run coder-eval run  tasks/hello_date.yaml   # run your first evaluation
@@ -129,11 +140,23 @@ The optional `[uipath]` extra (`uv sync --extra dev --extra uipath`) adds the in
 required). Without it the framework runs end-to-end; uipath-dependent features fail
 at dispatch with a clear hint.
 
-> **Using coder_eval in CI or another project?** Install the published package:
-> `pip install coder-eval` (or `uv add coder-eval`; extras install the same way —
-> `pip install "coder-eval[codex,antigravity]"`). In a real CI gate, pin to a
-> specific released version so a harness upgrade can't silently move your results.
-> See [Tutorial 02 — Running coder_eval in CI](docs/tutorials/02-ci-pipeline.md) for the full setup.
+**Using Coder Eval in CI or another project?** Install the published package
+instead of cloning:
+
+```bash
+uv tool install coder-eval    # puts the `coder-eval` CLI on your PATH,
+                              # in its own isolated environment
+
+uv tool install "coder-eval[codex,antigravity]"   # same, with agent extras
+coder-eval --version                              # verify the install
+```
+
+To add it as a project dependency instead: `uv add coder-eval` or
+`pip install coder-eval`. In a real CI gate, pin to a specific released version
+so a harness upgrade can't silently move your results. (The example `tasks/`
+live in this repo — clone it or point the CLI at your own task files.) See
+[Tutorial 02 — Running coder_eval in CI](docs/tutorials/02-ci-pipeline.md) for
+the full setup.
 
 ## Telemetry
 
@@ -161,17 +184,18 @@ at dispatch with a clear hint.
 
 ## How it compares
 
-- **vs. SWE-bench and fixed benchmarks** — SWE-bench is a fixed dataset of GitHub
-  issues; coder_eval is a *framework* for authoring your own tasks in declarative
-  YAML, so you evaluate the skills and workflows you care about (and can still wrap
-  a fixed dataset via [Bring Your Own Dataset](docs/BYOD.md)).
-- **vs. LLM-output eval harnesses (e.g. OpenAI Evals)** — those grade a model's text;
-  coder_eval runs a full **agent** in a **sandbox** with real tool use and multi-turn
-  dialog, then scores the files and commands it actually produced (continuous
-  0.0–1.0) — not just a judge over a string.
+- **vs. fixed benchmarks (SWE-bench, SkillsBench)** — they score a canonical dataset;
+  Coder Eval scores *your* tasks with continuous 0.0–1.0 weighted criteria (and can
+  still wrap a fixed dataset via [Bring Your Own Dataset](docs/BYOD.md)).
+- **vs. large-scale / RL harnesses (Harbor)** — Harbor targets scale and RL rollouts;
+  Coder Eval targets weighted, skill-aware suites gated in CI.
+- **vs. model-output eval tools (OpenAI Evals)** — they grade model text; Coder Eval
+  runs a full agent in a sandbox and scores the files and commands it produced.
 - **vs. hand-rolled scripts** — reproducible sandboxes, weighted criteria,
   cost/token telemetry, A/B experiments, and CI-ready pass/fail gates out of the box.
 
+See the full [comparison — with sources](https://uipath.github.io/coder_eval/comparison/).
+
 ## Task Definition
 
 A task is a YAML file: a prompt, the agent config, a sandbox, and success criteria.
@@ -221,12 +245,12 @@ extension points (new criteria, new agents).
 
 ## Known limits & non-goals
 
-- **Not a fixed benchmark or leaderboard** — coder_eval scores *your* tasks and ships
+- **Not a fixed benchmark or leaderboard** — Coder Eval scores *your* tasks and ships
   example tasks, not a canonical scored dataset.
 - **Tasks execute real code** — run untrusted tasks only under the container driver
   (see [Docker Isolation](docs/DOCKER_ISOLATION.md)); the `tempdir` driver is not a
   security boundary.
-- **Bring your own model credentials** — Anthropic, Bedrock, or Gemini keys; coder_eval
+- **Bring your own model credentials** — Anthropic, Bedrock, or Gemini keys; Coder Eval
   does not proxy or supply model access.
 - **Python 3.13+ only.**
 

{coder_eval-0.8.7rc10 → coder_eval-0.8.8}/README.md

@@ -1,25 +1,34 @@
-# coder_eval — evaluate AI coding agents & their skills
+# Coder Eval — evaluate & benchmark AI coding agents and Claude Code skills
 
 [![PyPI](https://img.shields.io/pypi/v/coder-eval.svg)](https://pypi.org/project/coder-eval/)
+[![Website](https://img.shields.io/badge/website-coder--eval.com-1f6feb.svg)](https://coder-eval.com)
+[![Docs](https://img.shields.io/badge/docs-uipath.github.io%2Fcoder__eval-1f6feb.svg)](https://uipath.github.io/coder_eval/)
 [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
 [![Python 3.13+](https://img.shields.io/badge/python-3.13%2B-blue.svg)](https://www.python.org/downloads/)
 [![CI](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml/badge.svg)](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml)
 [![Code style: Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
 
-A framework for evaluating AI coding agents **and their skills** — built for CLI
+**Coder Eval** (`pip install coder-eval` / `uv tool install coder-eval`) is an open-source framework for
+**evaluating and benchmarking AI coding agents and their skills** — built for CLI
 and skill builders — with sandboxing, reproducibility, and data-driven analysis.
-Not an "agentic coding" benchmark: it measures how effective your CLI and skills
-are when used by coding agents.
+It runs a real agent (**Claude Code**, **Codex**, or **Google Antigravity /
+Gemini**) in a sandbox against declarative YAML tasks, then scores the files and
+commands it actually produced. Not an "agentic coding" benchmark: it measures how
+effective your CLI and skills are when used by coding agents.
+
+Reach for it when you want to **test whether a Claude Code skill triggers**,
+**A/B-test Claude Code vs. Codex vs. Gemini** (or model vs. model, prompt vs.
+prompt), or **gate CI on coding-agent quality**. Unlike fixed datasets (SWE-bench,
+SkillsBench) that rank models on a shared leaderboard, Coder Eval evaluates the
+tasks, skills, and workflows *you* ship — with weighted 0.0–1.0 criteria, a
+`skill_triggered` activation check, an A/B experiment layer, and per-tool cost
+telemetry. See [How it compares](https://uipath.github.io/coder_eval/comparison/).
+📚 **Full docs:** **[uipath.github.io/coder_eval](https://uipath.github.io/coder_eval/)**.
 
 <p align="center">
-  <img src="docs/assets/hero.gif" alt="coder_eval running the hello_date task: a sandboxed agent writes and runs a script from a YAML task, then the scored result is browsed in evalboard" width="100%">
+  <img src="docs/assets/hero.gif" alt="Coder Eval running the hello_date task: a sandboxed agent writes and runs a script from a YAML task, then the scored result is browsed in evalboard" width="100%">
 </p>
 
-> **The Coding Agents Gym.** A sandboxed, reproducible framework to evaluate,
-> benchmark, and A/B-test AI coding agents — Claude Code, Codex, and Google
-> Antigravity (Gemini) today, any agent via a plugin SPI — with declarative
-> YAML tasks and weighted scoring.
-
 - **Declarative YAML tasks** with pinned dependencies and clear success criteria
 - **Sandboxed execution** in isolated environments with resource limits
 - **Weighted, continuous scoring** (0.0–1.0) with fractional credit and thresholds
@@ -30,14 +39,14 @@ are when used by coding agents.
 
 ## What you can do with it
 
-- **Benchmark coding agents** — score an agent across a suite of tasks with weighted, pass/fail thresholds
+- **Benchmark coding agents** — score an agent across a suite of tasks with weighted scoring and pass/fail thresholds
 - **Compare models & configs** — A/B-test Claude vs. Codex vs. Gemini, model vs. model, tool-on vs. tool-off, prompt vs. prompt
 - **Evaluate skills** — verify an agent actually engages a target skill (`skill_triggered`) and score skill-driven suites (SkillsBench-style)
 - **Keep skills up to date in CI** — re-validate your skills on every change or on a schedule; catch silent regressions when models, prompts, or the skills themselves drift
 - **Gate CI on agent quality** — run the suite in GitHub Actions and fail the build on regressions
 - **Bring your own dataset** — fan one task out over many rows for larger benchmark suites
 
-> **Keeping skills fresh?** Run coder_eval as a scheduled GitHub Actions job so your
+> **Keeping skills fresh?** Run Coder Eval as a scheduled GitHub Actions job so your
 > skills are continuously re-evaluated against the latest model — a skill that quietly
 > stops triggering surfaces as a failing criterion before your users hit it. See
 > **[Tutorial 02 — Running coder_eval in CI](docs/tutorials/02-ci-pipeline.md)**.
@@ -53,7 +62,9 @@ git clone https://github.com/UiPath/coder_eval.git
 cd coder_eval
 
 uv sync --extra dev          # install core + dev tools
-cp .env.example .env         # then set ANTHROPIC_API_KEY
+cp .env.example .env         # then set ANTHROPIC_API_KEY — or skip that: an
+                             # existing Claude Code login (`claude login`) is
+                             # picked up automatically
 
 uv run coder-eval plan tasks/hello_date.yaml   # validate (no tokens spent)
 uv run coder-eval run  tasks/hello_date.yaml   # run your first evaluation
@@ -67,11 +78,23 @@ The optional `[uipath]` extra (`uv sync --extra dev --extra uipath`) adds the in
 required). Without it the framework runs end-to-end; uipath-dependent features fail
 at dispatch with a clear hint.
 
-> **Using coder_eval in CI or another project?** Install the published package:
-> `pip install coder-eval` (or `uv add coder-eval`; extras install the same way —
-> `pip install "coder-eval[codex,antigravity]"`). In a real CI gate, pin to a
-> specific released version so a harness upgrade can't silently move your results.
-> See [Tutorial 02 — Running coder_eval in CI](docs/tutorials/02-ci-pipeline.md) for the full setup.
+**Using Coder Eval in CI or another project?** Install the published package
+instead of cloning:
+
+```bash
+uv tool install coder-eval    # puts the `coder-eval` CLI on your PATH,
+                              # in its own isolated environment
+
+uv tool install "coder-eval[codex,antigravity]"   # same, with agent extras
+coder-eval --version                              # verify the install
+```
+
+To add it as a project dependency instead: `uv add coder-eval` or
+`pip install coder-eval`. In a real CI gate, pin to a specific released version
+so a harness upgrade can't silently move your results. (The example `tasks/`
+live in this repo — clone it or point the CLI at your own task files.) See
+[Tutorial 02 — Running coder_eval in CI](docs/tutorials/02-ci-pipeline.md) for
+the full setup.
 
 ## Telemetry
 
@@ -99,17 +122,18 @@ at dispatch with a clear hint.
 
 ## How it compares
 
-- **vs. SWE-bench and fixed benchmarks** — SWE-bench is a fixed dataset of GitHub
-  issues; coder_eval is a *framework* for authoring your own tasks in declarative
-  YAML, so you evaluate the skills and workflows you care about (and can still wrap
-  a fixed dataset via [Bring Your Own Dataset](docs/BYOD.md)).
-- **vs. LLM-output eval harnesses (e.g. OpenAI Evals)** — those grade a model's text;
-  coder_eval runs a full **agent** in a **sandbox** with real tool use and multi-turn
-  dialog, then scores the files and commands it actually produced (continuous
-  0.0–1.0) — not just a judge over a string.
+- **vs. fixed benchmarks (SWE-bench, SkillsBench)** — they score a canonical dataset;
+  Coder Eval scores *your* tasks with continuous 0.0–1.0 weighted criteria (and can
+  still wrap a fixed dataset via [Bring Your Own Dataset](docs/BYOD.md)).
+- **vs. large-scale / RL harnesses (Harbor)** — Harbor targets scale and RL rollouts;
+  Coder Eval targets weighted, skill-aware suites gated in CI.
+- **vs. model-output eval tools (OpenAI Evals)** — they grade model text; Coder Eval
+  runs a full agent in a sandbox and scores the files and commands it produced.
 - **vs. hand-rolled scripts** — reproducible sandboxes, weighted criteria,
   cost/token telemetry, A/B experiments, and CI-ready pass/fail gates out of the box.
 
+See the full [comparison — with sources](https://uipath.github.io/coder_eval/comparison/).
+
 ## Task Definition
 
 A task is a YAML file: a prompt, the agent config, a sandbox, and success criteria.
@@ -159,12 +183,12 @@ extension points (new criteria, new agents).
 
 ## Known limits & non-goals
 
-- **Not a fixed benchmark or leaderboard** — coder_eval scores *your* tasks and ships
+- **Not a fixed benchmark or leaderboard** — Coder Eval scores *your* tasks and ships
   example tasks, not a canonical scored dataset.
 - **Tasks execute real code** — run untrusted tasks only under the container driver
   (see [Docker Isolation](docs/DOCKER_ISOLATION.md)); the `tempdir` driver is not a
   security boundary.
-- **Bring your own model credentials** — Anthropic, Bedrock, or Gemini keys; coder_eval
+- **Bring your own model credentials** — Anthropic, Bedrock, or Gemini keys; Coder Eval
   does not proxy or supply model access.
 - **Python 3.13+ only.**
 

{coder_eval-0.8.7rc10 → coder_eval-0.8.8}/docs/AB_EXPERIMENTS.md

@@ -1,3 +1,10 @@
+---
+description: >-
+  A/B-test AI coding agents with coder_eval's experiment layer — Claude Code vs.
+  Codex vs. Gemini, model vs. model, skill on vs. off, prompt vs. prompt — on
+  identical tasks.
+---
+
 # A/B Experiment Guide
 
 How to run the same tasks across multiple configuration variants ("arms") and

{coder_eval-0.8.7rc10 → coder_eval-0.8.8}/docs/BYOD.md

@@ -1,3 +1,10 @@
+---
+description: >-
+  Use a custom Docker image with coder_eval — extend the base coder-eval-agent
+  image with your own dependencies and tools, then point task configuration at
+  it.
+---
+
 # Bring Your Own Docker (BYOD)
 
 The BYOD feature allows customers to use custom Docker images that extend the base `coder-eval-agent` image, enabling them to add custom dependencies and tools while maintaining the latest coder-eval codebase.

{coder_eval-0.8.7rc10 → coder_eval-0.8.8}/docs/CODEX_AGENT_GUIDE.md

@@ -1,8 +1,15 @@
-# Codex Agent Implementation
+---
+description: >-
+  Run OpenAI Codex as the agent under evaluation in coder_eval — installation,
+  authentication, task configuration, and how Codex telemetry maps to sandboxed,
+  weighted scoring.
+---
+
+# Running OpenAI Codex in coder_eval
 
 ## Overview
 
-A new `CodexAgent` has been added to coder_eval that integrates OpenAI's Codex SDK. The implementation mirrors the structure of `ClaudeCodeAgent` and provides seamless integration with the evaluation framework.
+coder_eval can run OpenAI's Codex as the agent under evaluation, via the official Codex SDK. The `CodexAgent` mirrors the structure of `ClaudeCodeAgent` and plugs into the same sandbox, scoring, and telemetry pipeline — set `agent.type: codex` in a task and the rest of the framework works unchanged.
 
 ## Setup
 

{coder_eval-0.8.7rc10 → coder_eval-0.8.8}/docs/DOCKER_ISOLATION.md

@@ -1,9 +1,14 @@
+---
+description: >-
+  Run each coder_eval task in its own fresh Docker container — strong host
+  isolation, a pinned reproducible agent runtime, and custom images for
+  task-specific dependencies.
+---
+
 # Docker Isolation
 
 Run each evaluation task inside its own fresh container. Strong host isolation and a pinned, reproducible agent runtime.
 
-> Supersedes the agent-side FS perimeter flag from #199 (reverted in 9fe4320). The container boundary subsumes what that flag tried to do at the agent level.
-
 ## When to use
 
 Set `sandbox.driver: docker` on a task (or pass `--driver docker` on the CLI —

{coder_eval-0.8.7rc10 → coder_eval-0.8.8}/docs/TASK_DEFINITION_GUIDE.md

@@ -1,3 +1,10 @@
+---
+description: >-
+  Full schema reference for coder_eval task YAML — agent config, sandboxes, run
+  limits, dataset fan-out, and all 14 success criterion types with weighted
+  0.0–1.0 scoring.
+---
+
 # Task Definition Guide
 
 Complete reference for defining evaluation tasks in coder_eval.
@@ -123,11 +130,15 @@ an error.
 - `plan` — Agent proposes changes, waits for approval
 - `bypassPermissions` — No permission checks (use with caution)
 
+> **Codex note:** `permission_mode` confines the **`claude-code`** agent only. The **`codex`** agent always runs full-access regardless of the mode — its in-process OS sandbox is redundant given coder_eval's docker/tempdir isolation and unusable on our CI hosts (and on Windows). Run adversarial or untrusted Codex evals under the **docker driver**, which is the OS-level write boundary; the tempdir/host driver is a working directory, not a confinement boundary.
+
 **Agent Types:**
 - `claude-code` (default) — Claude Code SDK agent. Supports `sdk_options`, `claude_settings`, and all permission modes.
 - `codex` — OpenAI Codex agent (requires `[codex]` extra; set `CODEX_API_KEY` and optional `CODEX_BASE_URL` environment variables).
 - `none` — No-op agent: no coding agent runs and no model API call is made. See [No-op / System Tasks](#no-op--system-tasks-type-none) below.
 
+<a id="no-op--system-tasks-type-none"></a>
+
 ### No-op / System Tasks (`type: none`)
 
 Set `agent: {type: none}` to run a task with **no coding agent** — "coder-eval
@@ -159,7 +170,7 @@ Contract (enforced at load): a `type: none` task must declare no `initial_prompt
 every criterion must be agent-independent — criteria that inspect the agent
 trajectory (`command_executed`, `skill_triggered`, `reference_comparison`,
 `commands_efficiency`) are rejected. A worked example lives at
-[`tasks/agentless_smoke_test.yaml`](../tasks/agentless_smoke_test.yaml).
+[`tasks/agentless_smoke_test.yaml`](https://github.com/UiPath/coder_eval/blob/main/tasks/agentless_smoke_test.yaml).
 
 ### `max_turns`, `task_timeout`, `turn_timeout` location
 
@@ -590,7 +601,7 @@ Checks whether the agent executed specific tools/commands during evaluation. Ins
 
 Evaluates a UiPath agent against a named evaluation set. **Fractional scoring:** metrics passed / total metrics.
 
-> The `uipath` CLI must be available **inside the sandbox** (typically declared in the task's own Python deps). This is independent of the host's optional `coder-eval[uipath]` extra — see the install matrix in [README.md](../README.md#installation).
+> The `uipath` CLI must be available **inside the sandbox** (typically declared in the task's own Python deps). This is independent of the host's optional `coder-eval[uipath]` extra — see the install matrix in [README.md](https://github.com/UiPath/coder_eval/blob/main/README.md#quick-start).
 
 ```yaml
 - type: "uipath_eval"

{coder_eval-0.8.7rc10 → coder_eval-0.8.8}/docs/USER_GUIDE.md

@@ -1,3 +1,10 @@
+---
+description: >-
+  Complete coder_eval reference — every CLI command and flag, configuration
+  layers and -D overrides, environment variables, run outputs, and reports for
+  evaluating AI coding agents.
+---
+
 # coder_eval User Guide
 
 The full command, configuration, and output reference. For a gentle introduction
@@ -81,6 +88,8 @@ in `.claude/commands/`, available when using Claude Code in this repository:
 | `/coder-eval-run-analysis <path>` | Analyze evaluation runs and suggest improvements to tasks, config, and prompts. Works at task, variant, or run scope. |
 | `/coder-eval-task-create` | Create evaluation task YAML files from a natural language description. |
 
+<a id="api-routing--benchmarking"></a>
+
 ## API Routing & Benchmarking
 
 `coder-eval` supports two API routing modes, selected via `--backend` or the