PyPI - coder-eval - Versions diffs - 0.11.6__tar.gz → 0.11.7__tar.gz - Mend

coder-eval 0.11.6tar.gz → 0.11.7tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (667) hide show

{coder_eval-0.11.6 → coder_eval-0.11.7}/.github/pages-stub/index.html RENAMED Viewed

@@ -3,7 +3,7 @@
   <head>
     <meta charset="utf-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1" />
-    <title>Coder Eval — evaluate AI coding agents and Claude Code skills</title>
+    <title>Coder Eval — evaluate AI coding agents and their skills</title>
     <!--
       The description is here for link previews and for anyone reading source,
       not for ranking: this page is noindex (see below), so search engines will
@@ -12,7 +12,7 @@
     -->
     <meta
       name="description"
-      content="Coder Eval is an open-source framework for evaluating and benchmarking AI coding agents and Claude Code skills — sandboxed runs of Claude Code, Codex, and Gemini against declarative YAML tasks, with weighted scoring and CI gates. Documentation: coder-eval.com/docs"
+      content="Coder Eval is Playwright for coding agents: an open-source, agent-agnostic framework for evaluating and benchmarking AI coding agents and their skills — sandboxed runs of Claude Code, Codex, Antigravity (Gemini), or OpenCode against declarative YAML tasks, with weighted scoring and CI gates. Documentation: coder-eval.com/docs"
     />
     <!--
@@ -225,9 +225,10 @@
       -->
       <h1 class="sr-only">Coder Eval</h1>
       <p class="lead">
-        An open-source framework for evaluating and benchmarking AI coding agents and their Claude
-        Code skills: it runs a real agent — Claude Code, Codex, or Gemini — in a sandbox against
-        declarative YAML tasks, then scores the files and commands the agent actually produced.
+        <strong>Playwright for coding agents.</strong> An open-source, agent-agnostic framework for
+        evaluating and benchmarking AI coding agents and their skills: it runs a real agent — Claude
+        Code, Codex, Antigravity (Gemini), or OpenCode — in a sandbox against declarative YAML
+        tasks, then scores the files and commands the agent actually produced.
       </p>
       <p class="notice">
         The documentation has moved to

{coder_eval-0.11.6 → coder_eval-0.11.7}/CHANGELOG.md RENAMED Viewed

@@ -2,6 +2,97 @@
 <!-- version list -->
+## v0.11.7 (2026-09-08)
+### Bug Fixes
+- **deps**: Bump google-antigravity 0.1.7 -> 0.1.8 (Defender FP on the harness)
+  ([#158](https://github.com/UiPath/coder_eval/pull/158),
+  [`48a4d53`](https://github.com/UiPath/coder_eval/commit/48a4d5375f5cf0ce12f956ba77cbe4d48e4be543))
+- **deps**: Bump google-antigravity to 0.1.8 to clear a Defender false positive on the bundled
+  harness [PILOT-7463] ([#158](https://github.com/UiPath/coder_eval/pull/158),
+  [`48a4d53`](https://github.com/UiPath/coder_eval/commit/48a4d5375f5cf0ce12f956ba77cbe4d48e4be543))
+- **docs**: Update Antigravity version in docs
+  ([#158](https://github.com/UiPath/coder_eval/pull/158),
+  [`48a4d53`](https://github.com/UiPath/coder_eval/commit/48a4d5375f5cf0ce12f956ba77cbe4d48e4be543))
+- **evalboard**: Always show every known harness in the filter
+  ([#152](https://github.com/UiPath/coder_eval/pull/152),
+  [`5e7d2b6`](https://github.com/UiPath/coder_eval/commit/5e7d2b6e69de43fbfc0c961e43c8b540da5b403f))
+- **evalboard**: Stop counting one model as two in the run header
+  ([#152](https://github.com/UiPath/coder_eval/pull/152),
+  [`5e7d2b6`](https://github.com/UiPath/coder_eval/commit/5e7d2b6e69de43fbfc0c961e43c8b540da5b403f))
+- **pricing**: Refresh the rate card and add gemini 3.7/3.8 Flash
+  ([#155](https://github.com/UiPath/coder_eval/pull/155),
+  [`be98f9d`](https://github.com/UiPath/coder_eval/commit/be98f9d4bd02e12482f16cec3a57ab5c54be8149))
+### Documentation
+- Add intro video and make the README agent-agnostic
+  ([#157](https://github.com/UiPath/coder_eval/pull/157),
+  [`d715f85`](https://github.com/UiPath/coder_eval/commit/d715f856e32289be068dda7049251486660496ec))
+- Widen the framing past skills-only and guard the agent roster with CE047
+  ([#157](https://github.com/UiPath/coder_eval/pull/157),
+  [`d715f85`](https://github.com/UiPath/coder_eval/commit/d715f856e32289be068dda7049251486660496ec))
+- **deps**: Drop the Defender rationale from the antigravity pin comment
+  ([#158](https://github.com/UiPath/coder_eval/pull/158),
+  [`48a4d53`](https://github.com/UiPath/coder_eval/commit/48a4d5375f5cf0ce12f956ba77cbe4d48e4be543))
+- **evalboard**: Trim the comments added by this branch
+  ([#152](https://github.com/UiPath/coder_eval/pull/152),
+  [`5e7d2b6`](https://github.com/UiPath/coder_eval/commit/5e7d2b6e69de43fbfc0c961e43c8b540da5b403f))
+- **pricing**: Trim the rate-card comments to what affects an edit
+  ([#155](https://github.com/UiPath/coder_eval/pull/155),
+  [`be98f9d`](https://github.com/UiPath/coder_eval/commit/be98f9d4bd02e12482f16cec3a57ab5c54be8149))
+- **readme**: Add intro video and make the README agent-agnostic
+  ([#157](https://github.com/UiPath/coder_eval/pull/157),
+  [`d715f85`](https://github.com/UiPath/coder_eval/commit/d715f856e32289be068dda7049251486660496ec))
+- **readme**: Play the intro video inline, with YouTube as the fallback
+  ([#157](https://github.com/UiPath/coder_eval/pull/157),
+  [`d715f85`](https://github.com/UiPath/coder_eval/commit/d715f856e32289be068dda7049251486660496ec))
+- **readme**: Tell viewers to unmute the inline video
+  ([#157](https://github.com/UiPath/coder_eval/pull/157),
+  [`d715f85`](https://github.com/UiPath/coder_eval/commit/d715f856e32289be068dda7049251486660496ec))
+- **stub**: Make the Pages stub and package metadata agent-agnostic
+  ([#157](https://github.com/UiPath/coder_eval/pull/157),
+  [`d715f85`](https://github.com/UiPath/coder_eval/commit/d715f856e32289be068dda7049251486660496ec))
+- **tutorial**: Stop sending first-timers through the contributor toolchain
+  ([#157](https://github.com/UiPath/coder_eval/pull/157),
+  [`d715f85`](https://github.com/UiPath/coder_eval/commit/d715f856e32289be068dda7049251486660496ec))
+### Features
+- **evalboard**: Add loading skeletons to the slow routes
+  ([#152](https://github.com/UiPath/coder_eval/pull/152),
+  [`5e7d2b6`](https://github.com/UiPath/coder_eval/commit/5e7d2b6e69de43fbfc0c961e43c8b540da5b403f))
+### Performance Improvements
+- **evalboard**: Cut page load time and show loading state while pages load
+  ([#152](https://github.com/UiPath/coder_eval/pull/152),
+  [`5e7d2b6`](https://github.com/UiPath/coder_eval/commit/5e7d2b6e69de43fbfc0c961e43c8b540da5b403f))
+- **evalboard**: Move repeated per-row utility classes into the stylesheet
+  ([#152](https://github.com/UiPath/coder_eval/pull/152),
+  [`5e7d2b6`](https://github.com/UiPath/coder_eval/commit/5e7d2b6e69de43fbfc0c961e43c8b540da5b403f))
+- **evalboard**: Stop re-reading the whole run store on every render
+  ([#152](https://github.com/UiPath/coder_eval/pull/152),
+  [`5e7d2b6`](https://github.com/UiPath/coder_eval/commit/5e7d2b6e69de43fbfc0c961e43c8b540da5b403f))
 ## v0.11.6 (2026-09-01)
 ### Bug Fixes

{coder_eval-0.11.6 → coder_eval-0.11.7}/CLAUDE.md RENAMED Viewed

@@ -216,7 +216,7 @@ make plugin-reference  # the plugin's bundled criteria reference from the models
 Editing `src/coder_eval/pricing.py` means editing `evalboard/lib/pricing.ts` too — it is a hand-copied mirror, and `evalboard/lib/__tests__/pricing-parity.test.ts` fails the build on drift in either direction.
-Recent additions, each traceable to a shipped defect: **CE037** (no unreferenced module-level private helper in `src/` — a helper whose docstring documents a bug the live code still has is worse than none), **CE038** (in an `@asynccontextmanager`, the acquire must sit INSIDE the `try` whose `finally` releases it — `asyncio.shield` protects the inner task, NOT the await, so a cancel on `__aenter__` skips the unwind while the work completes), **CE039** (a criterion checker must not return a gating `score=0.0` from an `except OSError` over a path the *task author* named — that books an eval-config error as an agent failure; raise `CheckerMisuseError` instead, and `# noqa: CE039` the cases that really are the agent's).
+Recent additions, each traceable to a shipped defect: **CE037** (no unreferenced module-level private helper in `src/` — a helper whose docstring documents a bug the live code still has is worse than none), **CE038** (in an `@asynccontextmanager`, the acquire must sit INSIDE the `try` whose `finally` releases it — `asyncio.shield` protects the inner task, NOT the await, so a cancel on `__aenter__` skips the unwind while the work completes), **CE039** (a criterion checker must not return a gating `score=0.0` from an `except OSError` over a path the *task author* named — that books an eval-config error as an agent failure; raise `CheckerMisuseError` instead, and `# noqa: CE039` the cases that really are the agent's), **CE047** (every onboarding/marketing surface — README, `docs/index.md`, `docs/comparison.md`, `docs/llms.txt`, `mkdocs.yml`'s `site_description`, the Pages stub, and pyproject's `description`/`keywords` — must name every built-in `AgentKind`; OpenCode shipped while four of those seven still listed three harnesses, and nothing failed).
 When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time — CE026–CE031, CE033–CE036 — are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE036 enforces the `live_verdict` determinism + monotonicity contract (`criteria/base.py`) that `EarlyStopWatcher`'s latching, deferred fail-stop, and flip-attribution silently depend on: monotonicity over arbitrary Python is undecidable, so instead of a static check it REPLAYS each live criterion against every prefix of recorded trajectories (`tests/lint/live_verdict_contract.py::CASES`) — on the authored ordering AND under seeded shuffles (`permuted_violations`, which catch order-sensitive bugs the authored walk misses) — and asserts the property directly, plus registry-derived coverage — every `LiveSuccessCriterion` in the union must have cases, and every polarity its instances claim via `live_decidable_polarities()` must actually be reached by one (otherwise a single always-`undecided` fixture would "cover" a type while proving nothing). Adding a live criterion therefore means adding `ContractCase`s in the same change. CE035 resolves every `steps.<id>.outputs.<key>` / `needs.<job>.outputs.<key>` reference in `.github/workflows/**` to a writer that actually produces that key — GitHub expands an unwritten output to the empty string, so a typo degrades a gate silently and actionlint models `steps.*.outputs` as an open string map. CE034 scans `tasks/` and forces an armed, live-*passable* `command_executed` to set `require_success` — a crashed invocation would otherwise latch a live PASS, fire `on_pass: stop`, and let FIRED-ONLY armed gating report SUCCESS without ever consulting the unarmed criteria (negative assertions are fail-only and are exempt). CE033 keeps the plugin's bundled `reference/criteria.md` in parity with the `SuccessCriterion` union that generates it (`make plugin-reference` writes it; the rule re-renders and diffs — never hand-edit the file). CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name. CE026 keeps the GitHub Action's onboarding surfaces honest — `README.md`, `docs/CI_GATE.md`, `docs/tutorials/02-ci-pipeline.md`, and the plugin's `ci` skill, whose emitted workflow users copy into their own repos: a page's *first* Action snippet must show the agent-runtime prerequisite steps (pinned to the `action-dogfood` job that proves them in CI), a zero-install absolute next to such a snippet must name the channel it means, every `github.com/marketplace/actions/<slug>` link plus the shields badge label must match `action.yml`'s `name:`, and every `with:` key on a snippet's action step must be a real `action.yml` input (GitHub ignores unknown inputs, so a rename would silently degrade every copied workflow). Renaming an action input or changing its runtime prerequisites therefore means updating the skill too.)

{coder_eval-0.11.6 → coder_eval-0.11.7}/PKG-INFO RENAMED Viewed

@@ -1,7 +1,7 @@
 Metadata-Version: 2.5
 Name: coder-eval
-Version: 0.11.6
-Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity) with sandboxed, reproducible YAML task suites.
+Version: 0.11.7
+Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity, OpenCode) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval
 Project-URL: Documentation, https://coder-eval.com/docs
@@ -11,7 +11,7 @@ Author-email: UiPath <coder-eval@uipath.com>
 License-Expression: Apache-2.0
 License-File: LICENSE
 License-File: NOTICE
-Keywords: agent,agent-evaluation,agent-skills,agent-testing,ai,ai-evaluation,anthropic,antigravity,benchmark,claude,claude-code,claude-code-skills,claude-skills,code-generation,codex,coding-agent,eval,evals,evaluation,gemini,llm,llm-eval,llm-evaluation,llmops,sandbox,skills-evaluation,skillsbench,swe-bench
+Keywords: agent,agent-evaluation,agent-skills,agent-testing,ai,ai-evaluation,anthropic,antigravity,benchmark,claude,claude-code,claude-code-skills,claude-skills,code-generation,codex,coding-agent,eval,evals,evaluation,gemini,llm,llm-eval,llm-evaluation,llmops,opencode,sandbox,skills-evaluation,skillsbench,swe-bench
 Classifier: Development Status :: 4 - Beta
 Classifier: Environment :: Console
 Classifier: Intended Audience :: Developers
@@ -42,7 +42,7 @@ Requires-Dist: starlette>=1.3.1
 Requires-Dist: tqdm>=4.67.3
 Requires-Dist: typer>=0.24.1
 Provides-Extra: antigravity
-Requires-Dist: google-antigravity==0.1.7; extra == 'antigravity'
+Requires-Dist: google-antigravity==0.1.8; extra == 'antigravity'
 Provides-Extra: codex
 Requires-Dist: openai-codex>=0.144.4; extra == 'codex'
 Provides-Extra: dev
@@ -65,7 +65,7 @@ Provides-Extra: uipath
 Requires-Dist: uipath>=2.10.31; extra == 'uipath'
 Description-Content-Type: text/markdown
-# Coder Eval — evaluate & benchmark AI coding agents and Claude Code skills
+# Coder Eval — evaluate and benchmark AI coding agents and their skills
 [![PyPI](https://img.shields.io/pypi/v/coder-eval.svg)](https://pypi.org/project/coder-eval/)
 [![GitHub Marketplace](https://img.shields.io/badge/marketplace-coder__eval-2ea44f.svg)](https://github.com/marketplace/actions/coder_eval)
@@ -75,21 +75,28 @@ Description-Content-Type: text/markdown
 [![Python 3.13+](https://img.shields.io/badge/python-3.13%2B-blue.svg)](https://www.python.org/downloads/)
 [![CI](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml/badge.svg)](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml)
-**Coder Eval** (`pip install coder-eval` / `uv tool install coder-eval`) is an open-source framework for
-**evaluating and benchmarking AI coding agents and their skills** — built for CLI
-and skill builders — with sandboxing, reproducibility, and data-driven analysis.
-It runs a real agent (**Claude Code**, **Codex**, **Google Antigravity /
-Gemini**, or **OpenCode**) in a sandbox against declarative YAML tasks, then scores the files and
-commands it actually produced. Not an "agentic coding" benchmark: it measures how
-effective your CLI and skills are when used by coding agents.
-Reach for it when you want to **test whether a Claude Code skill triggers**,
-**A/B-test Claude Code vs. Codex vs. Gemini vs. OpenCode** (or model vs. model,
-prompt vs. prompt), or **gate CI on coding-agent quality**. Unlike fixed datasets (SWE-bench,
-SkillsBench) that rank models on a shared leaderboard, Coder Eval evaluates the
-tasks, skills, and workflows *you* ship — with weighted 0.0–1.0 criteria, a
-`skill_triggered` activation check, an A/B experiment layer, and per-tool cost
-telemetry. See [How it compares](https://coder-eval.com/docs/comparison).
+<p align="center">
+  <strong>Playwright for coding agents</strong> — one declarative test file, any agent
+  runtime, a real sandbox, and a pass/fail gate in CI.
+</p>
+**Coder Eval** (`pip install coder-eval` / `uv tool install coder-eval`) is an
+open-source, **agent-agnostic** framework for **evaluating and benchmarking AI coding
+agents and their skills** — built for benchmark authors, CLI builders, and skill
+builders — with sandboxing, reproducibility, and data-driven analysis. It runs a real
+agent — **Claude Code**, **OpenAI Codex**, **Google Antigravity (Gemini)**, or
+**OpenCode** — in a sandbox against declarative YAML tasks, then scores the files and
+commands it actually produced. Changing harness is one field (`agent.type`); the
+tasks, criteria, scoring, telemetry, and reports stay the same.
+Reach for it when you want to **benchmark agents on your own domain tasks**,
+**test whether a skill triggers** in the agent you ship for, **A/B-test Claude Code
+vs. Codex vs. Gemini vs. OpenCode** (or model vs. model, prompt vs. prompt), or
+**gate CI on coding-agent quality**. It is **not a fixed leaderboard**: unlike
+SWE-bench or SkillsBench, which rank models on a shared task set, you bring the tasks
+and you bring the scoring — weighted 0.0–1.0 criteria, a `skill_triggered` activation
+check, an A/B experiment layer, and per-tool cost telemetry, over whatever work *you*
+care about. See [How it compares](https://coder-eval.com/docs/comparison).
 📚 **Full docs:** **[coder-eval.com/docs](https://coder-eval.com/docs)**.
 <p align="center">
@@ -100,15 +107,30 @@ telemetry. See [How it compares](https://coder-eval.com/docs/comparison).
 - **Sandboxed execution** in isolated environments with resource limits
 - **Weighted, continuous scoring** (0.0–1.0) with fractional credit and thresholds
 - **Many criterion types** — from file checks to code similarity and LLM-graded rubrics
-- **Agent abstraction** — Claude Code, Codex, Antigravity (Gemini), and OpenCode today, extensible via a plugin SPI
+- **Agent-agnostic by design** — Claude Code, OpenAI Codex, Antigravity (Gemini), and OpenCode today; add your own harness through the plugin SPI
 - **Experiment layer** — A/B agent configs (models, tools, prompts) side-by-side
 - **Full telemetry** — every tool call, token counts, and cost, with real-time streaming
+## Watch the intro
+<div align="center">
+  <video src="https://github.com/user-attachments/assets/6dc1c88e-e5b7-4fc5-ba9c-4b117efc8b54" controls>
+    <a href="https://www.youtube.com/watch?v=Iyq-5m1CnuI">
+      <img src="https://img.youtube.com/vi/Iyq-5m1CnuI/maxresdefault.jpg" alt="Video: Coder Eval — UiPath open-source framework to test AI coding agents" width="70%">
+    </a>
+  </video>
+</div>
+🔊 **Turn the sound on** — GitHub's inline player always starts muted.
+▶ Also on YouTube: **[Coder Eval: UiPath open-source framework to test AI Coding Agents](https://www.youtube.com/watch?v=Iyq-5m1CnuI)**
+— what the framework does, and how a run works end to end.
 ## What you can do with it
 - **Benchmark coding agents** — score an agent across a suite of tasks with weighted scoring and pass/fail thresholds
 - **Compare models & configs** — A/B-test Claude vs. Codex vs. Gemini vs. OpenCode, model vs. model, tool-on vs. tool-off, prompt vs. prompt
-- **Evaluate skills** — verify an agent actually engages a target skill (`skill_triggered`) and score skill-driven suites (SkillsBench-style)
+- **Evaluate skills** — verify an agent actually engages a target skill (`skill_triggered`) and score skill-driven suites (SkillsBench-style), on whichever harness your users run
 - **Keep skills up to date in CI** — re-validate your skills on every change or on a schedule; catch silent regressions when models, prompts, or the skills themselves drift
 - **Gate CI on agent quality** — run the suite in GitHub Actions and fail the build on regressions
 - **Bring your own dataset** — fan one task out over many rows for larger benchmark suites
@@ -120,15 +142,27 @@ telemetry. See [How it compares](https://coder-eval.com/docs/comparison).
 ## Quick Start
-**Prerequisites:** Python 3.13+, [uv](https://docs.astral.sh/uv/) 0.8+, and the
-[Claude CLI](https://docs.anthropic.com/claude/docs/claude-code) (`brew install claude`).
-Developed on macOS; CI runs on Linux.
+**Prerequisites:** Python 3.13+, [uv](https://docs.astral.sh/uv/) 0.8+, and **the
+runtime of at least one coding agent** — plus that agent's own model credentials.
+Pick the agent you want to evaluate; two of the four ship with a Coder Eval extra,
+the other two are separate CLIs you install yourself:
+| Agent | `agent.type` | Runtime | Guide |
+| --- | --- | --- | --- |
+| Claude Code (default) | `claude-code` | `brew install claude` — separate CLI | [Claude Code](docs/agents/CLAUDE_CODE.md) |
+| OpenAI Codex | `codex` | `uv sync --extra codex` — the extra ships the Codex SDK + CLI | [Codex](docs/agents/CODEX.md) |
+| Google Antigravity (Gemini) | `antigravity` | `uv sync --extra antigravity` — the extra ships the harness binary | [Antigravity](docs/agents/ANTIGRAVITY.md) |
+| OpenCode (open-weight models) | `opencode` | `npm install -g opencode-ai` — separate CLI | [OpenCode](docs/agents/OPENCODE.md) |
+The examples below use the default `claude-code` agent. Developed on macOS; CI runs on
+Linux.
 ```bash
 git clone https://github.com/UiPath/coder_eval.git
 cd coder_eval
-uv sync --extra dev          # install core + dev tools
+uv sync                      # install the framework (add --extra codex /
+                             # --extra antigravity for those runtimes)
 cp .env.example .env         # then set ANTHROPIC_API_KEY — or skip that: an
                              # existing Claude Code login (`claude login`) is
                              # picked up automatically
@@ -140,13 +174,14 @@ uv run coder-eval report runs/latest           # view the result
 New here? Follow **[Tutorial 01 — Your First Evaluation](docs/tutorials/01-first-evaluation.md)**.
-The optional `[uipath]` extra (`uv sync --extra dev --extra uipath`) adds the in-host
-`uipath` SDK for local sandbox parity; it installs from public PyPI (no credentials
-required). Without it the framework runs end-to-end; uipath-dependent features fail
-at dispatch with a clear hint.
+Two more extras you only need on purpose: `--extra dev` adds the contributor
+toolchain (pytest, ruff, pyright, pre-commit — see
+[CONTRIBUTING.md](CONTRIBUTING.md)), and `--extra uipath` adds the in-host `uipath`
+SDK for local sandbox parity (public PyPI, no credentials). Without either, the
+framework still runs end-to-end.
-**Using Coder Eval in CI or another project?** Install the published package
-instead of cloning:
+**Just want the CLI, without cloning?** Install the published package — this is also
+what a CI job or another repo does:
 ```bash
 uv tool install coder-eval    # puts the `coder-eval` CLI on your PATH,
@@ -165,9 +200,10 @@ the full setup.
 ## Use inside Claude Code
-This repo is also a **Claude Code plugin marketplace**, so the whole loop —
-scaffold a suite, author a task, check whether a skill triggers, read the
-results — runs inside the agent:
+Coder Eval evaluates any of the supported agents, and it also ships an authoring
+front-end for one of them: this repo is a **Claude Code plugin marketplace**, so the
+whole loop — scaffold a suite, author a task, check whether a skill triggers, read the
+results — runs inside Claude Code. The suites you author this way run on every harness:
 ```
 /plugin marketplace add UiPath/coder_eval
@@ -342,9 +378,13 @@ success_criteria:
     description: "Script must execute successfully"
 ```
-Tasks can omit the `agent` section entirely — defaults resolve from the experiment
-layer (`experiments/default.yaml`). For the full schema and every criterion type,
-see the [Task Definition Guide](docs/TASK_DEFINITION_GUIDE.md).
+`agent.type` is the only harness-specific line: swap it for `codex`, `antigravity`, or
+`opencode` — or override it per run with `coder-eval run … -D agent.type=opencode` — and
+the same criteria score the same way. Tasks can omit the `agent` section entirely —
+defaults resolve from the experiment layer (`experiments/default.yaml`). For the full
+schema and every criterion type, see the
+[Task Definition Guide](docs/TASK_DEFINITION_GUIDE.md); for what each `run_limits` field
+means on each harness, see [Run-Limit Parity](docs/agents/HARNESS_PARITY.md).
 > **Tip:** With the [Claude Code plugin](docs/PLUGIN.md) installed, use
 > `/coder-eval:task` to scaffold a task from a natural-language description, and
@@ -368,8 +408,11 @@ extension points (new criteria, new agents).
 - **Tasks execute real code** — run untrusted tasks only under the container driver
   (see [Docker Isolation](docs/DOCKER_ISOLATION.md)); the `tempdir` driver is not a
   security boundary.
-- **Bring your own model credentials** — Anthropic, Bedrock, or Gemini keys; Coder Eval
-  does not proxy or supply model access.
+- **Bring your own agent runtime and model credentials** — Coder Eval never supplies
+  model access, and it only ships an agent runtime where an extra says so (`codex`,
+  `antigravity`); Claude Code and OpenCode are separate CLIs. Supply the runtime (see
+  [Quick Start](#quick-start)) and the keys it needs — Anthropic, Bedrock, OpenAI,
+  Gemini, or an OpenRouter key for open-weight models via OpenCode.
 - **Python 3.13+ only.**
 ## Support & security
@@ -385,6 +428,8 @@ extension points (new criteria, new agents).
 ## Acknowledgments
-Built with the [Claude Agent SDK](https://github.com/anthropics/claude-agent-sdk),
-[Pydantic](https://pydantic.dev/), [Typer](https://typer.tiangolo.com/), and
-[Rich](https://rich.readthedocs.io/).
+Built with [Pydantic](https://pydantic.dev/), [Typer](https://typer.tiangolo.com/),
+and [Rich](https://rich.readthedocs.io/), on top of the harnesses it drives — the
+[Claude Agent SDK](https://github.com/anthropics/claude-agent-sdk), the
+[Codex SDK](https://github.com/openai/codex), [Google Antigravity](https://antigravity.google/),
+and [OpenCode](https://opencode.ai).

{coder_eval-0.11.6 → coder_eval-0.11.7}/README.md RENAMED Viewed

@@ -1,4 +1,4 @@
-# Coder Eval — evaluate & benchmark AI coding agents and Claude Code skills
+# Coder Eval — evaluate and benchmark AI coding agents and their skills
 [![PyPI](https://img.shields.io/pypi/v/coder-eval.svg)](https://pypi.org/project/coder-eval/)
 [![GitHub Marketplace](https://img.shields.io/badge/marketplace-coder__eval-2ea44f.svg)](https://github.com/marketplace/actions/coder_eval)
@@ -8,21 +8,28 @@
 [![Python 3.13+](https://img.shields.io/badge/python-3.13%2B-blue.svg)](https://www.python.org/downloads/)
 [![CI](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml/badge.svg)](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml)
-**Coder Eval** (`pip install coder-eval` / `uv tool install coder-eval`) is an open-source framework for
-**evaluating and benchmarking AI coding agents and their skills** — built for CLI
-and skill builders — with sandboxing, reproducibility, and data-driven analysis.
-It runs a real agent (**Claude Code**, **Codex**, **Google Antigravity /
-Gemini**, or **OpenCode**) in a sandbox against declarative YAML tasks, then scores the files and
-commands it actually produced. Not an "agentic coding" benchmark: it measures how
-effective your CLI and skills are when used by coding agents.
-Reach for it when you want to **test whether a Claude Code skill triggers**,
-**A/B-test Claude Code vs. Codex vs. Gemini vs. OpenCode** (or model vs. model,
-prompt vs. prompt), or **gate CI on coding-agent quality**. Unlike fixed datasets (SWE-bench,
-SkillsBench) that rank models on a shared leaderboard, Coder Eval evaluates the
-tasks, skills, and workflows *you* ship — with weighted 0.0–1.0 criteria, a
-`skill_triggered` activation check, an A/B experiment layer, and per-tool cost
-telemetry. See [How it compares](https://coder-eval.com/docs/comparison).
+<p align="center">
+  <strong>Playwright for coding agents</strong> — one declarative test file, any agent
+  runtime, a real sandbox, and a pass/fail gate in CI.
+</p>
+**Coder Eval** (`pip install coder-eval` / `uv tool install coder-eval`) is an
+open-source, **agent-agnostic** framework for **evaluating and benchmarking AI coding
+agents and their skills** — built for benchmark authors, CLI builders, and skill
+builders — with sandboxing, reproducibility, and data-driven analysis. It runs a real
+agent — **Claude Code**, **OpenAI Codex**, **Google Antigravity (Gemini)**, or
+**OpenCode** — in a sandbox against declarative YAML tasks, then scores the files and
+commands it actually produced. Changing harness is one field (`agent.type`); the
+tasks, criteria, scoring, telemetry, and reports stay the same.
+Reach for it when you want to **benchmark agents on your own domain tasks**,
+**test whether a skill triggers** in the agent you ship for, **A/B-test Claude Code
+vs. Codex vs. Gemini vs. OpenCode** (or model vs. model, prompt vs. prompt), or
+**gate CI on coding-agent quality**. It is **not a fixed leaderboard**: unlike
+SWE-bench or SkillsBench, which rank models on a shared task set, you bring the tasks
+and you bring the scoring — weighted 0.0–1.0 criteria, a `skill_triggered` activation
+check, an A/B experiment layer, and per-tool cost telemetry, over whatever work *you*
+care about. See [How it compares](https://coder-eval.com/docs/comparison).
 📚 **Full docs:** **[coder-eval.com/docs](https://coder-eval.com/docs)**.
 <p align="center">
@@ -33,15 +40,30 @@ telemetry. See [How it compares](https://coder-eval.com/docs/comparison).
 - **Sandboxed execution** in isolated environments with resource limits
 - **Weighted, continuous scoring** (0.0–1.0) with fractional credit and thresholds
 - **Many criterion types** — from file checks to code similarity and LLM-graded rubrics
-- **Agent abstraction** — Claude Code, Codex, Antigravity (Gemini), and OpenCode today, extensible via a plugin SPI
+- **Agent-agnostic by design** — Claude Code, OpenAI Codex, Antigravity (Gemini), and OpenCode today; add your own harness through the plugin SPI
 - **Experiment layer** — A/B agent configs (models, tools, prompts) side-by-side
 - **Full telemetry** — every tool call, token counts, and cost, with real-time streaming
+## Watch the intro
+<div align="center">
+  <video src="https://github.com/user-attachments/assets/6dc1c88e-e5b7-4fc5-ba9c-4b117efc8b54" controls>
+    <a href="https://www.youtube.com/watch?v=Iyq-5m1CnuI">
+      <img src="https://img.youtube.com/vi/Iyq-5m1CnuI/maxresdefault.jpg" alt="Video: Coder Eval — UiPath open-source framework to test AI coding agents" width="70%">
+    </a>
+  </video>
+</div>
+🔊 **Turn the sound on** — GitHub's inline player always starts muted.
+▶ Also on YouTube: **[Coder Eval: UiPath open-source framework to test AI Coding Agents](https://www.youtube.com/watch?v=Iyq-5m1CnuI)**
+— what the framework does, and how a run works end to end.
 ## What you can do with it
 - **Benchmark coding agents** — score an agent across a suite of tasks with weighted scoring and pass/fail thresholds
 - **Compare models & configs** — A/B-test Claude vs. Codex vs. Gemini vs. OpenCode, model vs. model, tool-on vs. tool-off, prompt vs. prompt
-- **Evaluate skills** — verify an agent actually engages a target skill (`skill_triggered`) and score skill-driven suites (SkillsBench-style)
+- **Evaluate skills** — verify an agent actually engages a target skill (`skill_triggered`) and score skill-driven suites (SkillsBench-style), on whichever harness your users run
 - **Keep skills up to date in CI** — re-validate your skills on every change or on a schedule; catch silent regressions when models, prompts, or the skills themselves drift
 - **Gate CI on agent quality** — run the suite in GitHub Actions and fail the build on regressions
 - **Bring your own dataset** — fan one task out over many rows for larger benchmark suites
@@ -53,15 +75,27 @@ telemetry. See [How it compares](https://coder-eval.com/docs/comparison).
 ## Quick Start
-**Prerequisites:** Python 3.13+, [uv](https://docs.astral.sh/uv/) 0.8+, and the
-[Claude CLI](https://docs.anthropic.com/claude/docs/claude-code) (`brew install claude`).
-Developed on macOS; CI runs on Linux.
+**Prerequisites:** Python 3.13+, [uv](https://docs.astral.sh/uv/) 0.8+, and **the
+runtime of at least one coding agent** — plus that agent's own model credentials.
+Pick the agent you want to evaluate; two of the four ship with a Coder Eval extra,
+the other two are separate CLIs you install yourself:
+| Agent | `agent.type` | Runtime | Guide |
+| --- | --- | --- | --- |
+| Claude Code (default) | `claude-code` | `brew install claude` — separate CLI | [Claude Code](docs/agents/CLAUDE_CODE.md) |
+| OpenAI Codex | `codex` | `uv sync --extra codex` — the extra ships the Codex SDK + CLI | [Codex](docs/agents/CODEX.md) |
+| Google Antigravity (Gemini) | `antigravity` | `uv sync --extra antigravity` — the extra ships the harness binary | [Antigravity](docs/agents/ANTIGRAVITY.md) |
+| OpenCode (open-weight models) | `opencode` | `npm install -g opencode-ai` — separate CLI | [OpenCode](docs/agents/OPENCODE.md) |
+The examples below use the default `claude-code` agent. Developed on macOS; CI runs on
+Linux.
 ```bash
 git clone https://github.com/UiPath/coder_eval.git
 cd coder_eval
-uv sync --extra dev          # install core + dev tools
+uv sync                      # install the framework (add --extra codex /
+                             # --extra antigravity for those runtimes)
 cp .env.example .env         # then set ANTHROPIC_API_KEY — or skip that: an
                              # existing Claude Code login (`claude login`) is
                              # picked up automatically
@@ -73,13 +107,14 @@ uv run coder-eval report runs/latest           # view the result
 New here? Follow **[Tutorial 01 — Your First Evaluation](docs/tutorials/01-first-evaluation.md)**.
-The optional `[uipath]` extra (`uv sync --extra dev --extra uipath`) adds the in-host
-`uipath` SDK for local sandbox parity; it installs from public PyPI (no credentials
-required). Without it the framework runs end-to-end; uipath-dependent features fail
-at dispatch with a clear hint.
+Two more extras you only need on purpose: `--extra dev` adds the contributor
+toolchain (pytest, ruff, pyright, pre-commit — see
+[CONTRIBUTING.md](CONTRIBUTING.md)), and `--extra uipath` adds the in-host `uipath`
+SDK for local sandbox parity (public PyPI, no credentials). Without either, the
+framework still runs end-to-end.
-**Using Coder Eval in CI or another project?** Install the published package
-instead of cloning:
+**Just want the CLI, without cloning?** Install the published package — this is also
+what a CI job or another repo does:
 ```bash
 uv tool install coder-eval    # puts the `coder-eval` CLI on your PATH,
@@ -98,9 +133,10 @@ the full setup.
 ## Use inside Claude Code
-This repo is also a **Claude Code plugin marketplace**, so the whole loop —
-scaffold a suite, author a task, check whether a skill triggers, read the
-results — runs inside the agent:
+Coder Eval evaluates any of the supported agents, and it also ships an authoring
+front-end for one of them: this repo is a **Claude Code plugin marketplace**, so the
+whole loop — scaffold a suite, author a task, check whether a skill triggers, read the
+results — runs inside Claude Code. The suites you author this way run on every harness:
 ```
 /plugin marketplace add UiPath/coder_eval
@@ -275,9 +311,13 @@ success_criteria:
     description: "Script must execute successfully"
 ```
-Tasks can omit the `agent` section entirely — defaults resolve from the experiment
-layer (`experiments/default.yaml`). For the full schema and every criterion type,
-see the [Task Definition Guide](docs/TASK_DEFINITION_GUIDE.md).
+`agent.type` is the only harness-specific line: swap it for `codex`, `antigravity`, or
+`opencode` — or override it per run with `coder-eval run … -D agent.type=opencode` — and
+the same criteria score the same way. Tasks can omit the `agent` section entirely —
+defaults resolve from the experiment layer (`experiments/default.yaml`). For the full
+schema and every criterion type, see the
+[Task Definition Guide](docs/TASK_DEFINITION_GUIDE.md); for what each `run_limits` field
+means on each harness, see [Run-Limit Parity](docs/agents/HARNESS_PARITY.md).
 > **Tip:** With the [Claude Code plugin](docs/PLUGIN.md) installed, use
 > `/coder-eval:task` to scaffold a task from a natural-language description, and
@@ -301,8 +341,11 @@ extension points (new criteria, new agents).
 - **Tasks execute real code** — run untrusted tasks only under the container driver
   (see [Docker Isolation](docs/DOCKER_ISOLATION.md)); the `tempdir` driver is not a
   security boundary.
-- **Bring your own model credentials** — Anthropic, Bedrock, or Gemini keys; Coder Eval
-  does not proxy or supply model access.
+- **Bring your own agent runtime and model credentials** — Coder Eval never supplies
+  model access, and it only ships an agent runtime where an extra says so (`codex`,
+  `antigravity`); Claude Code and OpenCode are separate CLIs. Supply the runtime (see
+  [Quick Start](#quick-start)) and the keys it needs — Anthropic, Bedrock, OpenAI,
+  Gemini, or an OpenRouter key for open-weight models via OpenCode.
 - **Python 3.13+ only.**
 ## Support & security
@@ -318,6 +361,8 @@ extension points (new criteria, new agents).
 ## Acknowledgments
-Built with the [Claude Agent SDK](https://github.com/anthropics/claude-agent-sdk),
-[Pydantic](https://pydantic.dev/), [Typer](https://typer.tiangolo.com/), and
-[Rich](https://rich.readthedocs.io/).
+Built with [Pydantic](https://pydantic.dev/), [Typer](https://typer.tiangolo.com/),
+and [Rich](https://rich.readthedocs.io/), on top of the harnesses it drives — the
+[Claude Agent SDK](https://github.com/anthropics/claude-agent-sdk), the
+[Codex SDK](https://github.com/openai/codex), [Google Antigravity](https://antigravity.google/),
+and [OpenCode](https://opencode.ai).

{coder_eval-0.11.6 → coder_eval-0.11.7}/action.yml RENAMED Viewed

@@ -25,7 +25,7 @@ inputs:
   version:
     description: coder-eval version to install from PyPI, or "local" to install from the action checkout
     required: false
-    default: "0.11.6"   # <-- kept in sync with releases by release.yml
+    default: "0.11.7"   # <-- kept in sync with releases by release.yml
   extras:
     description: >-
       Comma-separated coder-eval extras (`codex`, `antigravity,litellm`), composed

{coder_eval-0.11.6 → coder_eval-0.11.7}/docs/agents/ANTIGRAVITY.md RENAMED Viewed

@@ -27,7 +27,7 @@ working directory — both required for an unattended eval.
 pip install 'coder-eval[antigravity]'
 ```
-This pulls in `google-antigravity` (pinned to `0.1.7`), whose wheel bundles the
+This pulls in `google-antigravity` (pinned to `0.1.8`), whose wheel bundles the
 platform `localharness` binary. As with the other agents the SDK is imported lazily
 — a base install without the extra still runs end-to-end; Antigravity tasks fail at
 dispatch with a clear hint to install the extra.

coder-eval 0.11.6__tar.gz → 0.11.7__tar.gz

coder-eval 0.11.6tar.gz → 0.11.7tar.gz