coderace 0.9.0__tar.gz → 1.3.0__tar.gz

{coderace-0.9.0 → coderace-1.3.0}/CHANGELOG.md

@@ -1,5 +1,51 @@
 # Changelog
 
+## [1.3.0] - 2026-03-05
+
+### Added
+- **Model selection**: Per-agent model override via `agent:model` syntax in `--agents` / `--agent` flags
+  - Example: `coderace run task.yaml --agent codex:gpt-5.4 --agent codex:gpt-5.3-codex`
+  - Example: `coderace benchmark --agents claude:opus-4-6,claude:sonnet-4-6`
+- `BaseAdapter.__init__(model=None)`: all adapters accept optional model at construction
+- `BaseAdapter.build_command(task, model=None)`: model parameter flows to CLI flag
+- `parse_agent_spec()`, `make_display_name()`, `instantiate_adapter()` in `coderace.adapters`
+- All adapters (codex, claude, aider, gemini, opencode) append `--model <name>` when specified
+- Benchmark and race commands handle model-specific agents; display names flow to results, store, ELO, dashboard
+- Task YAML: `agents` list accepts `agent:model` entries (e.g. `- codex:gpt-5.4`)
+
+### Changed
+- `AgentResult.agent` is now the display name (`codex (gpt-5.4)`) when a model is specified
+- ELO ratings, leaderboard, and dashboard automatically track model variants as separate entries
+- Branch names sanitized to be git-compatible (colons replaced with dashes)
+
+## [1.2.0] - 2026-03-03
+
+### Added
+
+- **`coderace race` command** - New first-to-pass race mode with early-stop semantics. Agents run in parallel worktrees and the race ends when the first winner is found.
+- **Live race UI** - Rich `Live` panel with per-agent status and timers:
+  - `🔨 coding...`
+  - `🧪 testing...`
+  - `✅ WINNER!`
+  - `❌ failed`
+  - `⏰ timed out`
+  - `🛑 stopped`
+- **Winner announcement and runner-up delta** - Prints race winner and optional runner-up timing delta after the live panel closes.
+- **Race result persistence (JSON fallback)** - Saves race summaries to `.coderace/race-results.json` including `race_id`, winner metadata, participant statuses, exit codes, and wall times. Supports `--no-save`.
+- **Race test suite** - Added 21 race-focused tests (`tests/test_race.py`) covering winner logic, cancellation, timeout/no-winner paths, verification modes, live updates, serialization, and Ctrl+C cleanup.
+
+## [1.0.0] - 2026-02-28
+
+### Added
+
+- **Benchmark trials mode** — `coderace benchmark --trials N` now runs each `(task, agent)` pair repeatedly and stores each trial with `trial_number` in SQLite.
+- **Statistical benchmarking module** — New `coderace/statistics.py` computes per-pair and per-agent aggregates: mean/stddev, 95% confidence intervals, pass rate, consistency, win rate, cost efficiency, and reliability.
+- **Persistent ELO ratings** — New `coderace/elo.py` plus `elo_ratings` store table. Ratings update automatically after each benchmark using pairwise task outcomes and persist across runs.
+- **`coderace ratings` command** — View persistent ELO rankings, output as JSON (`--json`), and reset all ratings (`--reset`).
+- **Standardized benchmark export** — `coderace benchmark --export <path>` writes shareable JSON with run metadata, system info, per-trial details, aggregate stats, and current ELO ratings.
+- **Enhanced benchmark report rendering** — Multi-trial reports now show statistical columns (`mean +/- stddev`, CI, consistency, reliability) and include ELO ratings in terminal/markdown/html output.
+- **Integration and edge-case coverage for v1.0 flow** — Added tests for full `--trials 3` benchmark + export + ELO pipeline and edge cases (single trial/agent/task and always-failing agent).
+
 ## [0.7.0] - 2026-02-26
 
 ### Added

coderace-1.3.0/DONE.txt

@@ -0,0 +1,70 @@
+Context Eval Build Contract: COMPLETE
+======================================
+
+Date: 2026-03-02
+All deliverables (D1-D4) implemented and validated.
+
+## What Was Built
+
+### D1: context-eval CLI Command (core + CLI)
+- `coderace/context_eval.py`: Core A/B evaluation engine
+  - ContextEvalResult and TrialResult data classes
+  - Context file backup/restore/placement/removal for baseline vs treatment
+  - KNOWN_CONTEXT_FILES list (CLAUDE.md, AGENTS.md, .cursorrules, etc.)
+  - run_context_eval() orchestrator running N trials per condition
+- `coderace/commands/context_eval.py`: CLI subcommand with:
+  --context-file PATH, --task PATH, --benchmark, --agents, --trials N,
+  --output PATH, --task-dir PATH
+- Full input validation (missing files, invalid agents, trials < 2, etc.)
+
+### D2: Statistical Comparison Report
+- `coderace/context_eval_report.py`: Statistical analysis and rendering
+  - Delta with 95% CI using Welch's t-test
+  - Cohen's d effect size
+  - Per-agent summary: baseline vs treatment pass rates and scores
+  - Per-task breakdown: which tasks improved, which degraded
+  - Summary verdict: "improved", "degraded", or "no significant improvement"
+  - Rich terminal table output
+  - JSON output format
+
+### D3: Dashboard Integration
+- Extended `coderace/dashboard.py` with context-eval A/B section:
+  - Bar chart: baseline vs treatment scores per agent
+  - Delta table with CI (95%) and effect size
+  - Verdict display
+  - CSS for A/B visualization (.ab-baseline, .ab-treatment, .positive, .negative)
+- Added --context-eval PATH flag to `coderace dashboard` command
+
+### D4: Documentation + Examples
+- README.md: Added "Context Evaluation" and "Measuring Context Engineering Impact" sections
+  with usage examples, output format, CLI flags table, and effect size interpretation guide
+- examples/context-eval-demo.sh: Executable demo script
+- Clear help text on `coderace context-eval --help` and `coderace --help`
+
+## Test Results
+
+- 58 new tests added (41 for D1+D2, 17 for D3)
+- All 505 tests pass (447 original + 58 new)
+- No regressions in existing test suite
+
+## Commits
+
+1. feat(context-eval): add context-eval command with A/B statistical comparison (D1+D2)
+2. feat(context-eval): add dashboard A/B comparison section (D3)
+3. docs(context-eval): add README section, examples, and interpretation guide (D4)
+
+## Files Created/Modified
+
+New files:
+- coderace/context_eval.py
+- coderace/commands/context_eval.py
+- coderace/context_eval_report.py
+- tests/test_context_eval.py
+- tests/test_context_eval_dashboard.py
+- examples/context-eval-demo.sh
+
+Modified files:
+- coderace/cli.py (registered context-eval subcommand + --context-eval dashboard flag)
+- coderace/dashboard.py (added A/B comparison section)
+- README.md (added context-eval documentation)
+- progress-log.md (added D1-D4 progress entries)

{coderace-0.9.0 → coderace-1.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: coderace
-Version: 0.9.0
+Version: 1.3.0
 Summary: Race coding agents against each other on real tasks
 Project-URL: Homepage, https://github.com/mikiships/coderace
 Project-URL: Repository, https://github.com/mikiships/coderace
@@ -30,6 +30,11 @@ Description-Content-Type: text/markdown
 
 # coderace
 
+[![PyPI](https://img.shields.io/pypi/v/coderace)](https://pypi.org/project/coderace/)
+[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](#install)
+[![Tests](https://img.shields.io/badge/tests-526%20passing-brightgreen)](#)
+[![License](https://img.shields.io/badge/license-MIT-lightgrey)](#license)
+
 Stop reading blog comparisons. Race coding agents against each other on real tasks in *your* repo with *your* code.
 
 Every week there's a new "Claude Code vs Codex vs Cursor" post. They test on toy problems with cherry-picked examples. coderace gives you automated, reproducible, scored comparisons on the tasks you actually care about.
@@ -340,6 +345,41 @@ Keys can be agent names (`claude`, `codex`, `aider`, `gemini`, `opencode`) or mo
 
 Pricing is easy to update: the table lives in `coderace/cost.py` as a plain dict.
 
+## Model Selection
+
+Compare different models of the same agent head-to-head using the `agent:model` syntax:
+
+```bash
+# Compare two Codex models on the same task
+coderace run task.yaml --agent codex:gpt-5.4 --agent codex:gpt-5.3-codex
+
+# Mix agents and models
+coderace run task.yaml --agent codex:gpt-5.4 --agent claude:opus-4-6 --agent claude:sonnet-4-6
+
+# Benchmark multiple model variants across built-in tasks
+coderace benchmark --agents codex:gpt-5.4,codex:gpt-5.3-codex,claude:opus-4-6
+
+# Race with model variants (parallel)
+coderace race task.yaml
+```
+
+In task YAML files:
+
+```yaml
+agents:
+  - codex:gpt-5.4
+  - codex:gpt-5.3-codex
+  - claude:opus-4-6
+  - claude:sonnet-4-6
+```
+
+**How it works:**
+- `agent:model` splits on the first colon: `codex:gpt-5.4` → agent `codex`, model `gpt-5.4`
+- The model is passed via `--model <name>` to the underlying CLI
+- Results display as `codex (gpt-5.4)` vs `codex (gpt-5.3-codex)` for easy comparison
+- ELO ratings, leaderboard, and dashboard track each model variant separately
+- The same agent can appear multiple times with different models in one run
+
 ## Leaderboard & History
 
 Every `coderace run` automatically saves results to a local SQLite database (`~/.coderace/results.db`). Two new commands aggregate this data.
@@ -471,6 +511,37 @@ coderace run task.yaml --parallel
 
 Sequential mode (default) runs agents one at a time on the same repo.
 
+## Race Mode
+
+Use `coderace race` for first-to-pass execution. Unlike `coderace run --parallel`, race mode stops as soon as one agent passes the win condition:
+
+- If verification is configured, winner = first agent that passes verification.
+- If verification is not configured, winner = first agent that exits cleanly.
+- Remaining agents are stopped after a short graceful shutdown window.
+
+```bash
+coderace race task.yaml --agent claude --agent codex
+```
+
+Example terminal output:
+
+```text
+🏁 coderace race - fix-auth-bug
+Running 3 agents in parallel...
+
+Agent   Status                 Time
+claude  🔨 coding...           0:00:23
+codex   🧪 testing...          0:00:31
+aider   🛑 stopped             0:00:18
+
+🏆 Winner: codex - completed in 1:23 (first to pass verification)
+Runner-up: claude - finished 0:12 later
+```
+
+When to use each mode:
+- Use `coderace race` when you want the fastest successful patch and can stop early.
+- Use `coderace run --parallel` when you want full scoring across all agents before deciding.
+
 ## Why coderace?
 
 **Blog posts compare models. coderace compares agents on your work.**
@@ -606,9 +677,15 @@ coderace benchmark --agents claude --difficulty easy,medium
 # Dry-run: see what would run without executing
 coderace benchmark --agents claude,codex --dry-run
 
+# Statistical mode: run repeated trials per pair
+coderace benchmark --agents claude,codex --tasks fibonacci,json-parser --trials 5
+
 # Save report to file
 coderace benchmark --agents claude,codex --output report.md
 coderace benchmark --agents claude,codex --output report.html
+
+# Export standardized JSON (shareable benchmark artifact)
+coderace benchmark --agents claude,codex --trials 5 --export benchmark.json
 ```
 
 ### Example Terminal Output
@@ -652,7 +729,171 @@ coderace benchmark show bench-20260227-143022
 | `--difficulty` | Filter by difficulty: `easy`, `medium`, `hard` | all |
 | `--timeout` | Per-task timeout in seconds | `300` |
 | `--parallel N` | Run N agents in parallel | `1` (sequential) |
+| `--trials N` | Repeat each `(task, agent)` pair N times | `1` |
 | `--dry-run` | List combinations without running | `false` |
 | `--format` | Output format: `terminal`, `markdown`, `html` | `terminal` |
 | `--output` | Save report to file | — |
+| `--export` | Write standardized benchmark JSON file | — |
 | `--no-save` | Skip saving results to the store | `false` |
+
+### Statistical Reports (`--trials > 1`)
+
+When `--trials` is greater than 1, benchmark reports switch to statistical mode:
+
+- Task cells show `mean score +/- stddev` (plus mean wall time)
+- Report includes `CI (95%)`, `Consistency`, and `Reliability` columns
+- Summary includes per-agent mean score, confidence interval, win rate, and reliability
+- ELO ratings are rendered at the bottom of terminal/markdown/html reports
+
+### ELO Ratings
+
+Every benchmark run updates persistent ELO ratings across all benchmark history.
+
+```bash
+# Show ratings
+coderace ratings
+
+# JSON output
+coderace ratings --json
+
+# Reset all ratings to 1500
+coderace ratings --reset
+```
+
+ELO rules:
+- Initial rating: `1500`
+- K-factor: `32`
+- Each task is treated as a round-robin set of pairwise matches
+- Winner per pair is based on higher mean trial score (draw when within 1 point)
+
+### Export Format (`--export`)
+
+`coderace benchmark --export benchmark.json` writes a standardized JSON artifact:
+
+```json
+{
+  "coderace_version": "1.0.0",
+  "benchmark_id": "bench-20260228-133000",
+  "timestamp": "2026-02-28T13:30:00Z",
+  "system": { "os": "...", "python": "...", "cpu": "..." },
+  "config": { "trials": 5, "timeout": 300, "tasks": ["..."], "agents": ["..."] },
+  "results": [
+    {
+      "task": "fibonacci",
+      "agent": "claude",
+      "trials": 5,
+      "mean_score": 87.5,
+      "stddev_score": 3.2,
+      "ci_95": [83.1, 91.9],
+      "mean_time": 45.2,
+      "mean_cost": 0.03,
+      "pass_rate": 1.0,
+      "consistency_score": 0.96,
+      "per_trial": []
+    }
+  ],
+  "elo_ratings": { "claude": 1523, "codex": 1488 },
+  "summary": {}
+}
+```
+
+## Context Evaluation
+
+The `coderace context-eval` command measures whether a context file (CLAUDE.md, AGENTS.md, .cursorrules, etc.) actually improves agent performance. It runs A/B trials — baseline (no context file) vs treatment (with context file) — and produces statistical comparisons.
+
+```bash
+# Evaluate whether CLAUDE.md improves claude's performance on a task
+coderace context-eval --context-file CLAUDE.md --task fix-auth-bug.yaml --agents claude --trials 5
+
+# Evaluate across all built-in benchmark tasks
+coderace context-eval --context-file CLAUDE.md --benchmark --agents claude,codex
+
+# Save results as JSON
+coderace context-eval --context-file CLAUDE.md --task task.yaml --agents claude --output results.json
+
+# Use a custom task directory
+coderace context-eval --context-file CLAUDE.md --benchmark --task-dir ./my-tasks --agents claude
+```
+
+### How It Works
+
+For each agent × task combination:
+1. Run N trials **without** the context file (baseline condition)
+2. Run N trials **with** the context file placed in the task directory (treatment condition)
+3. Compare pass rates, mean scores, and compute statistical significance
+
+### Output
+
+The terminal report shows:
+- **Per-agent summary**: baseline vs treatment pass rates and scores, delta with 95% CI, Cohen's d effect size
+- **Per-task breakdown**: which tasks improved, which degraded
+- **Verdict**: whether the context file significantly improved performance
+
+```
+┌────────┬───────────────────┬────────────────────┬────────────────┬─────────────────┬────────┬──────────────────┬─────────────┐
+│ Agent  │ Baseline Pass Rate│ Treatment Pass Rate │ Baseline Score │ Treatment Score │ Delta  │ CI (95%)         │ Effect Size │
+├────────┼───────────────────┼────────────────────┼────────────────┼─────────────────┼────────┼──────────────────┼─────────────┤
+│ claude │              67%  │              100%  │           55.0 │            81.0 │ +26.0  │ [10.5, 41.5]     │        2.10 │
+│ codex  │              33%  │               67%  │           45.0 │            70.0 │ +25.0  │ [8.0, 42.0]      │        1.80 │
+└────────┴───────────────────┴────────────────────┴────────────────┴─────────────────┴────────┴──────────────────┴─────────────┘
+
+Context file improved performance by +25.5 points (CI: [12.0, 39.0])
+```
+
+### Context-Eval CLI Flags
+
+| Flag | Description | Default |
+|------|-------------|---------|
+| `--context-file` | Path to the context file to evaluate (required) | — |
+| `--task` | Path to a single task YAML | — |
+| `--benchmark` | Run against built-in benchmark tasks | `false` |
+| `--agents` | Comma-separated agent names (required) | — |
+| `--trials` | Trials per condition (min: 2) | `3` |
+| `--output` | Save JSON results to file | — |
+| `--task-dir` | Custom task directory for benchmark mode | — |
+
+### Dashboard Integration
+
+Include context-eval results in the HTML dashboard:
+
+```bash
+# Run context-eval and save JSON
+coderace context-eval --context-file CLAUDE.md --task task.yaml --agents claude --output eval.json
+
+# Generate dashboard with A/B comparison section
+coderace dashboard --context-eval eval.json
+```
+
+## Measuring Context Engineering Impact
+
+Context engineering — crafting CLAUDE.md, AGENTS.md, .cursorrules, and similar files — is becoming a core developer skill. But until now, there was no way to empirically measure whether your context files actually help.
+
+**The problem:** You write a CLAUDE.md with coding conventions, architectural guidelines, and project-specific instructions. But does it actually make agents produce better code? Or is it cargo-cult configuration?
+
+**The solution:** `coderace context-eval` gives you data:
+
+1. **Write your context file** (e.g., CLAUDE.md with project conventions)
+2. **Run A/B evaluation** against real coding tasks
+3. **Get statistical evidence** of improvement (or lack thereof)
+
+```bash
+# Iterate on your context file with data
+coderace context-eval --context-file CLAUDE.md --benchmark --agents claude --trials 5
+
+# Compare different context files
+coderace context-eval --context-file v1-claude.md --task task.yaml --agents claude --output v1.json
+coderace context-eval --context-file v2-claude.md --task task.yaml --agents claude --output v2.json
+```
+
+**Interpreting results:**
+- **Effect size > 0.8**: Large improvement — your context file is helping significantly
+- **Effect size 0.2–0.8**: Moderate improvement — some benefit, room to iterate
+- **Effect size < 0.2**: Negligible — your context file isn't making a measurable difference
+- **CI crosses zero**: Not statistically significant — need more trials or a better context file
+
+## See Also
+
+- **[agentmd](https://github.com/mikiships/agentmd)** — Generate and score context files (CLAUDE.md, AGENTS.md, .cursorrules) for AI coding agents. Pair with coderace: generate context with agentmd, measure agent performance with coderace, iterate with data instead of vibes.
+- **[agentlint](https://github.com/mikiships/agentlint)** — Lint AI agent git diffs for risky patterns (scope drift, secret leaks, test regression). Static analysis, no LLM required.
+
+Measure (coderace) → Optimize (agentmd) → Guard (agentlint).

{coderace-0.9.0 → coderace-1.3.0}/README.md

@@ -1,5 +1,10 @@
 # coderace
 
+[![PyPI](https://img.shields.io/pypi/v/coderace)](https://pypi.org/project/coderace/)
+[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](#install)
+[![Tests](https://img.shields.io/badge/tests-526%20passing-brightgreen)](#)
+[![License](https://img.shields.io/badge/license-MIT-lightgrey)](#license)
+
 Stop reading blog comparisons. Race coding agents against each other on real tasks in *your* repo with *your* code.
 
 Every week there's a new "Claude Code vs Codex vs Cursor" post. They test on toy problems with cherry-picked examples. coderace gives you automated, reproducible, scored comparisons on the tasks you actually care about.
@@ -310,6 +315,41 @@ Keys can be agent names (`claude`, `codex`, `aider`, `gemini`, `opencode`) or mo
 
 Pricing is easy to update: the table lives in `coderace/cost.py` as a plain dict.
 
+## Model Selection
+
+Compare different models of the same agent head-to-head using the `agent:model` syntax:
+
+```bash
+# Compare two Codex models on the same task
+coderace run task.yaml --agent codex:gpt-5.4 --agent codex:gpt-5.3-codex
+
+# Mix agents and models
+coderace run task.yaml --agent codex:gpt-5.4 --agent claude:opus-4-6 --agent claude:sonnet-4-6
+
+# Benchmark multiple model variants across built-in tasks
+coderace benchmark --agents codex:gpt-5.4,codex:gpt-5.3-codex,claude:opus-4-6
+
+# Race with model variants (parallel)
+coderace race task.yaml
+```
+
+In task YAML files:
+
+```yaml
+agents:
+  - codex:gpt-5.4
+  - codex:gpt-5.3-codex
+  - claude:opus-4-6
+  - claude:sonnet-4-6
+```
+
+**How it works:**
+- `agent:model` splits on the first colon: `codex:gpt-5.4` → agent `codex`, model `gpt-5.4`
+- The model is passed via `--model <name>` to the underlying CLI
+- Results display as `codex (gpt-5.4)` vs `codex (gpt-5.3-codex)` for easy comparison
+- ELO ratings, leaderboard, and dashboard track each model variant separately
+- The same agent can appear multiple times with different models in one run
+
 ## Leaderboard & History
 
 Every `coderace run` automatically saves results to a local SQLite database (`~/.coderace/results.db`). Two new commands aggregate this data.
@@ -441,6 +481,37 @@ coderace run task.yaml --parallel
 
 Sequential mode (default) runs agents one at a time on the same repo.
 
+## Race Mode
+
+Use `coderace race` for first-to-pass execution. Unlike `coderace run --parallel`, race mode stops as soon as one agent passes the win condition:
+
+- If verification is configured, winner = first agent that passes verification.
+- If verification is not configured, winner = first agent that exits cleanly.
+- Remaining agents are stopped after a short graceful shutdown window.
+
+```bash
+coderace race task.yaml --agent claude --agent codex
+```
+
+Example terminal output:
+
+```text
+🏁 coderace race - fix-auth-bug
+Running 3 agents in parallel...
+
+Agent   Status                 Time
+claude  🔨 coding...           0:00:23
+codex   🧪 testing...          0:00:31
+aider   🛑 stopped             0:00:18
+
+🏆 Winner: codex - completed in 1:23 (first to pass verification)
+Runner-up: claude - finished 0:12 later
+```
+
+When to use each mode:
+- Use `coderace race` when you want the fastest successful patch and can stop early.
+- Use `coderace run --parallel` when you want full scoring across all agents before deciding.
+
 ## Why coderace?
 
 **Blog posts compare models. coderace compares agents on your work.**
@@ -576,9 +647,15 @@ coderace benchmark --agents claude --difficulty easy,medium
 # Dry-run: see what would run without executing
 coderace benchmark --agents claude,codex --dry-run
 
+# Statistical mode: run repeated trials per pair
+coderace benchmark --agents claude,codex --tasks fibonacci,json-parser --trials 5
+
 # Save report to file
 coderace benchmark --agents claude,codex --output report.md
 coderace benchmark --agents claude,codex --output report.html
+
+# Export standardized JSON (shareable benchmark artifact)
+coderace benchmark --agents claude,codex --trials 5 --export benchmark.json
 ```
 
 ### Example Terminal Output
@@ -622,7 +699,171 @@ coderace benchmark show bench-20260227-143022
 | `--difficulty` | Filter by difficulty: `easy`, `medium`, `hard` | all |
 | `--timeout` | Per-task timeout in seconds | `300` |
 | `--parallel N` | Run N agents in parallel | `1` (sequential) |
+| `--trials N` | Repeat each `(task, agent)` pair N times | `1` |
 | `--dry-run` | List combinations without running | `false` |
 | `--format` | Output format: `terminal`, `markdown`, `html` | `terminal` |
 | `--output` | Save report to file | — |
+| `--export` | Write standardized benchmark JSON file | — |
 | `--no-save` | Skip saving results to the store | `false` |
+
+### Statistical Reports (`--trials > 1`)
+
+When `--trials` is greater than 1, benchmark reports switch to statistical mode:
+
+- Task cells show `mean score +/- stddev` (plus mean wall time)
+- Report includes `CI (95%)`, `Consistency`, and `Reliability` columns
+- Summary includes per-agent mean score, confidence interval, win rate, and reliability
+- ELO ratings are rendered at the bottom of terminal/markdown/html reports
+
+### ELO Ratings
+
+Every benchmark run updates persistent ELO ratings across all benchmark history.
+
+```bash
+# Show ratings
+coderace ratings
+
+# JSON output
+coderace ratings --json
+
+# Reset all ratings to 1500
+coderace ratings --reset
+```
+
+ELO rules:
+- Initial rating: `1500`
+- K-factor: `32`
+- Each task is treated as a round-robin set of pairwise matches
+- Winner per pair is based on higher mean trial score (draw when within 1 point)
+
+### Export Format (`--export`)
+
+`coderace benchmark --export benchmark.json` writes a standardized JSON artifact:
+
+```json
+{
+  "coderace_version": "1.0.0",
+  "benchmark_id": "bench-20260228-133000",
+  "timestamp": "2026-02-28T13:30:00Z",
+  "system": { "os": "...", "python": "...", "cpu": "..." },
+  "config": { "trials": 5, "timeout": 300, "tasks": ["..."], "agents": ["..."] },
+  "results": [
+    {
+      "task": "fibonacci",
+      "agent": "claude",
+      "trials": 5,
+      "mean_score": 87.5,
+      "stddev_score": 3.2,
+      "ci_95": [83.1, 91.9],
+      "mean_time": 45.2,
+      "mean_cost": 0.03,
+      "pass_rate": 1.0,
+      "consistency_score": 0.96,
+      "per_trial": []
+    }
+  ],
+  "elo_ratings": { "claude": 1523, "codex": 1488 },
+  "summary": {}
+}
+```
+
+## Context Evaluation
+
+The `coderace context-eval` command measures whether a context file (CLAUDE.md, AGENTS.md, .cursorrules, etc.) actually improves agent performance. It runs A/B trials — baseline (no context file) vs treatment (with context file) — and produces statistical comparisons.
+
+```bash
+# Evaluate whether CLAUDE.md improves claude's performance on a task
+coderace context-eval --context-file CLAUDE.md --task fix-auth-bug.yaml --agents claude --trials 5
+
+# Evaluate across all built-in benchmark tasks
+coderace context-eval --context-file CLAUDE.md --benchmark --agents claude,codex
+
+# Save results as JSON
+coderace context-eval --context-file CLAUDE.md --task task.yaml --agents claude --output results.json
+
+# Use a custom task directory
+coderace context-eval --context-file CLAUDE.md --benchmark --task-dir ./my-tasks --agents claude
+```
+
+### How It Works
+
+For each agent × task combination:
+1. Run N trials **without** the context file (baseline condition)
+2. Run N trials **with** the context file placed in the task directory (treatment condition)
+3. Compare pass rates, mean scores, and compute statistical significance
+
+### Output
+
+The terminal report shows:
+- **Per-agent summary**: baseline vs treatment pass rates and scores, delta with 95% CI, Cohen's d effect size
+- **Per-task breakdown**: which tasks improved, which degraded
+- **Verdict**: whether the context file significantly improved performance
+
+```
+┌────────┬───────────────────┬────────────────────┬────────────────┬─────────────────┬────────┬──────────────────┬─────────────┐
+│ Agent  │ Baseline Pass Rate│ Treatment Pass Rate │ Baseline Score │ Treatment Score │ Delta  │ CI (95%)         │ Effect Size │
+├────────┼───────────────────┼────────────────────┼────────────────┼─────────────────┼────────┼──────────────────┼─────────────┤
+│ claude │              67%  │              100%  │           55.0 │            81.0 │ +26.0  │ [10.5, 41.5]     │        2.10 │
+│ codex  │              33%  │               67%  │           45.0 │            70.0 │ +25.0  │ [8.0, 42.0]      │        1.80 │
+└────────┴───────────────────┴────────────────────┴────────────────┴─────────────────┴────────┴──────────────────┴─────────────┘
+
+Context file improved performance by +25.5 points (CI: [12.0, 39.0])
+```
+
+### Context-Eval CLI Flags
+
+| Flag | Description | Default |
+|------|-------------|---------|
+| `--context-file` | Path to the context file to evaluate (required) | — |
+| `--task` | Path to a single task YAML | — |
+| `--benchmark` | Run against built-in benchmark tasks | `false` |
+| `--agents` | Comma-separated agent names (required) | — |
+| `--trials` | Trials per condition (min: 2) | `3` |
+| `--output` | Save JSON results to file | — |
+| `--task-dir` | Custom task directory for benchmark mode | — |
+
+### Dashboard Integration
+
+Include context-eval results in the HTML dashboard:
+
+```bash
+# Run context-eval and save JSON
+coderace context-eval --context-file CLAUDE.md --task task.yaml --agents claude --output eval.json
+
+# Generate dashboard with A/B comparison section
+coderace dashboard --context-eval eval.json
+```
+
+## Measuring Context Engineering Impact
+
+Context engineering — crafting CLAUDE.md, AGENTS.md, .cursorrules, and similar files — is becoming a core developer skill. But until now, there was no way to empirically measure whether your context files actually help.
+
+**The problem:** You write a CLAUDE.md with coding conventions, architectural guidelines, and project-specific instructions. But does it actually make agents produce better code? Or is it cargo-cult configuration?
+
+**The solution:** `coderace context-eval` gives you data:
+
+1. **Write your context file** (e.g., CLAUDE.md with project conventions)
+2. **Run A/B evaluation** against real coding tasks
+3. **Get statistical evidence** of improvement (or lack thereof)
+
+```bash
+# Iterate on your context file with data
+coderace context-eval --context-file CLAUDE.md --benchmark --agents claude --trials 5
+
+# Compare different context files
+coderace context-eval --context-file v1-claude.md --task task.yaml --agents claude --output v1.json
+coderace context-eval --context-file v2-claude.md --task task.yaml --agents claude --output v2.json
+```
+
+**Interpreting results:**
+- **Effect size > 0.8**: Large improvement — your context file is helping significantly
+- **Effect size 0.2–0.8**: Moderate improvement — some benefit, room to iterate
+- **Effect size < 0.2**: Negligible — your context file isn't making a measurable difference
+- **CI crosses zero**: Not statistically significant — need more trials or a better context file
+
+## See Also
+
+- **[agentmd](https://github.com/mikiships/agentmd)** — Generate and score context files (CLAUDE.md, AGENTS.md, .cursorrules) for AI coding agents. Pair with coderace: generate context with agentmd, measure agent performance with coderace, iterate with data instead of vibes.
+- **[agentlint](https://github.com/mikiships/agentlint)** — Lint AI agent git diffs for risky patterns (scope drift, secret leaks, test regression). Static analysis, no LLM required.
+
+Measure (coderace) → Optimize (agentmd) → Guard (agentlint).