coderace 0.8.1__tar.gz → 1.2.0__tar.gz

{coderace-0.8.1 → coderace-1.2.0}/CHANGELOG.md

@@ -1,5 +1,33 @@
 # Changelog
 
+## [1.2.0] - 2026-03-03
+
+### Added
+
+- **`coderace race` command** - New first-to-pass race mode with early-stop semantics. Agents run in parallel worktrees and the race ends when the first winner is found.
+- **Live race UI** - Rich `Live` panel with per-agent status and timers:
+  - `🔨 coding...`
+  - `🧪 testing...`
+  - `✅ WINNER!`
+  - `❌ failed`
+  - `⏰ timed out`
+  - `🛑 stopped`
+- **Winner announcement and runner-up delta** - Prints race winner and optional runner-up timing delta after the live panel closes.
+- **Race result persistence (JSON fallback)** - Saves race summaries to `.coderace/race-results.json` including `race_id`, winner metadata, participant statuses, exit codes, and wall times. Supports `--no-save`.
+- **Race test suite** - Added 21 race-focused tests (`tests/test_race.py`) covering winner logic, cancellation, timeout/no-winner paths, verification modes, live updates, serialization, and Ctrl+C cleanup.
+
+## [1.0.0] - 2026-02-28
+
+### Added
+
+- **Benchmark trials mode** — `coderace benchmark --trials N` now runs each `(task, agent)` pair repeatedly and stores each trial with `trial_number` in SQLite.
+- **Statistical benchmarking module** — New `coderace/statistics.py` computes per-pair and per-agent aggregates: mean/stddev, 95% confidence intervals, pass rate, consistency, win rate, cost efficiency, and reliability.
+- **Persistent ELO ratings** — New `coderace/elo.py` plus `elo_ratings` store table. Ratings update automatically after each benchmark using pairwise task outcomes and persist across runs.
+- **`coderace ratings` command** — View persistent ELO rankings, output as JSON (`--json`), and reset all ratings (`--reset`).
+- **Standardized benchmark export** — `coderace benchmark --export <path>` writes shareable JSON with run metadata, system info, per-trial details, aggregate stats, and current ELO ratings.
+- **Enhanced benchmark report rendering** — Multi-trial reports now show statistical columns (`mean +/- stddev`, CI, consistency, reliability) and include ELO ratings in terminal/markdown/html output.
+- **Integration and edge-case coverage for v1.0 flow** — Added tests for full `--trials 3` benchmark + export + ELO pipeline and edge cases (single trial/agent/task and always-failing agent).
+
 ## [0.7.0] - 2026-02-26
 
 ### Added

coderace-1.2.0/DONE.txt

@@ -0,0 +1,70 @@
+Context Eval Build Contract: COMPLETE
+======================================
+
+Date: 2026-03-02
+All deliverables (D1-D4) implemented and validated.
+
+## What Was Built
+
+### D1: context-eval CLI Command (core + CLI)
+- `coderace/context_eval.py`: Core A/B evaluation engine
+  - ContextEvalResult and TrialResult data classes
+  - Context file backup/restore/placement/removal for baseline vs treatment
+  - KNOWN_CONTEXT_FILES list (CLAUDE.md, AGENTS.md, .cursorrules, etc.)
+  - run_context_eval() orchestrator running N trials per condition
+- `coderace/commands/context_eval.py`: CLI subcommand with:
+  --context-file PATH, --task PATH, --benchmark, --agents, --trials N,
+  --output PATH, --task-dir PATH
+- Full input validation (missing files, invalid agents, trials < 2, etc.)
+
+### D2: Statistical Comparison Report
+- `coderace/context_eval_report.py`: Statistical analysis and rendering
+  - Delta with 95% CI using Welch's t-test
+  - Cohen's d effect size
+  - Per-agent summary: baseline vs treatment pass rates and scores
+  - Per-task breakdown: which tasks improved, which degraded
+  - Summary verdict: "improved", "degraded", or "no significant improvement"
+  - Rich terminal table output
+  - JSON output format
+
+### D3: Dashboard Integration
+- Extended `coderace/dashboard.py` with context-eval A/B section:
+  - Bar chart: baseline vs treatment scores per agent
+  - Delta table with CI (95%) and effect size
+  - Verdict display
+  - CSS for A/B visualization (.ab-baseline, .ab-treatment, .positive, .negative)
+- Added --context-eval PATH flag to `coderace dashboard` command
+
+### D4: Documentation + Examples
+- README.md: Added "Context Evaluation" and "Measuring Context Engineering Impact" sections
+  with usage examples, output format, CLI flags table, and effect size interpretation guide
+- examples/context-eval-demo.sh: Executable demo script
+- Clear help text on `coderace context-eval --help` and `coderace --help`
+
+## Test Results
+
+- 58 new tests added (41 for D1+D2, 17 for D3)
+- All 505 tests pass (447 original + 58 new)
+- No regressions in existing test suite
+
+## Commits
+
+1. feat(context-eval): add context-eval command with A/B statistical comparison (D1+D2)
+2. feat(context-eval): add dashboard A/B comparison section (D3)
+3. docs(context-eval): add README section, examples, and interpretation guide (D4)
+
+## Files Created/Modified
+
+New files:
+- coderace/context_eval.py
+- coderace/commands/context_eval.py
+- coderace/context_eval_report.py
+- tests/test_context_eval.py
+- tests/test_context_eval_dashboard.py
+- examples/context-eval-demo.sh
+
+Modified files:
+- coderace/cli.py (registered context-eval subcommand + --context-eval dashboard flag)
+- coderace/dashboard.py (added A/B comparison section)
+- README.md (added context-eval documentation)
+- progress-log.md (added D1-D4 progress entries)

{coderace-0.8.1 → coderace-1.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: coderace
-Version: 0.8.1
+Version: 1.2.0
 Summary: Race coding agents against each other on real tasks
 Project-URL: Homepage, https://github.com/mikiships/coderace
 Project-URL: Repository, https://github.com/mikiships/coderace
@@ -106,6 +106,12 @@ description: |
 repo: .
 test_command: pytest tests/test_auth.py -x
 lint_command: ruff check .
+# Optional: independent verification suite written after agent completes
+# verify_command: python3 -m pytest verify_auth.py -x -q
+# verify_files:
+#   verify_auth.py: |
+#     def test_real_contract():
+#       assert True
 timeout: 300
 agents:
   - claude
@@ -133,6 +139,34 @@ For each agent in the task:
 | Wall time | 15% | Faster is better (normalized across agents) |
 | Lines changed | 10% | Fewer is better (normalized across agents) |
 
+## Verification Tests
+
+For stronger evaluation, tasks can define an independent verification suite that the agent does not control.
+
+```yaml
+verify_command: python3 -m pytest verify_api_contract.py -x -q
+verify_files:
+  verify_api_contract.py: |
+    def test_contract_behavior():
+      assert True
+```
+
+Flow for verification-enabled tasks:
+1. Agent completes implementation.
+2. `test_command` runs (agent-authored tests).
+3. `verify_files` are written into the workspace (overwriting same-path files).
+4. `verify_command` runs.
+
+Default scoring when `verify_command` is present:
+- tests: 25%
+- verify: 30%
+- exit: 20%
+- lint: 15%
+- time: 5%
+- lines: 5%
+
+Tasks without `verify_command` keep the legacy default scoring (40/20/15/15/10).
+
 ## Output
 
 Terminal table with Rich formatting:
@@ -172,6 +206,14 @@ coderace run --builtin fibonacci
 | `csv-analyzer` | Medium | CLI tool for CSV summary statistics |
 | `http-server` | Medium-Hard | HTTP/1.1 server using only stdlib socket |
 | `binary-search-tree` | Hard | AVL tree with insert, delete, search, and balancing |
+| `regex-engine` | Hard | Regex engine with custom matcher + verification suite |
+| `lru-cache` | Hard | Thread-safe LRU + TTL correctness verification |
+| `expression-evaluator` | Hard | Expression parser/evaluator with precedence and functions |
+| `url-router` | Hard | HTTP-style router with params, wildcard, and 405/404 logic |
+| `diff-algorithm` | Hard | Unified diff + patch application roundtrip checks |
+| `task-scheduler` | Hard | Dependency-aware priority scheduler with timeout handling |
+
+`coderace tasks list` now includes a `Verify` column so you can see which built-ins ship with verification suites.
 
 ## Try It Now
 
@@ -429,6 +471,37 @@ coderace run task.yaml --parallel
 
 Sequential mode (default) runs agents one at a time on the same repo.
 
+## Race Mode
+
+Use `coderace race` for first-to-pass execution. Unlike `coderace run --parallel`, race mode stops as soon as one agent passes the win condition:
+
+- If verification is configured, winner = first agent that passes verification.
+- If verification is not configured, winner = first agent that exits cleanly.
+- Remaining agents are stopped after a short graceful shutdown window.
+
+```bash
+coderace race task.yaml --agent claude --agent codex
+```
+
+Example terminal output:
+
+```text
+🏁 coderace race - fix-auth-bug
+Running 3 agents in parallel...
+
+Agent   Status                 Time
+claude  🔨 coding...           0:00:23
+codex   🧪 testing...          0:00:31
+aider   🛑 stopped             0:00:18
+
+🏆 Winner: codex - completed in 1:23 (first to pass verification)
+Runner-up: claude - finished 0:12 later
+```
+
+When to use each mode:
+- Use `coderace race` when you want the fastest successful patch and can stop early.
+- Use `coderace run --parallel` when you want full scoring across all agents before deciding.
+
 ## Why coderace?
 
 **Blog posts compare models. coderace compares agents on your work.**
@@ -564,9 +637,15 @@ coderace benchmark --agents claude --difficulty easy,medium
 # Dry-run: see what would run without executing
 coderace benchmark --agents claude,codex --dry-run
 
+# Statistical mode: run repeated trials per pair
+coderace benchmark --agents claude,codex --tasks fibonacci,json-parser --trials 5
+
 # Save report to file
 coderace benchmark --agents claude,codex --output report.md
 coderace benchmark --agents claude,codex --output report.html
+
+# Export standardized JSON (shareable benchmark artifact)
+coderace benchmark --agents claude,codex --trials 5 --export benchmark.json
 ```
 
 ### Example Terminal Output
@@ -610,7 +689,168 @@ coderace benchmark show bench-20260227-143022
 | `--difficulty` | Filter by difficulty: `easy`, `medium`, `hard` | all |
 | `--timeout` | Per-task timeout in seconds | `300` |
 | `--parallel N` | Run N agents in parallel | `1` (sequential) |
+| `--trials N` | Repeat each `(task, agent)` pair N times | `1` |
 | `--dry-run` | List combinations without running | `false` |
 | `--format` | Output format: `terminal`, `markdown`, `html` | `terminal` |
 | `--output` | Save report to file | — |
+| `--export` | Write standardized benchmark JSON file | — |
 | `--no-save` | Skip saving results to the store | `false` |
+
+### Statistical Reports (`--trials > 1`)
+
+When `--trials` is greater than 1, benchmark reports switch to statistical mode:
+
+- Task cells show `mean score +/- stddev` (plus mean wall time)
+- Report includes `CI (95%)`, `Consistency`, and `Reliability` columns
+- Summary includes per-agent mean score, confidence interval, win rate, and reliability
+- ELO ratings are rendered at the bottom of terminal/markdown/html reports
+
+### ELO Ratings
+
+Every benchmark run updates persistent ELO ratings across all benchmark history.
+
+```bash
+# Show ratings
+coderace ratings
+
+# JSON output
+coderace ratings --json
+
+# Reset all ratings to 1500
+coderace ratings --reset
+```
+
+ELO rules:
+- Initial rating: `1500`
+- K-factor: `32`
+- Each task is treated as a round-robin set of pairwise matches
+- Winner per pair is based on higher mean trial score (draw when within 1 point)
+
+### Export Format (`--export`)
+
+`coderace benchmark --export benchmark.json` writes a standardized JSON artifact:
+
+```json
+{
+  "coderace_version": "1.0.0",
+  "benchmark_id": "bench-20260228-133000",
+  "timestamp": "2026-02-28T13:30:00Z",
+  "system": { "os": "...", "python": "...", "cpu": "..." },
+  "config": { "trials": 5, "timeout": 300, "tasks": ["..."], "agents": ["..."] },
+  "results": [
+    {
+      "task": "fibonacci",
+      "agent": "claude",
+      "trials": 5,
+      "mean_score": 87.5,
+      "stddev_score": 3.2,
+      "ci_95": [83.1, 91.9],
+      "mean_time": 45.2,
+      "mean_cost": 0.03,
+      "pass_rate": 1.0,
+      "consistency_score": 0.96,
+      "per_trial": []
+    }
+  ],
+  "elo_ratings": { "claude": 1523, "codex": 1488 },
+  "summary": {}
+}
+```
+
+## Context Evaluation
+
+The `coderace context-eval` command measures whether a context file (CLAUDE.md, AGENTS.md, .cursorrules, etc.) actually improves agent performance. It runs A/B trials — baseline (no context file) vs treatment (with context file) — and produces statistical comparisons.
+
+```bash
+# Evaluate whether CLAUDE.md improves claude's performance on a task
+coderace context-eval --context-file CLAUDE.md --task fix-auth-bug.yaml --agents claude --trials 5
+
+# Evaluate across all built-in benchmark tasks
+coderace context-eval --context-file CLAUDE.md --benchmark --agents claude,codex
+
+# Save results as JSON
+coderace context-eval --context-file CLAUDE.md --task task.yaml --agents claude --output results.json
+
+# Use a custom task directory
+coderace context-eval --context-file CLAUDE.md --benchmark --task-dir ./my-tasks --agents claude
+```
+
+### How It Works
+
+For each agent × task combination:
+1. Run N trials **without** the context file (baseline condition)
+2. Run N trials **with** the context file placed in the task directory (treatment condition)
+3. Compare pass rates, mean scores, and compute statistical significance
+
+### Output
+
+The terminal report shows:
+- **Per-agent summary**: baseline vs treatment pass rates and scores, delta with 95% CI, Cohen's d effect size
+- **Per-task breakdown**: which tasks improved, which degraded
+- **Verdict**: whether the context file significantly improved performance
+
+```
+┌────────┬───────────────────┬────────────────────┬────────────────┬─────────────────┬────────┬──────────────────┬─────────────┐
+│ Agent  │ Baseline Pass Rate│ Treatment Pass Rate │ Baseline Score │ Treatment Score │ Delta  │ CI (95%)         │ Effect Size │
+├────────┼───────────────────┼────────────────────┼────────────────┼─────────────────┼────────┼──────────────────┼─────────────┤
+│ claude │              67%  │              100%  │           55.0 │            81.0 │ +26.0  │ [10.5, 41.5]     │        2.10 │
+│ codex  │              33%  │               67%  │           45.0 │            70.0 │ +25.0  │ [8.0, 42.0]      │        1.80 │
+└────────┴───────────────────┴────────────────────┴────────────────┴─────────────────┴────────┴──────────────────┴─────────────┘
+
+Context file improved performance by +25.5 points (CI: [12.0, 39.0])
+```
+
+### Context-Eval CLI Flags
+
+| Flag | Description | Default |
+|------|-------------|---------|
+| `--context-file` | Path to the context file to evaluate (required) | — |
+| `--task` | Path to a single task YAML | — |
+| `--benchmark` | Run against built-in benchmark tasks | `false` |
+| `--agents` | Comma-separated agent names (required) | — |
+| `--trials` | Trials per condition (min: 2) | `3` |
+| `--output` | Save JSON results to file | — |
+| `--task-dir` | Custom task directory for benchmark mode | — |
+
+### Dashboard Integration
+
+Include context-eval results in the HTML dashboard:
+
+```bash
+# Run context-eval and save JSON
+coderace context-eval --context-file CLAUDE.md --task task.yaml --agents claude --output eval.json
+
+# Generate dashboard with A/B comparison section
+coderace dashboard --context-eval eval.json
+```
+
+## Measuring Context Engineering Impact
+
+Context engineering — crafting CLAUDE.md, AGENTS.md, .cursorrules, and similar files — is becoming a core developer skill. But until now, there was no way to empirically measure whether your context files actually help.
+
+**The problem:** You write a CLAUDE.md with coding conventions, architectural guidelines, and project-specific instructions. But does it actually make agents produce better code? Or is it cargo-cult configuration?
+
+**The solution:** `coderace context-eval` gives you data:
+
+1. **Write your context file** (e.g., CLAUDE.md with project conventions)
+2. **Run A/B evaluation** against real coding tasks
+3. **Get statistical evidence** of improvement (or lack thereof)
+
+```bash
+# Iterate on your context file with data
+coderace context-eval --context-file CLAUDE.md --benchmark --agents claude --trials 5
+
+# Compare different context files
+coderace context-eval --context-file v1-claude.md --task task.yaml --agents claude --output v1.json
+coderace context-eval --context-file v2-claude.md --task task.yaml --agents claude --output v2.json
+```
+
+**Interpreting results:**
+- **Effect size > 0.8**: Large improvement — your context file is helping significantly
+- **Effect size 0.2–0.8**: Moderate improvement — some benefit, room to iterate
+- **Effect size < 0.2**: Negligible — your context file isn't making a measurable difference
+- **CI crosses zero**: Not statistically significant — need more trials or a better context file
+
+## See Also
+
+- **[agentmd](https://github.com/mikiships/agentmd)** — Generate and score context files (CLAUDE.md, AGENTS.md, .cursorrules) for AI coding agents. Pair with coderace: generate context with agentmd, measure agent performance with coderace, iterate with data instead of vibes.

{coderace-0.8.1 → coderace-1.2.0}/README.md

@@ -76,6 +76,12 @@ description: |
 repo: .
 test_command: pytest tests/test_auth.py -x
 lint_command: ruff check .
+# Optional: independent verification suite written after agent completes
+# verify_command: python3 -m pytest verify_auth.py -x -q
+# verify_files:
+#   verify_auth.py: |
+#     def test_real_contract():
+#       assert True
 timeout: 300
 agents:
   - claude
@@ -103,6 +109,34 @@ For each agent in the task:
 | Wall time | 15% | Faster is better (normalized across agents) |
 | Lines changed | 10% | Fewer is better (normalized across agents) |
 
+## Verification Tests
+
+For stronger evaluation, tasks can define an independent verification suite that the agent does not control.
+
+```yaml
+verify_command: python3 -m pytest verify_api_contract.py -x -q
+verify_files:
+  verify_api_contract.py: |
+    def test_contract_behavior():
+      assert True
+```
+
+Flow for verification-enabled tasks:
+1. Agent completes implementation.
+2. `test_command` runs (agent-authored tests).
+3. `verify_files` are written into the workspace (overwriting same-path files).
+4. `verify_command` runs.
+
+Default scoring when `verify_command` is present:
+- tests: 25%
+- verify: 30%
+- exit: 20%
+- lint: 15%
+- time: 5%
+- lines: 5%
+
+Tasks without `verify_command` keep the legacy default scoring (40/20/15/15/10).
+
 ## Output
 
 Terminal table with Rich formatting:
@@ -142,6 +176,14 @@ coderace run --builtin fibonacci
 | `csv-analyzer` | Medium | CLI tool for CSV summary statistics |
 | `http-server` | Medium-Hard | HTTP/1.1 server using only stdlib socket |
 | `binary-search-tree` | Hard | AVL tree with insert, delete, search, and balancing |
+| `regex-engine` | Hard | Regex engine with custom matcher + verification suite |
+| `lru-cache` | Hard | Thread-safe LRU + TTL correctness verification |
+| `expression-evaluator` | Hard | Expression parser/evaluator with precedence and functions |
+| `url-router` | Hard | HTTP-style router with params, wildcard, and 405/404 logic |
+| `diff-algorithm` | Hard | Unified diff + patch application roundtrip checks |
+| `task-scheduler` | Hard | Dependency-aware priority scheduler with timeout handling |
+
+`coderace tasks list` now includes a `Verify` column so you can see which built-ins ship with verification suites.
 
 ## Try It Now
 
@@ -399,6 +441,37 @@ coderace run task.yaml --parallel
 
 Sequential mode (default) runs agents one at a time on the same repo.
 
+## Race Mode
+
+Use `coderace race` for first-to-pass execution. Unlike `coderace run --parallel`, race mode stops as soon as one agent passes the win condition:
+
+- If verification is configured, winner = first agent that passes verification.
+- If verification is not configured, winner = first agent that exits cleanly.
+- Remaining agents are stopped after a short graceful shutdown window.
+
+```bash
+coderace race task.yaml --agent claude --agent codex
+```
+
+Example terminal output:
+
+```text
+🏁 coderace race - fix-auth-bug
+Running 3 agents in parallel...
+
+Agent   Status                 Time
+claude  🔨 coding...           0:00:23
+codex   🧪 testing...          0:00:31
+aider   🛑 stopped             0:00:18
+
+🏆 Winner: codex - completed in 1:23 (first to pass verification)
+Runner-up: claude - finished 0:12 later
+```
+
+When to use each mode:
+- Use `coderace race` when you want the fastest successful patch and can stop early.
+- Use `coderace run --parallel` when you want full scoring across all agents before deciding.
+
 ## Why coderace?
 
 **Blog posts compare models. coderace compares agents on your work.**
@@ -534,9 +607,15 @@ coderace benchmark --agents claude --difficulty easy,medium
 # Dry-run: see what would run without executing
 coderace benchmark --agents claude,codex --dry-run
 
+# Statistical mode: run repeated trials per pair
+coderace benchmark --agents claude,codex --tasks fibonacci,json-parser --trials 5
+
 # Save report to file
 coderace benchmark --agents claude,codex --output report.md
 coderace benchmark --agents claude,codex --output report.html
+
+# Export standardized JSON (shareable benchmark artifact)
+coderace benchmark --agents claude,codex --trials 5 --export benchmark.json
 ```
 
 ### Example Terminal Output
@@ -580,7 +659,168 @@ coderace benchmark show bench-20260227-143022
 | `--difficulty` | Filter by difficulty: `easy`, `medium`, `hard` | all |
 | `--timeout` | Per-task timeout in seconds | `300` |
 | `--parallel N` | Run N agents in parallel | `1` (sequential) |
+| `--trials N` | Repeat each `(task, agent)` pair N times | `1` |
 | `--dry-run` | List combinations without running | `false` |
 | `--format` | Output format: `terminal`, `markdown`, `html` | `terminal` |
 | `--output` | Save report to file | — |
+| `--export` | Write standardized benchmark JSON file | — |
 | `--no-save` | Skip saving results to the store | `false` |
+
+### Statistical Reports (`--trials > 1`)
+
+When `--trials` is greater than 1, benchmark reports switch to statistical mode:
+
+- Task cells show `mean score +/- stddev` (plus mean wall time)
+- Report includes `CI (95%)`, `Consistency`, and `Reliability` columns
+- Summary includes per-agent mean score, confidence interval, win rate, and reliability
+- ELO ratings are rendered at the bottom of terminal/markdown/html reports
+
+### ELO Ratings
+
+Every benchmark run updates persistent ELO ratings across all benchmark history.
+
+```bash
+# Show ratings
+coderace ratings
+
+# JSON output
+coderace ratings --json
+
+# Reset all ratings to 1500
+coderace ratings --reset
+```
+
+ELO rules:
+- Initial rating: `1500`
+- K-factor: `32`
+- Each task is treated as a round-robin set of pairwise matches
+- Winner per pair is based on higher mean trial score (draw when within 1 point)
+
+### Export Format (`--export`)
+
+`coderace benchmark --export benchmark.json` writes a standardized JSON artifact:
+
+```json
+{
+  "coderace_version": "1.0.0",
+  "benchmark_id": "bench-20260228-133000",
+  "timestamp": "2026-02-28T13:30:00Z",
+  "system": { "os": "...", "python": "...", "cpu": "..." },
+  "config": { "trials": 5, "timeout": 300, "tasks": ["..."], "agents": ["..."] },
+  "results": [
+    {
+      "task": "fibonacci",
+      "agent": "claude",
+      "trials": 5,
+      "mean_score": 87.5,
+      "stddev_score": 3.2,
+      "ci_95": [83.1, 91.9],
+      "mean_time": 45.2,
+      "mean_cost": 0.03,
+      "pass_rate": 1.0,
+      "consistency_score": 0.96,
+      "per_trial": []
+    }
+  ],
+  "elo_ratings": { "claude": 1523, "codex": 1488 },
+  "summary": {}
+}
+```
+
+## Context Evaluation
+
+The `coderace context-eval` command measures whether a context file (CLAUDE.md, AGENTS.md, .cursorrules, etc.) actually improves agent performance. It runs A/B trials — baseline (no context file) vs treatment (with context file) — and produces statistical comparisons.
+
+```bash
+# Evaluate whether CLAUDE.md improves claude's performance on a task
+coderace context-eval --context-file CLAUDE.md --task fix-auth-bug.yaml --agents claude --trials 5
+
+# Evaluate across all built-in benchmark tasks
+coderace context-eval --context-file CLAUDE.md --benchmark --agents claude,codex
+
+# Save results as JSON
+coderace context-eval --context-file CLAUDE.md --task task.yaml --agents claude --output results.json
+
+# Use a custom task directory
+coderace context-eval --context-file CLAUDE.md --benchmark --task-dir ./my-tasks --agents claude
+```
+
+### How It Works
+
+For each agent × task combination:
+1. Run N trials **without** the context file (baseline condition)
+2. Run N trials **with** the context file placed in the task directory (treatment condition)
+3. Compare pass rates, mean scores, and compute statistical significance
+
+### Output
+
+The terminal report shows:
+- **Per-agent summary**: baseline vs treatment pass rates and scores, delta with 95% CI, Cohen's d effect size
+- **Per-task breakdown**: which tasks improved, which degraded
+- **Verdict**: whether the context file significantly improved performance
+
+```
+┌────────┬───────────────────┬────────────────────┬────────────────┬─────────────────┬────────┬──────────────────┬─────────────┐
+│ Agent  │ Baseline Pass Rate│ Treatment Pass Rate │ Baseline Score │ Treatment Score │ Delta  │ CI (95%)         │ Effect Size │
+├────────┼───────────────────┼────────────────────┼────────────────┼─────────────────┼────────┼──────────────────┼─────────────┤
+│ claude │              67%  │              100%  │           55.0 │            81.0 │ +26.0  │ [10.5, 41.5]     │        2.10 │
+│ codex  │              33%  │               67%  │           45.0 │            70.0 │ +25.0  │ [8.0, 42.0]      │        1.80 │
+└────────┴───────────────────┴────────────────────┴────────────────┴─────────────────┴────────┴──────────────────┴─────────────┘
+
+Context file improved performance by +25.5 points (CI: [12.0, 39.0])
+```
+
+### Context-Eval CLI Flags
+
+| Flag | Description | Default |
+|------|-------------|---------|
+| `--context-file` | Path to the context file to evaluate (required) | — |
+| `--task` | Path to a single task YAML | — |
+| `--benchmark` | Run against built-in benchmark tasks | `false` |
+| `--agents` | Comma-separated agent names (required) | — |
+| `--trials` | Trials per condition (min: 2) | `3` |
+| `--output` | Save JSON results to file | — |
+| `--task-dir` | Custom task directory for benchmark mode | — |
+
+### Dashboard Integration
+
+Include context-eval results in the HTML dashboard:
+
+```bash
+# Run context-eval and save JSON
+coderace context-eval --context-file CLAUDE.md --task task.yaml --agents claude --output eval.json
+
+# Generate dashboard with A/B comparison section
+coderace dashboard --context-eval eval.json
+```
+
+## Measuring Context Engineering Impact
+
+Context engineering — crafting CLAUDE.md, AGENTS.md, .cursorrules, and similar files — is becoming a core developer skill. But until now, there was no way to empirically measure whether your context files actually help.
+
+**The problem:** You write a CLAUDE.md with coding conventions, architectural guidelines, and project-specific instructions. But does it actually make agents produce better code? Or is it cargo-cult configuration?
+
+**The solution:** `coderace context-eval` gives you data:
+
+1. **Write your context file** (e.g., CLAUDE.md with project conventions)
+2. **Run A/B evaluation** against real coding tasks
+3. **Get statistical evidence** of improvement (or lack thereof)
+
+```bash
+# Iterate on your context file with data
+coderace context-eval --context-file CLAUDE.md --benchmark --agents claude --trials 5
+
+# Compare different context files
+coderace context-eval --context-file v1-claude.md --task task.yaml --agents claude --output v1.json
+coderace context-eval --context-file v2-claude.md --task task.yaml --agents claude --output v2.json
+```
+
+**Interpreting results:**
+- **Effect size > 0.8**: Large improvement — your context file is helping significantly
+- **Effect size 0.2–0.8**: Moderate improvement — some benefit, room to iterate
+- **Effect size < 0.2**: Negligible — your context file isn't making a measurable difference
+- **CI crosses zero**: Not statistically significant — need more trials or a better context file
+
+## See Also
+
+- **[agentmd](https://github.com/mikiships/agentmd)** — Generate and score context files (CLAUDE.md, AGENTS.md, .cursorrules) for AI coding agents. Pair with coderace: generate context with agentmd, measure agent performance with coderace, iterate with data instead of vibes.