coderace 0.9.0__tar.gz → 1.2.0__tar.gz

{coderace-0.9.0 → coderace-1.2.0}/CHANGELOG.md

@@ -1,5 +1,33 @@
 # Changelog
 
+## [1.2.0] - 2026-03-03
+
+### Added
+
+- **`coderace race` command** - New first-to-pass race mode with early-stop semantics. Agents run in parallel worktrees and the race ends when the first winner is found.
+- **Live race UI** - Rich `Live` panel with per-agent status and timers:
+  - `🔨 coding...`
+  - `🧪 testing...`
+  - `✅ WINNER!`
+  - `❌ failed`
+  - `⏰ timed out`
+  - `🛑 stopped`
+- **Winner announcement and runner-up delta** - Prints race winner and optional runner-up timing delta after the live panel closes.
+- **Race result persistence (JSON fallback)** - Saves race summaries to `.coderace/race-results.json` including `race_id`, winner metadata, participant statuses, exit codes, and wall times. Supports `--no-save`.
+- **Race test suite** - Added 21 race-focused tests (`tests/test_race.py`) covering winner logic, cancellation, timeout/no-winner paths, verification modes, live updates, serialization, and Ctrl+C cleanup.
+
+## [1.0.0] - 2026-02-28
+
+### Added
+
+- **Benchmark trials mode** — `coderace benchmark --trials N` now runs each `(task, agent)` pair repeatedly and stores each trial with `trial_number` in SQLite.
+- **Statistical benchmarking module** — New `coderace/statistics.py` computes per-pair and per-agent aggregates: mean/stddev, 95% confidence intervals, pass rate, consistency, win rate, cost efficiency, and reliability.
+- **Persistent ELO ratings** — New `coderace/elo.py` plus `elo_ratings` store table. Ratings update automatically after each benchmark using pairwise task outcomes and persist across runs.
+- **`coderace ratings` command** — View persistent ELO rankings, output as JSON (`--json`), and reset all ratings (`--reset`).
+- **Standardized benchmark export** — `coderace benchmark --export <path>` writes shareable JSON with run metadata, system info, per-trial details, aggregate stats, and current ELO ratings.
+- **Enhanced benchmark report rendering** — Multi-trial reports now show statistical columns (`mean +/- stddev`, CI, consistency, reliability) and include ELO ratings in terminal/markdown/html output.
+- **Integration and edge-case coverage for v1.0 flow** — Added tests for full `--trials 3` benchmark + export + ELO pipeline and edge cases (single trial/agent/task and always-failing agent).
+
 ## [0.7.0] - 2026-02-26
 
 ### Added

coderace-1.2.0/DONE.txt

@@ -0,0 +1,70 @@
+Context Eval Build Contract: COMPLETE
+======================================
+
+Date: 2026-03-02
+All deliverables (D1-D4) implemented and validated.
+
+## What Was Built
+
+### D1: context-eval CLI Command (core + CLI)
+- `coderace/context_eval.py`: Core A/B evaluation engine
+  - ContextEvalResult and TrialResult data classes
+  - Context file backup/restore/placement/removal for baseline vs treatment
+  - KNOWN_CONTEXT_FILES list (CLAUDE.md, AGENTS.md, .cursorrules, etc.)
+  - run_context_eval() orchestrator running N trials per condition
+- `coderace/commands/context_eval.py`: CLI subcommand with:
+  --context-file PATH, --task PATH, --benchmark, --agents, --trials N,
+  --output PATH, --task-dir PATH
+- Full input validation (missing files, invalid agents, trials < 2, etc.)
+
+### D2: Statistical Comparison Report
+- `coderace/context_eval_report.py`: Statistical analysis and rendering
+  - Delta with 95% CI using Welch's t-test
+  - Cohen's d effect size
+  - Per-agent summary: baseline vs treatment pass rates and scores
+  - Per-task breakdown: which tasks improved, which degraded
+  - Summary verdict: "improved", "degraded", or "no significant improvement"
+  - Rich terminal table output
+  - JSON output format
+
+### D3: Dashboard Integration
+- Extended `coderace/dashboard.py` with context-eval A/B section:
+  - Bar chart: baseline vs treatment scores per agent
+  - Delta table with CI (95%) and effect size
+  - Verdict display
+  - CSS for A/B visualization (.ab-baseline, .ab-treatment, .positive, .negative)
+- Added --context-eval PATH flag to `coderace dashboard` command
+
+### D4: Documentation + Examples
+- README.md: Added "Context Evaluation" and "Measuring Context Engineering Impact" sections
+  with usage examples, output format, CLI flags table, and effect size interpretation guide
+- examples/context-eval-demo.sh: Executable demo script
+- Clear help text on `coderace context-eval --help` and `coderace --help`
+
+## Test Results
+
+- 58 new tests added (41 for D1+D2, 17 for D3)
+- All 505 tests pass (447 original + 58 new)
+- No regressions in existing test suite
+
+## Commits
+
+1. feat(context-eval): add context-eval command with A/B statistical comparison (D1+D2)
+2. feat(context-eval): add dashboard A/B comparison section (D3)
+3. docs(context-eval): add README section, examples, and interpretation guide (D4)
+
+## Files Created/Modified
+
+New files:
+- coderace/context_eval.py
+- coderace/commands/context_eval.py
+- coderace/context_eval_report.py
+- tests/test_context_eval.py
+- tests/test_context_eval_dashboard.py
+- examples/context-eval-demo.sh
+
+Modified files:
+- coderace/cli.py (registered context-eval subcommand + --context-eval dashboard flag)
+- coderace/dashboard.py (added A/B comparison section)
+- README.md (added context-eval documentation)
+- progress-log.md (added D1-D4 progress entries)

{coderace-0.9.0 → coderace-1.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: coderace
-Version: 0.9.0
+Version: 1.2.0
 Summary: Race coding agents against each other on real tasks
 Project-URL: Homepage, https://github.com/mikiships/coderace
 Project-URL: Repository, https://github.com/mikiships/coderace
@@ -471,6 +471,37 @@ coderace run task.yaml --parallel
 
 Sequential mode (default) runs agents one at a time on the same repo.
 
+## Race Mode
+
+Use `coderace race` for first-to-pass execution. Unlike `coderace run --parallel`, race mode stops as soon as one agent passes the win condition:
+
+- If verification is configured, winner = first agent that passes verification.
+- If verification is not configured, winner = first agent that exits cleanly.
+- Remaining agents are stopped after a short graceful shutdown window.
+
+```bash
+coderace race task.yaml --agent claude --agent codex
+```
+
+Example terminal output:
+
+```text
+🏁 coderace race - fix-auth-bug
+Running 3 agents in parallel...
+
+Agent   Status                 Time
+claude  🔨 coding...           0:00:23
+codex   🧪 testing...          0:00:31
+aider   🛑 stopped             0:00:18
+
+🏆 Winner: codex - completed in 1:23 (first to pass verification)
+Runner-up: claude - finished 0:12 later
+```
+
+When to use each mode:
+- Use `coderace race` when you want the fastest successful patch and can stop early.
+- Use `coderace run --parallel` when you want full scoring across all agents before deciding.
+
 ## Why coderace?
 
 **Blog posts compare models. coderace compares agents on your work.**
@@ -606,9 +637,15 @@ coderace benchmark --agents claude --difficulty easy,medium
 # Dry-run: see what would run without executing
 coderace benchmark --agents claude,codex --dry-run
 
+# Statistical mode: run repeated trials per pair
+coderace benchmark --agents claude,codex --tasks fibonacci,json-parser --trials 5
+
 # Save report to file
 coderace benchmark --agents claude,codex --output report.md
 coderace benchmark --agents claude,codex --output report.html
+
+# Export standardized JSON (shareable benchmark artifact)
+coderace benchmark --agents claude,codex --trials 5 --export benchmark.json
 ```
 
 ### Example Terminal Output
@@ -652,7 +689,168 @@ coderace benchmark show bench-20260227-143022
 | `--difficulty` | Filter by difficulty: `easy`, `medium`, `hard` | all |
 | `--timeout` | Per-task timeout in seconds | `300` |
 | `--parallel N` | Run N agents in parallel | `1` (sequential) |
+| `--trials N` | Repeat each `(task, agent)` pair N times | `1` |
 | `--dry-run` | List combinations without running | `false` |
 | `--format` | Output format: `terminal`, `markdown`, `html` | `terminal` |
 | `--output` | Save report to file | — |
+| `--export` | Write standardized benchmark JSON file | — |
 | `--no-save` | Skip saving results to the store | `false` |
+
+### Statistical Reports (`--trials > 1`)
+
+When `--trials` is greater than 1, benchmark reports switch to statistical mode:
+
+- Task cells show `mean score +/- stddev` (plus mean wall time)
+- Report includes `CI (95%)`, `Consistency`, and `Reliability` columns
+- Summary includes per-agent mean score, confidence interval, win rate, and reliability
+- ELO ratings are rendered at the bottom of terminal/markdown/html reports
+
+### ELO Ratings
+
+Every benchmark run updates persistent ELO ratings across all benchmark history.
+
+```bash
+# Show ratings
+coderace ratings
+
+# JSON output
+coderace ratings --json
+
+# Reset all ratings to 1500
+coderace ratings --reset
+```
+
+ELO rules:
+- Initial rating: `1500`
+- K-factor: `32`
+- Each task is treated as a round-robin set of pairwise matches
+- Winner per pair is based on higher mean trial score (draw when within 1 point)
+
+### Export Format (`--export`)
+
+`coderace benchmark --export benchmark.json` writes a standardized JSON artifact:
+
+```json
+{
+  "coderace_version": "1.0.0",
+  "benchmark_id": "bench-20260228-133000",
+  "timestamp": "2026-02-28T13:30:00Z",
+  "system": { "os": "...", "python": "...", "cpu": "..." },
+  "config": { "trials": 5, "timeout": 300, "tasks": ["..."], "agents": ["..."] },
+  "results": [
+    {
+      "task": "fibonacci",
+      "agent": "claude",
+      "trials": 5,
+      "mean_score": 87.5,
+      "stddev_score": 3.2,
+      "ci_95": [83.1, 91.9],
+      "mean_time": 45.2,
+      "mean_cost": 0.03,
+      "pass_rate": 1.0,
+      "consistency_score": 0.96,
+      "per_trial": []
+    }
+  ],
+  "elo_ratings": { "claude": 1523, "codex": 1488 },
+  "summary": {}
+}
+```
+
+## Context Evaluation
+
+The `coderace context-eval` command measures whether a context file (CLAUDE.md, AGENTS.md, .cursorrules, etc.) actually improves agent performance. It runs A/B trials — baseline (no context file) vs treatment (with context file) — and produces statistical comparisons.
+
+```bash
+# Evaluate whether CLAUDE.md improves claude's performance on a task
+coderace context-eval --context-file CLAUDE.md --task fix-auth-bug.yaml --agents claude --trials 5
+
+# Evaluate across all built-in benchmark tasks
+coderace context-eval --context-file CLAUDE.md --benchmark --agents claude,codex
+
+# Save results as JSON
+coderace context-eval --context-file CLAUDE.md --task task.yaml --agents claude --output results.json
+
+# Use a custom task directory
+coderace context-eval --context-file CLAUDE.md --benchmark --task-dir ./my-tasks --agents claude
+```
+
+### How It Works
+
+For each agent × task combination:
+1. Run N trials **without** the context file (baseline condition)
+2. Run N trials **with** the context file placed in the task directory (treatment condition)
+3. Compare pass rates, mean scores, and compute statistical significance
+
+### Output
+
+The terminal report shows:
+- **Per-agent summary**: baseline vs treatment pass rates and scores, delta with 95% CI, Cohen's d effect size
+- **Per-task breakdown**: which tasks improved, which degraded
+- **Verdict**: whether the context file significantly improved performance
+
+```
+┌────────┬───────────────────┬────────────────────┬────────────────┬─────────────────┬────────┬──────────────────┬─────────────┐
+│ Agent  │ Baseline Pass Rate│ Treatment Pass Rate │ Baseline Score │ Treatment Score │ Delta  │ CI (95%)         │ Effect Size │
+├────────┼───────────────────┼────────────────────┼────────────────┼─────────────────┼────────┼──────────────────┼─────────────┤
+│ claude │              67%  │              100%  │           55.0 │            81.0 │ +26.0  │ [10.5, 41.5]     │        2.10 │
+│ codex  │              33%  │               67%  │           45.0 │            70.0 │ +25.0  │ [8.0, 42.0]      │        1.80 │
+└────────┴───────────────────┴────────────────────┴────────────────┴─────────────────┴────────┴──────────────────┴─────────────┘
+
+Context file improved performance by +25.5 points (CI: [12.0, 39.0])
+```
+
+### Context-Eval CLI Flags
+
+| Flag | Description | Default |
+|------|-------------|---------|
+| `--context-file` | Path to the context file to evaluate (required) | — |
+| `--task` | Path to a single task YAML | — |
+| `--benchmark` | Run against built-in benchmark tasks | `false` |
+| `--agents` | Comma-separated agent names (required) | — |
+| `--trials` | Trials per condition (min: 2) | `3` |
+| `--output` | Save JSON results to file | — |
+| `--task-dir` | Custom task directory for benchmark mode | — |
+
+### Dashboard Integration
+
+Include context-eval results in the HTML dashboard:
+
+```bash
+# Run context-eval and save JSON
+coderace context-eval --context-file CLAUDE.md --task task.yaml --agents claude --output eval.json
+
+# Generate dashboard with A/B comparison section
+coderace dashboard --context-eval eval.json
+```
+
+## Measuring Context Engineering Impact
+
+Context engineering — crafting CLAUDE.md, AGENTS.md, .cursorrules, and similar files — is becoming a core developer skill. But until now, there was no way to empirically measure whether your context files actually help.
+
+**The problem:** You write a CLAUDE.md with coding conventions, architectural guidelines, and project-specific instructions. But does it actually make agents produce better code? Or is it cargo-cult configuration?
+
+**The solution:** `coderace context-eval` gives you data:
+
+1. **Write your context file** (e.g., CLAUDE.md with project conventions)
+2. **Run A/B evaluation** against real coding tasks
+3. **Get statistical evidence** of improvement (or lack thereof)
+
+```bash
+# Iterate on your context file with data
+coderace context-eval --context-file CLAUDE.md --benchmark --agents claude --trials 5
+
+# Compare different context files
+coderace context-eval --context-file v1-claude.md --task task.yaml --agents claude --output v1.json
+coderace context-eval --context-file v2-claude.md --task task.yaml --agents claude --output v2.json
+```
+
+**Interpreting results:**
+- **Effect size > 0.8**: Large improvement — your context file is helping significantly
+- **Effect size 0.2–0.8**: Moderate improvement — some benefit, room to iterate
+- **Effect size < 0.2**: Negligible — your context file isn't making a measurable difference
+- **CI crosses zero**: Not statistically significant — need more trials or a better context file
+
+## See Also
+
+- **[agentmd](https://github.com/mikiships/agentmd)** — Generate and score context files (CLAUDE.md, AGENTS.md, .cursorrules) for AI coding agents. Pair with coderace: generate context with agentmd, measure agent performance with coderace, iterate with data instead of vibes.

{coderace-0.9.0 → coderace-1.2.0}/README.md

@@ -441,6 +441,37 @@ coderace run task.yaml --parallel
 
 Sequential mode (default) runs agents one at a time on the same repo.
 
+## Race Mode
+
+Use `coderace race` for first-to-pass execution. Unlike `coderace run --parallel`, race mode stops as soon as one agent passes the win condition:
+
+- If verification is configured, winner = first agent that passes verification.
+- If verification is not configured, winner = first agent that exits cleanly.
+- Remaining agents are stopped after a short graceful shutdown window.
+
+```bash
+coderace race task.yaml --agent claude --agent codex
+```
+
+Example terminal output:
+
+```text
+🏁 coderace race - fix-auth-bug
+Running 3 agents in parallel...
+
+Agent   Status                 Time
+claude  🔨 coding...           0:00:23
+codex   🧪 testing...          0:00:31
+aider   🛑 stopped             0:00:18
+
+🏆 Winner: codex - completed in 1:23 (first to pass verification)
+Runner-up: claude - finished 0:12 later
+```
+
+When to use each mode:
+- Use `coderace race` when you want the fastest successful patch and can stop early.
+- Use `coderace run --parallel` when you want full scoring across all agents before deciding.
+
 ## Why coderace?
 
 **Blog posts compare models. coderace compares agents on your work.**
@@ -576,9 +607,15 @@ coderace benchmark --agents claude --difficulty easy,medium
 # Dry-run: see what would run without executing
 coderace benchmark --agents claude,codex --dry-run
 
+# Statistical mode: run repeated trials per pair
+coderace benchmark --agents claude,codex --tasks fibonacci,json-parser --trials 5
+
 # Save report to file
 coderace benchmark --agents claude,codex --output report.md
 coderace benchmark --agents claude,codex --output report.html
+
+# Export standardized JSON (shareable benchmark artifact)
+coderace benchmark --agents claude,codex --trials 5 --export benchmark.json
 ```
 
 ### Example Terminal Output
@@ -622,7 +659,168 @@ coderace benchmark show bench-20260227-143022
 | `--difficulty` | Filter by difficulty: `easy`, `medium`, `hard` | all |
 | `--timeout` | Per-task timeout in seconds | `300` |
 | `--parallel N` | Run N agents in parallel | `1` (sequential) |
+| `--trials N` | Repeat each `(task, agent)` pair N times | `1` |
 | `--dry-run` | List combinations without running | `false` |
 | `--format` | Output format: `terminal`, `markdown`, `html` | `terminal` |
 | `--output` | Save report to file | — |
+| `--export` | Write standardized benchmark JSON file | — |
 | `--no-save` | Skip saving results to the store | `false` |
+
+### Statistical Reports (`--trials > 1`)
+
+When `--trials` is greater than 1, benchmark reports switch to statistical mode:
+
+- Task cells show `mean score +/- stddev` (plus mean wall time)
+- Report includes `CI (95%)`, `Consistency`, and `Reliability` columns
+- Summary includes per-agent mean score, confidence interval, win rate, and reliability
+- ELO ratings are rendered at the bottom of terminal/markdown/html reports
+
+### ELO Ratings
+
+Every benchmark run updates persistent ELO ratings across all benchmark history.
+
+```bash
+# Show ratings
+coderace ratings
+
+# JSON output
+coderace ratings --json
+
+# Reset all ratings to 1500
+coderace ratings --reset
+```
+
+ELO rules:
+- Initial rating: `1500`
+- K-factor: `32`
+- Each task is treated as a round-robin set of pairwise matches
+- Winner per pair is based on higher mean trial score (draw when within 1 point)
+
+### Export Format (`--export`)
+
+`coderace benchmark --export benchmark.json` writes a standardized JSON artifact:
+
+```json
+{
+  "coderace_version": "1.0.0",
+  "benchmark_id": "bench-20260228-133000",
+  "timestamp": "2026-02-28T13:30:00Z",
+  "system": { "os": "...", "python": "...", "cpu": "..." },
+  "config": { "trials": 5, "timeout": 300, "tasks": ["..."], "agents": ["..."] },
+  "results": [
+    {
+      "task": "fibonacci",
+      "agent": "claude",
+      "trials": 5,
+      "mean_score": 87.5,
+      "stddev_score": 3.2,
+      "ci_95": [83.1, 91.9],
+      "mean_time": 45.2,
+      "mean_cost": 0.03,
+      "pass_rate": 1.0,
+      "consistency_score": 0.96,
+      "per_trial": []
+    }
+  ],
+  "elo_ratings": { "claude": 1523, "codex": 1488 },
+  "summary": {}
+}
+```
+
+## Context Evaluation
+
+The `coderace context-eval` command measures whether a context file (CLAUDE.md, AGENTS.md, .cursorrules, etc.) actually improves agent performance. It runs A/B trials — baseline (no context file) vs treatment (with context file) — and produces statistical comparisons.
+
+```bash
+# Evaluate whether CLAUDE.md improves claude's performance on a task
+coderace context-eval --context-file CLAUDE.md --task fix-auth-bug.yaml --agents claude --trials 5
+
+# Evaluate across all built-in benchmark tasks
+coderace context-eval --context-file CLAUDE.md --benchmark --agents claude,codex
+
+# Save results as JSON
+coderace context-eval --context-file CLAUDE.md --task task.yaml --agents claude --output results.json
+
+# Use a custom task directory
+coderace context-eval --context-file CLAUDE.md --benchmark --task-dir ./my-tasks --agents claude
+```
+
+### How It Works
+
+For each agent × task combination:
+1. Run N trials **without** the context file (baseline condition)
+2. Run N trials **with** the context file placed in the task directory (treatment condition)
+3. Compare pass rates, mean scores, and compute statistical significance
+
+### Output
+
+The terminal report shows:
+- **Per-agent summary**: baseline vs treatment pass rates and scores, delta with 95% CI, Cohen's d effect size
+- **Per-task breakdown**: which tasks improved, which degraded
+- **Verdict**: whether the context file significantly improved performance
+
+```
+┌────────┬───────────────────┬────────────────────┬────────────────┬─────────────────┬────────┬──────────────────┬─────────────┐
+│ Agent  │ Baseline Pass Rate│ Treatment Pass Rate │ Baseline Score │ Treatment Score │ Delta  │ CI (95%)         │ Effect Size │
+├────────┼───────────────────┼────────────────────┼────────────────┼─────────────────┼────────┼──────────────────┼─────────────┤
+│ claude │              67%  │              100%  │           55.0 │            81.0 │ +26.0  │ [10.5, 41.5]     │        2.10 │
+│ codex  │              33%  │               67%  │           45.0 │            70.0 │ +25.0  │ [8.0, 42.0]      │        1.80 │
+└────────┴───────────────────┴────────────────────┴────────────────┴─────────────────┴────────┴──────────────────┴─────────────┘
+
+Context file improved performance by +25.5 points (CI: [12.0, 39.0])
+```
+
+### Context-Eval CLI Flags
+
+| Flag | Description | Default |
+|------|-------------|---------|
+| `--context-file` | Path to the context file to evaluate (required) | — |
+| `--task` | Path to a single task YAML | — |
+| `--benchmark` | Run against built-in benchmark tasks | `false` |
+| `--agents` | Comma-separated agent names (required) | — |
+| `--trials` | Trials per condition (min: 2) | `3` |
+| `--output` | Save JSON results to file | — |
+| `--task-dir` | Custom task directory for benchmark mode | — |
+
+### Dashboard Integration
+
+Include context-eval results in the HTML dashboard:
+
+```bash
+# Run context-eval and save JSON
+coderace context-eval --context-file CLAUDE.md --task task.yaml --agents claude --output eval.json
+
+# Generate dashboard with A/B comparison section
+coderace dashboard --context-eval eval.json
+```
+
+## Measuring Context Engineering Impact
+
+Context engineering — crafting CLAUDE.md, AGENTS.md, .cursorrules, and similar files — is becoming a core developer skill. But until now, there was no way to empirically measure whether your context files actually help.
+
+**The problem:** You write a CLAUDE.md with coding conventions, architectural guidelines, and project-specific instructions. But does it actually make agents produce better code? Or is it cargo-cult configuration?
+
+**The solution:** `coderace context-eval` gives you data:
+
+1. **Write your context file** (e.g., CLAUDE.md with project conventions)
+2. **Run A/B evaluation** against real coding tasks
+3. **Get statistical evidence** of improvement (or lack thereof)
+
+```bash
+# Iterate on your context file with data
+coderace context-eval --context-file CLAUDE.md --benchmark --agents claude --trials 5
+
+# Compare different context files
+coderace context-eval --context-file v1-claude.md --task task.yaml --agents claude --output v1.json
+coderace context-eval --context-file v2-claude.md --task task.yaml --agents claude --output v2.json
+```
+
+**Interpreting results:**
+- **Effect size > 0.8**: Large improvement — your context file is helping significantly
+- **Effect size 0.2–0.8**: Moderate improvement — some benefit, room to iterate
+- **Effect size < 0.2**: Negligible — your context file isn't making a measurable difference
+- **CI crosses zero**: Not statistically significant — need more trials or a better context file
+
+## See Also
+
+- **[agentmd](https://github.com/mikiships/agentmd)** — Generate and score context files (CLAUDE.md, AGENTS.md, .cursorrules) for AI coding agents. Pair with coderace: generate context with agentmd, measure agent performance with coderace, iterate with data instead of vibes.

coderace-1.2.0/all-day-build-contract-context-eval.md

@@ -0,0 +1,120 @@
+# All-Day Build Contract: Context Eval
+
+Status: In Progress
+Date: 2026-03-02
+Owner: Codex execution pass
+Scope type: Deliverable-gated (no hour promises)
+
+## 1. Objective
+
+Add a `coderace context-eval` command that measures whether context files (CLAUDE.md, AGENTS.md, .cursorrules, etc.) actually improve agent performance on coding tasks. Runs A/B trials: baseline (no context file) vs treatment (with context file), produces statistical comparison with confidence intervals. This is the first tool that lets developers empirically measure context engineering impact.
+
+This contract is considered complete only when every deliverable and validation gate below is satisfied.
+
+## 2. Non-Negotiable Build Rules
+
+1. No time-based completion claims.
+2. Completion is allowed only when all checklist items are checked.
+3. Full test suite must pass at the end.
+4. New features must ship with docs and report addendum updates in the same pass.
+5. CLI outputs must be deterministic and schema-backed where specified.
+6. Never modify files outside the project directory.
+7. Commit after each completed deliverable (not at the end).
+8. If stuck on same issue for 3 attempts, stop and write a blocker report.
+9. Do NOT refactor, restyle, or "improve" code outside the deliverables.
+10. Read existing tests and docs before writing new code.
+
+## 3. Feature Deliverables
+
+### D1. `context-eval` CLI Command (core + CLI)
+
+Add a new `context-eval` subcommand to the coderace CLI that:
+- Accepts `--context-file PATH` (the file to evaluate, e.g., CLAUDE.md)
+- Accepts `--task PATH` (a single task YAML) or `--benchmark` (run built-in tasks)
+- Accepts `--agents AGENT1,AGENT2` (which agents to test, default: all configured)
+- Accepts `--trials N` (number of trials per condition, default: 3, min: 2)
+- Accepts `--output PATH` (optional JSON output path)
+- Accepts `--task-dir PATH` (optional, custom task directory for benchmark mode)
+
+Workflow:
+1. For each agent × task combination:
+   a. Run N trials WITHOUT the context file present (baseline condition)
+   b. Run N trials WITH the context file placed in the task working directory (treatment condition)
+2. Collect pass/fail + timing for each trial
+3. Produce comparison report
+
+The context file placement: copy the file to the task's working directory before treatment trials, remove it after. If a context file already exists in the task dir, back it up and restore after.
+
+Required files:
+- `coderace/context_eval.py` (core logic)
+- `coderace/cli.py` (add subcommand)
+
+- [ ] CLI argument parsing with validation
+- [ ] Baseline runner (strips context files from task dir)
+- [ ] Treatment runner (places context file in task dir)
+- [ ] Backup/restore logic for pre-existing context files
+- [ ] Integration with existing `run` infrastructure (reuse task loading, agent execution, scoring)
+- [ ] Tests for D1
+
+### D2. Statistical Comparison Report
+
+After running both conditions, produce a comparison report that includes:
+- Per-agent: baseline pass rate vs treatment pass rate
+- Delta (treatment - baseline) with 95% confidence interval
+- Effect size (Cohen's d or similar)
+- Per-task breakdown: which tasks improved, which degraded
+- Summary verdict: "Context file improved performance by X% (CI: [lo, hi])" or "No significant improvement detected"
+
+Output formats:
+- Rich terminal table (default, using existing table formatting)
+- JSON (with `--output`)
+
+Reuse existing statistical infrastructure from `benchmark` command where possible (confidence intervals, etc.).
+
+- [ ] Statistical comparison logic (delta, CI, effect size)
+- [ ] Terminal table report
+- [ ] JSON output
+- [ ] Summary verdict logic
+- [ ] Tests for D2
+
+### D3. Dashboard Integration
+
+Extend the existing `dashboard` command to include context-eval results when available:
+- New section in HTML dashboard showing A/B comparison
+- Bar chart: baseline vs treatment pass rates per agent
+- Delta chart with confidence intervals
+
+Reuse existing dashboard infrastructure (Jinja templates, chart generation).
+
+- [ ] Dashboard data model for context-eval results
+- [ ] HTML template section for A/B comparison
+- [ ] Charts (bar chart + delta chart)
+- [ ] Tests for D3
+
+### D4. Documentation + Examples
+
+- [ ] Update README.md with context-eval section (usage, examples, interpretation guide)
+- [ ] Add example in `examples/` directory: `context-eval-demo.yaml` or script
+- [ ] Update `coderace --help` / `coderace context-eval --help` with clear descriptions
+- [ ] Add a section on "Measuring Context Engineering Impact" to README
+
+## 4. Test Requirements
+
+- [ ] Unit tests for context file placement/removal/backup logic
+- [ ] Unit tests for statistical comparison (known inputs → known outputs)
+- [ ] Integration test: mock agent that passes more with context vs without
+- [ ] Edge cases: no agents configured, context file doesn't exist, task dir already has context file, trials=1 (should error), 0 tasks selected
+- [ ] All existing tests (447) must still pass
+
+## 5. Reports
+
+- Write progress to `progress-log.md` after each deliverable
+- Include: what was built, what tests pass, what's next, any blockers
+- Final summary when all deliverables done or stopped
+
+## 6. Stop Conditions
+
+- All deliverables checked and all tests passing → DONE
+- 3 consecutive failed attempts on same issue → STOP, write blocker report
+- Scope creep detected (new requirements discovered) → STOP, report what's new
+- All tests passing but deliverables remain → continue to next deliverable