coderace 1.3.0__tar.gz → 1.4.0__tar.gz

coderace-1.4.0/.claude-task.md

@@ -0,0 +1 @@
+Read the build contract in all-day-build-contract-benchmark-tasks-v2.md. Read existing tasks in src/coderace/tasks/ to understand the format. Then execute the contract: build all 5 deliverables (D1-D5). Follow the non-negotiable rules exactly. Commit after each deliverable. Run pytest tests/ -x after each commit to verify nothing breaks.

{coderace-1.3.0 → coderace-1.4.0}/CHANGELOG.md

@@ -1,5 +1,15 @@
 # Changelog
 
+## [1.4.0] - 2026-03-05
+
+### Added
+- **4 new benchmark tasks** testing real-world coding skills beyond "build from scratch":
+  - `bug-hunt`: Find and fix 5 planted bugs in a calculator module (debugging)
+  - `refactor`: Improve messy code while keeping existing tests passing (refactoring)
+  - `concurrent-queue`: Thread-safe priority queue with producer/consumer pattern (concurrency)
+  - `api-client`: HTTP client with retry, rate limiting, and circuit breaker (API design)
+- Total built-in tasks: 20 (up from 16)
+
 ## [1.3.0] - 2026-03-05
 
 ### Added

{coderace-1.3.0 → coderace-1.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: coderace
-Version: 1.3.0
+Version: 1.4.0
 Summary: Race coding agents against each other on real tasks
 Project-URL: Homepage, https://github.com/mikiships/coderace
 Project-URL: Repository, https://github.com/mikiships/coderace
@@ -217,6 +217,10 @@ coderace run --builtin fibonacci
 | `url-router` | Hard | HTTP-style router with params, wildcard, and 405/404 logic |
 | `diff-algorithm` | Hard | Unified diff + patch application roundtrip checks |
 | `task-scheduler` | Hard | Dependency-aware priority scheduler with timeout handling |
+| `bug-hunt` | Hard | Find and fix 5 planted bugs in a calculator module |
+| `refactor` | Hard | Refactor messy code while keeping tests passing |
+| `concurrent-queue` | Hard | Thread-safe priority queue with producer/consumer |
+| `api-client` | Hard | HTTP client with retry, rate limiting, circuit breaker |
 
 `coderace tasks list` now includes a `Verify` column so you can see which built-ins ship with verification suites.
 

{coderace-1.3.0 → coderace-1.4.0}/README.md

@@ -187,6 +187,10 @@ coderace run --builtin fibonacci
 | `url-router` | Hard | HTTP-style router with params, wildcard, and 405/404 logic |
 | `diff-algorithm` | Hard | Unified diff + patch application roundtrip checks |
 | `task-scheduler` | Hard | Dependency-aware priority scheduler with timeout handling |
+| `bug-hunt` | Hard | Find and fix 5 planted bugs in a calculator module |
+| `refactor` | Hard | Refactor messy code while keeping tests passing |
+| `concurrent-queue` | Hard | Thread-safe priority queue with producer/consumer |
+| `api-client` | Hard | HTTP client with retry, rate limiting, circuit breaker |
 
 `coderace tasks list` now includes a `Verify` column so you can see which built-ins ship with verification suites.
 

coderace-1.4.0/all-day-build-contract-benchmark-tasks-v2.md

@@ -0,0 +1,184 @@
+# All-Day Build Contract: New Benchmark Tasks (Batch 2)
+
+Status: In Progress
+Date: 2026-03-05
+Owner: Claude Code execution pass
+Scope type: Deliverable-gated (no hour promises)
+
+## 1. Objective
+
+Add 4 new built-in benchmark tasks to coderace that test **real-world coding skills** beyond "build from scratch." Current 16 tasks all ask agents to implement something new. These 4 test debugging, refactoring, concurrency, and API design — skills that matter in production but aren't benchmarked.
+
+This contract is considered complete only when every deliverable and validation gate below is satisfied.
+
+## 2. Non-Negotiable Build Rules
+
+1. No time-based completion claims.
+2. Completion is allowed only when all checklist items are checked.
+3. Full test suite must pass at the end (`pytest tests/ -x`).
+4. New tasks must follow the exact same format as existing built-in tasks (see `src/coderace/tasks/` for examples).
+5. CLI outputs must be deterministic and schema-backed where specified.
+6. Never modify files outside the project directory.
+7. Commit after each completed deliverable (not at the end).
+8. If stuck on same issue for 3 attempts, stop and write a blocker report.
+9. Do NOT refactor, restyle, or "improve" existing code or tasks.
+10. Read existing tasks in `src/coderace/tasks/` before writing new ones. Match the exact YAML format, naming conventions, and verification test style.
+
+## 3. Feature Deliverables
+
+### D1. Task: `bug-hunt` (debugging — fix planted bugs in existing code)
+
+**Concept:** Provide a working-looking module (`buggy_calculator.py`) with 5 planted bugs. The agent must find and fix them without being told exactly what's wrong.
+
+The task description gives the agent:
+- A pre-written `buggy_calculator.py` with an advanced calculator (expressions, variables, functions like `sqrt`, `abs`, `min`, `max`)
+- A failing test file that demonstrates the bugs exist
+- Instructions: "Fix ALL bugs. Do not rewrite from scratch."
+
+Planted bugs (specific, not vague):
+1. Division rounds to int instead of float (`//` instead of `/`)
+2. Negative number parsing fails (misses unary minus in expressions)
+3. Variable assignment overwrites built-in functions
+4. `min(a, b)` returns max (arguments reversed)
+5. Parenthesized expressions off-by-one in closing paren detection
+
+Hidden verification tests:
+- All 5 bugs must be fixed
+- Original working functionality must still work
+- Code must not be a complete rewrite (verify specific function signatures still exist)
+- Edge cases: nested expressions with negatives, chained function calls
+
+Required files:
+- `src/coderace/tasks/bug-hunt.yaml`
+- The YAML must include `setup_files:` with the buggy source and failing tests
+
+- [ ] Write buggy_calculator.py with exactly 5 planted bugs
+- [ ] Write test_buggy_calculator.py that exposes the bugs
+- [ ] Write verify_bug_hunt.py with hidden tests for all 5 fixes + non-rewrite check
+- [ ] Write bug-hunt.yaml task definition
+- [ ] Tests for D1 (task loads, runs dry-run, YAML validates)
+
+### D2. Task: `refactor` (improve existing messy code while keeping tests passing)
+
+**Concept:** Provide a working but messy module (`data_store.py`) — a key-value store with bad naming, duplicated logic, no type hints, mixed concerns, and poor error handling. The agent must refactor it while keeping the existing test suite passing.
+
+The task description gives the agent:
+- `data_store.py` — a working but ugly key-value store (~150 lines, deliberately messy)
+- `test_data_store.py` — passing tests (these must STILL pass after refactoring)
+- Instructions: "Refactor for readability, maintainability, and best practices. All existing tests must pass. Do not change test file."
+
+Quality dimensions to verify:
+1. Type hints added (verify with mypy or ast inspection)
+2. Functions under 25 lines each
+3. No duplicated logic (measure before/after)
+4. Consistent naming convention
+5. Proper error handling (specific exceptions, not bare except)
+
+Hidden verification tests:
+- All original test_data_store.py tests still pass
+- AST analysis: type hints present on all public functions
+- AST analysis: no function exceeds 25 lines
+- New edge cases that the refactored code should handle more gracefully
+- Import check: the module still exports the same public API
+
+Required files:
+- `src/coderace/tasks/refactor.yaml`
+
+- [ ] Write messy data_store.py (~150 lines, working but ugly)
+- [ ] Write test_data_store.py with comprehensive passing tests
+- [ ] Write verify_refactor.py with hidden quality checks (AST-based + functional)
+- [ ] Write refactor.yaml task definition
+- [ ] Tests for D2
+
+### D3. Task: `concurrent-queue` (thread-safe producer/consumer with priority)
+
+**Concept:** Build a thread-safe priority queue with producer/consumer pattern. Tests concurrency understanding — a known weak spot for LLMs.
+
+The task description tells the agent to create `concurrent_queue.py` with:
+- `PriorityTaskQueue` class
+- `submit(task, priority)` — thread-safe task submission
+- `worker(callback)` — starts a worker thread that processes tasks by priority
+- `shutdown(wait=True)` — graceful shutdown
+- Support for: task cancellation, timeout on get, max queue size with backpressure
+- Thread-safe stats: tasks_submitted, tasks_completed, tasks_failed
+
+And `test_concurrent_queue.py` with tests for:
+- Basic submit and process
+- Priority ordering (higher priority processed first)
+- Multiple workers
+- Graceful shutdown (pending tasks complete)
+- Task cancellation
+- Backpressure when queue is full
+- Thread safety under concurrent access
+
+Hidden verification tests:
+- Stress test: 1000 tasks, 10 workers, verify all complete
+- Priority ordering under load (submit mixed priorities, verify processing order)
+- Shutdown with pending tasks (verify all complete before shutdown returns)
+- Cancellation of in-flight vs queued tasks
+- Deadlock detection (timeout-based, the test itself must complete in <10s)
+
+Required files:
+- `src/coderace/tasks/concurrent-queue.yaml`
+
+- [ ] Write concurrent-queue.yaml with full spec
+- [ ] Write verify_concurrent_queue.py with stress tests and deadlock detection
+- [ ] Tests for D3
+
+### D4. Task: `api-client` (HTTP client with retry, rate limiting, circuit breaker)
+
+**Concept:** Build a production-grade HTTP API client with resilience patterns. Tests real-world engineering patterns.
+
+The task description tells the agent to create `api_client.py` with:
+- `APIClient(base_url, max_retries=3, rate_limit_per_sec=10)`
+- Methods: `get`, `post`, `put`, `delete` — all async-compatible but sync by default
+- Retry with exponential backoff (jitter) on 429 and 5xx
+- Rate limiting (token bucket)
+- Circuit breaker (open after N consecutive failures, half-open after timeout)
+- Request/response logging
+- Configurable timeout per request
+
+And `test_api_client.py` with tests using unittest.mock to simulate server responses.
+
+Hidden verification tests:
+- Retry behavior: mock server returns 503 twice then 200, verify 3 attempts made
+- Rate limiting: fire 20 requests, verify spacing respects rate limit
+- Circuit breaker: force 5 failures, verify circuit opens, then half-opens after cooldown
+- Backoff jitter: verify retry delays aren't deterministic (statistical test)
+- Timeout handling: mock slow server, verify timeout raised
+
+Required files:
+- `src/coderace/tasks/api-client.yaml`
+
+- [ ] Write api-client.yaml with full spec
+- [ ] Write verify_api_client.py with resilience pattern tests
+- [ ] Tests for D4
+
+### D5. Integration + Documentation
+
+- [ ] All 4 new tasks appear in `coderace tasks list` output
+- [ ] `coderace benchmark --dry-run --tasks bug-hunt,refactor,concurrent-queue,api-client` works
+- [ ] Update CHANGELOG.md with new tasks
+- [ ] Version bump to 1.4.0 in pyproject.toml
+- [ ] README: add "20 built-in tasks" (update from 16) and mention the new task categories
+- [ ] All existing tests still pass (`pytest tests/ -x`)
+
+## 4. Test Requirements
+
+- [ ] Unit tests for each new task (YAML loads, validates, dry-run works)
+- [ ] Integration test: `coderace benchmark --dry-run --tasks bug-hunt,refactor,concurrent-queue,api-client` completes
+- [ ] All existing 574 tests must still pass
+- [ ] New tests bring total to 590+
+
+## 5. Reports
+
+- Write progress to `progress-log.md` after each deliverable
+- Include: what was built, what tests pass, what's next, any blockers
+- Final summary when all deliverables done or stopped
+
+## 6. Stop Conditions
+
+- All deliverables checked and all tests passing -> DONE
+- 3 consecutive failed attempts on same issue -> STOP, write blocker report
+- Scope creep detected (new requirements discovered) -> STOP, report what's new
+- All tests passing but deliverables remain -> continue to next deliverable

coderace-1.4.0/coderace/builtins/tasks/api-client.yaml

@@ -0,0 +1,335 @@
+name: api-client
+difficulty: hard
+description: |
+  Build a production-grade HTTP API client with resilience patterns.
+
+  Create a file called `api_client.py` with the following class:
+
+  ```python
+  class APIClient:
+      def __init__(
+          self,
+          base_url: str,
+          max_retries: int = 3,
+          rate_limit_per_sec: float = 10,
+          timeout: float = 30.0,
+          circuit_breaker_threshold: int = 5,
+          circuit_breaker_timeout: float = 30.0,
+      ):
+          """Initialize the API client.
+
+          Args:
+              base_url: Base URL for all requests (e.g., "https://api.example.com")
+              max_retries: Max retry attempts on 429/5xx responses
+              rate_limit_per_sec: Max requests per second (token bucket)
+              timeout: Default timeout per request in seconds
+              circuit_breaker_threshold: Consecutive failures before circuit opens
+              circuit_breaker_timeout: Seconds before circuit transitions to half-open
+          """
+
+      def get(self, path: str, **kwargs) -> requests.Response:
+          """Send a GET request."""
+
+      def post(self, path: str, **kwargs) -> requests.Response:
+          """Send a POST request."""
+
+      def put(self, path: str, **kwargs) -> requests.Response:
+          """Send a PUT request."""
+
+      def delete(self, path: str, **kwargs) -> requests.Response:
+          """Send a DELETE request."""
+  ```
+
+  Requirements:
+  - All HTTP methods support `timeout`, `headers`, `params`, `json`, `data` kwargs
+  - Retry with exponential backoff + jitter on 429 and 5xx status codes
+  - Rate limiting using token bucket algorithm (max N requests per second)
+  - Circuit breaker pattern:
+    - CLOSED: requests flow normally
+    - OPEN: after N consecutive failures, immediately raise `CircuitBreakerOpen`
+    - HALF-OPEN: after timeout, allow one test request through
+    - Success in HALF-OPEN -> CLOSED; failure -> back to OPEN
+  - Custom exceptions: `CircuitBreakerOpen`, `APIClientError`
+  - `request_log` attribute: list of dicts with keys `method`, `url`, `status_code`, `timestamp`
+  - Uses the `requests` library for HTTP
+
+  Also create `test_api_client.py` with tests using `unittest.mock` to mock HTTP responses:
+  - Basic GET/POST/PUT/DELETE
+  - Retry on 503 then success
+  - Rate limiting (verify delays between rapid requests)
+  - Circuit breaker opens after consecutive failures
+  - Circuit breaker transitions from OPEN to HALF-OPEN to CLOSED
+  - Timeout handling
+  - Request logging
+
+  Do NOT modify any files other than `api_client.py` and `test_api_client.py`.
+repo: .
+test_command: python3 -m pytest test_api_client.py -x -q
+verify_command: python3 -m pytest verify_api_client.py -x -q
+verify_files:
+  verify_api_client.py: |
+    """Hidden verification tests for api-client task."""
+
+    import time
+    from unittest.mock import MagicMock, patch, PropertyMock
+
+    import pytest
+
+    from api_client import APIClient, CircuitBreakerOpen
+
+
+    def _mock_response(status_code=200, json_data=None):
+        resp = MagicMock()
+        resp.status_code = status_code
+        resp.json.return_value = json_data or {}
+        resp.ok = 200 <= status_code < 300
+        resp.raise_for_status = MagicMock()
+        if status_code >= 400:
+            resp.raise_for_status.side_effect = Exception(f"HTTP {status_code}")
+        return resp
+
+
+    class TestRetryBehavior:
+        @patch("api_client.requests.Session.request")
+        def test_retries_on_503_then_succeeds(self, mock_request):
+            mock_request.side_effect = [
+                _mock_response(503),
+                _mock_response(503),
+                _mock_response(200, {"ok": True}),
+            ]
+            client = APIClient(
+                "http://test.com",
+                max_retries=3,
+                rate_limit_per_sec=1000,
+            )
+            resp = client.get("/endpoint")
+            assert resp.status_code == 200
+            assert mock_request.call_count == 3
+
+        @patch("api_client.requests.Session.request")
+        def test_retries_on_429(self, mock_request):
+            mock_request.side_effect = [
+                _mock_response(429),
+                _mock_response(200),
+            ]
+            client = APIClient(
+                "http://test.com",
+                max_retries=3,
+                rate_limit_per_sec=1000,
+            )
+            resp = client.get("/path")
+            assert resp.status_code == 200
+            assert mock_request.call_count == 2
+
+        @patch("api_client.requests.Session.request")
+        def test_retries_exhausted_returns_last_response(self, mock_request):
+            mock_request.side_effect = [
+                _mock_response(503),
+                _mock_response(503),
+                _mock_response(503),
+            ]
+            client = APIClient(
+                "http://test.com",
+                max_retries=3,
+                rate_limit_per_sec=1000,
+            )
+            resp = client.get("/fail")
+            assert resp.status_code == 503
+
+
+    class TestBackoffJitter:
+        @patch("api_client.requests.Session.request")
+        def test_retry_delays_have_jitter(self, mock_request):
+            """Retry delays should not be deterministic."""
+            mock_request.side_effect = [
+                _mock_response(503),
+                _mock_response(503),
+                _mock_response(200),
+            ]
+            delays = []
+
+            original_sleep = time.sleep
+            def capture_sleep(duration):
+                delays.append(duration)
+                # Don't actually sleep in tests
+
+            client = APIClient(
+                "http://test.com",
+                max_retries=3,
+                rate_limit_per_sec=1000,
+            )
+            with patch("time.sleep", side_effect=capture_sleep):
+                client.get("/test")
+
+            # Should have at least 1 retry delay
+            retry_delays = [d for d in delays if d > 0.001]
+            assert len(retry_delays) >= 1
+
+
+    class TestRateLimiting:
+        @patch("api_client.requests.Session.request")
+        def test_rate_limit_spacing(self, mock_request):
+            mock_request.return_value = _mock_response(200)
+            # Very low rate limit to make spacing measurable
+            client = APIClient(
+                "http://test.com",
+                max_retries=0,
+                rate_limit_per_sec=5,
+            )
+            start = time.time()
+            for _ in range(6):
+                client.get("/test")
+            elapsed = time.time() - start
+            # 6 requests at 5/sec should take at least ~1 second
+            assert elapsed >= 0.8
+
+
+    class TestCircuitBreaker:
+        @patch("api_client.requests.Session.request")
+        def test_circuit_opens_after_consecutive_failures(self, mock_request):
+            mock_request.return_value = _mock_response(503)
+            client = APIClient(
+                "http://test.com",
+                max_retries=1,
+                rate_limit_per_sec=1000,
+                circuit_breaker_threshold=3,
+                circuit_breaker_timeout=60,
+            )
+
+            # Cause consecutive failures to open the circuit
+            for _ in range(3):
+                try:
+                    client.get("/fail")
+                except CircuitBreakerOpen:
+                    pass
+
+            # Now circuit should be open
+            with pytest.raises(CircuitBreakerOpen):
+                client.get("/blocked")
+
+        @patch("api_client.requests.Session.request")
+        def test_circuit_half_open_after_timeout(self, mock_request):
+            mock_request.return_value = _mock_response(503)
+            client = APIClient(
+                "http://test.com",
+                max_retries=1,
+                rate_limit_per_sec=1000,
+                circuit_breaker_threshold=3,
+                circuit_breaker_timeout=0.1,
+            )
+
+            # Open the circuit
+            for _ in range(5):
+                try:
+                    client.get("/fail")
+                except CircuitBreakerOpen:
+                    pass
+
+            # Wait for timeout to elapse
+            time.sleep(0.15)
+
+            # Should allow one request through (half-open)
+            mock_request.return_value = _mock_response(200)
+            resp = client.get("/recover")
+            assert resp.status_code == 200
+
+        @patch("api_client.requests.Session.request")
+        def test_circuit_closes_on_half_open_success(self, mock_request):
+            mock_request.return_value = _mock_response(503)
+            client = APIClient(
+                "http://test.com",
+                max_retries=1,
+                rate_limit_per_sec=1000,
+                circuit_breaker_threshold=3,
+                circuit_breaker_timeout=0.1,
+            )
+
+            # Open the circuit
+            for _ in range(5):
+                try:
+                    client.get("/fail")
+                except CircuitBreakerOpen:
+                    pass
+
+            time.sleep(0.15)
+
+            # Half-open success
+            mock_request.return_value = _mock_response(200)
+            client.get("/recover")
+
+            # Circuit should be closed now — further requests should work
+            resp = client.get("/another")
+            assert resp.status_code == 200
+
+
+    class TestHTTPMethods:
+        @patch("api_client.requests.Session.request")
+        def test_post(self, mock_request):
+            mock_request.return_value = _mock_response(201)
+            client = APIClient("http://test.com", rate_limit_per_sec=1000)
+            resp = client.post("/data", json={"key": "value"})
+            assert resp.status_code == 201
+
+        @patch("api_client.requests.Session.request")
+        def test_put(self, mock_request):
+            mock_request.return_value = _mock_response(200)
+            client = APIClient("http://test.com", rate_limit_per_sec=1000)
+            resp = client.put("/data/1", json={"key": "updated"})
+            assert resp.status_code == 200
+
+        @patch("api_client.requests.Session.request")
+        def test_delete(self, mock_request):
+            mock_request.return_value = _mock_response(204)
+            client = APIClient("http://test.com", rate_limit_per_sec=1000)
+            resp = client.delete("/data/1")
+            assert resp.status_code == 204
+
+
+    class TestRequestLogging:
+        @patch("api_client.requests.Session.request")
+        def test_requests_are_logged(self, mock_request):
+            mock_request.return_value = _mock_response(200)
+            client = APIClient("http://test.com", rate_limit_per_sec=1000)
+            client.get("/a")
+            client.post("/b")
+
+            log = client.request_log
+            assert len(log) >= 2
+            assert log[0]["method"].upper() == "GET"
+            assert "/a" in log[0]["url"]
+            assert log[0]["status_code"] == 200
+            assert "timestamp" in log[0]
+
+
+    class TestTimeout:
+        @patch("api_client.requests.Session.request")
+        def test_timeout_passed_to_request(self, mock_request):
+            mock_request.return_value = _mock_response(200)
+            client = APIClient(
+                "http://test.com",
+                rate_limit_per_sec=1000,
+                timeout=5.0,
+            )
+            client.get("/test")
+            call_kwargs = mock_request.call_args
+            # Timeout should be passed to the underlying request
+            assert call_kwargs.kwargs.get("timeout") == 5.0 or (
+                len(call_kwargs.args) > 0 and
+                any(isinstance(a, (int, float)) and a == 5.0
+                    for a in call_kwargs.args)
+            ) or "timeout" in str(call_kwargs)
+lint_command: ruff check api_client.py
+timeout: 600
+agents:
+  - claude
+  - codex
+  - gemini
+  - aider
+  - opencode
+scoring:
+  tests: 25
+  verify: 30
+  exit: 20
+  lint: 15
+  time: 5
+  lines: 5