coderace 0.1.0__tar.gz

coderace-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,30 @@
+name: Publish to PyPI
+
+on:
+  release:
+    types: [published]
+
+permissions:
+  id-token: write
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: https://pypi.org/p/coderace
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install build tools
+        run: pip install build
+
+      - name: Build package
+        run: python -m build
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

coderace-0.1.0/.gitignore

@@ -0,0 +1,13 @@
+.venv/
+.coderace/
+.agent-relay/
+__pycache__/
+*.egg-info/
+*.pyc
+dist/
+build/
+
+AGENTS.md
+CLAUDE.md
+GEMINI.md
+prpm.lock

coderace-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 mikiships
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

coderace-0.1.0/PKG-INFO

@@ -0,0 +1,142 @@
+Metadata-Version: 2.4
+Name: coderace
+Version: 0.1.0
+Summary: Race coding agents against each other on real tasks
+Project-URL: Homepage, https://github.com/mikiships/coderace
+Project-URL: Repository, https://github.com/mikiships/coderace
+Author: mikiships
+License-Expression: MIT
+License-File: LICENSE
+Keywords: ai,aider,benchmark,claude,codex,coding-agents
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0
+Requires-Dist: typer>=0.9.0
+Provides-Extra: dev
+Requires-Dist: pytest-mock>=3.0; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Requires-Dist: ruff>=0.4.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# coderace
+
+Race coding agents against each other on real tasks in your repo.
+
+Define a task. Run it against Claude Code, Codex, and Aider. Get a scored comparison table.
+
+## Install
+
+```bash
+pip install coderace
+```
+
+## Quick Start
+
+```bash
+# Create a task template
+coderace init fix-auth-bug
+
+# Edit the task file (describe the bug, set test command)
+# Then race the agents:
+coderace run fix-auth-bug.yaml
+
+# Or race them in parallel (uses git worktrees):
+coderace run fix-auth-bug.yaml --parallel
+
+# View results from the last run
+coderace results fix-auth-bug.yaml
+```
+
+## Task Format
+
+```yaml
+name: fix-auth-bug
+description: |
+  The login endpoint returns 500 when email contains a plus sign.
+  Fix the email validation in auth/validators.py.
+repo: .
+test_command: pytest tests/test_auth.py -x
+lint_command: ruff check .
+timeout: 300
+agents:
+  - claude
+  - codex
+  - aider
+```
+
+## What It Does
+
+For each agent in the task:
+
+1. Creates a fresh git branch (`coderace/<agent>-<task>`)
+2. Invokes the agent CLI with the task description
+3. Runs your test command
+4. Runs your lint command (optional)
+5. Computes a composite score
+
+## Scoring
+
+| Metric | Weight | Description |
+|--------|--------|-------------|
+| Tests pass | 40% | Did the test command exit 0? |
+| Exit clean | 20% | Did the agent itself exit 0 without timeout? |
+| Lint clean | 15% | Did the lint command exit 0? |
+| Wall time | 15% | Faster is better (normalized across agents) |
+| Lines changed | 10% | Fewer is better (normalized across agents) |
+
+## Output
+
+Terminal table with Rich formatting:
+
+```
+┌──────┬────────┬───────┬───────┬──────┬──────┬──────────┬───────┐
+│ Rank │ Agent  │ Score │ Tests │ Exit │ Lint │ Time (s) │ Lines │
+├──────┼────────┼───────┼───────┼──────┼──────┼──────────┼───────┤
+│  1   │ claude │  85.0 │ PASS  │ PASS │ PASS │     10.5 │    42 │
+│  2   │ codex  │  70.0 │ PASS  │ PASS │ FAIL │     15.2 │    98 │
+│  3   │ aider  │  55.0 │ FAIL  │ PASS │ PASS │      8.1 │    31 │
+└──────┴────────┴───────┴───────┴──────┴──────┴──────────┴───────┘
+```
+
+Results also saved as JSON in `.coderace/<task>-results.json`.
+
+## Supported Agents
+
+| Agent | CLI | Command |
+|-------|-----|---------|
+| Claude Code | `claude` | `claude --print --output-format json -p "<task>"` |
+| Codex | `codex` | `codex --quiet --full-auto -p "<task>"` |
+| Aider | `aider` | `aider --message "<task>" --yes --no-auto-commits` |
+| Gemini CLI | `gemini` | `gemini --non-interactive -p "<task>"` |
+
+Each agent must be installed and authenticated separately.
+
+## Parallel Mode
+
+Use `--parallel` (or `-p`) to run all agents simultaneously using git worktrees. Each agent gets its own isolated working directory, so they don't interfere with each other.
+
+```bash
+coderace run task.yaml --parallel
+```
+
+Sequential mode (default) runs agents one at a time on the same repo.
+
+## Requirements
+
+- Python 3.10+
+- Git
+- At least one coding agent CLI installed
+
+## License
+
+MIT

coderace-0.1.0/README.md

@@ -0,0 +1,112 @@
+# coderace
+
+Race coding agents against each other on real tasks in your repo.
+
+Define a task. Run it against Claude Code, Codex, and Aider. Get a scored comparison table.
+
+## Install
+
+```bash
+pip install coderace
+```
+
+## Quick Start
+
+```bash
+# Create a task template
+coderace init fix-auth-bug
+
+# Edit the task file (describe the bug, set test command)
+# Then race the agents:
+coderace run fix-auth-bug.yaml
+
+# Or race them in parallel (uses git worktrees):
+coderace run fix-auth-bug.yaml --parallel
+
+# View results from the last run
+coderace results fix-auth-bug.yaml
+```
+
+## Task Format
+
+```yaml
+name: fix-auth-bug
+description: |
+  The login endpoint returns 500 when email contains a plus sign.
+  Fix the email validation in auth/validators.py.
+repo: .
+test_command: pytest tests/test_auth.py -x
+lint_command: ruff check .
+timeout: 300
+agents:
+  - claude
+  - codex
+  - aider
+```
+
+## What It Does
+
+For each agent in the task:
+
+1. Creates a fresh git branch (`coderace/<agent>-<task>`)
+2. Invokes the agent CLI with the task description
+3. Runs your test command
+4. Runs your lint command (optional)
+5. Computes a composite score
+
+## Scoring
+
+| Metric | Weight | Description |
+|--------|--------|-------------|
+| Tests pass | 40% | Did the test command exit 0? |
+| Exit clean | 20% | Did the agent itself exit 0 without timeout? |
+| Lint clean | 15% | Did the lint command exit 0? |
+| Wall time | 15% | Faster is better (normalized across agents) |
+| Lines changed | 10% | Fewer is better (normalized across agents) |
+
+## Output
+
+Terminal table with Rich formatting:
+
+```
+┌──────┬────────┬───────┬───────┬──────┬──────┬──────────┬───────┐
+│ Rank │ Agent  │ Score │ Tests │ Exit │ Lint │ Time (s) │ Lines │
+├──────┼────────┼───────┼───────┼──────┼──────┼──────────┼───────┤
+│  1   │ claude │  85.0 │ PASS  │ PASS │ PASS │     10.5 │    42 │
+│  2   │ codex  │  70.0 │ PASS  │ PASS │ FAIL │     15.2 │    98 │
+│  3   │ aider  │  55.0 │ FAIL  │ PASS │ PASS │      8.1 │    31 │
+└──────┴────────┴───────┴───────┴──────┴──────┴──────────┴───────┘
+```
+
+Results also saved as JSON in `.coderace/<task>-results.json`.
+
+## Supported Agents
+
+| Agent | CLI | Command |
+|-------|-----|---------|
+| Claude Code | `claude` | `claude --print --output-format json -p "<task>"` |
+| Codex | `codex` | `codex --quiet --full-auto -p "<task>"` |
+| Aider | `aider` | `aider --message "<task>" --yes --no-auto-commits` |
+| Gemini CLI | `gemini` | `gemini --non-interactive -p "<task>"` |
+
+Each agent must be installed and authenticated separately.
+
+## Parallel Mode
+
+Use `--parallel` (or `-p`) to run all agents simultaneously using git worktrees. Each agent gets its own isolated working directory, so they don't interfere with each other.
+
+```bash
+coderace run task.yaml --parallel
+```
+
+Sequential mode (default) runs agents one at a time on the same repo.
+
+## Requirements
+
+- Python 3.10+
+- Git
+- At least one coding agent CLI installed
+
+## License
+
+MIT

coderace-0.1.0/coderace/__init__.py

@@ -0,0 +1,3 @@
+"""coderace - Race coding agents against each other on real tasks."""
+
+__version__ = "0.1.0"

coderace-0.1.0/coderace/adapters/__init__.py

@@ -0,0 +1,23 @@
+"""Agent adapters for coderace."""
+
+from coderace.adapters.aider import AiderAdapter
+from coderace.adapters.base import BaseAdapter
+from coderace.adapters.claude import ClaudeAdapter
+from coderace.adapters.codex import CodexAdapter
+from coderace.adapters.gemini import GeminiAdapter
+
+ADAPTERS: dict[str, type[BaseAdapter]] = {
+    "claude": ClaudeAdapter,
+    "codex": CodexAdapter,
+    "aider": AiderAdapter,
+    "gemini": GeminiAdapter,
+}
+
+__all__ = [
+    "ADAPTERS",
+    "BaseAdapter",
+    "ClaudeAdapter",
+    "CodexAdapter",
+    "AiderAdapter",
+    "GeminiAdapter",
+]

coderace-0.1.0/coderace/adapters/aider.py

@@ -0,0 +1,20 @@
+"""Aider adapter."""
+
+from __future__ import annotations
+
+from coderace.adapters.base import BaseAdapter
+
+
+class AiderAdapter(BaseAdapter):
+    """Adapter for Aider coding assistant."""
+
+    name = "aider"
+
+    def build_command(self, task_description: str) -> list[str]:
+        return [
+            "aider",
+            "--message",
+            task_description,
+            "--yes",
+            "--no-auto-commits",
+        ]

coderace-0.1.0/coderace/adapters/base.py

@@ -0,0 +1,60 @@
+"""Base adapter interface."""
+
+from __future__ import annotations
+
+import subprocess
+import time
+from abc import ABC, abstractmethod
+from pathlib import Path
+
+from coderace.types import AgentResult
+
+
+class BaseAdapter(ABC):
+    """Abstract base class for coding agent adapters."""
+
+    name: str = "base"
+
+    @abstractmethod
+    def build_command(self, task_description: str) -> list[str]:
+        """Build the CLI command to invoke this agent."""
+        ...
+
+    def run(self, task_description: str, workdir: Path, timeout: int) -> AgentResult:
+        """Run the agent on a task and capture results."""
+        cmd = self.build_command(task_description)
+        start = time.monotonic()
+        timed_out = False
+
+        try:
+            proc = subprocess.run(
+                cmd,
+                cwd=workdir,
+                capture_output=True,
+                text=True,
+                timeout=timeout,
+            )
+            exit_code = proc.returncode
+            stdout = proc.stdout
+            stderr = proc.stderr
+        except subprocess.TimeoutExpired as e:
+            timed_out = True
+            exit_code = -1
+            stdout = (e.stdout or b"").decode("utf-8", errors="replace") if e.stdout else ""
+            stderr = (e.stderr or b"").decode("utf-8", errors="replace") if e.stderr else ""
+        except FileNotFoundError:
+            timed_out = False
+            exit_code = 127
+            stdout = ""
+            stderr = f"Command not found: {cmd[0]}"
+
+        wall_time = time.monotonic() - start
+
+        return AgentResult(
+            agent=self.name,
+            exit_code=exit_code,
+            stdout=stdout,
+            stderr=stderr,
+            wall_time=wall_time,
+            timed_out=timed_out,
+        )

coderace-0.1.0/coderace/adapters/claude.py

@@ -0,0 +1,21 @@
+"""Claude Code adapter."""
+
+from __future__ import annotations
+
+from coderace.adapters.base import BaseAdapter
+
+
+class ClaudeAdapter(BaseAdapter):
+    """Adapter for Claude Code CLI."""
+
+    name = "claude"
+
+    def build_command(self, task_description: str) -> list[str]:
+        return [
+            "claude",
+            "--print",
+            "--output-format",
+            "json",
+            "-p",
+            task_description,
+        ]

coderace-0.1.0/coderace/adapters/codex.py

@@ -0,0 +1,20 @@
+"""Codex CLI adapter."""
+
+from __future__ import annotations
+
+from coderace.adapters.base import BaseAdapter
+
+
+class CodexAdapter(BaseAdapter):
+    """Adapter for OpenAI Codex CLI."""
+
+    name = "codex"
+
+    def build_command(self, task_description: str) -> list[str]:
+        return [
+            "codex",
+            "--quiet",
+            "--full-auto",
+            "-p",
+            task_description,
+        ]

coderace-0.1.0/coderace/adapters/gemini.py

@@ -0,0 +1,19 @@
+"""Gemini CLI adapter."""
+
+from __future__ import annotations
+
+from coderace.adapters.base import BaseAdapter
+
+
+class GeminiAdapter(BaseAdapter):
+    """Adapter for Google Gemini CLI."""
+
+    name = "gemini"
+
+    def build_command(self, task_description: str) -> list[str]:
+        return [
+            "gemini",
+            "--non-interactive",
+            "-p",
+            task_description,
+        ]