commitarena 0.1.1__tar.gz

commitarena-0.1.1/.gitignore

@@ -0,0 +1,15 @@
+__pycache__/
+*.py[cod]
+.venv/
+dist/
+build/
+*.egg-info/
+.pytest_cache/
+.ruff_cache/
+.env
+.env.*
+!.env.example
+runs/
+tasks/
+commitarena-demo.html
+.DS_Store

commitarena-0.1.1/CHANGELOG.md

@@ -0,0 +1,31 @@
+# Changelog
+
+## 0.1.1 - 2026-11-03
+
+Restore sqlparse license notice verbatim; fix leaderboard overflow.
+
+- `src/commitarena/demo/LICENSE-sqlparse` is again a byte-for-byte copy of the upstream BSD-3-Clause license,
+  copyright notice included, as the license requires.
+- Report: the leaderboard fits at 1280 px (the "Ended by" column wraps instead of pushing "Cutoff" out of the
+  card), and the page no longer overflows horizontally at 390 px; wide tables scroll inside their card.
+
+## 0.1.0 - 2026-11-03
+
+First release.
+
+- `commitarena mine`: finds commits that change source and tests (Python + pytest), validates in Docker that
+  the new tests fail without the fix and pass with it (FAIL_TO_PASS) without breaking others (PASS_TO_PASS).
+  The task prompt is built from the failing tests only; the commit message is never used.
+- `commitarena run`: one minimal agent loop with a single `bash` tool for every model, via one
+  OpenAI-compatible client (OpenRouter, OpenAI, Mistral, Ollama). Every command runs in a `--network none`
+  container with CPU, memory and time limits. Test files are restored before scoring. `--max-cost`
+  (default 1 USD) is checked before every call. Resumable run directories.
+- `commitarena report`: static HTML report (EN/ES, light/dark, mobile): leaderboard, task x model matrix,
+  pass rate before vs after each model's training cutoff, model diff next to the human fix.
+- `commitarena demo`: recorded real run of 11 sqlparse (BSD-3-Clause) fixes with Mistral Large 4, Ministral 8B
+  and Codestral, no keys or Docker needed.
+- Agent loop details: older tool outputs are elided to fit a 16K window, exact repeated commands get a
+  notice, truncated outputs say how to read specific lines, commands with a null byte are a failed step,
+  empty or cut-off replies get up to two nudges.
+- Python API (`run_task`, `load_tasks`, `ContextFile`, ...) with context-file injection for AGENTS.md /
+  CLAUDE.md experiments.

commitarena-0.1.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Diego Pérez
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

commitarena-0.1.1/PKG-INFO

@@ -0,0 +1,242 @@
+Metadata-Version: 2.5
+Name: commitarena
+Version: 0.1.1
+Summary: Your git history is the benchmark. Replay real bug fixes against any model, in a sandbox with no network.
+Project-URL: Homepage, https://diegoperezg7.github.io/commitarena/
+Project-URL: Source, https://github.com/diegoperezg7/commitarena
+Project-URL: Issues, https://github.com/diegoperezg7/commitarena/issues
+Author: Diego Pérez
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agents,benchmark,docker,evaluation,git,llm,swe-bench
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Requires-Dist: openai==3.42.1
+Requires-Dist: rich==15.0.0
+Requires-Dist: typer==0.27.3
+Description-Content-Type: text/markdown
+
+<p align="center"><img src="docs/assets/banner.svg" alt="commitarena" width="640"></p>
+
+<p align="center">
+<a href="https://github.com/diegoperezg7/commitarena/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/diegoperezg7/commitarena/actions/workflows/ci.yml/badge.svg"></a>
+<a href="LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-3b5bd4"></a>
+<a href="https://github.com/diegoperezg7/commitarena/releases"><img alt="Release" src="https://img.shields.io/github/v/release/diegoperezg7/commitarena?color=3b5bd4"></a>
+<a href="https://github.com/diegoperezg7/commitarena/stargazers"><img alt="Stars" src="https://img.shields.io/github/stars/diegoperezg7/commitarena?color=3b5bd4"></a>
+</p>
+
+<p align="center"><b>Your git history is the benchmark. Replay real bug fixes against any model, in a sandbox with no network.</b></p>
+
+<p align="center"><a href="README.es.md">Español</a> · <a href="https://diegoperezg7.github.io/commitarena/">Example report</a></p>
+
+<p align="center"><img src="docs/assets/demo.gif" alt="commitarena demo in a terminal" width="760"></p>
+
+commitarena mines commits from your repository where a bug fix came with a test, checks in Docker that the
+test fails without the fix and passes with it, and then asks each model to make that test pass. Every
+model gets the same small agent loop with one `bash` tool, and every command it runs executes in a
+container started with `--network none`. The report splits the pass rate into fixes committed before and
+after each model's training cutoff, so you can see whether a model solves the bug or remembers the patch.
+
+## Why
+
+There are good tools close to this. What I wanted and did not find in one place:
+
+| | What it does | Difference |
+|---|---|---|
+| [sjarmak/codeprobe](https://github.com/sjarmak/codeprobe) (10 stars) | Mines merged PRs, runs agent CLIs, ranks with cost and an HTML report. | Its README says each agent runs "inside a `--network=bridge` container (the agent needs the model API)". In commitarena the model is called from the host and the code the model writes never has a network. |
+| [PozziTiv4ik/Repo-Trials](https://github.com/PozziTiv4ik/Repo-Trials) (0 stars) | FAIL_TO_PASS tasks from your git history. | commitarena adds the same harness for every model, local ones included, and the before/after-cutoff split. |
+| [harbor-framework/harbor](https://github.com/harbor-framework/harbor) (6,755 stars) | A general framework for agent evaluations. | commitarena is one command for one question: which model fixes the bugs of this repo. |
+
+Star counts from the GitHub API on 2026-11-03.
+
+Three things commitarena does on purpose:
+
+1. **No network for model-written code.** The agent loop runs on your machine and only sends chat
+   requests. Each `bash` call goes through `docker exec` into a container with `--network none`,
+   `--cap-drop ALL`, `no-new-privileges`, 2 CPU, 2 GB, a PID limit and a per-command timeout.
+2. **One harness for every model.** Same system prompt, same single tool, same step and time limits,
+   whether it is a hosted model or a 4B model on Ollama. No vendor agent, no extra tools.
+3. **Solve or remember.** Public repos are in training data. The report shows pass rate on fixes dated
+   before each model's cutoff next to fixes dated after it. Cutoffs are configurable; by default the
+   model's release date is used, which is a safe upper bound.
+
+## Quickstart
+
+The demo needs no API keys and no Docker. It prints a recorded run and writes the HTML report.
+
+```bash
+uvx --from git+https://github.com/diegoperezg7/commitarena@v0.1.1 commitarena demo
+```
+
+Or from a clone:
+
+```bash
+git clone https://github.com/diegoperezg7/commitarena && cd commitarena && uv run commitarena demo
+```
+
+`uvx commitarena` is coming to PyPI.
+
+## Example output
+
+<!-- demo:start -->
+```text
+$ commitarena demo
+Recorded result (not a live run): 2026-11-03, commitarena 0.1.0, models mistral/codestral-latest,
+mistral/ministral-8b-latest, mistral/mistral-large-4, on Darwin arm64 (Apple M4, 16 GB RAM).
+Repo: https://github.com/andialbrecht/sqlparse (BSD-3-Clause)
+Leaderboard
+┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┓
+┃ model                       ┃  pass ┃ rate ┃ before cutoff ┃ after cutoff ┃ cost $ ┃    tokens ┃ avg s ┃ avg steps ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━┩
+│ mistral/mistral-large-4     │ 11/11 │ 100% │           8/8 │          3/3 │ 0.5401 │   672,848 │   118 │      11.1 │
+│ mistral/ministral-8b-latest │  1/11 │   9% │           0/5 │          1/6 │ 0.2445 │ 1,630,017 │    68 │      28.3 │
+│ mistral/codestral-latest    │  0/11 │   0% │           0/4 │          0/7 │ 0.6507 │ 2,038,536 │    75 │      27.8 │
+└─────────────────────────────┴───────┴──────┴───────────────┴──────────────┴────────┴───────────┴───────┴───────────┘
+```
+<!-- demo:end -->
+
+<p align="center"><a href="https://diegoperezg7.github.io/commitarena/"><img src="docs/assets/report-light.png" alt="report" width="760"></a></p>
+
+How to read it: with the same prompt, tool, limits and sandbox, Mistral Large 4 fixed all 11 bugs,
+including the 3 committed after its release (October 2026), so it was solving, not recalling. Ministral 8B
+fixed one, and that one is dated after its own cutoff. Codestral fixed none. Three tasks after a cutoff is
+a small sample: treat the split as a signal, not a verdict.
+
+**Why Codestral scores 0/11.** It is the model, not the scoring: it repeated an earlier command verbatim
+in 170 of its 304 tool calls, kept doing so after 163 "you already ran this exact command" notices from
+the harness, ran out of its 30 steps in 9 of 11 attempts, left the code untouched in 6, and never edited a
+test file. This is the start of the `PRIMARY KEY` task, whose human fix is one regex line in
+`sqlparse/keywords.py` (full trajectories are in [docs/demo-run/trajectories](docs/demo-run/trajectories)):
+
+```text
+12 grep -n 'PRIMARY' sqlparse/keywords.py
+13 sed -n '440,450p' sqlparse/keywords.py
+14 sed -n '440,450p' sqlparse/keywords.py      <- repeat notice
+15 grep -n 'KEY' sqlparse/keywords.py
+...
+23 grep -n 'PRIMARY' sqlparse/keywords.py      <- repeat notice
+24 sed -n '440,450p' sqlparse/keywords.py      <- repeat notice
+25 grep -n 'KEY' sqlparse/keywords.py          <- repeat notice
+```
+
+## Use it on your repo
+
+You need Docker running, plus Ollama or an API key.
+
+```bash
+# 1. Mine tasks: commits that change source and tests, validated in Docker
+commitarena mine https://github.com/you/yourrepo --max-tasks 10
+
+# 2. Run models (a local one and a hosted one)
+export MISTRAL_API_KEY=...
+commitarena run --tasks tasks/you-yourrepo/tasks.json \
+  --models ollama/qwen3:4b-16k,mistral/codestral-latest --max-cost 1
+
+# 3. The report is written to runs/<timestamp>/report.html; re-render it with other cutoffs
+commitarena report runs/<timestamp> --cutoff mistral/codestral-latest=2025-07-31 --open
+```
+
+## How it works
+
+```mermaid
+flowchart LR
+  subgraph host[Your machine]
+    G[git history] --> M[mine: commits touching src + tests]
+    M --> V{Docker, no network:<br/>new tests fail before,<br/>pass after?}
+    V -->|yes| T[tasks.json<br/>failing tests + test diff<br/>no commit message]
+    T --> L[agent loop<br/>one bash tool]
+    L <-->|chat API| P[(model provider<br/>or Ollama)]
+  end
+  L -->|docker exec| S[sandbox<br/>--network none<br/>2 CPU, 2 GB]
+  S --> R[copy out the workspace]
+  R --> X[keep non-test changes only,<br/>restore original tests]
+  X --> Y[fresh no-network container:<br/>full test suite]
+  Y --> Z[report.html<br/>leaderboard, matrix,<br/>cutoff split, diffs]
+```
+
+- **Mining.** Non-merge commits that change at least one test file and at least one `.py` source file
+  (small fixes only by default). For each one the test suite runs twice in a no-network container: the
+  parent commit plus the new test files, and the fix commit. A task is kept when some tests go from
+  failing to passing (FAIL_TO_PASS) and nothing that passed before breaks (PASS_TO_PASS). The test image
+  is built once per repo with your install command, which is the only step with network.
+- **The prompt** lists the failing tests, the test diff and the current failure output. The commit
+  message, the issue and the human fix are never shown. The workspace gets a fresh one-commit git history
+  so the real history cannot leak the fix.
+- **Scoring.** After the attempt the workspace is copied out. Only changes to non-test files are applied
+  onto a pristine copy, so edits to tests or `conftest.py` are discarded, and the full suite runs in a
+  fresh container. Pass means every FAIL_TO_PASS test passes and no PASS_TO_PASS test fails.
+- **Metrics.** Pass, cost (provider-reported for OpenRouter, otherwise tokens times list price; local
+  models cost 0), input and output tokens, wall time and steps (model calls).
+
+## Configuration
+
+| Flag | Default | |
+|---|---|---|
+| `run --models` | | Comma-separated `provider/model`. Providers: `openrouter`, `openai`, `mistral`, `ollama`. |
+| `run --max-cost` | `1.0` | USD. Checked before every model call; the run stops with partial results. |
+| `run --max-steps` / `--max-time` | `30` / `1200` | Model calls and seconds per attempt. |
+| `run --cutoff MODEL=DATE` | release date | Training cutoff used for the before/after split (repeatable). |
+| `run --price MODEL=IN,OUT` | built-in | USD per million tokens for models without a known price. |
+| `run --context-file F` | | Inject an AGENTS.md / CLAUDE.md: `--context-mode both\|repo\|prompt`. |
+| `run --strip-context-files` | off | Remove the repo's own AGENTS.md / CLAUDE.md before the attempt. |
+| `run --cpus` / `--memory` / `--command-timeout` | `2` / `2g` / `120` | Sandbox limits. |
+| `mine --since` / `--until` / `--scan` / `--max-tasks` | | Which commits to consider. |
+| `mine --python` / `--setup` | `3.12` / `pip install -e .` | How the test image is built. |
+
+Environment variables: `OPENROUTER_API_KEY`, `OPENAI_API_KEY`, `MISTRAL_API_KEY`, `OLLAMA_HOST`,
+`COMMITARENA_CACHE` (clones, default `~/.cache/commitarena`). See [.env.example](.env.example). There is
+no telemetry.
+
+**Ollama context window.** Ollama's OpenAI-compatible endpoint ignores `num_ctx`, and its default
+window can be 4096 tokens, which silently drops the task text and gives worse results. commitarena warns
+about it. Use a variant with `num_ctx 16384` (the one used in development is `qwen3:4b-16k`); the agent
+loop elides older tool outputs to stay inside 16K.
+
+```bash
+printf 'FROM qwen3:4b\nPARAMETER num_ctx 16384\n' > Modelfile && ollama create qwen3:4b-16k -f Modelfile
+commitarena run --tasks tasks.json --models ollama/qwen3:4b-16k
+```
+
+Local models go through exactly the same code path (qwen3:4b-16k was run during development), but it is not
+in the recorded demo. On this machine it spent every 4096-token reply reasoning and never called the
+tool, and its thinking could not be turned off through the OpenAI endpoint (`think: false`,
+`reasoning_effort: "none"` and `/no_think` were all tried). Raising the reply limit for one model would
+break the same-harness rule, so it was left out rather than shown with a misleading 0.
+
+## Python API
+
+```python
+from commitarena import load_tasks, run_task, Budget, ContextFile
+
+task = load_tasks("tasks.json")[0]
+res = run_task(task, "ollama/qwen3:4b-16k", budget=Budget(max_cost=1.0),
+               context=ContextFile(open("AGENTS.md").read(), in_repo=True, in_prompt=True))
+print(res.passed, res.cost_usd, res.tokens_in, res.steps, res.wall_time_s)
+```
+
+## Demo data
+
+The recorded demo uses 11 tasks mined from [sqlparse](https://github.com/andialbrecht/sqlparse)
+(BSD-3-Clause, Copyright (c) 2016, Andi Albrecht). The bundled files contain the failing tests, the test
+diffs and the upstream fixes as diffs, each linked to its commit. The upstream license is bundled verbatim in
+[src/commitarena/demo/LICENSE-sqlparse](src/commitarena/demo/LICENSE-sqlparse).
+
+## Roadmap
+
+- Other languages and test runners
+- Claude Code and Codex as contestants
+- GitHub issue text as an optional prompt
+- pass@k and repeated attempts
+- Parallel runs
+- Export to Harbor
+
+## License
+
+MIT. See [LICENSE](LICENSE).

commitarena-0.1.1/README.es.md

@@ -0,0 +1,219 @@
+<p align="center"><img src="docs/assets/banner.svg" alt="commitarena" width="640"></p>
+
+<p align="center">
+<a href="https://github.com/diegoperezg7/commitarena/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/diegoperezg7/commitarena/actions/workflows/ci.yml/badge.svg"></a>
+<a href="LICENSE"><img alt="Licencia: MIT" src="https://img.shields.io/badge/license-MIT-3b5bd4"></a>
+<a href="https://github.com/diegoperezg7/commitarena/releases"><img alt="Release" src="https://img.shields.io/github/v/release/diegoperezg7/commitarena?color=3b5bd4"></a>
+<a href="https://github.com/diegoperezg7/commitarena/stargazers"><img alt="Estrellas" src="https://img.shields.io/github/stars/diegoperezg7/commitarena?color=3b5bd4"></a>
+</p>
+
+<p align="center"><b>Tu historial de git es el benchmark. Reproduce fixes reales contra cualquier modelo, en un sandbox sin red.</b></p>
+
+<p align="center"><a href="README.md">English</a> · <a href="https://diegoperezg7.github.io/commitarena/">Informe de ejemplo</a></p>
+
+<p align="center"><img src="docs/assets/demo.gif" alt="demo de commitarena en una terminal" width="760"></p>
+
+commitarena busca en tu repositorio los commits en los que un fix vino con un test, comprueba en Docker
+que el test falla sin el fix y pasa con él, y le pide a cada modelo que haga pasar ese test. Todos los
+modelos usan el mismo bucle de agente pequeño con una sola herramienta `bash`, y cada comando que ejecutan
+corre en un contenedor arrancado con `--network none`. El informe separa el acierto en fixes anteriores y
+posteriores a la fecha de corte de entrenamiento de cada modelo, para ver si el modelo resuelve el bug o
+recuerda el parche.
+
+## Por qué
+
+Hay herramientas buenas cerca de esto. Lo que yo quería y no encontré junto:
+
+| | Qué hace | Diferencia |
+|---|---|---|
+| [sjarmak/codeprobe](https://github.com/sjarmak/codeprobe) (10 estrellas) | Mina PRs fusionadas, corre CLIs de agentes y saca ranking con coste e informe HTML. | Su README dice que cada agente corre "inside a `--network=bridge` container (the agent needs the model API)". En commitarena el modelo se llama desde el host y el código que escribe el modelo nunca tiene red. |
+| [PozziTiv4ik/Repo-Trials](https://github.com/PozziTiv4ik/Repo-Trials) (0 estrellas) | Tareas FAIL_TO_PASS a partir de tu historial de git. | commitarena añade el mismo arnés para todos los modelos, también los locales, y el corte antes/después de la fecha de entrenamiento. |
+| [harbor-framework/harbor](https://github.com/harbor-framework/harbor) (6.755 estrellas) | Un framework general de evaluación de agentes. | commitarena es un comando para una pregunta: qué modelo arregla los bugs de este repo. |
+
+Estrellas sacadas de la API de GitHub el 3 nov 2026.
+
+Tres cosas que commitarena hace a propósito:
+
+1. **Sin red para el código del modelo.** El bucle del agente corre en tu máquina y solo envía peticiones
+   de chat. Cada llamada a `bash` va por `docker exec` a un contenedor con `--network none`,
+   `--cap-drop ALL`, `no-new-privileges`, 2 CPU, 2 GB, límite de procesos y timeout por comando.
+2. **Un arnés para todos.** Mismo prompt de sistema, misma herramienta, mismos límites de pasos y tiempo,
+   sea un modelo en la nube o uno de 4B en Ollama. Sin agentes del fabricante ni herramientas extra.
+3. **¿Lo resuelve o lo recuerda?** Los repos públicos están en los datos de entrenamiento. El informe
+   pone el acierto en fixes anteriores a la fecha de corte de cada modelo junto al de los posteriores. Las
+   fechas se configuran; por defecto se usa la fecha de lanzamiento del modelo, que es una cota segura.
+
+## Inicio rápido
+
+La demo no necesita claves ni Docker. Muestra una ejecución grabada y escribe el informe HTML.
+
+```bash
+uvx --from git+https://github.com/diegoperezg7/commitarena@v0.1.1 commitarena demo
+```
+
+O desde un clon:
+
+```bash
+git clone https://github.com/diegoperezg7/commitarena && cd commitarena && uv run commitarena demo
+```
+
+`uvx commitarena` llegará pronto a PyPI.
+
+## Ejemplo de salida
+
+<!-- demo:start -->
+```text
+$ commitarena demo
+Recorded result (not a live run): 2026-11-03, commitarena 0.1.0, models mistral/codestral-latest,
+mistral/ministral-8b-latest, mistral/mistral-large-4, on Darwin arm64 (Apple M4, 16 GB RAM).
+Repo: https://github.com/andialbrecht/sqlparse (BSD-3-Clause)
+Leaderboard
+┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┓
+┃ model                       ┃  pass ┃ rate ┃ before cutoff ┃ after cutoff ┃ cost $ ┃    tokens ┃ avg s ┃ avg steps ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━┩
+│ mistral/mistral-large-4     │ 11/11 │ 100% │           8/8 │          3/3 │ 0.5401 │   672,848 │   118 │      11.1 │
+│ mistral/ministral-8b-latest │  1/11 │   9% │           0/5 │          1/6 │ 0.2445 │ 1,630,017 │    68 │      28.3 │
+│ mistral/codestral-latest    │  0/11 │   0% │           0/4 │          0/7 │ 0.6507 │ 2,038,536 │    75 │      27.8 │
+└─────────────────────────────┴───────┴──────┴───────────────┴──────────────┴────────┴───────────┴───────┴───────────┘
+```
+<!-- demo:end -->
+
+<p align="center"><a href="https://diegoperezg7.github.io/commitarena/"><img src="docs/assets/report-dark.png" alt="report" width="760"></a></p>
+
+Cómo leerlo: con el mismo prompt, herramienta, límites y sandbox, Mistral Large 4 arregló los 11 bugs,
+incluidos los 3 posteriores a su lanzamiento (octubre de 2026), así que los resolvió, no los recordó.
+Ministral 8B arregló uno, y ese es posterior a su propio corte. Codestral no arregló ninguno. Tres tareas
+después del corte es una muestra pequeña: es una señal, no un veredicto.
+
+**Por qué Codestral saca 0/11.** Es el modelo, no la puntuación: repitió literalmente un comando anterior
+en 170 de sus 304 llamadas, siguió haciéndolo tras 163 avisos del arnés ("you already ran this exact
+command"), agotó sus 30 pasos en 9 de 11 intentos, no cambió el código en 6 y no tocó ningún test. Así
+empieza la tarea de `PRIMARY KEY`, cuyo fix humano es una línea de regex en `sqlparse/keywords.py`
+(trayectorias completas en [docs/demo-run/trajectories](docs/demo-run/trajectories)):
+
+```text
+12 grep -n 'PRIMARY' sqlparse/keywords.py
+13 sed -n '440,450p' sqlparse/keywords.py
+14 sed -n '440,450p' sqlparse/keywords.py      <- aviso de repetición
+15 grep -n 'KEY' sqlparse/keywords.py
+...
+23 grep -n 'PRIMARY' sqlparse/keywords.py      <- aviso de repetición
+24 sed -n '440,450p' sqlparse/keywords.py      <- aviso de repetición
+25 grep -n 'KEY' sqlparse/keywords.py          <- aviso de repetición
+```
+
+## Úsalo en tu repo
+
+Necesitas Docker en marcha y Ollama o una API key.
+
+```bash
+# 1. Minar tareas: commits que cambian código y tests, validados en Docker
+commitarena mine https://github.com/tu/turepo --max-tasks 10
+
+# 2. Ejecutar modelos (uno local y uno en la nube)
+export MISTRAL_API_KEY=...
+commitarena run --tasks tasks/tu-turepo/tasks.json \
+  --models ollama/qwen3:4b-16k,mistral/codestral-latest --max-cost 1
+
+# 3. El informe queda en runs/<fecha>/report.html; se puede regenerar con otras fechas de corte
+commitarena report runs/<fecha> --cutoff mistral/codestral-latest=2025-07-31 --open
+```
+
+## Cómo funciona
+
+```mermaid
+flowchart LR
+  subgraph host[Tu máquina]
+    G[historial de git] --> M[mine: commits que tocan src + tests]
+    M --> V{Docker sin red:<br/>¿los tests nuevos fallan antes<br/>y pasan después?}
+    V -->|sí| T[tasks.json<br/>tests que fallan + diff de tests<br/>sin mensaje de commit]
+    T --> L[bucle de agente<br/>una herramienta bash]
+    L <-->|API de chat| P[(proveedor del modelo<br/>u Ollama)]
+  end
+  L -->|docker exec| S[sandbox<br/>--network none<br/>2 CPU, 2 GB]
+  S --> R[se copia el workspace]
+  R --> X[solo cambios fuera de tests,<br/>tests originales restaurados]
+  X --> Y[contenedor nuevo sin red:<br/>suite completa]
+  Y --> Z[report.html<br/>ranking, matriz,<br/>corte por fecha, diffs]
+```
+
+- **Minado.** Commits sin merge que cambian al menos un fichero de test y al menos un `.py` de código
+  (por defecto solo fixes pequeños). Para cada uno la suite corre dos veces en un contenedor sin red: el
+  commit padre con los tests nuevos, y el commit del fix. La tarea se queda si algún test pasa de fallar a
+  pasar (FAIL_TO_PASS) y no se rompe nada de lo que pasaba (PASS_TO_PASS). La imagen de tests se construye
+  una vez por repo con tu comando de instalación, que es el único paso con red.
+- **El prompt** lista los tests que fallan, el diff de los tests y la salida del fallo. Nunca se enseña
+  el mensaje del commit, la issue ni el fix humano. El workspace lleva un historial de git nuevo de un solo
+  commit, para que el historial real no filtre el fix.
+- **Puntuación.** Al acabar se copia el workspace. Solo se aplican los cambios en ficheros que no son de
+  test sobre una copia limpia, así que lo que toque en tests o `conftest.py` se descarta, y la suite
+  completa corre en un contenedor nuevo. Acierto = pasan todos los FAIL_TO_PASS y no falla ningún
+  PASS_TO_PASS.
+- **Métricas.** Acierto, coste (el que informa OpenRouter, si no tokens por precio de lista; los modelos
+  locales cuestan 0), tokens de entrada y salida, tiempo y pasos (llamadas al modelo).
+
+## Configuración
+
+| Opción | Por defecto | |
+|---|---|---|
+| `run --models` | | `proveedor/modelo` separados por comas. Proveedores: `openrouter`, `openai`, `mistral`, `ollama`. |
+| `run --max-cost` | `1.0` | USD. Se comprueba antes de cada llamada; la ejecución para con resultados parciales. |
+| `run --max-steps` / `--max-time` | `30` / `1200` | Llamadas al modelo y segundos por intento. |
+| `run --cutoff MODELO=FECHA` | fecha de lanzamiento | Fecha de corte para el reparto antes/después (repetible). |
+| `run --price MODELO=IN,OUT` | integrado | USD por millón de tokens para modelos sin precio conocido. |
+| `run --context-file F` | | Inyecta un AGENTS.md / CLAUDE.md: `--context-mode both\|repo\|prompt`. |
+| `run --strip-context-files` | no | Quita el AGENTS.md / CLAUDE.md propio del repo antes del intento. |
+| `run --cpus` / `--memory` / `--command-timeout` | `2` / `2g` / `120` | Límites del sandbox. |
+| `mine --since` / `--until` / `--scan` / `--max-tasks` | | Qué commits se miran. |
+| `mine --python` / `--setup` | `3.12` / `pip install -e .` | Cómo se construye la imagen de tests. |
+
+Variables de entorno: `OPENROUTER_API_KEY`, `OPENAI_API_KEY`, `MISTRAL_API_KEY`, `OLLAMA_HOST`,
+`COMMITARENA_CACHE` (clones, por defecto `~/.cache/commitarena`). Ver [.env.example](.env.example). No hay
+telemetría.
+
+**Ventana de contexto en Ollama.** El endpoint compatible con OpenAI de Ollama ignora `num_ctx`, y la
+ventana por defecto puede ser de 4096 tokens, lo que tira el texto de la tarea sin avisar y da peores
+resultados. commitarena avisa. Usa una variante con `num_ctx 16384` (la usada en el desarrollo es
+`qwen3:4b-16k`); el bucle del agente elide las salidas más antiguas para caber en 16K.
+
+```bash
+printf 'FROM qwen3:4b\nPARAMETER num_ctx 16384\n' > Modelfile && ollama create qwen3:4b-16k -f Modelfile
+commitarena run --tasks tasks.json --models ollama/qwen3:4b-16k
+```
+
+Los modelos locales pasan exactamente por el mismo código (qwen3:4b-16k se ejecutó durante el desarrollo),
+pero no está en la demo grabada. En esta máquina gastaba los 4096 tokens de cada respuesta
+razonando y nunca llamaba a la herramienta, y no hubo forma de desactivar el razonamiento por el endpoint
+de OpenAI (se probaron `think: false`, `reasoning_effort: "none"` y `/no_think`). Subir el límite solo para
+un modelo rompería la regla de arnés único, así que se dejó fuera en vez de mostrar un 0 engañoso.
+
+## API de Python
+
+```python
+from commitarena import load_tasks, run_task, Budget, ContextFile
+
+task = load_tasks("tasks.json")[0]
+res = run_task(task, "ollama/qwen3:4b-16k", budget=Budget(max_cost=1.0),
+               context=ContextFile(open("AGENTS.md").read(), in_repo=True, in_prompt=True))
+print(res.passed, res.cost_usd, res.tokens_in, res.steps, res.wall_time_s)
+```
+
+## Datos de la demo
+
+La demo grabada usa 11 tareas minadas de [sqlparse](https://github.com/andialbrecht/sqlparse)
+(BSD-3-Clause, Copyright (c) 2016, Andi Albrecht). Los ficheros incluidos contienen los tests que fallan,
+los diffs de los tests y los fixes originales como diffs, cada uno enlazado a su commit. La licencia original va incluida literal en
+[src/commitarena/demo/LICENSE-sqlparse](src/commitarena/demo/LICENSE-sqlparse).
+
+## Hoja de ruta
+
+- Otros lenguajes y runners de tests
+- Claude Code y Codex como concursantes
+- Texto de la issue de GitHub como prompt opcional
+- pass@k e intentos repetidos
+- Ejecución en paralelo
+- Exportar a Harbor
+
+## Licencia
+
+MIT. Ver [LICENSE](LICENSE).