bugbout 0.1.0__tar.gz

bugbout-0.1.0/.gitignore

@@ -0,0 +1,27 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+build/
+dist/
+.venv/
+venv/
+
+# Tooling
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+coverage.xml
+htmlcov/
+
+# bugbout's own results
+.bugbout/
+
+# Secrets
+.env
+.env.*
+*.pem
+
+# OS
+.DS_Store

bugbout-0.1.0/CHANGELOG.md

@@ -0,0 +1,15 @@
+# Changelog
+
+## 0.1.0 - 2026-10-20
+
+First release.
+
+- Mine tasks from a project's own fixed bugs, with ground truth from the maintainers'
+  tests.
+- Race Codex, Claude Code, Gemini CLI, mini-swe-agent, or any agent with a command
+  line, in sandboxed, history-free checkouts.
+- Cross-examine every patch with every agent's tests, and pick one per bug under a
+  judge policy.
+- Report in the terminal, Markdown and JSON.
+- `bugbout init`, `doctor`, `tasks`, `run` and `report`; and the same pipeline from
+  Python through `Bout`.

bugbout-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Keerthirajan58
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

bugbout-0.1.0/PKG-INFO

@@ -0,0 +1,120 @@
+Metadata-Version: 2.5
+Name: bugbout
+Version: 0.1.0
+Summary: Find out which coding agent actually fixes bugs in your codebase.
+Project-URL: Homepage, https://github.com/Keerthirajan58/bugbout
+Project-URL: Issues, https://github.com/Keerthirajan58/bugbout/issues
+Author-email: Keerthirajan58 <keerthirajan19122001@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: benchmark,coding-agents,evaluation,llm,swe-bench,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: MacOS
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Requires-Dist: pydantic>=2.10
+Requires-Dist: rich>=13.9
+Requires-Dist: typer>=0.15
+Provides-Extra: dev
+Requires-Dist: mypy==2.4.0; extra == 'dev'
+Requires-Dist: pytest-cov>=6.0; extra == 'dev'
+Requires-Dist: pytest>=8.3; extra == 'dev'
+Requires-Dist: ruff==0.17.1; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# bugbout
+
+**Which coding agent actually fixes bugs in *your* codebase?**
+
+Public benchmarks tell you how coding agents do on someone else's code. bugbout tells
+you how they do on yours. It replays real bugs from your project's own history, lets
+agents race to fix them, and scores every patch with the tests your maintainers wrote
+alongside the real fix - tests no agent ever gets to see.
+
+## Why
+
+- **Your code, not a leaderboard's.** An agent that tops a benchmark can still stumble
+  on your conventions, your framework and your test suite. Measure it where you will
+  use it.
+- **Scored by your own tests.** Every task is a fix your team already shipped with a
+  new test, and that test decides whether an agent's patch really fixes the bug.
+- **Bring any agent.** Codex, Claude Code, Gemini CLI and mini-swe-agent work out of
+  the box, and any other agent with a command line takes a few lines of config - so
+  you can also compare models, including local ones.
+- **Cross-examination.** Agents' tests are run against each other's patches, so you
+  also learn whether an "agent jury" could pick, or gate, a patch on its own before you
+  trust one in your workflow.
+- **Safe to run.** Agents work in history-free checkouts, inside an OS sandbox, with
+  web access switched off.
+
+## Quickstart
+
+You need Python 3.12+, git, [uv](https://docs.astral.sh/uv/), and at least one coding
+agent CLI installed and logged in.
+
+```bash
+pip install git+https://github.com/Keerthirajan58/bugbout
+
+cd your-project
+bugbout init      # writes bugbout.toml from your layout and the agents you have
+bugbout doctor    # checks everything before any agent spends quota
+bugbout run       # mines tasks, races the agents, judges and scores
+bugbout report --format markdown
+```
+
+A bout is resumable: stop it at any point and run it again to carry on. Add an agent
+later and only that agent runs.
+
+## How it works
+
+1. **Mine.** Fixes in your git history that name an issue and add a test become tasks.
+2. **Ground truth.** The maintainers' tests run before and after their fix; the tests
+   that flip from failing to passing define what "fixed" means.
+3. **Race.** Each agent gets the issue and a checkout of the code before the fix - no
+   git history, none of the fix's tests - and works inside a sandbox.
+4. **Cross-examine.** Every test the agents wrote runs against every patch.
+5. **Judge and score.** A policy picks one patch per bug from the executed evidence,
+   then the hidden tests score every patch and every pick.
+
+More in [How it works](docs/how-it-works.md).
+
+## Configuration
+
+`bugbout init` writes a commented `bugbout.toml`. A minimal one looks like this:
+
+```toml
+[project]
+source = "src/mypkg"
+tests = "tests"
+
+[[contestants]]
+agent = "codex"
+
+[[contestants]]
+agent = "claude-code"
+model = "sonnet"
+```
+
+Every setting is described in [Configuration](docs/configuration.md); the built-in
+agents, sandboxing and adding your own agent are covered in [Agents](docs/agents.md).
+
+## From Python
+
+```python
+from bugbout import Bout, render_markdown
+
+report = Bout.load(".").run()
+print(render_markdown(report))
+```
+
+## License
+
+[MIT](LICENSE)

bugbout-0.1.0/README.md

@@ -0,0 +1,88 @@
+# bugbout
+
+**Which coding agent actually fixes bugs in *your* codebase?**
+
+Public benchmarks tell you how coding agents do on someone else's code. bugbout tells
+you how they do on yours. It replays real bugs from your project's own history, lets
+agents race to fix them, and scores every patch with the tests your maintainers wrote
+alongside the real fix - tests no agent ever gets to see.
+
+## Why
+
+- **Your code, not a leaderboard's.** An agent that tops a benchmark can still stumble
+  on your conventions, your framework and your test suite. Measure it where you will
+  use it.
+- **Scored by your own tests.** Every task is a fix your team already shipped with a
+  new test, and that test decides whether an agent's patch really fixes the bug.
+- **Bring any agent.** Codex, Claude Code, Gemini CLI and mini-swe-agent work out of
+  the box, and any other agent with a command line takes a few lines of config - so
+  you can also compare models, including local ones.
+- **Cross-examination.** Agents' tests are run against each other's patches, so you
+  also learn whether an "agent jury" could pick, or gate, a patch on its own before you
+  trust one in your workflow.
+- **Safe to run.** Agents work in history-free checkouts, inside an OS sandbox, with
+  web access switched off.
+
+## Quickstart
+
+You need Python 3.12+, git, [uv](https://docs.astral.sh/uv/), and at least one coding
+agent CLI installed and logged in.
+
+```bash
+pip install git+https://github.com/Keerthirajan58/bugbout
+
+cd your-project
+bugbout init      # writes bugbout.toml from your layout and the agents you have
+bugbout doctor    # checks everything before any agent spends quota
+bugbout run       # mines tasks, races the agents, judges and scores
+bugbout report --format markdown
+```
+
+A bout is resumable: stop it at any point and run it again to carry on. Add an agent
+later and only that agent runs.
+
+## How it works
+
+1. **Mine.** Fixes in your git history that name an issue and add a test become tasks.
+2. **Ground truth.** The maintainers' tests run before and after their fix; the tests
+   that flip from failing to passing define what "fixed" means.
+3. **Race.** Each agent gets the issue and a checkout of the code before the fix - no
+   git history, none of the fix's tests - and works inside a sandbox.
+4. **Cross-examine.** Every test the agents wrote runs against every patch.
+5. **Judge and score.** A policy picks one patch per bug from the executed evidence,
+   then the hidden tests score every patch and every pick.
+
+More in [How it works](docs/how-it-works.md).
+
+## Configuration
+
+`bugbout init` writes a commented `bugbout.toml`. A minimal one looks like this:
+
+```toml
+[project]
+source = "src/mypkg"
+tests = "tests"
+
+[[contestants]]
+agent = "codex"
+
+[[contestants]]
+agent = "claude-code"
+model = "sonnet"
+```
+
+Every setting is described in [Configuration](docs/configuration.md); the built-in
+agents, sandboxing and adding your own agent are covered in [Agents](docs/agents.md).
+
+## From Python
+
+```python
+from bugbout import Bout, render_markdown
+
+report = Bout.load(".").run()
+print(render_markdown(report))
+```
+
+## License
+
+[MIT](LICENSE)

bugbout-0.1.0/docs/agents.md

@@ -0,0 +1,162 @@
+# Agents
+
+bugbout reimplements no agent. Each adapter runs the same command line a developer
+would, headlessly, and judges the agent only by what it leaves on disk. Two rules hold
+for every adapter:
+
+- **Nothing personal leaks in.** Your own agent configuration - plugins, memories, MCP
+  servers, house rules - is left out wherever the agent allows it, so the comparison is
+  between agents, not set-ups.
+- **Nothing from the future leaks in.** The tasks are real bugs with public fixes, so web
+  tools are off, commands run in a sandbox, and the checkout has no git history.
+
+## Built-in agents
+
+### `codex` - OpenAI's Codex CLI
+
+Runs `codex exec` with your user config and rules ignored, an ephemeral session, web
+search and every feature that reaches beyond the checkout switched off. Shell commands
+run in Codex's own `workspace-write` sandbox: no network, writes only in the checkout.
+
+| Option | Meaning |
+|---|---|
+| `reasoning-effort` | Passed through as `model_reasoning_effort`, e.g. `"high"`. |
+| `local-provider` | `ollama` or `lmstudio`, to run an open model locally. |
+
+### `claude-code` - Anthropic's Claude Code
+
+Runs `claude --print` in safe mode, which leaves out CLAUDE.md, skills, plugins, hooks
+and MCP servers. Edits are accepted only inside the checkout, shell commands run in
+Claude Code's own sandbox, web tools are disallowed, and anything that would need a
+person's approval is refused. `model` takes a full name or an alias such as `sonnet`.
+
+| Option | Meaning |
+|---|---|
+| `effort` | Claude Code's effort level, e.g. `"high"`. |
+
+### `gemini-cli` - Google's Gemini CLI
+
+Runs `gemini --prompt` in auto-approve mode, with web tools excluded through workspace
+settings. The whole process runs inside bugbout's sandbox: it can reach the Gemini API,
+but write only in the checkout, its logs and `~/.gemini`.
+
+### `mini-swe-agent` - any LiteLLM model
+
+[mini-swe-agent](https://mini-swe-agent.com) is the SWE-bench team's minimal agent, and
+the easiest way to race models rather than products - hosted or local. `model` is a
+LiteLLM name such as `ollama/qwen3-coder:30b` and is required. Every shell command it
+runs goes through bugbout's sandbox, with no network.
+
+| Option | Meaning |
+|---|---|
+| `text-actions` | `true` for models whose server cannot parse tool calls reliably. |
+| `step-limit` | The most steps per task; 75 by default. |
+| `model-kwargs` | Extra model arguments, e.g. `{ temperature = "0" }`. |
+
+For local models served by Ollama, give the server enough context for an agent, e.g.
+`OLLAMA_CONTEXT_LENGTH=65536 ollama serve`.
+
+### `command` - anything else with a command line
+
+Any agent that can run unattended can join through configuration alone:
+
+```toml
+[[contestants]]
+agent = "command"
+name = "aider"
+model = "gpt-5"
+command = ["aider", "--yes-always", "--no-git", "--model", "{model}", "--message-file", "{prompt_file}"]
+state-dirs = ["~/.aider"]
+```
+
+In `command`, these placeholders are filled in for each run:
+
+| Placeholder | Becomes |
+|---|---|
+| `{prompt}` | The task prompt itself. |
+| `{prompt_file}` | A file holding the prompt. |
+| `{workdir}` | The checkout the agent works in (also its working directory). |
+| `{logdir}` | A directory for the agent's own logs. |
+| `{model}` | The configured model. |
+
+Without `{prompt}` or `{prompt_file}`, the prompt arrives on standard input.
+
+| Option | Meaning |
+|---|---|
+| `command` | The command line, as a list. Required. |
+| `name` | Required, since there is no agent type to name it after. |
+| `env` | Extra environment variables for the agent. |
+| `state-dirs` | Directories outside the checkout the agent must write to, such as its own config. |
+| `sandboxed` | `false` to run without bugbout's sandbox - only for agents that sandbox themselves. |
+
+## Sandboxing
+
+On macOS, bugbout uses seatbelt (`sandbox-exec`); on Linux, bubblewrap (`bwrap`, e.g.
+`apt install bubblewrap`). Either way a sandboxed process can read anything, but write
+only in the directories it was given and the system's temporary directories.
+
+- An agent's **shell commands** get no network at all.
+- An agent's **whole process**, where bugbout confines it (Gemini CLI, `command`
+  agents), keeps the network so it can reach its model.
+- The **tests** bugbout runs - including tests an agent wrote - get no network and may
+  write only in their own checkout.
+
+`sandbox = "none"` runs everything unconfined. `bugbout doctor` reports which sandbox
+is in use, or why none can start. On Ubuntu 24.04 and later, AppArmor restricts the
+unprivileged user namespaces bubblewrap needs; allow them for `bwrap` with an AppArmor
+profile, or system-wide with `sudo sysctl kernel.apparmor_restrict_unprivileged_userns=0`.
+
+## Quotas and lanes
+
+Agents are rate-limited by their plans, not by your machine, so a bout queues runs per
+**quota**: runs sharing a quota take turns, and different quotas run in parallel. By
+default an agent's quota is its type (all `codex` agents share a ChatGPT plan, say),
+and one run per quota runs at a time. Raise that with `[run] quotas`, or move an agent
+to its own lane with `quota = "..."`.
+
+When an agent stops on a usage limit without producing anything, the attempt is marked
+`quota`. `bugbout run --retry-failed` runs such attempts again once the limit resets;
+finished attempts are never re-run.
+
+## Writing an adapter in Python
+
+An adapter says how to start the agent; bugbout does the rest - checkouts, timeouts,
+logs, patches and scoring.
+
+```python
+from pathlib import Path
+
+from bugbout import CodingAgent, Invocation, Session
+
+
+class MyAgent(CodingAgent):
+    harness = "my-agent"  # the default name prefix
+    executable = "my-agent"  # what `bugbout doctor` looks for
+
+    def invocation(self, session: Session) -> Invocation:
+        argv = [self.command(), "--model", self.model, "--yes"]
+        return Invocation(argv, stdin=session.prompt)
+
+    def parse(self, logdir: Path) -> tuple[str, dict[str, int]]:
+        # Optional: the agent's closing summary and token usage, from its logs.
+        return (logdir / "stdout.log").read_text()[-2000:], {}
+```
+
+`Session` carries the prompt, the checkout (`workdir`), a log directory and the
+sandbox; if the agent does not confine itself, wrap its command with
+`session.sandbox.wrap(argv, Confinement(...))` (from `bugbout.sandbox`). Pass agents to a bout directly:
+
+```python
+from bugbout import Bout, Project
+
+report = Bout(Project.load("."), contestants=[MyAgent("my-model")]).run()
+```
+
+To make an adapter available to `bugbout.toml`, publish it with an entry point:
+
+```toml
+[project.entry-points."bugbout.agents"]
+my-agent = "my_package.agent:MyAgent"
+```
+
+and use it as `agent = "my-agent"`.

bugbout-0.1.0/docs/configuration.md

@@ -0,0 +1,113 @@
+# Configuration
+
+A bout is configured by `bugbout.toml`, usually at the root of your repository.
+`bugbout init` writes one for you; every key below is optional except
+`project.source`. Keys use dashes, as in `pyproject.toml`.
+
+## `[project]` - the code under test
+
+| Key | Default | Meaning |
+|---|---|---|
+| `source` | *required* | The package directory, relative to the repository root, e.g. `src/mypkg`. |
+| `tests` | `"tests"` | The test directory. |
+| `repository` | `"."` | The repository: a path relative to `bugbout.toml`, or a git URL to clone. |
+| `github` | from `origin` | `owner/name` to read issues from, when `origin` is not on GitHub. |
+| `python` | `"3.12"` | The Python version of the test environment. |
+| `test-dependencies` | `["pytest"]` | What the tests need besides the project's own dependencies. |
+| `pytest-args` | `[]` | Extra arguments for every test run. |
+| `env` | `{}` | Extra environment variables for the tests, such as a database URL. |
+| `install-project` | `true` | Install the project once to pull in its runtime dependencies. |
+| `interpreter` | none | An existing Python to run tests with, instead of building an environment. |
+
+bugbout builds one environment per project with uv. It holds the test dependencies and
+the project's runtime dependencies but not the project itself: every checkout is put on
+`PYTHONPATH`, so the code under test is always the tree being tested.
+
+The project's own `addopts` are ignored, because they often need plugins the test
+environment does not have (coverage, xdist). Put anything your tests really need in
+`pytest-args`.
+
+## `[tasks]` - which fixes become tasks
+
+| Key | Default | Meaning |
+|---|---|---|
+| `since` | all history | Only consider changes after this date, e.g. `"2023-01-01"`. |
+| `limit` | `20` | The most tasks to keep, newest first. |
+| `max-source-files` | `2` | A fix that touches more source files than this is too broad to be one bug. |
+
+## `[run]` - how agents run
+
+| Key | Default | Meaning |
+|---|---|---|
+| `sandbox` | `"auto"` | `auto`, `seatbelt` (macOS), `bubblewrap` (Linux) or `none`. |
+| `timeout` | `1800` | Seconds one agent may spend on one task. |
+| `quotas` | `{}` | How many runs may share a quota at once, e.g. `{ codex = 2 }`. One by default. |
+
+## `[judge]` - how a patch is picked
+
+| Key | Default | Meaning |
+|---|---|---|
+| `policy` | `"majority"` | `majority`, `unanimous`, `strict` or `cross-vendor`; see [How it works](how-it-works.md#the-judge). |
+| `run-suite` | `true` | Also run the repository's own tests against every patch, so breakage shows. Turn it off for very slow suites. |
+
+## `[[contestants]]` - the agents that fix bugs
+
+Each entry is one agent with one model.
+
+| Key | Meaning |
+|---|---|
+| `agent` | The agent type: `codex`, `claude-code`, `gemini-cli`, `mini-swe-agent` or `command`. |
+| `model` | The model to use. Leave it out to use the agent's own default (not allowed for `mini-swe-agent`). |
+| `name` | How the agent appears in reports. Defaults to `agent/model`, or just `agent`. |
+| `vendor` | Who made the model. Guessed from the model name; set it when the guess is wrong. |
+| `quota` | The rate limit it shares with other agents. Defaults to the agent type. |
+
+Any other key is passed to the agent's adapter; see [Agents](agents.md) for each one's
+options. For example:
+
+```toml
+[[contestants]]
+agent = "codex"
+model = "gpt-5.1-codex"
+reasoning-effort = "high"
+```
+
+## `[[examiners]]` - the agents that write tests
+
+Examiners never fix anything. From the issue alone, they write tests that a correct fix
+must pass, and those tests cross-examine every contestant's patch. An examiner takes
+the same keys as a contestant, plus:
+
+| Key | Default | Meaning |
+|---|---|---|
+| `assignment` | `"reproduce"` | `reproduce`: tests for exactly what the issue reports. `specify`: also the general rule behind it, and nearby behaviour a fix must keep. |
+
+Examiners are optional. Without them, the contestants' own tests still cross-check each
+other's patches.
+
+## Environment variables
+
+| Variable | Meaning |
+|---|---|
+| `BUGBOUT_HOME` | Where environments and checkouts are cached; `~/.cache/bugbout` by default. |
+| `GITHUB_TOKEN` | Read issues with a token. Without one, bugbout uses the GitHub CLI's login, or anonymous access at 60 requests an hour. |
+
+Agents read their own credentials as usual - for example `GEMINI_API_KEY` for Gemini
+CLI, or whatever a LiteLLM model needs for mini-swe-agent.
+
+## Where things are kept
+
+Results live with your project, in `.bugbout/` beside `bugbout.toml`. That directory
+ignores itself in git; delete its `.gitignore` if you want to commit results.
+
+| Path | Holds |
+|---|---|
+| `tasks.json` | The mined tasks, and why other fixes were left out. |
+| `oracles.json` | Each task's ground truth. |
+| `attempts/` | One file per agent per task: how the run went, and its patch. |
+| `matrices/` | Every test's outcome on every patch, per task. |
+| `verdicts.json`, `resolutions.json` | The judge's picks, and every patch's score. |
+| `report.md`, `report.json` | The report from the last `bugbout run`. |
+
+Heavy, disposable state - test environments, checkouts and the agents' working copies
+with their logs - lives in `BUGBOUT_HOME`, never in your repository.

bugbout-0.1.0/docs/how-it-works.md

@@ -0,0 +1,108 @@
+# How it works
+
+A bout replays bugs your project has already fixed. Because the fix and its tests are
+known, every agent's patch can be scored by the maintainers' own judgement - without
+asking a model whether it looks right.
+
+## 1. Mining tasks
+
+bugbout walks the first-parent history of your default branch, so each change that
+landed is seen once: a squashed commit, a rebased one, or a pull request's merge commit.
+A change becomes a task when:
+
+- its title reads like a fix (*fix*, *bug*, *crash*, *regression*, ...);
+- it changes at most `max-source-files` Python files in the package;
+- it adds at least one test function to a test module;
+- it names an issue or pull request, such as `(#123)` or `Fixes #45`.
+
+The task's problem statement is the text of that issue. When a pull request links the
+issue it closes, the issue's own text is used instead, because it describes the problem
+without knowing the answer. Sections in which an author describes their fix
+(*Solution*, *Changes*, ...) and diff blocks are removed. If several changes point at
+the same issue, only the newest one is kept.
+
+## 2. Ground truth
+
+For each task, the test modules the fix touched are run twice: on the code just before
+the fix, with the fix's versions of those tests installed, and on the fix itself. The
+same definitions as SWE-bench follow:
+
+- **fail-to-pass** tests fail before the fix and pass after it: a correct patch must
+  make them all pass;
+- **pass-to-pass** tests pass on both sides: a correct patch must keep them passing.
+
+The fixed tree is run twice, and any test whose outcome differs between the two runs is
+set aside as unstable. A task where the fix made no test go from failing to passing
+proves nothing and is not scored.
+
+## 3. The race
+
+Every contestant gets the same prompt: the issue's text, a request to fix it and add a
+regression test, and a command that runs the project's tests. Its working copy is the
+code just before the fix, with:
+
+- no `.git` directory, so neither the fix nor any later commit can be read;
+- none of the fix's test files;
+- web tools off and commands sandboxed (see [Agents](agents.md#sandboxing)).
+
+Each attempt runs in a fresh copy, and what changed on disk *is* the patch - whatever
+the agent says it did. Attempts are saved as they finish, so a bout can stop and resume
+at any time without spending an agent's quota twice.
+
+Examiners, if any are configured, get the same checkout and the issue, but are told to
+fix nothing: only to write tests that capture the bug - checked to fail on the unfixed
+code for the reason the issue describes.
+
+## 4. Scoring
+
+A patch is scored by applying its source changes - never its edits to the tests - to the
+code before the fix, installing the maintainers' test files from the fix, and running
+them. It **resolves** the task when every fail-to-pass test passes and no pass-to-pass
+test fails. The report also separates patches that fixed the bug but broke something
+else, patches that left the bug in place, and attempts that produced no fix.
+
+## 5. Cross-examination
+
+Before anyone looks at the hidden tests, every test the agents wrote is run against
+every patch:
+
+- each contestant's regression test, against everyone else's fix;
+- each examiner's tests, which never saw a patch;
+- and the repository's existing suite, so breakage shows without anyone predicting it.
+
+Each author's tests are installed under a module name of their own, so none can shadow
+another's. The result is a matrix of executed outcomes - no model is asked anything.
+
+## The judge
+
+The judge reads the matrix and picks one patch per task, or refuses to. Every rule is
+about executed tests, so every verdict can be traced to cells in the matrix:
+
+1. **A reproduction must reproduce.** A test counts as evidence about the bug only if it
+   fails on the unfixed code.
+2. **Nobody vouches for themselves.** A patch is supported only by other authors' tests.
+3. **Breaking something needs a quorum.** A repository test the patch breaks counts
+   against it only if most other patches keep it passing; when nearly every patch
+   changes it, the issue probably asked for that change.
+4. **Abstain rather than guess.** Without enough independent support, there is no pick.
+
+| Policy | Approves a patch that passes... |
+|---|---|
+| `majority` | at least half of the independent reproductions, on average across authors. |
+| `unanimous` | every independent reproduction. |
+| `strict` | every independent reproduction, and breaks none of the examiners' tests for behaviour the issue does not ask to change. |
+| `cross-vendor` | as `majority`, but no vendor's tests may support its own models. |
+
+## The report
+
+- **Agents** - each contestant's patches against the ground truth: resolved, fixed but
+  broke a test, wrong fix, no fix.
+- **Picking one patch per bug** - the judge's picks against the best single agent, an
+  agent chosen at random, and the best any choice could do.
+- **Every patch through a merge gate** - each patch judged alone, as if one agent had
+  just said it was done: how often an approval was right, and how many wrong patches
+  were stopped.
+- **Which tests tell a right fix from a wrong one** - for each author of tests, and for
+  your own suite, how well their outcomes separate correct patches from wrong ones.
+- **How the tasks were found** - how much history was examined, and why fixes were left
+  out.