whichagent 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- whichagent-0.1.0/.github/workflows/ci.yml +16 -0
- whichagent-0.1.0/.github/workflows/release.yml +18 -0
- whichagent-0.1.0/.gitignore +5 -0
- whichagent-0.1.0/ARCHITECTURE.md +448 -0
- whichagent-0.1.0/LICENSE +21 -0
- whichagent-0.1.0/PKG-INFO +262 -0
- whichagent-0.1.0/README.md +244 -0
- whichagent-0.1.0/action.yml +63 -0
- whichagent-0.1.0/pyproject.toml +31 -0
- whichagent-0.1.0/src/whichagent/__init__.py +2 -0
- whichagent-0.1.0/src/whichagent/__main__.py +5 -0
- whichagent-0.1.0/src/whichagent/agents.py +199 -0
- whichagent-0.1.0/src/whichagent/cli.py +244 -0
- whichagent-0.1.0/src/whichagent/config.py +157 -0
- whichagent-0.1.0/src/whichagent/git.py +137 -0
- whichagent-0.1.0/src/whichagent/grade.py +85 -0
- whichagent-0.1.0/src/whichagent/mine.py +159 -0
- whichagent-0.1.0/src/whichagent/report.py +312 -0
- whichagent-0.1.0/src/whichagent/run.py +153 -0
- whichagent-0.1.0/src/whichagent/sandbox.py +248 -0
- whichagent-0.1.0/src/whichagent/stacks.py +55 -0
- whichagent-0.1.0/src/whichagent/stats.py +78 -0
- whichagent-0.1.0/src/whichagent/store.py +46 -0
- whichagent-0.1.0/src/whichagent/testresults.py +91 -0
- whichagent-0.1.0/src/whichagent/workspace.py +94 -0
- whichagent-0.1.0/tests/__init__.py +4 -0
- whichagent-0.1.0/tests/fake_agent.py +49 -0
- whichagent-0.1.0/tests/junit_runner.py +50 -0
- whichagent-0.1.0/tests/repos.py +118 -0
- whichagent-0.1.0/tests/test_pipeline.py +201 -0
- whichagent-0.1.0/tests/test_units.py +188 -0
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
name: ci
|
|
2
|
+
on: [push, pull_request]
|
|
3
|
+
jobs:
|
|
4
|
+
test:
|
|
5
|
+
strategy:
|
|
6
|
+
fail-fast: false
|
|
7
|
+
matrix:
|
|
8
|
+
os: [ubuntu-latest, macos-latest, windows-latest]
|
|
9
|
+
python: ["3.11", "3.13"]
|
|
10
|
+
runs-on: ${{ matrix.os }}
|
|
11
|
+
steps:
|
|
12
|
+
- uses: actions/checkout@v4
|
|
13
|
+
- uses: actions/setup-python@v5
|
|
14
|
+
with:
|
|
15
|
+
python-version: ${{ matrix.python }}
|
|
16
|
+
- run: python -m unittest -v
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
name: release
|
|
2
|
+
on:
|
|
3
|
+
push:
|
|
4
|
+
tags: ["v*"]
|
|
5
|
+
|
|
6
|
+
jobs:
|
|
7
|
+
pypi:
|
|
8
|
+
runs-on: ubuntu-latest
|
|
9
|
+
environment: pypi
|
|
10
|
+
permissions:
|
|
11
|
+
id-token: write # trusted publishing: no PyPI token stored in GitHub
|
|
12
|
+
steps:
|
|
13
|
+
- uses: actions/checkout@v4
|
|
14
|
+
- uses: actions/setup-python@v5
|
|
15
|
+
with:
|
|
16
|
+
python-version: "3.12"
|
|
17
|
+
- run: python -m pip install --quiet build && python -m build
|
|
18
|
+
- uses: pypa/gh-action-pypi-publish@release/v1
|
|
@@ -0,0 +1,448 @@
|
|
|
1
|
+
# whichagent architecture
|
|
2
|
+
|
|
3
|
+
> Design for v0.1–v0.4, all implemented. Section 19 says what has been run against real repos and agents, and what hasn't yet.
|
|
4
|
+
|
|
5
|
+
## 1. Goal
|
|
6
|
+
|
|
7
|
+
One command that answers: **which coding agent, with which setup, solves real tasks from my repo, how often, at what cost, and is the difference real?**
|
|
8
|
+
|
|
9
|
+
Principles:
|
|
10
|
+
|
|
11
|
+
- **Real tasks only.** Every task is a commit from your history, checked by your own tests. No synthetic bugs.
|
|
12
|
+
- **Local.** No service, no account, no upload. Results stay in your repo.
|
|
13
|
+
- **Honest numbers.** No answer leaks, no test tampering, an interval on every rate, a significance test on every comparison.
|
|
14
|
+
- **Zero dependencies.** Python standard library plus `git`. `gh` and Docker are optional.
|
|
15
|
+
|
|
16
|
+
Not goals: a hosted leaderboard, building agents, generating training data.
|
|
17
|
+
|
|
18
|
+
## 2. Pipeline
|
|
19
|
+
|
|
20
|
+
```mermaid
|
|
21
|
+
flowchart TB
|
|
22
|
+
subgraph mine ["whichagent mine"]
|
|
23
|
+
L["git log"] --> F["filter candidates"]
|
|
24
|
+
F --> V["validate: tests fail before the fix, pass after"]
|
|
25
|
+
V --> S["task statement: issue, PR or commit message"]
|
|
26
|
+
end
|
|
27
|
+
S --> T[("tasks.jsonl")]
|
|
28
|
+
T --> Q
|
|
29
|
+
subgraph run ["whichagent run"]
|
|
30
|
+
Q["scheduler: tasks × agents × attempts"] --> W["workspace: leak-free clone at the base commit"]
|
|
31
|
+
W --> O["overlays + setup"]
|
|
32
|
+
O --> A["agent CLI, headless"]
|
|
33
|
+
A --> G["grade: reset tests, add the real tests, run"]
|
|
34
|
+
end
|
|
35
|
+
G --> R[("results.jsonl")]
|
|
36
|
+
R --> P["whichagent report: terminal, HTML, Markdown"]
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
`mine` and `run` append to JSONL files as they go, so every command can be stopped and resumed without losing work.
|
|
40
|
+
|
|
41
|
+
## 3. Where things live
|
|
42
|
+
|
|
43
|
+
| What | Where |
|
|
44
|
+
|---|---|
|
|
45
|
+
| whichagent's code | this repo, `src/whichagent/` |
|
|
46
|
+
| Your config | `whichagent.toml` at your repo root (commit it) |
|
|
47
|
+
| Tasks, results, logs, diffs, reports | `.whichagent/` in your repo (add it to `.gitignore`) |
|
|
48
|
+
| Leak-free base clones | `.whichagent/bases/<task>.git`, one per task, made during `mine` |
|
|
49
|
+
| Workspaces | the system temp dir, one per run, deleted after grading (`--keep` to inspect) |
|
|
50
|
+
| Dependency cache | the system temp dir, `whichagent-cache/`: the 3 most recently used dependency sets per repo |
|
|
51
|
+
| Model prices | models.dev public JSON, cached for 24 hours in `.whichagent/prices.json` |
|
|
52
|
+
| whichagent's own CI | GitHub Actions on Linux, macOS and Windows, with fixture repos and fake agents (no API spend) |
|
|
53
|
+
| Releases | PyPI, published from GitHub Actions on version tags (trusted publishing) |
|
|
54
|
+
|
|
55
|
+
Workspaces sit in the temp dir on purpose: inside a synced folder (OneDrive, Dropbox) every run would trigger a sync of thousands of files.
|
|
56
|
+
|
|
57
|
+
## 4. Data model
|
|
58
|
+
|
|
59
|
+
### Task: one line in `.whichagent/tasks.jsonl`
|
|
60
|
+
|
|
61
|
+
```jsonc
|
|
62
|
+
{
|
|
63
|
+
"id": "3f9c2e1", // short sha of the fix commit
|
|
64
|
+
"base": "<sha>", // parent of the fix: where agents start
|
|
65
|
+
"fix": "<sha>", // the real fix
|
|
66
|
+
"date": "2026-09-14",
|
|
67
|
+
"statement": "CSV reader drops the last row when the file has no trailing newline ...",
|
|
68
|
+
"statement_source": "pr", // issue | pr | commit
|
|
69
|
+
"source_files": ["src/csvkit/reader.py"],
|
|
70
|
+
"test_files": ["tests/test_reader.py"],
|
|
71
|
+
"fix_lines": 14, // changed source lines in the real fix
|
|
72
|
+
"mode": "hidden", // hidden | visible (see 5.3)
|
|
73
|
+
"suite_green": true, // full suite passes at the fix commit
|
|
74
|
+
"fail_to_pass": ["tests/test_reader.py::test_no_trailing_newline"], // per-test grading, section 8
|
|
75
|
+
"pass_to_pass": ["tests/test_reader.py::test_basic"],
|
|
76
|
+
"suite_failing": [], // tests already failing in the full suite at the fix
|
|
77
|
+
"test_secs": 8.2 // how long the task's tests take
|
|
78
|
+
}
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
### Result: one line in `.whichagent/results.jsonl`
|
|
82
|
+
|
|
83
|
+
```jsonc
|
|
84
|
+
{
|
|
85
|
+
"task": "3f9c2e1", "agent": "codex", "attempt": 1,
|
|
86
|
+
"status": "solved", // solved | failed | timeout | agent_error | setup_error
|
|
87
|
+
"tests_pass": true,
|
|
88
|
+
"f2p": "1/1", "p2p": "12/12", // per-test grading: flipped tests passing, kept tests passing
|
|
89
|
+
"regression": false, // a test that passed at the fix fails now
|
|
90
|
+
"broke": [], // which ones
|
|
91
|
+
"cost_usd": 0.31, "tokens_in": 182000, "tokens_out": 9100,
|
|
92
|
+
"secs": 245, "turns": 18,
|
|
93
|
+
"files_changed": 2,
|
|
94
|
+
"extra_files": 0, // changed files the real fix didn't touch
|
|
95
|
+
"wrote_tests": true, // added tests of its own (reported, not graded)
|
|
96
|
+
"tampered": false, // edited existing tests or test config
|
|
97
|
+
"agent_version": "codex-cli 0.x",
|
|
98
|
+
"model": "..."
|
|
99
|
+
}
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
Each run also keeps `prompt.txt`, `install.log`, `setup.log`, `stdout.log`, `stderr.log`, `agent.diff`, `tests.log` and `suite.log` in `.whichagent/runs/<task>/<agent>@<version>/<attempt>/`.
|
|
103
|
+
|
|
104
|
+
## 5. Mining
|
|
105
|
+
|
|
106
|
+
### 5.1 Candidate filter (cheap: reads history, runs nothing)
|
|
107
|
+
|
|
108
|
+
`git log --no-merges --numstat` over the window (`since`, default 12 months) on the current branch. A commit is a candidate when:
|
|
109
|
+
|
|
110
|
+
- it changes at least one test file and at least one source file (decided by `test_patterns`)
|
|
111
|
+
- the source change is small: at most `max_files` (5) files and `max_lines` (200) changed lines
|
|
112
|
+
- it doesn't touch dependency files (`skip_if_changed`: lockfiles, `requirements*.txt`, ...). A dependency change means the base commit's environment can't run the new tests.
|
|
113
|
+
- it has a usable statement (5.4)
|
|
114
|
+
|
|
115
|
+
Candidates are sorted newest first and capped at `max_candidates` (300).
|
|
116
|
+
|
|
117
|
+
### 5.2 Validation (expensive: runs tests; parallel; cached in `tasks.jsonl`)
|
|
118
|
+
|
|
119
|
+
For each candidate, in a fresh workspace at `base` (section 6):
|
|
120
|
+
|
|
121
|
+
1. run `setup`
|
|
122
|
+
2. copy in the fix commit's versions of the task's test files and run the test command on them: it must **fail**
|
|
123
|
+
3. copy in the fix commit's source files and run again: it must **pass**
|
|
124
|
+
4. run the full suite once and store `suite_green` (this enables the regression check)
|
|
125
|
+
5. repeat steps 2–3 `verify_runs` times (default 2); any disagreement means flaky, and the candidate is dropped
|
|
126
|
+
|
|
127
|
+
Mining stops when `max_tasks` (25) valid tasks are found.
|
|
128
|
+
|
|
129
|
+
When the test runner reports per-test results (section 8), steps 2–3 record which tests fail before the fix and pass after (`fail_to_pass`, at least one) and which pass both times (`pass_to_pass`); repeated runs must agree on both lists. Step 4 records the tests that already fail in the full suite (`suite_failing`), so regressions can be checked even when the suite isn't fully green.
|
|
130
|
+
|
|
131
|
+
### 5.3 Hidden vs visible tests
|
|
132
|
+
|
|
133
|
+
- **hidden** (default, SWE-bench style): the agent gets the statement only, and the real tests appear at grading time. This is the realistic case: real tasks don't come with tests.
|
|
134
|
+
- **visible**: the agent also gets the new failing tests in its workspace, with an instruction not to edit them.
|
|
135
|
+
|
|
136
|
+
A task switches to visible when the failure in step 2 shows the tests can't even load: per-stack patterns such as `ImportError`, `errors during collection` or `Cannot find module`. That usually means the fix added a new function or module, and without seeing the tests an agent would have to guess its exact name.
|
|
137
|
+
|
|
138
|
+
### 5.4 Task statements
|
|
139
|
+
|
|
140
|
+
The first available of:
|
|
141
|
+
|
|
142
|
+
1. the linked issue (`Fixes #123` in the commit or PR), via `gh`
|
|
143
|
+
2. the PR that merged the commit, via `gh api repos/{owner}/{repo}/commits/{sha}/pulls`
|
|
144
|
+
3. the commit message, if it has at least `min_statement_words` (8) words
|
|
145
|
+
|
|
146
|
+
The prompt is identical for every agent:
|
|
147
|
+
|
|
148
|
+
```text
|
|
149
|
+
Resolve the following task in this repository.
|
|
150
|
+
|
|
151
|
+
<statement>
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
Visible mode appends the list of failing test files and "Don't edit these files."
|
|
155
|
+
|
|
156
|
+
Commit messages sometimes describe the fix itself ("use <= instead of <"). That inflates absolute scores but helps every agent equally, so comparisons stay fair. `statement_source` is shown in the report so you can filter.
|
|
157
|
+
|
|
158
|
+
### 5.5 Contamination
|
|
159
|
+
|
|
160
|
+
Models may have seen public repos in training. `--since <date>` keeps only commits after that date: use the latest training cutoff among the models you compare. Private repos don't have this problem.
|
|
161
|
+
|
|
162
|
+
## 6. Workspaces
|
|
163
|
+
|
|
164
|
+
Every validation and every agent run gets a fresh directory in the system temp dir.
|
|
165
|
+
|
|
166
|
+
**Leak-free base clones.** During mining, each task gets one bare clone that contains only the history up to `base`:
|
|
167
|
+
|
|
168
|
+
```bash
|
|
169
|
+
git -C <repo> branch whichagent/tmp-<id> <base>
|
|
170
|
+
git clone --bare --single-branch --no-tags --depth <history_depth> \
|
|
171
|
+
--branch whichagent/tmp-<id> file://<repo> .whichagent/bases/<id>.git
|
|
172
|
+
git -C <repo> branch -D whichagent/tmp-<id>
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
The fix commit and everything after it aren't reachable from `base`, so they never enter the clone. Each workspace is then a fast local clone of that bare repo, with the branch renamed to the repo's default branch and the remote removed. Agents get real `git log` history (default depth 200) and nothing from the future.
|
|
176
|
+
|
|
177
|
+
**Why not `git worktree`?** Worktrees share the main repo's object store: `git log --all` inside one lists every future commit, including the fix. Agents run `git log` all the time.
|
|
178
|
+
|
|
179
|
+
**Why outside the repo?** Agents read CLAUDE.md and AGENTS.md from parent directories and sometimes explore `..`. A workspace inside your repo would sit next to today's code, which contains the answer.
|
|
180
|
+
|
|
181
|
+
**Overlays.** Files from the agent's `overlay` map are copied in after cloning. This is how you A/B a CLAUDE.md, AGENTS.md, MCP config or skill.
|
|
182
|
+
|
|
183
|
+
**Install and setup.** `install` (dependencies) and then `setup` (builds, codegen) run in every workspace (`setup_timeout`, default 10 minutes). A failure is recorded as `setup_error` and isn't counted against the agent. With `cache = true`, everything `install` created is cached in the temp dir, keyed by the content of the lockfiles, and copied into later workspaces instead of installing again; Node stacks turn this on by default. Python virtualenvs aren't relocatable, and uv's own cache already makes `uv sync` fast.
|
|
184
|
+
|
|
185
|
+
**Docker mode.** With `--sandbox docker`, install, setup, the agent and the tests run in a container (section 14). git operations stay on the host; the container sees only the run's temp dir, mounted at `/wa`.
|
|
186
|
+
|
|
187
|
+
## 7. Agent adapters
|
|
188
|
+
|
|
189
|
+
An agent is data, not code:
|
|
190
|
+
|
|
191
|
+
```toml
|
|
192
|
+
[[agent]]
|
|
193
|
+
name = "claude"
|
|
194
|
+
cmd = "claude -p --output-format json --dangerously-skip-permissions"
|
|
195
|
+
stdin = true # the prompt goes to stdin
|
|
196
|
+
parse = "claude-json" # how to read cost, tokens and turns from the output
|
|
197
|
+
timeout = "30m"
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
- **Prompt delivery.** Through stdin (`stdin = true`) or a file (`{prompt_file}` in `cmd`), never as a command-line argument. That avoids length limits and the quoting bugs of `.cmd` shims on Windows.
|
|
201
|
+
- **Variants.** `extends = "claude"` copies an agent. Add `args = "--model claude-opus-5-5"` to test a model, or `overlay = {...}` to test a setup.
|
|
202
|
+
- **Parsers.** One small function per output format: `claude-json`, `codex-jsonl`, `gemini-json`, `aider-text`, and `none` (time and pass/fail only).
|
|
203
|
+
- **Prices.** Tokens are converted to dollars with models.dev prices, overridable in `[prices]`. Unknown prices show as `—`, never a guess. On subscription plans, costs are still shown at API prices, which keeps agents comparable on efficiency.
|
|
204
|
+
- **Versions.** `<binary> --version` is recorded on every result. Reports flag an agent whose runs span several versions; `--by-version` reports each version separately, and `run --new-version` reruns the tasks an agent has only done with another version.
|
|
205
|
+
- **Your global config.** By default agents read your usual global setup (`~/.claude`, `~/.codex`, ...): you're measuring your setup. To isolate an agent, point its config-dir variable elsewhere in its `env`.
|
|
206
|
+
|
|
207
|
+
Built-in adapters. `whichagent doctor` checks each adapter's flags against the installed CLI's `--help`.
|
|
208
|
+
|
|
209
|
+
| Agent | Command | Prompt | Cost source |
|
|
210
|
+
|---|---|---|---|
|
|
211
|
+
| Claude Code | `claude -p --output-format json --dangerously-skip-permissions` | stdin | `total_cost_usd` in the output |
|
|
212
|
+
| Codex CLI | `codex exec --json --sandbox workspace-write -` (Codex's own sandbox stays on) | stdin | token usage events × prices |
|
|
213
|
+
| Gemini CLI | `gemini --output-format json --yolo` | stdin | `stats` tokens × prices |
|
|
214
|
+
| Aider | `aider --message-file {prompt_file} --yes-always --no-auto-commits --no-gitignore` (plus quiet flags) | file | the "Cost: ... session" line; $0 for local models (ollama, lm_studio, llamafile) |
|
|
215
|
+
| Qwen Code | `qwen --output-format json --yolo` | stdin | `stats` tokens × prices |
|
|
216
|
+
| OpenCode | `opencode run --auto --format json` | stdin | `cost` on each step_finish event |
|
|
217
|
+
| Goose | `goose run --no-session --instructions {prompt_file}` | file | none yet |
|
|
218
|
+
| Amp | `amp --execute --dangerously-allow-all` | stdin | none yet |
|
|
219
|
+
| GitHub Copilot CLI | `copilot --allow-all-tools` | stdin | none yet |
|
|
220
|
+
| Cursor CLI | `cursor-agent --print --force --output-format json` | stdin | none yet |
|
|
221
|
+
|
|
222
|
+
Claude Code, Codex, OpenCode and Aider have been run end to end; every other adapter is marked unverified until it has. An agent's `env` values starting with `$` are read from your shell (`env = { GH_TOKEN = "$GH_TOKEN" }`), so credentials outside the allowlist never have to be written into the TOML.
|
|
223
|
+
|
|
224
|
+
## 8. Grading
|
|
225
|
+
|
|
226
|
+
After the agent exits or is killed at its timeout:
|
|
227
|
+
|
|
228
|
+
1. **Capture.** Stage everything (`git add -A`) and save the diff against the tree the agent started from (`base` plus overlays and, in visible mode, the given tests) as `agent.diff`. Commits the agent made are included; ignored files are not.
|
|
229
|
+
2. **Measure.** `files_changed`, `extra_files`, `wrote_tests` (added test lines or files) and `tampered` (removed lines from existing test files, deleted them, or touched test config: `conftest.py`, `pytest.ini`, `jest.config.*`, `vitest.config.*`, ...). Adding a test to an existing file isn't tampering. Edits to `pyproject.toml` or `package.json` are flagged for review rather than reset, since they can be legitimate.
|
|
230
|
+
3. **Reset.** Every path matching `test_patterns` or test config goes back to its `base` state (new test files are deleted), then the fix commit's test files are copied in.
|
|
231
|
+
4. **Test.** Run the test command on the task's test files (`test_timeout`, default 10 minutes). With per-test results, `tests_pass` means every `fail_to_pass` and `pass_to_pass` test passed; without them, that the command exited 0.
|
|
232
|
+
5. **Regression.** If the task's tests pass, run the full suite too. A test that passed at the fix and fails now is a regression (`broke` lists them); without per-test results, any failure counts when the suite was green at the fix. (When the task's tests fail, the run has failed anyway.)
|
|
233
|
+
6. **Verdict.** `solved = tests_pass and not regression`.
|
|
234
|
+
|
|
235
|
+
**Per-test results**, as SWE-bench grades: the test command writes a report to `{report}` — JUnit XML (pytest `--junitxml`, Vitest's and `node --test`'s junit reporters, Maven Surefire), Jest's JSON, or `go test -json`. Gradle's fixed report dir is set with `report`, and cargo's own output is read from the log. Test ids are made relative to the workspace, so they match across runs. Without a report, grading falls back to the exit code of the task's test files.
|
|
236
|
+
|
|
237
|
+
## 9. Scheduler
|
|
238
|
+
|
|
239
|
+
- **Matrix.** Tasks × agents × attempts, interleaved task by task. Stopping early (budget, Ctrl-C) still leaves every agent with the same tasks done, so partial results stay comparable.
|
|
240
|
+
- **Concurrency.** `jobs` worker threads (all heavy work is subprocesses, so threads are enough) and `jobs_per_agent` to respect subscription rate limits.
|
|
241
|
+
- **Resume.** Runs already in `results.jsonl` are skipped. A result is written only when its run finishes. `--retry-errors` reruns runs that ended in `agent_error` or `setup_error` (say, after fixing a login); the newest result counts.
|
|
242
|
+
- **Timeouts.** The whole process tree is killed: a new session plus `killpg` on POSIX, `taskkill /T /F` on Windows.
|
|
243
|
+
- **Environment.** Agents get an allowlist: PATH, home and temp dirs, locale, the OS variables Windows needs, the provider keys (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GEMINI_API_KEY`, ...) and the agent's own `env`. Nothing else from your shell. In Docker mode a container gets only the credentials and proxy variables, by name, plus the agent's `env`.
|
|
244
|
+
- **Budget.** Before starting, whichagent prints the run count and a cost range based on earlier results (or conservative defaults) and asks to continue (`--yes` skips the prompt). During the run, no new run starts once spent money plus the in-flight estimate would pass `max_cost`.
|
|
245
|
+
|
|
246
|
+
## 10. Statistics
|
|
247
|
+
|
|
248
|
+
Standard library only (`math`, `random`).
|
|
249
|
+
|
|
250
|
+
- **Solve rate.** Solved runs ÷ runs, averaged per task. 95% **Wilson interval** when `attempts = 1`, because it stays sane with small samples and near 0% or 100%. With more attempts, a **bootstrap over tasks**, because runs on the same task aren't independent, never narrower than the task-level Wilson interval (on a few tasks that all score 100%, a plain bootstrap would claim 100%–100%).
|
|
251
|
+
- **Solved every time.** The share of tasks solved in all `attempts` runs (pass^k): reliability for unattended use.
|
|
252
|
+
- **Cost per solved task.** Total cost ÷ solved runs: the number budgets are made of.
|
|
253
|
+
- **Paired comparison.** For agents A and B on shared tasks, A wins a task if its solve rate on it is higher. An exact two-sided **sign test** on wins vs losses (ties dropped) gives the verdict: "A is better (p = 0.002)" or "no real difference (p = 0.45)".
|
|
254
|
+
- **Small samples.** Under 15 tasks the report says plainly that only large gaps can be detected.
|
|
255
|
+
|
|
256
|
+
## 11. Reports
|
|
257
|
+
|
|
258
|
+
- **Terminal**: the leaderboard and the pairwise verdicts.
|
|
259
|
+
- **`report.html`**: one self-contained file (inline CSS, JS and SVG, no CDN, works offline, light and dark). It has the leaderboard, a cost-vs-solve-rate chart showing which agents are on the efficient frontier, and a task × agent grid: click a cell to see the agent's diff next to the real fix.
|
|
260
|
+
- **`report.md`**: paste into a PR, an issue or a post.
|
|
261
|
+
- **`card.svg`**: a small leaderboard to share or put in a README; it follows the viewer's light or dark theme.
|
|
262
|
+
- **Baseline**: `--baseline <results.jsonl>` adds each agent's change against an earlier run, with a paired sign test. The GitHub Action uses it for its PR comment.
|
|
263
|
+
- **JSONL**: `tasks.jsonl` and `results.jsonl` for your own analysis.
|
|
264
|
+
|
|
265
|
+
## 12. Configuration
|
|
266
|
+
|
|
267
|
+
`whichagent.toml` sits at the repo root. `whichagent init` writes it from the detected stack, and CLI flags override any key.
|
|
268
|
+
|
|
269
|
+
```toml
|
|
270
|
+
[repo]
|
|
271
|
+
install = "uv sync" # dependencies; cache = true caches them
|
|
272
|
+
setup = "" # builds, codegen
|
|
273
|
+
test = "uv run pytest -q {files} --junitxml={report}" # {files} = the task's test files
|
|
274
|
+
suite = "uv run pytest -q --junitxml={report}" # the whole suite
|
|
275
|
+
test_patterns = ["tests/**", "**/test_*.py", "**/*_test.py"]
|
|
276
|
+
skip_if_changed = ["uv.lock", "requirements*.txt"]
|
|
277
|
+
|
|
278
|
+
[mine]
|
|
279
|
+
since = "2026-01-01" # or "12 months"
|
|
280
|
+
max_tasks = 25
|
|
281
|
+
max_files = 5
|
|
282
|
+
max_lines = 200
|
|
283
|
+
verify_runs = 2
|
|
284
|
+
|
|
285
|
+
[run]
|
|
286
|
+
agents = ["claude", "codex", "gemini"] # default: built-ins found on PATH
|
|
287
|
+
attempts = 1
|
|
288
|
+
jobs = 3
|
|
289
|
+
timeout = "30m"
|
|
290
|
+
max_cost = 25.0 # USD, hard stop
|
|
291
|
+
regression = true
|
|
292
|
+
sandbox = "host" # or "docker"
|
|
293
|
+
|
|
294
|
+
[docker] # for --sandbox docker
|
|
295
|
+
image = "" # default: built from .devcontainer/devcontainer.json or a Dockerfile
|
|
296
|
+
mounts = ["~/.claude:/root/.claude:ro"]
|
|
297
|
+
|
|
298
|
+
[[agent]]
|
|
299
|
+
name = "claude-opus"
|
|
300
|
+
extends = "claude"
|
|
301
|
+
args = "--model claude-opus-5-5"
|
|
302
|
+
|
|
303
|
+
[[agent]]
|
|
304
|
+
name = "claude + new CLAUDE.md"
|
|
305
|
+
extends = "claude"
|
|
306
|
+
overlay = { "CLAUDE.md" = "experiments/CLAUDE.v2.md" }
|
|
307
|
+
|
|
308
|
+
[prices] # USD per million tokens, overrides models.dev
|
|
309
|
+
"some-model" = { input = 1.25, output = 10.0 }
|
|
310
|
+
```
|
|
311
|
+
|
|
312
|
+
`{dirs}` (each test file's directory as `./dir`, for `go test`) and `{names}` (file names without extension, for `cargo test --test` and JUnit classes) work like `{files}`; `{names:,}` joins with commas instead of spaces.
|
|
313
|
+
|
|
314
|
+
Stack detection:
|
|
315
|
+
|
|
316
|
+
| Found | `install` | `test` | Per-test results |
|
|
317
|
+
|---|---|---|---|
|
|
318
|
+
| `uv.lock`, `poetry.lock` | `uv sync`, `poetry install` | `pytest -q {files}` | `--junitxml={report}` |
|
|
319
|
+
| `package-lock.json`, `pnpm-lock.yaml`, `yarn.lock` | `npm ci`, `pnpm install --frozen-lockfile`, `yarn install` (cached) | `vitest run`, `jest` or `node --test`, picked from `package.json` | junit reporter, Jest JSON |
|
|
320
|
+
| `go.mod` | `go mod download` | `go test -json {dirs}` | `go test -json` |
|
|
321
|
+
| `Cargo.toml` | `cargo fetch` | `cargo test --no-fail-fast --test {names: --test }` | cargo's output |
|
|
322
|
+
| `pom.xml` | `mvn test-compile` | `mvn test -Dtest={names:,}` | Surefire XML |
|
|
323
|
+
| `build.gradle(.kts)` | | `gradlew test --tests {names: --tests }` | `build/test-results/test` |
|
|
324
|
+
| anything else | `init` asks for setup and test commands | | exit code |
|
|
325
|
+
|
|
326
|
+
## 13. CLI
|
|
327
|
+
|
|
328
|
+
```text
|
|
329
|
+
whichagent quickstart: init → mine → estimate → run → report
|
|
330
|
+
whichagent init write whichagent.toml from the detected stack
|
|
331
|
+
whichagent doctor check git, the test command at HEAD, and each agent's flags
|
|
332
|
+
whichagent mine find and validate tasks → .whichagent/tasks.jsonl
|
|
333
|
+
whichagent run run the matrix (resumable) → .whichagent/results.jsonl
|
|
334
|
+
whichagent report terminal table, report.html, report.md, card.svg
|
|
335
|
+
whichagent compare A B paired comparison of two agents
|
|
336
|
+
whichagent show <task> statement, real fix, each agent's diff
|
|
337
|
+
|
|
338
|
+
--retry-errors rerun runs that ended in agent_error or setup_error
|
|
339
|
+
--new-version rerun tasks an agent has only done with another version of itself
|
|
340
|
+
--by-version report each agent version separately
|
|
341
|
+
--baseline PATH compare against an earlier results.jsonl
|
|
342
|
+
--sandbox docker run install, setup, agents and tests in a container
|
|
343
|
+
--keep keep workspaces for inspection
|
|
344
|
+
```
|
|
345
|
+
|
|
346
|
+
## 14. Safety model
|
|
347
|
+
|
|
348
|
+
Benchmarking means running agents unattended with approvals off. On the host they act with your user's permissions.
|
|
349
|
+
|
|
350
|
+
On the host, whichagent has:
|
|
351
|
+
|
|
352
|
+
- throwaway clones outside your repo, so your working tree is never touched
|
|
353
|
+
- the environment allowlist (section 9), so secrets in your shell aren't passed through
|
|
354
|
+
- timeouts with a process-tree kill, and a hard spend cap
|
|
355
|
+
- a one-time confirmation on each machine's first run that says all of this plainly
|
|
356
|
+
|
|
357
|
+
Not covered on the host: files in your home directory are still reachable, and so is the network. An agent could even find your real repo on disk.
|
|
358
|
+
|
|
359
|
+
**Docker mode** (`--sandbox docker`) closes the filesystem gap: install, setup, agent and grading run in a container from `[docker] image`, the repo's `.devcontainer/devcontainer.json` or a Dockerfile, with only the run's temp dir mounted. The agent CLIs must be in the image. Credentials reach the container by name only; logins stored in files can be mounted read-only with `[docker] mounts`. Containers are killed with `docker kill` at their timeout. The network stays open, since agents need their APIs.
|
|
360
|
+
|
|
361
|
+
## 15. Tech stack
|
|
362
|
+
|
|
363
|
+
- **Python ≥ 3.11, zero runtime dependencies.** The standard library covers it: `argparse`, `subprocess`, `tomllib`, `json`, `concurrent.futures`, `tempfile`, `shutil`, `math`, `random`, `html`. A tool that runs agents with your permissions should be small enough to audit, with nothing in its supply chain.
|
|
364
|
+
- **No install step.** `uvx whichagent` or `pipx run whichagent`.
|
|
365
|
+
- **git** through `subprocess` (no GitPython). **gh** is optional, for issue and PR text.
|
|
366
|
+
- **None of** async, a database, a plugin system or a web server: threads, subprocesses, JSONL, one HTML file.
|
|
367
|
+
- **Platforms.** Linux, macOS and Windows, all three in CI.
|
|
368
|
+
|
|
369
|
+
## 16. Code layout
|
|
370
|
+
|
|
371
|
+
```text
|
|
372
|
+
src/whichagent/
|
|
373
|
+
cli.py commands and the quickstart flow ~240 lines
|
|
374
|
+
config.py TOML loading, defaults, agents and extends ~160
|
|
375
|
+
stacks.py stack detection ~55
|
|
376
|
+
store.py .whichagent/ files: tasks, results, run dirs ~45
|
|
377
|
+
git.py history, base clones, workspaces, diffs ~140
|
|
378
|
+
sandbox.py host and Docker executors, timeouts, env ~250
|
|
379
|
+
workspace.py clone, install with cache, setup ~95
|
|
380
|
+
mine.py candidates, validation, statements, mode ~160
|
|
381
|
+
agents.py output parsers, prices, flag checks ~170
|
|
382
|
+
run.py scheduler, budget, resume ~155
|
|
383
|
+
grade.py placeholders, capture, reset, test, verdict ~80
|
|
384
|
+
testresults.py JUnit XML, Jest JSON, go test -json, cargo ~90
|
|
385
|
+
stats.py Wilson, bootstrap, pass^k, sign test ~80
|
|
386
|
+
report.py terminal, Markdown, HTML, SVG card, baseline ~310
|
|
387
|
+
tests/
|
|
388
|
+
repos.py builds tiny git repos with known fix commits
|
|
389
|
+
fake_agent.py fake agents (section 17)
|
|
390
|
+
junit_runner.py unittest with JUnit XML, so fixtures get per-test results
|
|
391
|
+
action.yml the GitHub Action
|
|
392
|
+
```
|
|
393
|
+
|
|
394
|
+
About 2,000 lines. Dependencies point one way: `cli` uses `mine` and `run`, which use `workspace` and `grade`, which use `git`, `sandbox`, `store` and `testresults`. Nothing imports `cli`, and the executors in `sandbox` share one interface, so mining, running and grading don't care whether commands run on the host or in a container.
|
|
395
|
+
|
|
396
|
+
## 17. Testing whichagent itself
|
|
397
|
+
|
|
398
|
+
CI spends nothing on APIs. Tests build small git repos with known bug-fix commits and run the full pipeline against **fake agents**: tiny scripts with known behavior.
|
|
399
|
+
|
|
400
|
+
| Fake agent | Does | Expected result |
|
|
401
|
+
|---|---|---|
|
|
402
|
+
| `perfect` | applies the real fix | solved |
|
|
403
|
+
| `noop` | nothing | failed |
|
|
404
|
+
| `tamper` | edits the test so it passes | failed, `tampered` |
|
|
405
|
+
| `breaker` | fixes the bug, breaks another test | failed, regression |
|
|
406
|
+
| `sleeper` | sleeps past the timeout | timeout, no processes left behind |
|
|
407
|
+
| `peeker` | runs `git log --all` and searches for the fix | finds nothing |
|
|
408
|
+
| `escaper` | reads a file outside the workspace | blocked in Docker mode (on the host it can) |
|
|
409
|
+
|
|
410
|
+
`peeker` is the test behind the "no leaks" claim. The fixtures use `tests/junit_runner.py`, a stdlib unittest runner that writes JUnit XML, so per-test grading is tested without pytest. The Docker test runs when a Docker daemon is available.
|
|
411
|
+
|
|
412
|
+
## 18. Build order for v0.1
|
|
413
|
+
|
|
414
|
+
1. `tests/repos.py` fixture builder and the fake agents (tests first)
|
|
415
|
+
2. `config.py` and `git.py`: base clones and workspaces, with `peeker` passing
|
|
416
|
+
3. `mine.py`: candidate filter, validation, statements → `tasks.jsonl`
|
|
417
|
+
4. `grade.py`, checked against the fake agents
|
|
418
|
+
5. `run.py`: matrix, jobs, timeouts, env allowlist, resume, budget
|
|
419
|
+
6. `agents.py`: the Claude Code adapter first, then Codex, Gemini CLI and Aider
|
|
420
|
+
7. `stats.py` and the terminal report
|
|
421
|
+
8. `report.py`: Markdown and HTML
|
|
422
|
+
9. `init` stack detection, `doctor` and the quickstart flow
|
|
423
|
+
10. Dogfooding on 3 open-source repos (at least one Python, one TypeScript): fix what breaks, then publish to PyPI
|
|
424
|
+
|
|
425
|
+
Steps 1–9 are done. Step 10 is under way: 24 tasks mined on pallets/click, tj/commander.js and remeda/remeda, and what broke is fixed; agent runs on them and the PyPI release are next.
|
|
426
|
+
|
|
427
|
+
## 19. Milestones
|
|
428
|
+
|
|
429
|
+
All four are built. The "Done when" checks need real agents, platforms and services:
|
|
430
|
+
|
|
431
|
+
| Version | Scope | Done when | Status |
|
|
432
|
+
|---|---|---|---|
|
|
433
|
+
| v0.1 "It runs" | everything in section 18 | 20+ valid tasks mined and 3 agents run end to end on 3 real repos, on Linux, macOS and Windows | 24 tasks mined on 3 repos (Windows); Claude Code run end to end on a fixture repo; Aider + local qwen2.5-coder:7b on commander.js's 6 tasks; grading checked against the real fixes on 16 tasks |
|
|
434
|
+
| v0.2 "Sharper" | per-test grading; Go, Rust and Java; dependency cache; shareable SVG results card; comparison across agent versions | the same 3 repos graded per test, plus a Go and a Rust repo | the 3 repos' runners checked with `doctor`; parsers tested on sample Go, Rust and Java output |
|
|
435
|
+
| v0.3 "Sandboxed" | Docker and devcontainer mode | `peeker` and a filesystem-escape fake agent are both contained | test written; it needs a Docker daemon |
|
|
436
|
+
| v0.4 "Always on" | GitHub Action: reruns on CLAUDE.md or AGENTS.md changes and on a schedule, with a PR comment showing the delta | the action runs on whichagent's own repo | needs GitHub Actions running on the repo |
|
|
437
|
+
|
|
438
|
+
Later, if people want it: opt-in pooled results across repos ("which agent wins on Django apps?"), rewriting statements to strip solution hints, and tasks that span several commits.
|
|
439
|
+
|
|
440
|
+
## 20. Known limits and open questions
|
|
441
|
+
|
|
442
|
+
- **Inline tests** (Rust `#[cfg(test)]`, doctests): tests and code share a file, so a fix can't be split into tests and source by file. v0.1 skips these commits.
|
|
443
|
+
- **Monorepos**: one test command may not fit every package. A possible answer is per-path `[repo]` sections; still open.
|
|
444
|
+
- **Slow suites**: validation runs tests at least four times per candidate. `max_candidates`, `since` and caching keep this bounded, but a 20-minute suite makes mining slow.
|
|
445
|
+
- **Vague statements**: short commit messages make poor tasks. They're filtered by length, and PR or issue text is preferred.
|
|
446
|
+
- **Host mode isn't a sandbox**: see section 14.
|
|
447
|
+
- **Agent CLIs change often**: adapters are data, versions are recorded, and `doctor` checks the flags.
|
|
448
|
+
- **Cost**: a session costs cents to a few dollars, so a full run can reach tens of dollars. The estimate and `max_cost` make that explicit before anything is spent.
|
whichagent-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 itzzdev09
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|