whichagent 0.1.0__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,262 @@
1
+ Metadata-Version: 2.5
2
+ Name: whichagent
3
+ Version: 0.1.0
4
+ Summary: Find out which coding agent actually works on your repo: a benchmark built from your own git history.
5
+ Project-URL: Homepage, https://github.com/itzzdev09/whichagent
6
+ Project-URL: Issues, https://github.com/itzzdev09/whichagent/issues
7
+ License-Expression: MIT
8
+ License-File: LICENSE
9
+ Keywords: aider,benchmark,claude code,codex,coding agents,gemini cli,swe-bench
10
+ Classifier: Development Status :: 3 - Alpha
11
+ Classifier: Environment :: Console
12
+ Classifier: Intended Audience :: Developers
13
+ Classifier: Operating System :: OS Independent
14
+ Classifier: Programming Language :: Python :: 3
15
+ Classifier: Topic :: Software Development :: Testing
16
+ Requires-Python: >=3.11
17
+ Description-Content-Type: text/markdown
18
+
19
+ # whichagent
20
+
21
+ **Find out which coding agent actually works on *your* repo.**
22
+
23
+ [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
24
+ [![Status: pre-release](https://img.shields.io/badge/status-pre--release-orange.svg)](#status)
25
+ [![Zero dependencies](https://img.shields.io/badge/dependencies-0-brightgreen.svg)](pyproject.toml)
26
+
27
+ whichagent turns your git history into a benchmark. It finds past commits where a fix landed together with a test, rewinds your repo to just before each fix, and gives the same task to Claude Code, Codex, Gemini CLI, Aider or any agent you add. Your own tests decide who solved it.
28
+
29
+ ```text
30
+ $ whichagent --max-tasks 40
31
+ # illustrative output: not real results
32
+
33
+ Mined 412 commits -> 40 tasks (tests fail before the fix, pass after)
34
+ Agents: claude, codex, gemini
35
+ 120 runs, estimated $25.00-$60.00. Continue? [y/N] y
36
+
37
+ agent solved 95% CI $/run $/solved median time issues
38
+ claude 30/40 60-86% $0.41 $0.55 3m 12s -
39
+ codex 27/40 52-80% $0.28 $0.41 4m 05s 1 timeout
40
+ gemini 18/40 31-60% $0.12 $0.27 2m 48s -
41
+
42
+ claude vs codex 5 wins, 2 losses no real difference (p = 0.45)
43
+ claude vs gemini 13 wins, 1 loss claude is better (p = 0.0018)
44
+
45
+ Report: .whichagent/report.html (also report.md and card.svg)
46
+ ```
47
+
48
+ That's the answer you get: who is actually better on your code, who is just cheaper, and which gaps are noise.
49
+
50
+ ## Why
51
+
52
+ - **Public benchmarks aren't your code.** SWE-bench and friends measure a fixed set of public repos that every vendor reports on and optimizes for.
53
+ - **A new "best model" ships every month.** Switching on hype costs money, and you can't tell whether it helped.
54
+ - **Your agent setup is untested.** You edit CLAUDE.md or AGENTS.md, add an MCP server or a skill, and *feel* the agent got better.
55
+
56
+ whichagent answers with numbers from your code and your tests.
57
+
58
+ ## How it works
59
+
60
+ ```mermaid
61
+ flowchart LR
62
+ A["Your git history"] --> B["Mine<br/>fix commits that came with tests"]
63
+ B --> C["Validate<br/>tests fail before, pass after"]
64
+ C --> D["Run<br/>each agent in a clean clone<br/>at the commit before the fix"]
65
+ D --> E["Grade<br/>restore the real tests, run them"]
66
+ E --> F["Report<br/>solve rate, cost per fix,<br/>is the gap real?"]
67
+ ```
68
+
69
+ 1. **Mine.** Scan your history for commits that change code and tests together. Skip merges, big refactors and dependency bumps.
70
+ 2. **Validate.** Keep a commit only if its tests fail on the code before the fix and pass after it, twice in a row. Record exactly which tests flip (fail-to-pass) and which must keep passing.
71
+ 3. **Run.** Each agent gets a fresh clone that ends at the commit before the fix, plus a task description from the linked issue, the PR or the commit message. The clone has no future history, so `git log --all` can't leak the answer, and the agent doesn't see the tests.
72
+ 4. **Grade.** Undo anything the agent did to tests, add the real tests from the fix, run them. Solved means the flipped tests pass, the others still pass, and nothing else in the suite broke.
73
+ 5. **Report.** Solve rate with confidence intervals, cost per solved task, time, and a paired significance test for every pair of agents.
74
+
75
+ Everything runs on your machine. Your code goes only to the agents you choose to run, the same as when you use them day to day.
76
+
77
+ ## Quickstart
78
+
79
+ Python 3.11+ and git. Nothing else to install: whichagent has no dependencies.
80
+
81
+ ```bash
82
+ pipx install git+https://github.com/itzzdev09/whichagent
83
+ ```
84
+
85
+ ```bash
86
+ cd your-repo
87
+ whichagent
88
+ ```
89
+
90
+ The first run detects your stack, writes `whichagent.toml`, mines tasks, shows the estimated cost, asks before spending anything, runs every agent it finds and writes the report. Every step resumes where it stopped.
91
+
92
+ | Command | Does |
93
+ |---|---|
94
+ | `whichagent` | quickstart: init, mine, estimate, run, report |
95
+ | `whichagent init` | write `whichagent.toml` from the detected stack |
96
+ | `whichagent doctor` | check git, the test suite at HEAD and each agent's flags |
97
+ | `whichagent mine` | find and validate tasks |
98
+ | `whichagent run` | run the matrix (resumable) |
99
+ | `whichagent report` | terminal table, `report.html`, `report.md`, `card.svg` |
100
+ | `whichagent compare A B` | paired comparison of two agents |
101
+ | `whichagent show <task>` | statement, real fix, each agent's diff |
102
+
103
+ Every config key is also a flag: `--max-tasks 40`, `--agents claude,gemini`, `--attempts 3`, `--max-cost 10`, `--since 2026-06-01`.
104
+
105
+ ## Measure your setup, not just the agent
106
+
107
+ Run one agent with two setups and let your tests decide:
108
+
109
+ ```toml
110
+ [[agent]]
111
+ name = "claude + new CLAUDE.md"
112
+ extends = "claude"
113
+ overlay = { "CLAUDE.md" = "experiments/CLAUDE.v2.md" }
114
+
115
+ [[agent]]
116
+ name = "claude-haiku" # cheap tiers are worth measuring too
117
+ extends = "claude"
118
+ args = "--model haiku"
119
+ ```
120
+
121
+ Overlay files are copied into every workspace before the agent starts: CLAUDE.md, AGENTS.md, MCP configs, skills, rules files, anything.
122
+
123
+ ## Built to be honest
124
+
125
+ | Trap | What whichagent does |
126
+ |---|---|
127
+ | Agent finds the fix in future commits | Every run gets a fresh clone that ends at the base commit |
128
+ | Agent edits tests until they pass | Tests and test config are reset before grading; tampering is flagged |
129
+ | Agent breaks something else | Tests that passed at the fix and fail now count as a regression |
130
+ | 30/40 vs 27/40 looks like a win | 95% intervals and a paired sign test on every comparison |
131
+ | Model trained on your public repo | `--since` uses only commits newer than the models' training cutoff |
132
+ | One lucky run | `--attempts 3` reports "solved" and "solved every time" |
133
+ | Flaky tests | Tasks are validated twice; flaky ones are dropped |
134
+ | Agent auto-updated mid-benchmark | Versions are recorded; mixed versions are flagged, `--by-version` splits them |
135
+
136
+ ## Agents
137
+
138
+ An adapter is a few lines of TOML, so adding your favorite agent makes a good first PR.
139
+
140
+ | Agent | Built-in command | Cost data | Status |
141
+ |---|---|---|---|
142
+ | Claude Code | `claude -p --output-format json` | reported by the CLI | verified |
143
+ | Codex CLI | `codex exec --json --sandbox workspace-write -` | tokens x models.dev prices | verified |
144
+ | Gemini CLI | `gemini --output-format json --yolo` | tokens x models.dev prices | unverified |
145
+ | Qwen Code | `qwen --output-format json --yolo` | tokens x models.dev prices | unverified |
146
+ | Aider | `aider --message-file ... --yes-always` | parsed from its output; $0 for local models | verified |
147
+ | OpenCode | `opencode run --auto --format json` | reported by the CLI | verified |
148
+ | Goose, Amp, Copilot CLI, Cursor CLI | headless mode, prompt on stdin or a file | time and pass/fail only | unverified |
149
+ | Anything else | any command; the prompt arrives on stdin or as `{prompt_file}` | optional | |
150
+
151
+ Unverified adapters follow each CLI's docs but haven't been run by whichagent yet; `whichagent doctor` checks their flags on your machine. Several of these agents have free tiers or free models, which makes them cheap to compare. Agents get only an allowlist of environment variables; pass anything else with `env = { TOKEN = "$MY_TOKEN" }`, which reads it from your shell.
152
+
153
+ ### Run it for $0
154
+
155
+ whichagent itself is free and local. To benchmark without paying anyone, use open-source agents with local models, or free tiers:
156
+
157
+ ```toml
158
+ # Aider + a local Ollama model: no account, no API bill
159
+ [[agent]]
160
+ name = "aider-qwen7b"
161
+ extends = "aider"
162
+ args = "--model ollama_chat/qwen2.5-coder:7b --no-show-model-warnings"
163
+ env = { OLLAMA_API_BASE = "http://127.0.0.1:11434" }
164
+
165
+ # OpenCode with one of its free models (`opencode models | grep free`): no account
166
+ [[agent]]
167
+ name = "opencode-free"
168
+ extends = "opencode"
169
+ args = "-m opencode/deepseek-v4.1-flash-free"
170
+
171
+ # Qwen Code against Ollama's OpenAI-compatible endpoint
172
+ [[agent]]
173
+ name = "qwen-local"
174
+ extends = "qwen"
175
+ args = "--auth-type openai"
176
+ env = { OPENAI_BASE_URL = "http://127.0.0.1:11434/v1", OPENAI_API_KEY = "ollama", OPENAI_MODEL = "qwen2.5-coder:7b" }
177
+
178
+ [prices]
179
+ "qwen2.5-coder:7b" = { input = 0, output = 0 } # so the report shows $0 instead of "no cost"
180
+ ```
181
+
182
+ Aider and OpenCode setups above have been run end to end. Small local models are weak agents: on commander.js, qwen2.5-coder:7b solved nothing through Aider and made no tool calls through Qwen Code, while OpenCode's free model solved 1 of 2 tasks. Gemini CLI and GitHub Copilot have free tiers when you sign in; free tiers are rate-limited, so try a few tasks first with `whichagent run --tasks ID1,ID2`. Model names starting with `ollama/`, `ollama_chat/`, `lm_studio/` or `llamafile/` are priced at $0 automatically; `[run] max_cost` only matters for paid agents.
183
+
184
+ ## Stacks
185
+
186
+ `whichagent init` detects the stack and sets up per-test results where the runner can produce them:
187
+
188
+ | Found | Install | Tests | Per-test results |
189
+ |---|---|---|---|
190
+ | `uv.lock`, `poetry.lock` | `uv sync`, `poetry install` | pytest | JUnit XML |
191
+ | `package-lock.json`, `pnpm-lock.yaml`, `yarn.lock` | `npm ci`, `pnpm install`, `yarn install` (cached) | Vitest, Jest, `node --test` | JUnit XML, Jest JSON |
192
+ | `go.mod` | `go mod download` | `go test` | `go test -json` |
193
+ | `Cargo.toml` | `cargo fetch` | `cargo test` (integration tests) | cargo's output |
194
+ | `pom.xml`, `build.gradle` | Maven, Gradle | JUnit | Surefire and Gradle XML |
195
+
196
+ Anything else: `init` asks for your test command, with `{files}` where the test files go. Without per-test results, tasks are graded by the exit code of their test files.
197
+
198
+ ## Sandbox
199
+
200
+ By default agents run on your machine, in throwaway clones in the temp dir, with an environment allowlist, timeouts that kill the whole process tree, and a hard spend cap. That isn't a sandbox: an agent can still reach whatever your user account can.
201
+
202
+ `--sandbox docker` runs setup, agents and tests in a container built from your `.devcontainer/devcontainer.json`, a `Dockerfile`, or `[docker] image`, with only the run's temp dir mounted. The agent CLIs must be installed in the image; credentials go in by name, or through `[docker] mounts`.
203
+
204
+ ## In CI
205
+
206
+ whichagent is also a GitHub Action. This workflow reruns the benchmark when CLAUDE.md or AGENTS.md changes, comments the result on the pull request, and compares it with the last run on the default branch:
207
+
208
+ ```yaml
209
+ name: agents
210
+ on:
211
+ pull_request:
212
+ paths: [CLAUDE.md, AGENTS.md, ".claude/**"]
213
+ schedule: [{ cron: "0 6 * * 1" }]
214
+ workflow_dispatch:
215
+
216
+ jobs:
217
+ whichagent:
218
+ runs-on: ubuntu-latest
219
+ permissions: { contents: read, pull-requests: write }
220
+ steps:
221
+ - uses: actions/checkout@v4
222
+ with: { fetch-depth: 0 } # mining needs the history
223
+ - run: npm install -g @anthropic-ai/claude-code
224
+ - uses: actions/cache/restore@v4
225
+ if: github.event_name == 'pull_request'
226
+ with: { path: baseline, key: whichagent-${{ github.run_id }}, restore-keys: whichagent- }
227
+ - uses: itzzdev09/whichagent@main
228
+ with:
229
+ agents: claude
230
+ baseline: baseline/results.jsonl
231
+ env:
232
+ ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
233
+ - if: github.event_name != 'pull_request'
234
+ run: mkdir -p baseline && cp .whichagent/results.jsonl baseline/
235
+ - uses: actions/cache/save@v4
236
+ if: github.event_name != 'pull_request'
237
+ with: { path: baseline, key: whichagent-${{ github.run_id }} }
238
+ ```
239
+
240
+ So that agents see the pull request's CLAUDE.md rather than the one in each old commit, add `overlay = { "CLAUDE.md" = "CLAUDE.md" }` to the agent in `whichagent.toml`.
241
+
242
+ ## FAQ
243
+
244
+ **How is this different from SWE-bench?** SWE-bench is a fixed research dataset of public Python repos. whichagent builds a benchmark from *your* repo, in any language with a test command, and grades it the same way: tests that must flip, tests that must keep passing.
245
+
246
+ **And from Harbor?** Harbor is a general framework for running agent benchmarks; you bring the tasks. whichagent finds and validates the tasks in your history for you.
247
+
248
+ **What does it cost?** Each run is one normal agent session. whichagent shows an estimate before it starts and stops at `--max-cost`. On subscription plans, costs are shown at API prices so agents can still be compared on efficiency.
249
+
250
+ **Will it work on my repo?** If your tests run from one command and your history has commits that change code and tests together, yes.
251
+
252
+ ## Status
253
+
254
+ Pre-release. Everything in [ARCHITECTURE.md](ARCHITECTURE.md) is built and covered by an offline test suite with fake agents; dogfooding on real repos is under way. Grading has been checked against real fixes: on 16 tasks mined from click and commander.js, the real fix is graded solved and no change is graded failed, every time. The Claude Code, Codex, OpenCode and Aider adapters have been run end to end (OpenCode with a free model and Aider with a local Ollama model, both at $0); the other adapters, Docker mode and the GitHub Action haven't yet. Star or watch the repo to hear when it lands on PyPI.
255
+
256
+ ## Contributing
257
+
258
+ Issues and PRs are welcome. Agent adapters are the easiest place to start. Run the tests with `python -m unittest`; they need no API keys.
259
+
260
+ ## License
261
+
262
+ [MIT](LICENSE)
@@ -0,0 +1,21 @@
1
+ whichagent/__init__.py,sha256=AU37e-nPh7RVe7wHbAkDfZzmNqjp-db0TjIXM6airlI,96
2
+ whichagent/__main__.py,sha256=E6Gls0DNz8GQK2K-kOUIx8cYhgANW_CH54VKrfCfs14,52
3
+ whichagent/agents.py,sha256=vQQ4jRcX0iYIvwhciYYaXUnXLr_17ZB78FJ8-a0A2n4,8124
4
+ whichagent/cli.py,sha256=7HcBMwLRVqwlaylKMoR5o4bPnq-dvldsRmoUuP6cS4E,11691
5
+ whichagent/config.py,sha256=qcH7bUK9iFtmQYN-0cUCRqKlAmcM96mbIUWe39MjrGg,8278
6
+ whichagent/git.py,sha256=6Ykr0X7CFeIG4aJPCA2rtsZxyNnRpCYgX_Mmr_eH5aQ,4828
7
+ whichagent/grade.py,sha256=DcenYXIYze0PN9ilUae0C-8L_q1oW2DSF_-uSZcKpO4,4802
8
+ whichagent/mine.py,sha256=Ibd3evJoa8qnAmZEkbj5oCSAmdrYI12nISQnxkYlqNo,7948
9
+ whichagent/report.py,sha256=_qDkFjNoXP6jduVnQrkKGKZ0WaKoZjpIVjI23j_DOSI,16378
10
+ whichagent/run.py,sha256=XQduZNUwx1lMQAw8bo8RsAC3ENSYnvHq0l9P5ytWZWY,7765
11
+ whichagent/sandbox.py,sha256=YnFz6n5EFWgS1qLe6uIx993u5JeJnCFb_k35OpBUqho,10521
12
+ whichagent/stacks.py,sha256=VyBbDGeiKdMRR3JLXKXlwoQp1UeU02XhdvkH3mIWYqg,2836
13
+ whichagent/stats.py,sha256=G12bA6UkKExQlLOgikmVN5qR00KRf85w5tIlfKZ8n04,3462
14
+ whichagent/store.py,sha256=VBKNbrw7ZDYtiJYa0_ZfymPu7-jyOm7HJLLc18KStbs,1465
15
+ whichagent/testresults.py,sha256=PYg3djloYakxdELHCycpUc0FVLRE-BuCv4tJGDy7QD4,3177
16
+ whichagent/workspace.py,sha256=f1y5OEF9sTT2zKLc9uoQURH0RA-QSj1ZfTH9JkmCmlM,3589
17
+ whichagent-0.1.0.dist-info/METADATA,sha256=Jv0ojmETqwsnIMf_Q0cT9AbCmNyPHX6uVnPQl8q_duE,13865
18
+ whichagent-0.1.0.dist-info/WHEEL,sha256=2iZrJkDIkeDh0k9SikKMqFuRP-ua6mb4Ir1AsER4BEw,87
19
+ whichagent-0.1.0.dist-info/entry_points.txt,sha256=xd0FSoY3sdOSY8vaFYKUH7o5NZLhMHbchhKe6irPl1w,51
20
+ whichagent-0.1.0.dist-info/licenses/LICENSE,sha256=WzJCXhcwOdHkZ58sf7EKCna1VJlokeX2hBW9hYqs2YM,1066
21
+ whichagent-0.1.0.dist-info/RECORD,,
@@ -0,0 +1,4 @@
1
+ Wheel-Version: 1.0
2
+ Generator: hatchling 1.33.0
3
+ Root-Is-Purelib: true
4
+ Tag: py3-none-any
@@ -0,0 +1,2 @@
1
+ [console_scripts]
2
+ whichagent = whichagent.cli:main
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 itzzdev09
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.