whobrokeit 0.0.0-stage → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +84 -2
- package/README.tr.md +85 -0
- package/dist/agent.js +183 -0
- package/dist/cli.js +230 -0
- package/dist/demo.js +225 -0
- package/dist/engine.js +374 -0
- package/dist/errors.js +9 -0
- package/dist/exec.js +69 -0
- package/dist/explain.js +140 -0
- package/dist/git.js +39 -0
- package/dist/report.js +392 -0
- package/dist/repro.js +134 -0
- package/dist/sandbox.js +179 -0
- package/package.json +40 -6
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Berkan Atik
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,85 @@
|
|
|
1
|
-
#
|
|
1
|
+
# whobrokeit
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
**Find the commit that broke it, and prove it.**
|
|
4
|
+
|
|
5
|
+
English · [Türkçe](README.tr.md)
|
|
6
|
+
|
|
7
|
+
Describe a bug in one sentence. An AI agent writes a test that shows it, whobrokeit walks your git history to the first commit where that test fails, and proves the result: the test fails at that commit and passes at its parent. Then it explains what went wrong and suggests a fix, which it shows only after the test passes with it.
|
|
8
|
+
|
|
9
|
+
<p align="center">
|
|
10
|
+
<img src="assets/demo.gif" alt="whobrokeit finds the commit that broke subtotal(), in 37 seconds" width="860">
|
|
11
|
+
</p>
|
|
12
|
+
|
|
13
|
+
```sh
|
|
14
|
+
npx whobrokeit demo # try it on a generated repo with a hidden bug (no AI needed)
|
|
15
|
+
npx whobrokeit demo --ai # the same, with an AI agent writing the test
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
## Use it
|
|
19
|
+
|
|
20
|
+
Describe the bug, from anywhere inside your repo:
|
|
21
|
+
|
|
22
|
+
```sh
|
|
23
|
+
whobrokeit "Adding 3 mugs costs the same as 1 mug."
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
This needs [Claude Code](https://github.com/anthropics/claude-code) or [Codex](https://github.com/openai/codex) installed and logged in. whobrokeit uses whichever it finds.
|
|
27
|
+
|
|
28
|
+
Or bring your own failing test, and no AI is involved:
|
|
29
|
+
|
|
30
|
+
```sh
|
|
31
|
+
whobrokeit --test "npm test -- cart.test.js"
|
|
32
|
+
whobrokeit --test "pytest tests/test_cart.py -x"
|
|
33
|
+
whobrokeit --test "node repro.mjs" --with repro.mjs # a test you haven't committed yet
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
Any command works. Exit code 0 means it works, 125 means this commit can't be tested (skip it), anything else means broken. Add `--explain` to get the explanation and a fix in this mode too.
|
|
37
|
+
|
|
38
|
+
## What it does
|
|
39
|
+
|
|
40
|
+
1. **Reproduce.** The agent reads your code and writes a small script that prints a `WHOBROKEIT_BUG:` line when it sees the bug. whobrokeit runs it at `HEAD` and only goes on if it fails there for that reason. A crash without that line counts as "can't tell", so a test that calls a function older commits don't have never blames the wrong commit.
|
|
41
|
+
2. **Find a commit where it still worked.** It walks back 1, 2, 4, 8… commits until the test passes. Or pass `--good <ref>`.
|
|
42
|
+
3. **Bisect.** `git bisect` narrows it down to the first bad commit, skipping commits the test can't tell about.
|
|
43
|
+
4. **Prove it.** It runs the test again at the culprit (fails) and at its parent (passes).
|
|
44
|
+
5. **Explain and fix.** The agent says why the change broke things and proposes an edit. whobrokeit applies it, runs the test, and shows the fix only if the test passes. It saves it as a patch you can `git apply`.
|
|
45
|
+
|
|
46
|
+
## Tested on real bugs
|
|
47
|
+
|
|
48
|
+
I ran it on 12 closed regression issues from commander, semver, postcss, luxon, marked, express, markdown-it, fastify and yargs. It got only the issue text, exactly as the reporter wrote it. It proved the commit the maintainers had identified in **11 of 12** cases, in about 100 seconds each. The cases, the one miss and how to rerun everything are in [bench/](bench).
|
|
49
|
+
|
|
50
|
+
## Safe to run on your repo
|
|
51
|
+
|
|
52
|
+
- **Your working copy is never touched.** Every check runs in a temporary `git worktree`, which is removed at the end. Uncommitted changes, open files and your current branch stay as they are.
|
|
53
|
+
- **The agent can only read.** Claude Code runs with the Read, Grep and Glob tools only, and with your hooks, plugins and MCP servers turned off. Codex runs in its read-only sandbox. whobrokeit runs the test the agent writes, so check it if you don't trust the repo.
|
|
54
|
+
- **Nothing is taken on faith.** The test has to fail at `HEAD` for the reported reason, the culprit has to be proven against its parent, and a fix has to make the test pass.
|
|
55
|
+
- **Dependencies are installed per commit** when `package.json` or a lockfile changes (npm, pnpm, yarn, bun). Use `--setup "<cmd>"` for anything else, or `--no-setup`.
|
|
56
|
+
|
|
57
|
+
## Options
|
|
58
|
+
|
|
59
|
+
| Option | |
|
|
60
|
+
|---|---|
|
|
61
|
+
| `-t, --test <cmd>` | A command that fails while the bug is present. |
|
|
62
|
+
| `--agent <name>` | Who writes the test: `claude`, `codex`, or any command that reads a prompt on stdin and answers on stdout. Default: `claude`, else `codex`. |
|
|
63
|
+
| `--model <name>` | Model for the agent, e.g. `sonnet`. |
|
|
64
|
+
| `--explain` / `--no-explain` | Explain the culprit and test a fix. On by default when you describe a bug, off with `--test`. |
|
|
65
|
+
| `-w, --with <file>` | Copy this file from your working copy into every commit tested. Repeatable. |
|
|
66
|
+
| `-g, --good <ref>` | A commit where it still worked. Default: found automatically. |
|
|
67
|
+
| `-b, --bad <ref>` | A commit where it is broken. Default: `HEAD`. |
|
|
68
|
+
| `--setup <cmd>` / `--no-setup` | Install command, re-run when dependency files change. Default: auto. |
|
|
69
|
+
| `--timeout <sec>` | Time limit for each test run. Default: 600, or 300 for agent-written tests. |
|
|
70
|
+
| `--report <file>` | Also write a Markdown report, ready to paste into an issue or PR. |
|
|
71
|
+
| `--json` | Print the result as JSON. |
|
|
72
|
+
|
|
73
|
+
Exit codes: `0` found and proven · `1` error · `2` bug not reproduced · `3` no working commit found · `4` inconclusive.
|
|
74
|
+
|
|
75
|
+
## Good to know
|
|
76
|
+
|
|
77
|
+
- Requires Node.js 20+ and git. No other dependencies.
|
|
78
|
+
- The test can be in any language your project uses. Automatic dependency install covers JavaScript package managers; for other stacks, use `--setup`.
|
|
79
|
+
- With an agent, a run usually makes two requests: one to write the test, one to explain the culprit (each is retried once if its result doesn't hold up). On the demo repo the whole run takes 30 to 60 seconds.
|
|
80
|
+
- If the project commits no lockfile, old commits get today's versions of their dependencies. A bug that depends on a dependency's version can then be traced to the wrong commit.
|
|
81
|
+
- After a run, the agent's test and the fix are in `.git/whobrokeit/`, outside your working tree.
|
|
82
|
+
|
|
83
|
+
## License
|
|
84
|
+
|
|
85
|
+
[MIT](LICENSE)
|
package/README.tr.md
ADDED
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
# whobrokeit
|
|
2
|
+
|
|
3
|
+
**Hatayı getiren commit'i bulur ve kanıtlar.**
|
|
4
|
+
|
|
5
|
+
[English](README.md) · Türkçe
|
|
6
|
+
|
|
7
|
+
Hatayı tek cümleyle anlatırsın. Bir yapay zekâ ajanı hatayı gösteren bir test yazar; whobrokeit git geçmişinde bu testin ilk başarısız olduğu commit'e kadar gider ve sonucu kanıtlar: test o commit'te başarısız, bir öncekinde başarılıdır. Ardından neyin bozulduğunu açıklar ve bir düzeltme önerir. Düzeltmeyi ancak test onunla geçtikten sonra gösterir.
|
|
8
|
+
|
|
9
|
+
<p align="center">
|
|
10
|
+
<img src="assets/demo.gif" alt="whobrokeit, subtotal() fonksiyonunu bozan commit'i 37 saniyede buluyor" width="860">
|
|
11
|
+
</p>
|
|
12
|
+
|
|
13
|
+
```sh
|
|
14
|
+
npx whobrokeit demo # içine hata gizlenmiş örnek bir repoda dene (yapay zekâ gerekmez)
|
|
15
|
+
npx whobrokeit demo --ai # aynısı, testi yapay zekâ ajanı yazar
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
## Kullanım
|
|
19
|
+
|
|
20
|
+
Reponun içindeyken hatayı anlat:
|
|
21
|
+
|
|
22
|
+
```sh
|
|
23
|
+
whobrokeit "Sepete 3 kupa ekleyince fiyat 1 kupa kadar çıkıyor."
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
Bunun için [Claude Code](https://github.com/anthropics/claude-code) ya da [Codex](https://github.com/openai/codex) kurulu ve giriş yapılmış olmalı. whobrokeit hangisini bulursa onu kullanır. Açıklama, hatayı hangi dilde yazdıysan o dilde gelir.
|
|
27
|
+
|
|
28
|
+
Ya da başarısız olan testi kendin ver, o zaman yapay zekâ hiç devreye girmez:
|
|
29
|
+
|
|
30
|
+
```sh
|
|
31
|
+
whobrokeit --test "npm test -- cart.test.js"
|
|
32
|
+
whobrokeit --test "pytest tests/test_cart.py -x"
|
|
33
|
+
whobrokeit --test "node repro.mjs" --with repro.mjs # henüz commit'lemediğin bir test
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
Her komut olur. Çıkış kodu 0 çalışıyor, 125 bu commit test edilemiyor (atla), diğer her şey bozuk demektir. Bu modda da açıklama ve düzeltme istersen `--explain` ekle.
|
|
37
|
+
|
|
38
|
+
## Ne yapar
|
|
39
|
+
|
|
40
|
+
1. **Hatayı yeniden üretir.** Ajan kodu okur ve hatayı görünce `WHOBROKEIT_BUG:` satırı basan küçük bir betik yazar. whobrokeit onu `HEAD`'de çalıştırır, yalnız bu sebeple başarısız olursa devam eder. Bu satır olmadan gelen bir çökme "belli değil" sayılır; eski commit'lerde olmayan bir fonksiyonu çağıran test yüzünden yanlış commit suçlanmaz.
|
|
41
|
+
2. **Hâlâ çalıştığı bir commit bulur.** Test geçene kadar 1, 2, 4, 8… commit geri gider. İstersen `--good <ref>` ile kendin verirsin.
|
|
42
|
+
3. **Bisect yapar.** `git bisect` ile ilk bozuk commit'e iner, testin karar veremediği commit'leri atlar.
|
|
43
|
+
4. **Kanıtlar.** Testi suçlu commit'te (başarısız) ve bir öncekinde (başarılı) yeniden çalıştırır.
|
|
44
|
+
5. **Açıklar ve düzeltir.** Ajan değişikliğin neyi bozduğunu anlatır ve bir düzenleme önerir. whobrokeit düzenlemeyi uygular, testi çalıştırır, yalnız test geçerse gösterir. `git apply` ile uygulanabilen bir patch olarak da kaydeder.
|
|
45
|
+
|
|
46
|
+
## Gerçek hatalarda denendi
|
|
47
|
+
|
|
48
|
+
commander, semver, postcss, luxon, marked, express, markdown-it, fastify ve yargs projelerinden kapanmış 12 regresyon issue'sunda çalıştırdım. Araca yalnız issue metni verildi, raporu yazan kişinin yazdığı haliyle. **12 vakanın 11'inde** proje sahiplerinin belirlediği commit'i kanıtladı; her biri yaklaşık 100 saniye sürdü. Vakalar, tutmayan tek vaka ve hepsini yeniden çalıştırma adımları [bench/](bench) klasöründe.
|
|
49
|
+
|
|
50
|
+
## Reponda güvenle çalışır
|
|
51
|
+
|
|
52
|
+
- **Çalışma kopyana dokunmaz.** Her kontrol geçici bir `git worktree` içinde yapılır ve sonunda silinir. Commit'lenmemiş değişikliklerin, açık dosyaların ve bulunduğun branch olduğu gibi kalır.
|
|
53
|
+
- **Ajan yalnız okuyabilir.** Claude Code yalnız Read, Grep ve Glob araçlarıyla, hook'ların, eklentilerin ve MCP sunucuların kapalıyken çalışır. Codex kendi salt okunur sandbox'ında çalışır. Ajanın yazdığı testi whobrokeit çalıştırır; güvenmediğin bir repoda bu testi kontrol et.
|
|
54
|
+
- **Hiçbir şey sözle kabul edilmez.** Test `HEAD`'de bildirilen sebeple başarısız olmalı, suçlu commit bir öncekiyle karşılaştırılarak kanıtlanmalı, düzeltme de testi geçirmeli.
|
|
55
|
+
- **Bağımlılıklar commit başına kurulur:** `package.json` ya da lockfile değiştiğinde (npm, pnpm, yarn, bun). Başka bir şey için `--setup "<komut>"`, hiç kurmamak için `--no-setup`.
|
|
56
|
+
|
|
57
|
+
## Seçenekler
|
|
58
|
+
|
|
59
|
+
| Seçenek | |
|
|
60
|
+
|---|---|
|
|
61
|
+
| `-t, --test <cmd>` | Hata varken başarısız olan bir komut. |
|
|
62
|
+
| `--agent <name>` | Testi kim yazar: `claude`, `codex` ya da stdin'den talimat alıp stdout'a cevap yazan herhangi bir komut. Varsayılan: `claude`, yoksa `codex`. |
|
|
63
|
+
| `--model <name>` | Ajanın modeli, örn. `sonnet`. |
|
|
64
|
+
| `--explain` / `--no-explain` | Suçlu commit'i açıkla ve düzeltme dene. Hatayı anlattığında açık, `--test` ile kapalı. |
|
|
65
|
+
| `-w, --with <file>` | Bu dosyayı çalışma kopyandan test edilen her commit'e kopyala. Birden çok verilebilir. |
|
|
66
|
+
| `-g, --good <ref>` | Hâlâ çalıştığı bir commit. Varsayılan: otomatik bulunur. |
|
|
67
|
+
| `-b, --bad <ref>` | Bozuk olduğu bir commit. Varsayılan: `HEAD`. |
|
|
68
|
+
| `--setup <cmd>` / `--no-setup` | Kurulum komutu, bağımlılık dosyaları değişince yeniden çalışır. Varsayılan: otomatik. |
|
|
69
|
+
| `--timeout <sec>` | Her test çalıştırması için süre sınırı. Varsayılan: 600, ajanın yazdığı testlerde 300. |
|
|
70
|
+
| `--report <file>` | Ayrıca issue ya da PR'a yapıştırmaya hazır bir Markdown rapor yaz. |
|
|
71
|
+
| `--json` | Sonucu JSON olarak bas. |
|
|
72
|
+
|
|
73
|
+
Çıkış kodları: `0` bulundu ve kanıtlandı · `1` hata · `2` hata yeniden üretilemedi · `3` çalışan commit bulunamadı · `4` sonuçsuz.
|
|
74
|
+
|
|
75
|
+
## Bilmekte fayda var
|
|
76
|
+
|
|
77
|
+
- Node.js 20+ ve git gerekir. Başka bağımlılık yok.
|
|
78
|
+
- Test, projenin kullandığı herhangi bir dilde olabilir. Otomatik bağımlılık kurulumu JavaScript paket yöneticilerini kapsar; diğerleri için `--setup` kullan.
|
|
79
|
+
- Ajanla bir çalıştırma genelde iki istek yapar: biri testi yazmak, biri suçlu commit'i açıklamak için (sonuç tutmazsa her biri bir kez daha denenir). Demo reposunda tüm süreç 30–60 saniye sürer.
|
|
80
|
+
- Proje lockfile commit'lemiyorsa eski commit'ler bağımlılıklarının bugünkü sürümleriyle kurulur. Bir bağımlılığın sürümüne bağlı hatalarda yanlış commit bulunabilir.
|
|
81
|
+
- İş bitince ajanın yazdığı test ve düzeltme `.git/whobrokeit/` içinde durur, çalışma klasörünün dışında.
|
|
82
|
+
|
|
83
|
+
## Lisans
|
|
84
|
+
|
|
85
|
+
[MIT](LICENSE)
|
package/dist/agent.js
ADDED
|
@@ -0,0 +1,183 @@
|
|
|
1
|
+
import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises';
|
|
2
|
+
import { tmpdir } from 'node:os';
|
|
3
|
+
import { isAbsolute, join, relative, sep } from 'node:path';
|
|
4
|
+
import { UserError } from './errors.js';
|
|
5
|
+
import { runCommand } from './exec.js';
|
|
6
|
+
const PRESETS = ['claude', 'codex'];
|
|
7
|
+
const LABELS = { claude: 'Claude Code', codex: 'Codex' };
|
|
8
|
+
/**
|
|
9
|
+
* `undefined` picks the first installed preset. A preset name picks that one.
|
|
10
|
+
* Anything else is a custom command: it gets the prompt on stdin and answers on stdout.
|
|
11
|
+
*/
|
|
12
|
+
export async function pickAgent(choice, cwd, model) {
|
|
13
|
+
if (model !== undefined && !/^[\w.:/@[\]-]+$/.test(model))
|
|
14
|
+
throw new UserError(`--model ${model} is not a valid model name.`);
|
|
15
|
+
if (choice !== undefined && !PRESETS.includes(choice)) {
|
|
16
|
+
if (model !== undefined)
|
|
17
|
+
throw new UserError('--model only works with --agent claude or --agent codex.');
|
|
18
|
+
return customAgent(choice);
|
|
19
|
+
}
|
|
20
|
+
const candidates = choice ? [choice] : [...PRESETS];
|
|
21
|
+
for (const name of candidates) {
|
|
22
|
+
const probe = await runCommand(`${name} --version`, cwd, 30_000);
|
|
23
|
+
if (probe.code === 0)
|
|
24
|
+
return name === 'claude' ? claudeAgent(model) : codexAgent(model);
|
|
25
|
+
}
|
|
26
|
+
if (choice) {
|
|
27
|
+
throw new UserError(`${LABELS[choice]} (\`${choice}\`) is not installed or not on your PATH.`);
|
|
28
|
+
}
|
|
29
|
+
throw new UserError('Describing a bug in plain words needs Claude Code or Codex installed.', 'Install one of them, use --agent "<command>" for another tool, or pass your own test with --test "<command>".');
|
|
30
|
+
}
|
|
31
|
+
function claudeAgent(model) {
|
|
32
|
+
const label = LABELS.claude;
|
|
33
|
+
const command = [
|
|
34
|
+
'claude -p',
|
|
35
|
+
'--output-format stream-json --verbose',
|
|
36
|
+
// Read-only tools, no hooks, plugins or MCP servers, nothing left in the session list.
|
|
37
|
+
'--tools Read,Grep,Glob',
|
|
38
|
+
'--permission-prompts none',
|
|
39
|
+
'--safe-mode --strict-mcp-config',
|
|
40
|
+
'--no-session-persistence',
|
|
41
|
+
model ? `--model ${model}` : '',
|
|
42
|
+
]
|
|
43
|
+
.filter(Boolean)
|
|
44
|
+
.join(' ');
|
|
45
|
+
return {
|
|
46
|
+
label,
|
|
47
|
+
async ask(prompt, cwd, { timeoutMs, onActivity }) {
|
|
48
|
+
let result;
|
|
49
|
+
const lines = lineSplitter((line) => {
|
|
50
|
+
if (!line.startsWith('{'))
|
|
51
|
+
return;
|
|
52
|
+
let event;
|
|
53
|
+
try {
|
|
54
|
+
event = JSON.parse(line);
|
|
55
|
+
}
|
|
56
|
+
catch {
|
|
57
|
+
return;
|
|
58
|
+
}
|
|
59
|
+
if (event.type === 'assistant') {
|
|
60
|
+
for (const part of event.message?.content ?? []) {
|
|
61
|
+
if (part.type === 'tool_use')
|
|
62
|
+
onActivity?.(describeTool(part.name, part.input ?? {}, cwd));
|
|
63
|
+
}
|
|
64
|
+
}
|
|
65
|
+
else if (event.type === 'result') {
|
|
66
|
+
result = { text: event.result ?? '', isError: Boolean(event.is_error), subtype: event.subtype };
|
|
67
|
+
}
|
|
68
|
+
});
|
|
69
|
+
// The user's `language` setting survives --safe-mode and beats the prompt's "write in the bug
|
|
70
|
+
// report's language" rule, so a Turkish setting turned English reports into Turkish answers.
|
|
71
|
+
const dir = await mkdtemp(join(tmpdir(), 'whobrokeit-claude-'));
|
|
72
|
+
const settings = join(dir, 'settings.json');
|
|
73
|
+
let run;
|
|
74
|
+
try {
|
|
75
|
+
await writeFile(settings, JSON.stringify({ language: '' }));
|
|
76
|
+
run = await runCommand(`${command} --settings "${settings}"`, cwd, timeoutMs, { input: prompt, onStdout: lines.push });
|
|
77
|
+
}
|
|
78
|
+
finally {
|
|
79
|
+
await rm(dir, { recursive: true, force: true }).catch(() => { });
|
|
80
|
+
}
|
|
81
|
+
lines.flush();
|
|
82
|
+
failIfBroken(label, run, timeoutMs, result === undefined);
|
|
83
|
+
if (result?.isError) {
|
|
84
|
+
throw new UserError(`${label} stopped with an error: ${result.text || result.subtype || 'unknown error'}`);
|
|
85
|
+
}
|
|
86
|
+
return result?.text ?? '';
|
|
87
|
+
},
|
|
88
|
+
};
|
|
89
|
+
}
|
|
90
|
+
function codexAgent(model) {
|
|
91
|
+
const label = LABELS.codex;
|
|
92
|
+
return {
|
|
93
|
+
label,
|
|
94
|
+
async ask(prompt, cwd, { timeoutMs }) {
|
|
95
|
+
const dir = await mkdtemp(join(tmpdir(), 'whobrokeit-codex-'));
|
|
96
|
+
const answerFile = join(dir, 'answer.txt');
|
|
97
|
+
try {
|
|
98
|
+
const command = [
|
|
99
|
+
'codex exec',
|
|
100
|
+
'--sandbox read-only --skip-git-repo-check --ephemeral --color never',
|
|
101
|
+
model ? `-m ${model}` : '',
|
|
102
|
+
`-o "${answerFile}"`,
|
|
103
|
+
'-',
|
|
104
|
+
]
|
|
105
|
+
.filter(Boolean)
|
|
106
|
+
.join(' ');
|
|
107
|
+
const run = await runCommand(command, cwd, timeoutMs, { input: prompt });
|
|
108
|
+
const answer = await readFile(answerFile, 'utf8').catch(() => null);
|
|
109
|
+
failIfBroken(label, run, timeoutMs, answer === null);
|
|
110
|
+
return answer ?? '';
|
|
111
|
+
}
|
|
112
|
+
finally {
|
|
113
|
+
await rm(dir, { recursive: true, force: true }).catch(() => { });
|
|
114
|
+
}
|
|
115
|
+
},
|
|
116
|
+
};
|
|
117
|
+
}
|
|
118
|
+
function customAgent(command) {
|
|
119
|
+
return {
|
|
120
|
+
label: command,
|
|
121
|
+
async ask(prompt, cwd, { timeoutMs }) {
|
|
122
|
+
let stdout = '';
|
|
123
|
+
const run = await runCommand(command, cwd, timeoutMs, { input: prompt, onStdout: (text) => (stdout += text) });
|
|
124
|
+
failIfBroken(command, run, timeoutMs, false);
|
|
125
|
+
return stdout;
|
|
126
|
+
},
|
|
127
|
+
};
|
|
128
|
+
}
|
|
129
|
+
function failIfBroken(label, run, timeoutMs, noAnswer) {
|
|
130
|
+
if (run.timedOut) {
|
|
131
|
+
throw new UserError(`${label} did not answer within ${Math.round(timeoutMs / 60_000)} minutes.`);
|
|
132
|
+
}
|
|
133
|
+
if (run.code === 0 && !noAnswer)
|
|
134
|
+
return;
|
|
135
|
+
// Drop the JSON event stream; keep what a human would read.
|
|
136
|
+
const said = run.output
|
|
137
|
+
.split('\n')
|
|
138
|
+
.map((line) => line.trim())
|
|
139
|
+
.filter((line) => line && !line.startsWith('{'))
|
|
140
|
+
.slice(-8)
|
|
141
|
+
.join('\n');
|
|
142
|
+
throw new UserError(`${label} failed (exit code ${run.code}).${said ? `\n${said}` : ''}`, 'Check that it works on its own in this terminal (is it logged in?), or pass your own test with --test.');
|
|
143
|
+
}
|
|
144
|
+
function describeTool(name, input, cwd) {
|
|
145
|
+
const str = (key) => (typeof input[key] === 'string' ? input[key] : '');
|
|
146
|
+
const where = (path) => {
|
|
147
|
+
if (!path)
|
|
148
|
+
return '';
|
|
149
|
+
const rel = isAbsolute(path) ? relative(cwd, path) : path;
|
|
150
|
+
if (!rel)
|
|
151
|
+
return '.';
|
|
152
|
+
return rel.startsWith('..') ? path : rel.split(sep).join('/');
|
|
153
|
+
};
|
|
154
|
+
switch (name) {
|
|
155
|
+
case 'Read':
|
|
156
|
+
return `reading ${where(str('file_path'))}`;
|
|
157
|
+
case 'Grep':
|
|
158
|
+
return `searching for "${str('pattern')}"${str('path') ? ` in ${where(str('path'))}` : ''}`;
|
|
159
|
+
case 'Glob':
|
|
160
|
+
return `looking for ${str('pattern')}${str('path') ? ` in ${where(str('path'))}` : ''}`;
|
|
161
|
+
default:
|
|
162
|
+
return `using ${name ?? 'a tool'}`;
|
|
163
|
+
}
|
|
164
|
+
}
|
|
165
|
+
/** Turns arbitrary stdout chunks into whole lines. */
|
|
166
|
+
function lineSplitter(onLine) {
|
|
167
|
+
let buffer = '';
|
|
168
|
+
return {
|
|
169
|
+
push: (text) => {
|
|
170
|
+
buffer += text;
|
|
171
|
+
let newline;
|
|
172
|
+
while ((newline = buffer.indexOf('\n')) !== -1) {
|
|
173
|
+
onLine(buffer.slice(0, newline).trim());
|
|
174
|
+
buffer = buffer.slice(newline + 1);
|
|
175
|
+
}
|
|
176
|
+
},
|
|
177
|
+
flush: () => {
|
|
178
|
+
if (buffer.trim())
|
|
179
|
+
onLine(buffer.trim());
|
|
180
|
+
buffer = '';
|
|
181
|
+
},
|
|
182
|
+
};
|
|
183
|
+
}
|
package/dist/cli.js
ADDED
|
@@ -0,0 +1,230 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
import { readFileSync } from 'node:fs';
|
|
3
|
+
import { mkdir, mkdtemp, writeFile } from 'node:fs/promises';
|
|
4
|
+
import { tmpdir } from 'node:os';
|
|
5
|
+
import { basename, dirname, join, relative, resolve, sep } from 'node:path';
|
|
6
|
+
import { parseArgs } from 'node:util';
|
|
7
|
+
import { pickAgent } from './agent.js';
|
|
8
|
+
import { DEMO_BUG_REPORT, DEMO_REPRO, DEMO_TEST, createDemoRepo } from './demo.js';
|
|
9
|
+
import { UserError, findCulprit } from './engine.js';
|
|
10
|
+
import { killActiveCommands } from './exec.js';
|
|
11
|
+
import { git } from './git.js';
|
|
12
|
+
import { colorFor, liveReporter, renderOutcome, toMarkdown } from './report.js';
|
|
13
|
+
const VERSION = JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8'))
|
|
14
|
+
.version;
|
|
15
|
+
const HELP = `whobrokeit: find the commit that broke it, and prove it.
|
|
16
|
+
|
|
17
|
+
Usage
|
|
18
|
+
whobrokeit "<what is broken>" [options] an AI agent writes a test that shows the bug
|
|
19
|
+
whobrokeit --test "<command>" [options] bring your own failing test
|
|
20
|
+
whobrokeit demo [dir] [--ai] try it on a generated repo with a hidden bug
|
|
21
|
+
|
|
22
|
+
Options
|
|
23
|
+
-t, --test <cmd> A command that fails while the bug is present.
|
|
24
|
+
Exit 0 = works, 125 = can't tell (skip), anything else = broken.
|
|
25
|
+
--agent <name> Who writes the test: claude, codex, or any command that reads a
|
|
26
|
+
prompt on stdin and answers on stdout. Default: claude, else codex.
|
|
27
|
+
--model <name> Model for the agent, e.g. sonnet.
|
|
28
|
+
--explain Also explain the culprit and test a fix when you use --test.
|
|
29
|
+
--no-explain Skip the explanation and fix when you describe a bug.
|
|
30
|
+
-w, --with <file> Copy this file from your working copy into every commit tested.
|
|
31
|
+
Use it for a repro test you haven't committed. Repeatable.
|
|
32
|
+
-g, --good <ref> A commit where it still worked. Default: found automatically.
|
|
33
|
+
-b, --bad <ref> A commit where it is broken. Default: HEAD.
|
|
34
|
+
--setup <cmd> Install command, re-run when dependency files change. Default: auto (npm/pnpm/yarn/bun).
|
|
35
|
+
--no-setup Never install dependencies.
|
|
36
|
+
--timeout <sec> Time limit for each test run. Default: 600, or 300 for agent-written tests.
|
|
37
|
+
--report <file> Also write a Markdown report.
|
|
38
|
+
--json Print the result as JSON.
|
|
39
|
+
-h, --help
|
|
40
|
+
-v, --version
|
|
41
|
+
|
|
42
|
+
Your working copy is never touched: every check runs in a temporary git worktree.
|
|
43
|
+
The agent can only read the code. whobrokeit runs the test it writes, and trusts it
|
|
44
|
+
only after seeing it fail where the bug is. A suggested fix is shown only after the
|
|
45
|
+
test passes with it.
|
|
46
|
+
|
|
47
|
+
Exit codes
|
|
48
|
+
0 found and proven · 1 error · 2 bug not reproduced · 3 no working commit found · 4 inconclusive
|
|
49
|
+
`;
|
|
50
|
+
/**
|
|
51
|
+
* Keeps the agent-written test and the fix in .git/whobrokeit/, next to git's own state: out
|
|
52
|
+
* of the working copy, but easy to open, copy or `git apply`. Returns a path to show, or
|
|
53
|
+
* undefined if it could not be saved.
|
|
54
|
+
*/
|
|
55
|
+
async function saveFile(cwd, name, content) {
|
|
56
|
+
try {
|
|
57
|
+
const gitDir = resolve(cwd, await git(['rev-parse', '--git-common-dir'], cwd));
|
|
58
|
+
const target = join(gitDir, 'whobrokeit', basename(name));
|
|
59
|
+
await mkdir(dirname(target), { recursive: true });
|
|
60
|
+
await writeFile(target, content);
|
|
61
|
+
const shown = relative(process.cwd(), target);
|
|
62
|
+
return shown.startsWith('..') ? target : shown.split(sep).join('/');
|
|
63
|
+
}
|
|
64
|
+
catch {
|
|
65
|
+
return undefined;
|
|
66
|
+
}
|
|
67
|
+
}
|
|
68
|
+
function exitCode(outcome) {
|
|
69
|
+
switch (outcome.kind) {
|
|
70
|
+
case 'found':
|
|
71
|
+
return outcome.proven ? 0 : 4;
|
|
72
|
+
case 'not-reproduced':
|
|
73
|
+
case 'cannot-test':
|
|
74
|
+
case 'no-repro':
|
|
75
|
+
return 2;
|
|
76
|
+
case 'no-good-commit':
|
|
77
|
+
case 'good-fails':
|
|
78
|
+
return 3;
|
|
79
|
+
case 'ambiguous':
|
|
80
|
+
return 4;
|
|
81
|
+
}
|
|
82
|
+
}
|
|
83
|
+
async function main(argv) {
|
|
84
|
+
const { values, positionals } = parseArgs({
|
|
85
|
+
args: argv,
|
|
86
|
+
allowPositionals: true,
|
|
87
|
+
options: {
|
|
88
|
+
test: { type: 'string', short: 't' },
|
|
89
|
+
with: { type: 'string', short: 'w', multiple: true },
|
|
90
|
+
good: { type: 'string', short: 'g' },
|
|
91
|
+
bad: { type: 'string', short: 'b' },
|
|
92
|
+
setup: { type: 'string' },
|
|
93
|
+
'no-setup': { type: 'boolean' },
|
|
94
|
+
agent: { type: 'string' },
|
|
95
|
+
model: { type: 'string' },
|
|
96
|
+
ai: { type: 'boolean' },
|
|
97
|
+
explain: { type: 'boolean' },
|
|
98
|
+
'no-explain': { type: 'boolean' },
|
|
99
|
+
timeout: { type: 'string' },
|
|
100
|
+
report: { type: 'string' },
|
|
101
|
+
json: { type: 'boolean' },
|
|
102
|
+
help: { type: 'boolean', short: 'h' },
|
|
103
|
+
version: { type: 'boolean', short: 'v' },
|
|
104
|
+
},
|
|
105
|
+
});
|
|
106
|
+
if (values.help) {
|
|
107
|
+
process.stdout.write(HELP);
|
|
108
|
+
return 0;
|
|
109
|
+
}
|
|
110
|
+
if (values.version) {
|
|
111
|
+
process.stdout.write(`${VERSION}\n`);
|
|
112
|
+
return 0;
|
|
113
|
+
}
|
|
114
|
+
const progress = colorFor(process.stderr);
|
|
115
|
+
let cwd = process.cwd();
|
|
116
|
+
let test = values.test;
|
|
117
|
+
let bug = positionals.join(' ').trim() || undefined;
|
|
118
|
+
let withFiles = values.with ?? [];
|
|
119
|
+
if (positionals[0] === 'demo') {
|
|
120
|
+
const dir = positionals[1] ? resolve(positionals[1]) : await mkdtemp(join(tmpdir(), 'whobrokeit-demo-'));
|
|
121
|
+
await createDemoRepo(dir);
|
|
122
|
+
process.stderr.write(`\n${progress.bold('Demo repo:')} ${dir}\n` +
|
|
123
|
+
`24 commits of a tiny shopping-cart library. A user reports: "${DEMO_BUG_REPORT}"\n`);
|
|
124
|
+
cwd = dir;
|
|
125
|
+
if (values.ai) {
|
|
126
|
+
bug = DEMO_BUG_REPORT;
|
|
127
|
+
test = undefined;
|
|
128
|
+
}
|
|
129
|
+
else {
|
|
130
|
+
process.stderr.write(progress.dim(`The repro test ${DEMO_REPRO} is not committed; --with brings it into every commit.\n` +
|
|
131
|
+
'Add --ai to have an AI agent write the test from the bug report instead.\n'));
|
|
132
|
+
bug = undefined;
|
|
133
|
+
test = DEMO_TEST;
|
|
134
|
+
withFiles = [DEMO_REPRO];
|
|
135
|
+
}
|
|
136
|
+
}
|
|
137
|
+
else if (values.ai) {
|
|
138
|
+
throw new UserError('--ai only works with `whobrokeit demo`. Elsewhere, just describe the bug in quotes.');
|
|
139
|
+
}
|
|
140
|
+
if (test && bug) {
|
|
141
|
+
throw new UserError('Give either a bug description or --test, not both.');
|
|
142
|
+
}
|
|
143
|
+
if (!test && !bug) {
|
|
144
|
+
process.stdout.write(HELP);
|
|
145
|
+
return 1;
|
|
146
|
+
}
|
|
147
|
+
let timeoutMs;
|
|
148
|
+
if (values.timeout !== undefined) {
|
|
149
|
+
const seconds = Number(values.timeout);
|
|
150
|
+
if (!Number.isFinite(seconds) || seconds <= 0)
|
|
151
|
+
throw new UserError('--timeout must be a positive number of seconds.');
|
|
152
|
+
timeoutMs = seconds * 1000;
|
|
153
|
+
}
|
|
154
|
+
const controller = new AbortController();
|
|
155
|
+
const onSigint = () => {
|
|
156
|
+
if (controller.signal.aborted)
|
|
157
|
+
process.exit(130);
|
|
158
|
+
process.stderr.write('\nStopping and cleaning up…\n');
|
|
159
|
+
controller.abort();
|
|
160
|
+
killActiveCommands();
|
|
161
|
+
};
|
|
162
|
+
process.on('SIGINT', onSigint);
|
|
163
|
+
if (values.explain && values['no-explain'])
|
|
164
|
+
throw new UserError('Give either --explain or --no-explain, not both.');
|
|
165
|
+
const wantsExplain = bug ? !values['no-explain'] : Boolean(values.explain);
|
|
166
|
+
try {
|
|
167
|
+
let describe;
|
|
168
|
+
let explain;
|
|
169
|
+
if (bug || wantsExplain) {
|
|
170
|
+
const agent = await pickAgent(values.agent, cwd, values.model);
|
|
171
|
+
if (bug)
|
|
172
|
+
describe = { bug, agent };
|
|
173
|
+
if (wantsExplain)
|
|
174
|
+
explain = { agent };
|
|
175
|
+
}
|
|
176
|
+
else if (values.agent || values.model) {
|
|
177
|
+
throw new UserError('--agent and --model need a bug description or --explain; with only --test, no agent is used.');
|
|
178
|
+
}
|
|
179
|
+
const outcome = await findCulprit({
|
|
180
|
+
cwd,
|
|
181
|
+
test,
|
|
182
|
+
describe,
|
|
183
|
+
explain,
|
|
184
|
+
withFiles,
|
|
185
|
+
good: values.good,
|
|
186
|
+
bad: values.bad,
|
|
187
|
+
setup: values['no-setup'] ? false : values.setup,
|
|
188
|
+
timeoutMs,
|
|
189
|
+
signal: controller.signal,
|
|
190
|
+
onEvent: values.json ? undefined : liveReporter(process.stderr, progress, test ? { test } : { bug: bug ?? '' }),
|
|
191
|
+
});
|
|
192
|
+
const fix = outcome.kind === 'found' ? outcome.explanation?.fix : undefined;
|
|
193
|
+
const saved = {
|
|
194
|
+
test: outcome.repro ? await saveFile(cwd, outcome.repro.path, outcome.repro.code) : undefined,
|
|
195
|
+
fix: fix?.verified ? await saveFile(cwd, 'fix.patch', fix.patch) : undefined,
|
|
196
|
+
};
|
|
197
|
+
if (values.report)
|
|
198
|
+
await writeFile(resolve(values.report), toMarkdown(outcome, test ?? ''));
|
|
199
|
+
if (values.json) {
|
|
200
|
+
process.stdout.write(`${JSON.stringify({ ...outcome, savedTest: saved.test, savedFix: saved.fix }, null, 2)}\n`);
|
|
201
|
+
}
|
|
202
|
+
else {
|
|
203
|
+
process.stdout.write(renderOutcome(outcome, colorFor(process.stdout), process.stdout.columns || 100, saved));
|
|
204
|
+
}
|
|
205
|
+
return exitCode(outcome);
|
|
206
|
+
}
|
|
207
|
+
catch (err) {
|
|
208
|
+
if (controller.signal.aborted)
|
|
209
|
+
return 130;
|
|
210
|
+
throw err;
|
|
211
|
+
}
|
|
212
|
+
finally {
|
|
213
|
+
process.off('SIGINT', onSigint);
|
|
214
|
+
}
|
|
215
|
+
}
|
|
216
|
+
main(process.argv.slice(2)).then((code) => {
|
|
217
|
+
process.exitCode = code;
|
|
218
|
+
}, (err) => {
|
|
219
|
+
const c = colorFor(process.stderr);
|
|
220
|
+
if (err instanceof UserError) {
|
|
221
|
+
process.stderr.write(`${c.red('error:')} ${err.message}\n${err.hint ? `${c.dim(err.hint)}\n` : ''}`);
|
|
222
|
+
}
|
|
223
|
+
else if (err instanceof Error && 'code' in err && String(err.code).startsWith('ERR_PARSE_ARGS')) {
|
|
224
|
+
process.stderr.write(`${c.red('error:')} ${err.message}\n${c.dim('Run whobrokeit --help for usage.')}\n`);
|
|
225
|
+
}
|
|
226
|
+
else {
|
|
227
|
+
process.stderr.write(`${c.red('Unexpected error:')} ${err instanceof Error ? (err.stack ?? err.message) : String(err)}\n`);
|
|
228
|
+
}
|
|
229
|
+
process.exitCode = 1;
|
|
230
|
+
});
|