bite-check 0.4.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- bite_check-0.4.1/LICENSE +21 -0
- bite_check-0.4.1/PKG-INFO +194 -0
- bite_check-0.4.1/README.md +169 -0
- bite_check-0.4.1/bite.py +817 -0
- bite_check-0.4.1/bite_check.egg-info/PKG-INFO +194 -0
- bite_check-0.4.1/bite_check.egg-info/SOURCES.txt +13 -0
- bite_check-0.4.1/bite_check.egg-info/dependency_links.txt +1 -0
- bite_check-0.4.1/bite_check.egg-info/entry_points.txt +2 -0
- bite_check-0.4.1/bite_check.egg-info/requires.txt +6 -0
- bite_check-0.4.1/bite_check.egg-info/top_level.txt +3 -0
- bite_check-0.4.1/pyproject.toml +41 -0
- bite_check-0.4.1/recent.py +1060 -0
- bite_check-0.4.1/repo.py +104 -0
- bite_check-0.4.1/setup.cfg +4 -0
- bite_check-0.4.1/tests/test_selftest.py +11 -0
bite_check-0.4.1/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 connabyss
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,194 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: bite-check
|
|
3
|
+
Version: 0.4.1
|
|
4
|
+
Summary: Checks whether the "fixed" in your coding agent's recent commits is actually true.
|
|
5
|
+
License-Expression: MIT
|
|
6
|
+
Project-URL: Homepage, https://github.com/connabyss/bite
|
|
7
|
+
Project-URL: Source, https://github.com/connabyss/bite
|
|
8
|
+
Project-URL: Issues, https://github.com/connabyss/bite/issues
|
|
9
|
+
Keywords: testing,ai,coding-agent,claude,pytest,regression,git
|
|
10
|
+
Classifier: Development Status :: 4 - Beta
|
|
11
|
+
Classifier: Environment :: Console
|
|
12
|
+
Classifier: Intended Audience :: Developers
|
|
13
|
+
Classifier: Operating System :: OS Independent
|
|
14
|
+
Classifier: Programming Language :: Python :: 3
|
|
15
|
+
Classifier: Topic :: Software Development :: Quality Assurance
|
|
16
|
+
Classifier: Topic :: Software Development :: Testing
|
|
17
|
+
Requires-Python: >=3.10
|
|
18
|
+
Description-Content-Type: text/markdown
|
|
19
|
+
License-File: LICENSE
|
|
20
|
+
Provides-Extra: paid
|
|
21
|
+
Requires-Dist: claude-agent-sdk; extra == "paid"
|
|
22
|
+
Provides-Extra: test
|
|
23
|
+
Requires-Dist: pytest; extra == "test"
|
|
24
|
+
Dynamic: license-file
|
|
25
|
+
|
|
26
|
+
<p>
|
|
27
|
+
<picture>
|
|
28
|
+
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/connabyss/bite/main/assets/bite-header-dark.png">
|
|
29
|
+
<img alt="Bite" src="https://raw.githubusercontent.com/connabyss/bite/main/assets/bite-header-light.png" width="300">
|
|
30
|
+
</picture>
|
|
31
|
+
</p>
|
|
32
|
+
|
|
33
|
+
[](https://github.com/marketplace/actions/bite-free-check)
|
|
34
|
+
|
|
35
|
+
**Bite checks whether an AI coding agent's "fixed" is actually true.**
|
|
36
|
+
|
|
37
|
+
> "Reverting the claimed fix and making the change prove itself under a real test is exactly the kind of check that
|
|
38
|
+
> should exist for every "fixed" commit."
|
|
39
|
+
>
|
|
40
|
+
> — [Ivan Zdravkov](https://zdravkov.info), [on DEV](https://dev.to/izdravkov/comment/3jdej)
|
|
41
|
+
|
|
42
|
+
When a commit says it fixed something, Bite writes one small test of that exact claim, runs it itself on the commit and on today's code, and shows the result only if a second, independent check agrees the test is fair. If the test fails, the "fixed" wasn't.
|
|
43
|
+
|
|
44
|
+
## Try it in a minute
|
|
45
|
+
|
|
46
|
+
Install once with [pipx](https://pipx.pypa.io), which gives Bite its own space so there's no venv to set up or
|
|
47
|
+
activate:
|
|
48
|
+
|
|
49
|
+
```
|
|
50
|
+
python -m pip install --user pipx
|
|
51
|
+
python -m pipx ensurepath
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
Close the terminal window and open a new one, then:
|
|
55
|
+
|
|
56
|
+
```
|
|
57
|
+
pipx install bite-check
|
|
58
|
+
cd /path/to/your-repo
|
|
59
|
+
bite --free
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
`bite` now works in any terminal window. `bite --free` needs no key and costs nothing. It takes each recent
|
|
63
|
+
change back out, runs your tests, and lists the changes no test guards, plus any new test that still passes
|
|
64
|
+
without its commit's code (it tests nothing). If something it needs is missing, like git or pytest, it says
|
|
65
|
+
so and gives you the line that installs it. Each run ends with a one-page report, `bite_proof/report.html`, to open in your browser. To update Bite later: `pipx upgrade bite-check`.
|
|
66
|
+
|
|
67
|
+
To test the claims themselves (the paid step), add one extra package and set `ANTHROPIC_API_KEY`, then run `bite`:
|
|
68
|
+
|
|
69
|
+
```
|
|
70
|
+
pipx inject bite-check claude-agent-sdk
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
This step uses your own API key, and `--budget` caps what a run spends ($1 by default). It runs on Anthropic's
|
|
74
|
+
Agent SDK, so it needs an API key and never uses a Claude login or plan (Anthropic's Agent SDK rule:
|
|
75
|
+
https://code.claude.com/docs/en/agent-sdk/overview). The commits Bite checks can come from any coding AI or person.
|
|
76
|
+
|
|
77
|
+
Works with pytest, npm test, cargo test, go test and dotnet test. Settings and limits: [AGENTS.md](https://github.com/connabyss/bite/blob/main/AGENTS.md),
|
|
78
|
+
written so your coding AI (Claude Code, Codex, Cursor, Copilot) can set Bite up for you. To work on Bite itself: clone it, `pip install -e .[test]`, then `python -m pytest`.
|
|
79
|
+
|
|
80
|
+
## Run it on every pull request
|
|
81
|
+
|
|
82
|
+
Add this file to your repo as `.github/workflows/bite.yml`. On each pull request it runs the free check on that
|
|
83
|
+
pull request's commits and shows a pass or fail check next to your other CI. It fails when a commit's change
|
|
84
|
+
can be taken back out and your tests still pass. No key, no cost. It's also listed on the GitHub Marketplace as
|
|
85
|
+
[Bite free check](https://github.com/marketplace/actions/bite-free-check).
|
|
86
|
+
|
|
87
|
+
```yaml
|
|
88
|
+
name: bite
|
|
89
|
+
on: pull_request
|
|
90
|
+
permissions:
|
|
91
|
+
contents: read
|
|
92
|
+
pull-requests: write
|
|
93
|
+
jobs:
|
|
94
|
+
bite:
|
|
95
|
+
runs-on: ubuntu-latest
|
|
96
|
+
steps:
|
|
97
|
+
- uses: actions/checkout@v4
|
|
98
|
+
with:
|
|
99
|
+
ref: ${{ github.event.pull_request.head.sha }}
|
|
100
|
+
fetch-depth: 0
|
|
101
|
+
# install what your tests need here, e.g. for Python:
|
|
102
|
+
- uses: actions/setup-python@v5
|
|
103
|
+
- run: python -m pip install pytest -r requirements.txt
|
|
104
|
+
- uses: connabyss/bite@v0.4.0
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
Options go under `with:`: `args` (extra `bite` options, e.g. `--serious`), `max-commits` (default 20),
|
|
108
|
+
`fail-on-gap` (set to `false` to report without failing) and `comment` (set to `false` for no comment). It posts one comment on the pull request that says, per
|
|
109
|
+
commit, whether a test noticed the change being taken out, and updates that same comment on later pushes. The
|
|
110
|
+
rerun steps for each flagged commit are on the job's summary page.
|
|
111
|
+
|
|
112
|
+
## Bite-proof badge
|
|
113
|
+
|
|
114
|
+
When every change Bite checked had a test that noticed it, Bite ends with "Bite-proof. Nice work." and prints a
|
|
115
|
+
badge line you can paste into your README:
|
|
116
|
+
|
|
117
|
+
[](https://github.com/connabyss/bite)
|
|
118
|
+
|
|
119
|
+
Prefer a gentler wording for the warnings? `bite --roast` gives the same results, phrased differently.
|
|
120
|
+
|
|
121
|
+
## The catch that shows what it's for
|
|
122
|
+
|
|
123
|
+
In [fellowapp/prosemirror-py](https://github.com/fellowapp/prosemirror-py), commit `3614c96` (co-authored by Claude) says:
|
|
124
|
+
|
|
125
|
+
> fix: add weakref callback to clean up resolve cache entries
|
|
126
|
+
>
|
|
127
|
+
> The cache used dict keyed by id(doc) with no cleanup, causing a slow memory leak in long-running processes. Added a weakref finalizer callback that removes the cache entry when the doc is garbage collected.
|
|
128
|
+
|
|
129
|
+
Bite wrote this test of that claim and ran it:
|
|
130
|
+
|
|
131
|
+
```python
|
|
132
|
+
def test_cache_entry_removed_after_doc_gc():
|
|
133
|
+
resolvedpos._resolve_cache.clear()
|
|
134
|
+
d = doc(p("hello"))
|
|
135
|
+
ResolvedPos.resolve_cached(d, 2)
|
|
136
|
+
assert len(resolvedpos._resolve_cache) == 1
|
|
137
|
+
del d
|
|
138
|
+
gc.collect()
|
|
139
|
+
assert len(resolvedpos._resolve_cache) == 0 # fails: 1 == 0
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
It fails on that commit and on today's code. The cache keeps each document alive through its cached positions, so the cleanup never runs. After 20,000 documents are resolved and dropped, the cache still holds all 20,000, using 35 MB. The commit looked right, read right, and passed its own tests. Reported with a repro in [#335](https://github.com/fellowapp/prosemirror-py/issues/335); a tested fix is open as [#336](https://github.com/fellowapp/prosemirror-py/pull/336).
|
|
143
|
+
|
|
144
|
+
## Fixed or merged upstream
|
|
145
|
+
|
|
146
|
+
Each bug was reproduced with a standalone script in a clean environment before it was reported.
|
|
147
|
+
|
|
148
|
+
| Project | What happened |
|
|
149
|
+
|---|---|
|
|
150
|
+
| pyclibrary | 7 bugs reported in [#92](https://github.com/MatthieuDartiailh/pyclibrary/issues/92). All 7 fixes reviewed and merged: [#93](https://github.com/MatthieuDartiailh/pyclibrary/pull/93), [#94](https://github.com/MatthieuDartiailh/pyclibrary/pull/94), [#95](https://github.com/MatthieuDartiailh/pyclibrary/pull/95), [#96](https://github.com/MatthieuDartiailh/pyclibrary/pull/96), [#97](https://github.com/MatthieuDartiailh/pyclibrary/pull/97), [#98](https://github.com/MatthieuDartiailh/pyclibrary/pull/98), [#99](https://github.com/MatthieuDartiailh/pyclibrary/pull/99). The maintainer [wrote](https://github.com/MatthieuDartiailh/pyclibrary/issues/92): "Thanks for the small and focused PR very nice to review." |
|
|
151
|
+
| django-tailwind-cli | With one stylesheet, `tailwind watch` hid rebuild messages and build errors. Fix [#232](https://github.com/django-commons/django-tailwind-cli/pull/232) merged and released in [4.8.2](https://github.com/django-commons/django-tailwind-cli/releases/tag/v4.8.2). The maintainer [wrote](https://github.com/django-commons/django-tailwind-cli/pull/232): "Just about to release 4.8.2 with the fix included. Thanks for the contribution." |
|
|
152
|
+
| srptools | 4 bugs reported in [#13](https://github.com/idlesign/srptools/issues/13). The maintainer fixed all 4 and closed it the same day. |
|
|
153
|
+
| resolve-robotics-uri-py | URI lookup depended on the hash seed ([#26](https://github.com/gbionics/resolve-robotics-uri-py/issues/26)). Fixed by the maintainers in [#27](https://github.com/gbionics/resolve-robotics-uri-py/pull/27). The maintainer [wrote](https://github.com/gbionics/resolve-robotics-uri-py/issues/26): "Thanks a lot for the detailed issue report!" |
|
|
154
|
+
| text-to-cad | A Claude commit added a Python 3.11 syntax check that can't catch the exact bug it was written for (nested same-quote f-strings). Bite sent a test that does catch it, [#708](https://github.com/earthtojake/text-to-cad/pull/708), was merged and released in [0.8.2](https://github.com/earthtojake/text-to-cad/releases/tag/v0.8.2); the maintainer then closed the remaining gaps in [#712](https://github.com/earthtojake/text-to-cad/pull/712). |
|
|
155
|
+
| west | Loading a manifest from data with a reserved project name crashed. Fix [#1013](https://github.com/zephyrproject-rtos/west/pull/1013) approved by two maintainers and merged. |
|
|
156
|
+
| midea-lan | A message encoder got the frame length wrong when no padding was needed. Fix [#251](https://github.com/wuwentao/midea-lan/pull/251) merged. The maintainer [wrote](https://github.com/wuwentao/midea-lan/pull/251): "Thank you @connabyss for submitting this PR and for finding and fixing the 8370 framing bug." |
|
|
157
|
+
| mizani | Number labels picked too few decimals when values were close together (`[0, 10, 10.5]` showed as `10, 10`). Fix [#90](https://github.com/has2k1/mizani/pull/90) merged. The maintainer [wrote](https://github.com/has2k1/mizani/pull/90): "@connabyss, thank you." |
|
|
158
|
+
| pamqp | Bite's free check found three past fixes that no test would notice breaking, including one where `Basic.Reject(requeue=False)` was flipped back to `True`. Tests for all three ([#86](https://github.com/gmr/pamqp/pull/86)) merged. |
|
|
159
|
+
|
|
160
|
+
## Sent, waiting for review
|
|
161
|
+
|
|
162
|
+
| Project | Link | What happened |
|
|
163
|
+
|---|---|---|
|
|
164
|
+
| prosemirror-py | [#335](https://github.com/fellowapp/prosemirror-py/issues/335), [#336](https://github.com/fellowapp/prosemirror-py/pull/336), [#337](https://github.com/fellowapp/prosemirror-py/pull/337) | The leak above. A second Claude commit (`1e3339e`) let `Mapping` change a list the caller still owned; fix in #337. |
|
|
165
|
+
| python-vcr-tests | [#18](https://github.com/keboola/python-vcr-tests/issues/18), [#19](https://github.com/keboola/python-vcr-tests/pull/19) | A Claude commit said its database recorder captures every query; only the last per cursor is saved. |
|
|
166
|
+
| ChatSBOM | [#217](https://github.com/WangYihang/ChatSBOM/issues/217) | A Claude commit's stale-result guard is reset on every walk page, so a repo can be collected twice. |
|
|
167
|
+
| llm | [#1828](https://github.com/simonw/llm/pull/1828) | A Claude commit fixed per-turn tool ids in `llm logs`, but its test never made a tool call, so it passed even with the bug. |
|
|
168
|
+
| treg | [#1237](https://github.com/superdesigndev/treg/pull/1237) | A Claude commit's test for an org header passes even with the fix removed, so it guards nothing. |
|
|
169
|
+
| claude-seo | [#392](https://github.com/AgriciDaniel/claude-seo/pull/392) | A Claude commit's `--release` check let a value with a trailing newline through. |
|
|
170
|
+
| marshmallow-fastoneofschema | [#1](https://github.com/Kalepa/marshmallow-fastoneofschema/issues/1), [#2](https://github.com/Kalepa/marshmallow-fastoneofschema/pull/2), [#3](https://github.com/Kalepa/marshmallow-fastoneofschema/pull/3), [#4](https://github.com/Kalepa/marshmallow-fastoneofschema/pull/4) | 3 bugs, one fix each. |
|
|
171
|
+
| django-modeltranslation | [#831](https://github.com/deschler/django-modeltranslation/pull/831) | Nested `auto_populate()` and `fallbacks()` blocks reset the setting to nothing on exit instead of restoring it. |
|
|
172
|
+
| mypy_boto3_builder | [#395](https://github.com/youtype/mypy_boto3_builder/issues/395) | Reported, no reply yet. |
|
|
173
|
+
| drf-pydantic | [#51](https://github.com/georgebv/drf-pydantic/issues/51) | Reported, no reply yet. |
|
|
174
|
+
|
|
175
|
+
## Limits, plainly
|
|
176
|
+
|
|
177
|
+
- **Most claims hold.** Across all rounds so far, Bite tested 137 "fixed" claims that it could run tests for (all from commits made with Claude Code, the AI those rounds looked at). 8 were false (about 1 in 17), and 4 of those were still broken on today's code. On well-reviewed projects it usually finds nothing, and that's the right answer.
|
|
178
|
+
- **Quiet is not a pass.** Bite only shows a failing test that the second check calls fair and real. That second check has thrown out more than 10 failing tests that weren't. A claim that holds proves that one claim, not that the commit has no other bugs.
|
|
179
|
+
- **It needs the project's tests to run.** If they don't run on a clean copy, Bite can't check that commit.
|
|
180
|
+
- **Bug hunting is weaker than claim checking.** In a 10-project overnight hunt, two blind AI judges rated 21 of 22 proven fixes as real bugs and none as wrong, but Bite found only 1 of the 10 bugs the maintainers fixed soon after.
|
|
181
|
+
- **Small samples, AI judges.** The judges read code without running it. The human checks are the maintainers' fixes and merges listed above.
|
|
182
|
+
|
|
183
|
+
## Contributing
|
|
184
|
+
|
|
185
|
+
Bug reports, fixes and AI-assisted PRs are all welcome. See [CONTRIBUTING.md](https://github.com/connabyss/bite/blob/main/CONTRIBUTING.md). If Bite flagged
|
|
186
|
+
something wrongly, or missed a claim you know was false, please [open an issue](https://github.com/connabyss/bite/issues/new/choose).
|
|
187
|
+
|
|
188
|
+
Thank you to Bite's first contributors:
|
|
189
|
+
|
|
190
|
+
- [@snehaapratap](https://github.com/snehaapratap) made the setup error messages point to steps that exist ([#18](https://github.com/connabyss/bite/pull/18)).
|
|
191
|
+
- [@gauravjangra3165](https://github.com/gauravjangra3165) added macOS to the tests ([#19](https://github.com/connabyss/bite/pull/19)).
|
|
192
|
+
- [@bfs2021](https://github.com/bfs2021) made the "check it yourself" steps work for commits that add a new file ([#37](https://github.com/connabyss/bite/pull/37), fixing [#27](https://github.com/connabyss/bite/issues/27)).
|
|
193
|
+
|
|
194
|
+
MIT licensed.
|
|
@@ -0,0 +1,169 @@
|
|
|
1
|
+
<p>
|
|
2
|
+
<picture>
|
|
3
|
+
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/connabyss/bite/main/assets/bite-header-dark.png">
|
|
4
|
+
<img alt="Bite" src="https://raw.githubusercontent.com/connabyss/bite/main/assets/bite-header-light.png" width="300">
|
|
5
|
+
</picture>
|
|
6
|
+
</p>
|
|
7
|
+
|
|
8
|
+
[](https://github.com/marketplace/actions/bite-free-check)
|
|
9
|
+
|
|
10
|
+
**Bite checks whether an AI coding agent's "fixed" is actually true.**
|
|
11
|
+
|
|
12
|
+
> "Reverting the claimed fix and making the change prove itself under a real test is exactly the kind of check that
|
|
13
|
+
> should exist for every "fixed" commit."
|
|
14
|
+
>
|
|
15
|
+
> — [Ivan Zdravkov](https://zdravkov.info), [on DEV](https://dev.to/izdravkov/comment/3jdej)
|
|
16
|
+
|
|
17
|
+
When a commit says it fixed something, Bite writes one small test of that exact claim, runs it itself on the commit and on today's code, and shows the result only if a second, independent check agrees the test is fair. If the test fails, the "fixed" wasn't.
|
|
18
|
+
|
|
19
|
+
## Try it in a minute
|
|
20
|
+
|
|
21
|
+
Install once with [pipx](https://pipx.pypa.io), which gives Bite its own space so there's no venv to set up or
|
|
22
|
+
activate:
|
|
23
|
+
|
|
24
|
+
```
|
|
25
|
+
python -m pip install --user pipx
|
|
26
|
+
python -m pipx ensurepath
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
Close the terminal window and open a new one, then:
|
|
30
|
+
|
|
31
|
+
```
|
|
32
|
+
pipx install bite-check
|
|
33
|
+
cd /path/to/your-repo
|
|
34
|
+
bite --free
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
`bite` now works in any terminal window. `bite --free` needs no key and costs nothing. It takes each recent
|
|
38
|
+
change back out, runs your tests, and lists the changes no test guards, plus any new test that still passes
|
|
39
|
+
without its commit's code (it tests nothing). If something it needs is missing, like git or pytest, it says
|
|
40
|
+
so and gives you the line that installs it. Each run ends with a one-page report, `bite_proof/report.html`, to open in your browser. To update Bite later: `pipx upgrade bite-check`.
|
|
41
|
+
|
|
42
|
+
To test the claims themselves (the paid step), add one extra package and set `ANTHROPIC_API_KEY`, then run `bite`:
|
|
43
|
+
|
|
44
|
+
```
|
|
45
|
+
pipx inject bite-check claude-agent-sdk
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
This step uses your own API key, and `--budget` caps what a run spends ($1 by default). It runs on Anthropic's
|
|
49
|
+
Agent SDK, so it needs an API key and never uses a Claude login or plan (Anthropic's Agent SDK rule:
|
|
50
|
+
https://code.claude.com/docs/en/agent-sdk/overview). The commits Bite checks can come from any coding AI or person.
|
|
51
|
+
|
|
52
|
+
Works with pytest, npm test, cargo test, go test and dotnet test. Settings and limits: [AGENTS.md](https://github.com/connabyss/bite/blob/main/AGENTS.md),
|
|
53
|
+
written so your coding AI (Claude Code, Codex, Cursor, Copilot) can set Bite up for you. To work on Bite itself: clone it, `pip install -e .[test]`, then `python -m pytest`.
|
|
54
|
+
|
|
55
|
+
## Run it on every pull request
|
|
56
|
+
|
|
57
|
+
Add this file to your repo as `.github/workflows/bite.yml`. On each pull request it runs the free check on that
|
|
58
|
+
pull request's commits and shows a pass or fail check next to your other CI. It fails when a commit's change
|
|
59
|
+
can be taken back out and your tests still pass. No key, no cost. It's also listed on the GitHub Marketplace as
|
|
60
|
+
[Bite free check](https://github.com/marketplace/actions/bite-free-check).
|
|
61
|
+
|
|
62
|
+
```yaml
|
|
63
|
+
name: bite
|
|
64
|
+
on: pull_request
|
|
65
|
+
permissions:
|
|
66
|
+
contents: read
|
|
67
|
+
pull-requests: write
|
|
68
|
+
jobs:
|
|
69
|
+
bite:
|
|
70
|
+
runs-on: ubuntu-latest
|
|
71
|
+
steps:
|
|
72
|
+
- uses: actions/checkout@v4
|
|
73
|
+
with:
|
|
74
|
+
ref: ${{ github.event.pull_request.head.sha }}
|
|
75
|
+
fetch-depth: 0
|
|
76
|
+
# install what your tests need here, e.g. for Python:
|
|
77
|
+
- uses: actions/setup-python@v5
|
|
78
|
+
- run: python -m pip install pytest -r requirements.txt
|
|
79
|
+
- uses: connabyss/bite@v0.4.0
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
Options go under `with:`: `args` (extra `bite` options, e.g. `--serious`), `max-commits` (default 20),
|
|
83
|
+
`fail-on-gap` (set to `false` to report without failing) and `comment` (set to `false` for no comment). It posts one comment on the pull request that says, per
|
|
84
|
+
commit, whether a test noticed the change being taken out, and updates that same comment on later pushes. The
|
|
85
|
+
rerun steps for each flagged commit are on the job's summary page.
|
|
86
|
+
|
|
87
|
+
## Bite-proof badge
|
|
88
|
+
|
|
89
|
+
When every change Bite checked had a test that noticed it, Bite ends with "Bite-proof. Nice work." and prints a
|
|
90
|
+
badge line you can paste into your README:
|
|
91
|
+
|
|
92
|
+
[](https://github.com/connabyss/bite)
|
|
93
|
+
|
|
94
|
+
Prefer a gentler wording for the warnings? `bite --roast` gives the same results, phrased differently.
|
|
95
|
+
|
|
96
|
+
## The catch that shows what it's for
|
|
97
|
+
|
|
98
|
+
In [fellowapp/prosemirror-py](https://github.com/fellowapp/prosemirror-py), commit `3614c96` (co-authored by Claude) says:
|
|
99
|
+
|
|
100
|
+
> fix: add weakref callback to clean up resolve cache entries
|
|
101
|
+
>
|
|
102
|
+
> The cache used dict keyed by id(doc) with no cleanup, causing a slow memory leak in long-running processes. Added a weakref finalizer callback that removes the cache entry when the doc is garbage collected.
|
|
103
|
+
|
|
104
|
+
Bite wrote this test of that claim and ran it:
|
|
105
|
+
|
|
106
|
+
```python
|
|
107
|
+
def test_cache_entry_removed_after_doc_gc():
|
|
108
|
+
resolvedpos._resolve_cache.clear()
|
|
109
|
+
d = doc(p("hello"))
|
|
110
|
+
ResolvedPos.resolve_cached(d, 2)
|
|
111
|
+
assert len(resolvedpos._resolve_cache) == 1
|
|
112
|
+
del d
|
|
113
|
+
gc.collect()
|
|
114
|
+
assert len(resolvedpos._resolve_cache) == 0 # fails: 1 == 0
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
It fails on that commit and on today's code. The cache keeps each document alive through its cached positions, so the cleanup never runs. After 20,000 documents are resolved and dropped, the cache still holds all 20,000, using 35 MB. The commit looked right, read right, and passed its own tests. Reported with a repro in [#335](https://github.com/fellowapp/prosemirror-py/issues/335); a tested fix is open as [#336](https://github.com/fellowapp/prosemirror-py/pull/336).
|
|
118
|
+
|
|
119
|
+
## Fixed or merged upstream
|
|
120
|
+
|
|
121
|
+
Each bug was reproduced with a standalone script in a clean environment before it was reported.
|
|
122
|
+
|
|
123
|
+
| Project | What happened |
|
|
124
|
+
|---|---|
|
|
125
|
+
| pyclibrary | 7 bugs reported in [#92](https://github.com/MatthieuDartiailh/pyclibrary/issues/92). All 7 fixes reviewed and merged: [#93](https://github.com/MatthieuDartiailh/pyclibrary/pull/93), [#94](https://github.com/MatthieuDartiailh/pyclibrary/pull/94), [#95](https://github.com/MatthieuDartiailh/pyclibrary/pull/95), [#96](https://github.com/MatthieuDartiailh/pyclibrary/pull/96), [#97](https://github.com/MatthieuDartiailh/pyclibrary/pull/97), [#98](https://github.com/MatthieuDartiailh/pyclibrary/pull/98), [#99](https://github.com/MatthieuDartiailh/pyclibrary/pull/99). The maintainer [wrote](https://github.com/MatthieuDartiailh/pyclibrary/issues/92): "Thanks for the small and focused PR very nice to review." |
|
|
126
|
+
| django-tailwind-cli | With one stylesheet, `tailwind watch` hid rebuild messages and build errors. Fix [#232](https://github.com/django-commons/django-tailwind-cli/pull/232) merged and released in [4.8.2](https://github.com/django-commons/django-tailwind-cli/releases/tag/v4.8.2). The maintainer [wrote](https://github.com/django-commons/django-tailwind-cli/pull/232): "Just about to release 4.8.2 with the fix included. Thanks for the contribution." |
|
|
127
|
+
| srptools | 4 bugs reported in [#13](https://github.com/idlesign/srptools/issues/13). The maintainer fixed all 4 and closed it the same day. |
|
|
128
|
+
| resolve-robotics-uri-py | URI lookup depended on the hash seed ([#26](https://github.com/gbionics/resolve-robotics-uri-py/issues/26)). Fixed by the maintainers in [#27](https://github.com/gbionics/resolve-robotics-uri-py/pull/27). The maintainer [wrote](https://github.com/gbionics/resolve-robotics-uri-py/issues/26): "Thanks a lot for the detailed issue report!" |
|
|
129
|
+
| text-to-cad | A Claude commit added a Python 3.11 syntax check that can't catch the exact bug it was written for (nested same-quote f-strings). Bite sent a test that does catch it, [#708](https://github.com/earthtojake/text-to-cad/pull/708), was merged and released in [0.8.2](https://github.com/earthtojake/text-to-cad/releases/tag/v0.8.2); the maintainer then closed the remaining gaps in [#712](https://github.com/earthtojake/text-to-cad/pull/712). |
|
|
130
|
+
| west | Loading a manifest from data with a reserved project name crashed. Fix [#1013](https://github.com/zephyrproject-rtos/west/pull/1013) approved by two maintainers and merged. |
|
|
131
|
+
| midea-lan | A message encoder got the frame length wrong when no padding was needed. Fix [#251](https://github.com/wuwentao/midea-lan/pull/251) merged. The maintainer [wrote](https://github.com/wuwentao/midea-lan/pull/251): "Thank you @connabyss for submitting this PR and for finding and fixing the 8370 framing bug." |
|
|
132
|
+
| mizani | Number labels picked too few decimals when values were close together (`[0, 10, 10.5]` showed as `10, 10`). Fix [#90](https://github.com/has2k1/mizani/pull/90) merged. The maintainer [wrote](https://github.com/has2k1/mizani/pull/90): "@connabyss, thank you." |
|
|
133
|
+
| pamqp | Bite's free check found three past fixes that no test would notice breaking, including one where `Basic.Reject(requeue=False)` was flipped back to `True`. Tests for all three ([#86](https://github.com/gmr/pamqp/pull/86)) merged. |
|
|
134
|
+
|
|
135
|
+
## Sent, waiting for review
|
|
136
|
+
|
|
137
|
+
| Project | Link | What happened |
|
|
138
|
+
|---|---|---|
|
|
139
|
+
| prosemirror-py | [#335](https://github.com/fellowapp/prosemirror-py/issues/335), [#336](https://github.com/fellowapp/prosemirror-py/pull/336), [#337](https://github.com/fellowapp/prosemirror-py/pull/337) | The leak above. A second Claude commit (`1e3339e`) let `Mapping` change a list the caller still owned; fix in #337. |
|
|
140
|
+
| python-vcr-tests | [#18](https://github.com/keboola/python-vcr-tests/issues/18), [#19](https://github.com/keboola/python-vcr-tests/pull/19) | A Claude commit said its database recorder captures every query; only the last per cursor is saved. |
|
|
141
|
+
| ChatSBOM | [#217](https://github.com/WangYihang/ChatSBOM/issues/217) | A Claude commit's stale-result guard is reset on every walk page, so a repo can be collected twice. |
|
|
142
|
+
| llm | [#1828](https://github.com/simonw/llm/pull/1828) | A Claude commit fixed per-turn tool ids in `llm logs`, but its test never made a tool call, so it passed even with the bug. |
|
|
143
|
+
| treg | [#1237](https://github.com/superdesigndev/treg/pull/1237) | A Claude commit's test for an org header passes even with the fix removed, so it guards nothing. |
|
|
144
|
+
| claude-seo | [#392](https://github.com/AgriciDaniel/claude-seo/pull/392) | A Claude commit's `--release` check let a value with a trailing newline through. |
|
|
145
|
+
| marshmallow-fastoneofschema | [#1](https://github.com/Kalepa/marshmallow-fastoneofschema/issues/1), [#2](https://github.com/Kalepa/marshmallow-fastoneofschema/pull/2), [#3](https://github.com/Kalepa/marshmallow-fastoneofschema/pull/3), [#4](https://github.com/Kalepa/marshmallow-fastoneofschema/pull/4) | 3 bugs, one fix each. |
|
|
146
|
+
| django-modeltranslation | [#831](https://github.com/deschler/django-modeltranslation/pull/831) | Nested `auto_populate()` and `fallbacks()` blocks reset the setting to nothing on exit instead of restoring it. |
|
|
147
|
+
| mypy_boto3_builder | [#395](https://github.com/youtype/mypy_boto3_builder/issues/395) | Reported, no reply yet. |
|
|
148
|
+
| drf-pydantic | [#51](https://github.com/georgebv/drf-pydantic/issues/51) | Reported, no reply yet. |
|
|
149
|
+
|
|
150
|
+
## Limits, plainly
|
|
151
|
+
|
|
152
|
+
- **Most claims hold.** Across all rounds so far, Bite tested 137 "fixed" claims that it could run tests for (all from commits made with Claude Code, the AI those rounds looked at). 8 were false (about 1 in 17), and 4 of those were still broken on today's code. On well-reviewed projects it usually finds nothing, and that's the right answer.
|
|
153
|
+
- **Quiet is not a pass.** Bite only shows a failing test that the second check calls fair and real. That second check has thrown out more than 10 failing tests that weren't. A claim that holds proves that one claim, not that the commit has no other bugs.
|
|
154
|
+
- **It needs the project's tests to run.** If they don't run on a clean copy, Bite can't check that commit.
|
|
155
|
+
- **Bug hunting is weaker than claim checking.** In a 10-project overnight hunt, two blind AI judges rated 21 of 22 proven fixes as real bugs and none as wrong, but Bite found only 1 of the 10 bugs the maintainers fixed soon after.
|
|
156
|
+
- **Small samples, AI judges.** The judges read code without running it. The human checks are the maintainers' fixes and merges listed above.
|
|
157
|
+
|
|
158
|
+
## Contributing
|
|
159
|
+
|
|
160
|
+
Bug reports, fixes and AI-assisted PRs are all welcome. See [CONTRIBUTING.md](https://github.com/connabyss/bite/blob/main/CONTRIBUTING.md). If Bite flagged
|
|
161
|
+
something wrongly, or missed a claim you know was false, please [open an issue](https://github.com/connabyss/bite/issues/new/choose).
|
|
162
|
+
|
|
163
|
+
Thank you to Bite's first contributors:
|
|
164
|
+
|
|
165
|
+
- [@snehaapratap](https://github.com/snehaapratap) made the setup error messages point to steps that exist ([#18](https://github.com/connabyss/bite/pull/18)).
|
|
166
|
+
- [@gauravjangra3165](https://github.com/gauravjangra3165) added macOS to the tests ([#19](https://github.com/connabyss/bite/pull/19)).
|
|
167
|
+
- [@bfs2021](https://github.com/bfs2021) made the "check it yourself" steps work for commits that add a new file ([#37](https://github.com/connabyss/bite/pull/37), fixing [#27](https://github.com/connabyss/bite/issues/27)).
|
|
168
|
+
|
|
169
|
+
MIT licensed.
|