bug-age 2.0.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- bug_age-2.0.0/LICENSE-APACHE +202 -0
- bug_age-2.0.0/LICENSE-MIT +21 -0
- bug_age-2.0.0/PKG-INFO +301 -0
- bug_age-2.0.0/README.md +284 -0
- bug_age-2.0.0/bug_age/__init__.py +13 -0
- bug_age-2.0.0/bug_age/__main__.py +7 -0
- bug_age-2.0.0/bug_age/_compat.py +155 -0
- bug_age-2.0.0/bug_age/bundle.py +381 -0
- bug_age-2.0.0/bug_age/catalog.py +68 -0
- bug_age-2.0.0/bug_age/cli.py +265 -0
- bug_age-2.0.0/bug_age/data/Dockerfile +14 -0
- bug_age-2.0.0/bug_age/data/bundle.schema.json +89 -0
- bug_age-2.0.0/bug_age/data/explorer.html +851 -0
- bug_age-2.0.0/bug_age/explorer.py +42 -0
- bug_age-2.0.0/bug_age/feasibility.py +339 -0
- bug_age-2.0.0/bug_age/git.py +71 -0
- bug_age-2.0.0/bug_age/mcp.py +211 -0
- bug_age-2.0.0/bug_age/pipeline.py +151 -0
- bug_age-2.0.0/bug_age/repros.py +178 -0
- bug_age-2.0.0/bug_age/runner.py +294 -0
- bug_age-2.0.0/bug_age/schema.py +102 -0
- bug_age-2.0.0/bug_age/sweep.py +280 -0
- bug_age-2.0.0/bug_age/szz.py +135 -0
- bug_age-2.0.0/bug_age/timetravel.py +108 -0
- bug_age-2.0.0/bug_age.egg-info/PKG-INFO +301 -0
- bug_age-2.0.0/bug_age.egg-info/SOURCES.txt +35 -0
- bug_age-2.0.0/bug_age.egg-info/dependency_links.txt +1 -0
- bug_age-2.0.0/bug_age.egg-info/entry_points.txt +2 -0
- bug_age-2.0.0/bug_age.egg-info/top_level.txt +1 -0
- bug_age-2.0.0/pyproject.toml +32 -0
- bug_age-2.0.0/setup.cfg +4 -0
- bug_age-2.0.0/tests/test_bundle.py +128 -0
- bug_age-2.0.0/tests/test_explorer_cli_mcp.py +132 -0
- bug_age-2.0.0/tests/test_method_a.py +96 -0
- bug_age-2.0.0/tests/test_oracle.py +90 -0
- bug_age-2.0.0/tests/test_sweep.py +156 -0
- bug_age-2.0.0/tests/test_sweep_scenarios.py +80 -0
|
@@ -0,0 +1,202 @@
|
|
|
1
|
+
|
|
2
|
+
Apache License
|
|
3
|
+
Version 2.0, January 2004
|
|
4
|
+
http://www.apache.org/licenses/
|
|
5
|
+
|
|
6
|
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
|
7
|
+
|
|
8
|
+
1. Definitions.
|
|
9
|
+
|
|
10
|
+
"License" shall mean the terms and conditions for use, reproduction,
|
|
11
|
+
and distribution as defined by Sections 1 through 9 of this document.
|
|
12
|
+
|
|
13
|
+
"Licensor" shall mean the copyright owner or entity authorized by
|
|
14
|
+
the copyright owner that is granting the License.
|
|
15
|
+
|
|
16
|
+
"Legal Entity" shall mean the union of the acting entity and all
|
|
17
|
+
other entities that control, are controlled by, or are under common
|
|
18
|
+
control with that entity. For the purposes of this definition,
|
|
19
|
+
"control" means (i) the power, direct or indirect, to cause the
|
|
20
|
+
direction or management of such entity, whether by contract or
|
|
21
|
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
|
22
|
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
|
23
|
+
|
|
24
|
+
"You" (or "Your") shall mean an individual or Legal Entity
|
|
25
|
+
exercising permissions granted by this License.
|
|
26
|
+
|
|
27
|
+
"Source" form shall mean the preferred form for making modifications,
|
|
28
|
+
including but not limited to software source code, documentation
|
|
29
|
+
source, and configuration files.
|
|
30
|
+
|
|
31
|
+
"Object" form shall mean any form resulting from mechanical
|
|
32
|
+
transformation or translation of a Source form, including but
|
|
33
|
+
not limited to compiled object code, generated documentation,
|
|
34
|
+
and conversions to other media types.
|
|
35
|
+
|
|
36
|
+
"Work" shall mean the work of authorship, whether in Source or
|
|
37
|
+
Object form, made available under the License, as indicated by a
|
|
38
|
+
copyright notice that is included in or attached to the work
|
|
39
|
+
(an example is provided in the Appendix below).
|
|
40
|
+
|
|
41
|
+
"Derivative Works" shall mean any work, whether in Source or Object
|
|
42
|
+
form, that is based on (or derived from) the Work and for which the
|
|
43
|
+
editorial revisions, annotations, elaborations, or other modifications
|
|
44
|
+
represent, as a whole, an original work of authorship. For the purposes
|
|
45
|
+
of this License, Derivative Works shall not include works that remain
|
|
46
|
+
separable from, or merely link (or bind by name) to the interfaces of,
|
|
47
|
+
the Work and Derivative Works thereof.
|
|
48
|
+
|
|
49
|
+
"Contribution" shall mean any work of authorship, including
|
|
50
|
+
the original version of the Work and any modifications or additions
|
|
51
|
+
to that Work or Derivative Works thereof, that is intentionally
|
|
52
|
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
|
53
|
+
or by an individual or Legal Entity authorized to submit on behalf of
|
|
54
|
+
the copyright owner. For the purposes of this definition, "submitted"
|
|
55
|
+
means any form of electronic, verbal, or written communication sent
|
|
56
|
+
to the Licensor or its representatives, including but not limited to
|
|
57
|
+
communication on electronic mailing lists, source code control systems,
|
|
58
|
+
and issue tracking systems that are managed by, or on behalf of, the
|
|
59
|
+
Licensor for the purpose of discussing and improving the Work, but
|
|
60
|
+
excluding communication that is conspicuously marked or otherwise
|
|
61
|
+
designated in writing by the copyright owner as "Not a Contribution."
|
|
62
|
+
|
|
63
|
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
|
64
|
+
on behalf of whom a Contribution has been received by Licensor and
|
|
65
|
+
subsequently incorporated within the Work.
|
|
66
|
+
|
|
67
|
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
|
68
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
69
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
70
|
+
copyright license to reproduce, prepare Derivative Works of,
|
|
71
|
+
publicly display, publicly perform, sublicense, and distribute the
|
|
72
|
+
Work and such Derivative Works in Source or Object form.
|
|
73
|
+
|
|
74
|
+
3. Grant of Patent License. Subject to the terms and conditions of
|
|
75
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
76
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
77
|
+
(except as stated in this section) patent license to make, have made,
|
|
78
|
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
|
79
|
+
where such license applies only to those patent claims licensable
|
|
80
|
+
by such Contributor that are necessarily infringed by their
|
|
81
|
+
Contribution(s) alone or by combination of their Contribution(s)
|
|
82
|
+
with the Work to which such Contribution(s) was submitted. If You
|
|
83
|
+
institute patent litigation against any entity (including a
|
|
84
|
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
|
85
|
+
or a Contribution incorporated within the Work constitutes direct
|
|
86
|
+
or contributory patent infringement, then any patent licenses
|
|
87
|
+
granted to You under this License for that Work shall terminate
|
|
88
|
+
as of the date such litigation is filed.
|
|
89
|
+
|
|
90
|
+
4. Redistribution. You may reproduce and distribute copies of the
|
|
91
|
+
Work or Derivative Works thereof in any medium, with or without
|
|
92
|
+
modifications, and in Source or Object form, provided that You
|
|
93
|
+
meet the following conditions:
|
|
94
|
+
|
|
95
|
+
(a) You must give any other recipients of the Work or
|
|
96
|
+
Derivative Works a copy of this License; and
|
|
97
|
+
|
|
98
|
+
(b) You must cause any modified files to carry prominent notices
|
|
99
|
+
stating that You changed the files; and
|
|
100
|
+
|
|
101
|
+
(c) You must retain, in the Source form of any Derivative Works
|
|
102
|
+
that You distribute, all copyright, patent, trademark, and
|
|
103
|
+
attribution notices from the Source form of the Work,
|
|
104
|
+
excluding those notices that do not pertain to any part of
|
|
105
|
+
the Derivative Works; and
|
|
106
|
+
|
|
107
|
+
(d) If the Work includes a "NOTICE" text file as part of its
|
|
108
|
+
distribution, then any Derivative Works that You distribute must
|
|
109
|
+
include a readable copy of the attribution notices contained
|
|
110
|
+
within such NOTICE file, excluding those notices that do not
|
|
111
|
+
pertain to any part of the Derivative Works, in at least one
|
|
112
|
+
of the following places: within a NOTICE text file distributed
|
|
113
|
+
as part of the Derivative Works; within the Source form or
|
|
114
|
+
documentation, if provided along with the Derivative Works; or,
|
|
115
|
+
within a display generated by the Derivative Works, if and
|
|
116
|
+
wherever such third-party notices normally appear. The contents
|
|
117
|
+
of the NOTICE file are for informational purposes only and
|
|
118
|
+
do not modify the License. You may add Your own attribution
|
|
119
|
+
notices within Derivative Works that You distribute, alongside
|
|
120
|
+
or as an addendum to the NOTICE text from the Work, provided
|
|
121
|
+
that such additional attribution notices cannot be construed
|
|
122
|
+
as modifying the License.
|
|
123
|
+
|
|
124
|
+
You may add Your own copyright statement to Your modifications and
|
|
125
|
+
may provide additional or different license terms and conditions
|
|
126
|
+
for use, reproduction, or distribution of Your modifications, or
|
|
127
|
+
for any such Derivative Works as a whole, provided Your use,
|
|
128
|
+
reproduction, and distribution of the Work otherwise complies with
|
|
129
|
+
the conditions stated in this License.
|
|
130
|
+
|
|
131
|
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
|
132
|
+
any Contribution intentionally submitted for inclusion in the Work
|
|
133
|
+
by You to the Licensor shall be under the terms and conditions of
|
|
134
|
+
this License, without any additional terms or conditions.
|
|
135
|
+
Notwithstanding the above, nothing herein shall supersede or modify
|
|
136
|
+
the terms of any separate license agreement you may have executed
|
|
137
|
+
with Licensor regarding such Contributions.
|
|
138
|
+
|
|
139
|
+
6. Trademarks. This License does not grant permission to use the trade
|
|
140
|
+
names, trademarks, service marks, or product names of the Licensor,
|
|
141
|
+
except as required for reasonable and customary use in describing the
|
|
142
|
+
origin of the Work and reproducing the content of the NOTICE file.
|
|
143
|
+
|
|
144
|
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
|
145
|
+
agreed to in writing, Licensor provides the Work (and each
|
|
146
|
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
|
147
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
|
148
|
+
implied, including, without limitation, any warranties or conditions
|
|
149
|
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
|
150
|
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
|
151
|
+
appropriateness of using or redistributing the Work and assume any
|
|
152
|
+
risks associated with Your exercise of permissions under this License.
|
|
153
|
+
|
|
154
|
+
8. Limitation of Liability. In no event and under no legal theory,
|
|
155
|
+
whether in tort (including negligence), contract, or otherwise,
|
|
156
|
+
unless required by applicable law (such as deliberate and grossly
|
|
157
|
+
negligent acts) or agreed to in writing, shall any Contributor be
|
|
158
|
+
liable to You for damages, including any direct, indirect, special,
|
|
159
|
+
incidental, or consequential damages of any character arising as a
|
|
160
|
+
result of this License or out of the use or inability to use the
|
|
161
|
+
Work (including but not limited to damages for loss of goodwill,
|
|
162
|
+
work stoppage, computer failure or malfunction, or any and all
|
|
163
|
+
other commercial damages or losses), even if such Contributor
|
|
164
|
+
has been advised of the possibility of such damages.
|
|
165
|
+
|
|
166
|
+
9. Accepting Warranty or Additional Liability. While redistributing
|
|
167
|
+
the Work or Derivative Works thereof, You may choose to offer,
|
|
168
|
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
|
169
|
+
or other liability obligations and/or rights consistent with this
|
|
170
|
+
License. However, in accepting such obligations, You may act only
|
|
171
|
+
on Your own behalf and on Your sole responsibility, not on behalf
|
|
172
|
+
of any other Contributor, and only if You agree to indemnify,
|
|
173
|
+
defend, and hold each Contributor harmless for any liability
|
|
174
|
+
incurred by, or claims asserted against, such Contributor by reason
|
|
175
|
+
of your accepting any such warranty or additional liability.
|
|
176
|
+
|
|
177
|
+
END OF TERMS AND CONDITIONS
|
|
178
|
+
|
|
179
|
+
APPENDIX: How to apply the Apache License to your work.
|
|
180
|
+
|
|
181
|
+
To apply the Apache License to your work, attach the following
|
|
182
|
+
boilerplate notice, with the fields enclosed by brackets "[]"
|
|
183
|
+
replaced with your own identifying information. (Don't include
|
|
184
|
+
the brackets!) The text should be enclosed in the appropriate
|
|
185
|
+
comment syntax for the file format. We also recommend that a
|
|
186
|
+
file or class name and description of purpose be included on the
|
|
187
|
+
same "printed page" as the copyright notice for easier
|
|
188
|
+
identification within third-party archives.
|
|
189
|
+
|
|
190
|
+
Copyright [yyyy] [name of copyright owner]
|
|
191
|
+
|
|
192
|
+
Licensed under the Apache License, Version 2.0 (the "License");
|
|
193
|
+
you may not use this file except in compliance with the License.
|
|
194
|
+
You may obtain a copy of the License at
|
|
195
|
+
|
|
196
|
+
http://www.apache.org/licenses/LICENSE-2.0
|
|
197
|
+
|
|
198
|
+
Unless required by applicable law or agreed to in writing, software
|
|
199
|
+
distributed under the License is distributed on an "AS IS" BASIS,
|
|
200
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
201
|
+
See the License for the specific language governing permissions and
|
|
202
|
+
limitations under the License.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 jacob-adm
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
bug_age-2.0.0/PKG-INFO
ADDED
|
@@ -0,0 +1,301 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: bug-age
|
|
3
|
+
Version: 2.0.0
|
|
4
|
+
Summary: Measure how long fixed bugs lived in a git repository, from repository data only.
|
|
5
|
+
License: MIT OR Apache-2.0
|
|
6
|
+
Project-URL: Source, https://github.com/jacob-adm/bug-age
|
|
7
|
+
Classifier: Programming Language :: Python :: 3
|
|
8
|
+
Classifier: Environment :: Console
|
|
9
|
+
Classifier: Topic :: Software Development :: Quality Assurance
|
|
10
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
11
|
+
Classifier: License :: OSI Approved :: Apache Software License
|
|
12
|
+
Requires-Python: >=3.11
|
|
13
|
+
Description-Content-Type: text/markdown
|
|
14
|
+
License-File: LICENSE-APACHE
|
|
15
|
+
License-File: LICENSE-MIT
|
|
16
|
+
Dynamic: license-file
|
|
17
|
+
|
|
18
|
+
# bug-age
|
|
19
|
+
|
|
20
|
+
Measures how long bugs lived in a git repository, using real repository data. For each fixed bug it reports commits elapsed, calendar days, releases shipped, and churn between the commit that introduced it and the fix.
|
|
21
|
+
|
|
22
|
+
It uses two independent methods and compares them:
|
|
23
|
+
|
|
24
|
+
- **Method A, blame-based (SZZ-style).** Runs `git blame` on the lines a fix changed and names the commit that last touched them as the likely introducer. This is fast and needs only git, but it can be wrong. Known failure cases: ghost commits, formatting-only changes, and introducers that were not the latest toucher.
|
|
25
|
+
- **Method B, behavioral sweep.** Takes a regression test that fails before the fix and passes after it, then replays that test across history in a hermetic per-commit Docker environment. The result is the first failing commit, the last passing commit, and every untestable commit with a reason.
|
|
26
|
+
|
|
27
|
+
Every number comes from scripts. AI helped write the code, the Method B regression tests and the article prose, but no model produces or changes a number. See [What is deterministic and what uses AI](#what-is-deterministic-and-what-uses-ai). bug-age supports Rust crates today.
|
|
28
|
+
|
|
29
|
+
What you get, and the three names this README uses for it:
|
|
30
|
+
|
|
31
|
+
- **Report.** What a run produces: the result files in `results/<name>/` and one bundle, `dist/<name>/bundle.json`, which holds every number. Each repository you run gets its own report.
|
|
32
|
+
- **Explorer.** A viewer. It is a static HTML page you build from one or more bundles to browse their reports. It holds no results of its own and runs no analysis.
|
|
33
|
+
- **Live example.** One explorer, built from the reports of the two [example runs](#example-runs-tokenizers-and-rust-sdk) and published online. It is a snapshot: running bug-age on other repositories does not change it.
|
|
34
|
+
|
|
35
|
+
To see the output first, [open the live example](https://claude.ai/artifact/UDw6jyeA4zMDGhnzL7Typb).
|
|
36
|
+
|
|
37
|
+
## Get started
|
|
38
|
+
|
|
39
|
+
Point bug-age at your own repository in six steps. A first run with Method A only needs git and Python; Docker is needed only for the feasibility check and Method B.
|
|
40
|
+
|
|
41
|
+
**1. Install.**
|
|
42
|
+
|
|
43
|
+
```sh
|
|
44
|
+
pip install bug-age # Python 3.11 or later, no dependencies
|
|
45
|
+
bug-age dockerfile | docker build -t bug-age-spike:4 - # only for feasibility and Method B
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
You also need git. The Docker image name `bug-age-spike:4` is the CLI default; pass `--image` to use another.
|
|
49
|
+
|
|
50
|
+
**2. Describe your repository in a config file.** Copy an existing one and edit it:
|
|
51
|
+
|
|
52
|
+
```sh
|
|
53
|
+
curl -sO https://raw.githubusercontent.com/jacob-adm/bug-age/main/targets/rust-sdk.toml
|
|
54
|
+
mkdir -p targets && mv rust-sdk.toml targets/myrepo.toml
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
| Key | Required | What to set |
|
|
58
|
+
|---|---|---|
|
|
59
|
+
| `name` | yes | Short name: lowercase letters, digits and `-`. Used in file names and the explorer. |
|
|
60
|
+
| `repo_url` | yes | Clone URL of the repository. |
|
|
61
|
+
| `head` | yes | Full SHA of the commit to analyse up to. Pinning it makes the run reproducible. |
|
|
62
|
+
| `crate_path` | yes | Directory of the crate inside the repository (`.` for the root). Only history under this path is analysed. |
|
|
63
|
+
| `work_dir`, `results_dir`, `spike_dir`, `dist_dir` | yes | Where the clone, results, feasibility reports and bundle go, relative to this repository. Follow the pattern `work/<name>`, `results/<name>`, `results/<name>/spike`, `dist/<name>`. |
|
|
64
|
+
| `release_tag_glob` | no | Pattern matching your release tags, for release counts. Default `v[0-9]*`. Set it if your tags look different, e.g. `mycrate-v[0-9]*`, or every release count will be 0. |
|
|
65
|
+
| `package` | no | Cargo package to build and test in a workspace. Default: the whole workspace. |
|
|
66
|
+
| `samples` | no | Commits sampled by the feasibility check. Default 20. |
|
|
67
|
+
| `szz_window_start` | no | `YYYY-MM-DD`. Flags fixes whose introducer falls on or after this date. |
|
|
68
|
+
| `out_of_scope` | no | `STAGE\|EVIDENCE\|DOES_NOT_PROVE`: a part of the repository you deliberately leave out, listed as NOT TESTED. |
|
|
69
|
+
|
|
70
|
+
**3. Run the pipeline.**
|
|
71
|
+
|
|
72
|
+
```sh
|
|
73
|
+
bug-age pipeline targets/myrepo.toml --skip-feasibility # Method A only, no Docker
|
|
74
|
+
bug-age pipeline targets/myrepo.toml # adds the Docker feasibility check
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
**4. Read the report.** `results/myrepo/PIPELINE.md` lists every stage with its status (OK, REUSED, FAILED or NOT TESTED), the reason, and the time taken. The bundle, `dist/myrepo/bundle.json`, is the report of record: every number for this repository is in it. Its sha256 and the exact command that built it are in `dist/myrepo/receipt.json`.
|
|
78
|
+
|
|
79
|
+
**5. Browse the report in your own explorer.** This builds a viewer for your bundle on your machine. The published live example is not affected and keeps showing only the two example runs. To browse several of your repositories together, list them all: `--targets a=dist/a,b=dist/b`.
|
|
80
|
+
|
|
81
|
+
```sh
|
|
82
|
+
bug-age explorer --targets myrepo=dist/myrepo --out dist/explorer-myrepo
|
|
83
|
+
open dist/explorer-myrepo/index.html # or open the file in any browser
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
**6. Optional: date selected bugs by testing (Method B).** Pick a fix from `results/myrepo/catalog.json` and add a regression test at `results/myrepo/repros/<first 10 chars of the fix SHA>/overlay/tests/bugage_repro.rs`. The overlay is copied into the crate at every commit tested, so the test can only use the crate's public API. It must fail at the commit just before the fix and pass at the fix; otherwise the sweep rejects it as `repro-invalid`. Run step 3 again: Method B runs whenever a regression test has no sweep yet. See `results/tokenizers/repros/` for six examples.
|
|
87
|
+
|
|
88
|
+
## Example runs: tokenizers and rust-sdk
|
|
89
|
+
|
|
90
|
+
The same pipeline was run on two Rust codebases. The first, the core crate of [huggingface/tokenizers](https://github.com/huggingface/tokenizers), got every stage, including Method B sweeps for six bugs. The second, the `rmcp` crate of [modelcontextprotocol/rust-sdk](https://github.com/modelcontextprotocol/rust-sdk), was added with a config file alone, to show the pipeline carries over unchanged.
|
|
91
|
+
|
|
92
|
+
**[▶ Open the live example](https://claude.ai/artifact/UDw6jyeA4zMDGhnzL7Typb)**: an explorer built from these two reports and published online. Switch between the two repositories at the top. It is a snapshot of the committed bundles and changes only when it is rebuilt and republished. Each number opens to its evidence: the SHAs, the test runs, the receipt hash and the command that reproduces it. The page also recomputes its headline numbers in your browser and flags any mismatch.
|
|
93
|
+
|
|
94
|
+
[](https://claude.ai/artifact/UDw6jyeA4zMDGhnzL7Typb)
|
|
95
|
+
|
|
96
|
+
| | tokenizers | rust-sdk |
|
|
97
|
+
|---|---|---|
|
|
98
|
+
| Config | [`targets/tokenizers.toml`](targets/tokenizers.toml) | [`targets/rust-sdk.toml`](targets/rust-sdk.toml) |
|
|
99
|
+
| Method A (every catalogued fix) | [`results/tokenizers/METHOD_A.md`](results/tokenizers/METHOD_A.md) | [`results/rust-sdk/szz.json`](results/rust-sdk/szz.json) |
|
|
100
|
+
| Method B (behavioural sweeps) | Six bugs: [`results/tokenizers/AB.md`](results/tokenizers/AB.md) | NOT TESTED: no regression tests written yet |
|
|
101
|
+
| Stage report | [`results/tokenizers/PIPELINE.md`](results/tokenizers/PIPELINE.md) | [`results/rust-sdk/PIPELINE.md`](results/rust-sdk/PIPELINE.md) |
|
|
102
|
+
| Report of record (bundle) | [`dist/tokenizers/bundle.json`](dist/tokenizers/bundle.json) | [`dist/rust-sdk/bundle.json`](dist/rust-sdk/bundle.json) |
|
|
103
|
+
|
|
104
|
+
### What the explorer shows
|
|
105
|
+
|
|
106
|
+
These screenshots are captured by `make article` from the explorer built for the two example runs, so they always match the committed bundles.
|
|
107
|
+
|
|
108
|
+
**Lifespans.** Each bar runs from the nominated introducing commit to the fix. A date scrubber shows which bugs were alive on any day.
|
|
109
|
+
|
|
110
|
+

|
|
111
|
+
|
|
112
|
+
**Commit × test results.** Every commit the pipeline built and tested in Docker without network: green passes, red fails, grey does not build there. Each square opens the run log.
|
|
113
|
+
|
|
114
|
+

|
|
115
|
+
|
|
116
|
+
### Reading Method B verdicts
|
|
117
|
+
|
|
118
|
+
Verdicts are honest about what could not be tested:
|
|
119
|
+
|
|
120
|
+
| Verdict | Meaning |
|
|
121
|
+
|---|---|
|
|
122
|
+
| `located` | The exact first failing commit was found |
|
|
123
|
+
| `censored-by-testability` | The bug is present at the first commit where the test compiles, so its age is a lower bound (`≥`) |
|
|
124
|
+
| `left-censored` | The bug is already present at the start of the sweep range (3 years before the fix), so its age is a lower bound |
|
|
125
|
+
|
|
126
|
+
The feasibility check, which measures toolchain and dependency drift across history, is in `spike/results/` for tokenizers and `results/rust-sdk/spike/` for rust-sdk.
|
|
127
|
+
|
|
128
|
+
## How bugs are found
|
|
129
|
+
|
|
130
|
+
bug-age does not discover new bugs. It works from bugs the project has already fixed, and uses testing only to date some of them.
|
|
131
|
+
|
|
132
|
+
1. **Find the fixes (no testing).** Read the first-parent commit history of the crate and keep commits whose subject says `fix`, `bug` or `hotfix`. Fixed rules exclude typo, docs, lint, CI, release and similar commits. Each remaining commit counts as one bug. Every decision is recorded with the rule that made it.
|
|
133
|
+
2. **Date every fix from history (Method A, no testing).** Blame the lines each fix changed; the commit that last touched them is the nominated introducer. Age is the time, commits and releases between that commit and the fix.
|
|
134
|
+
3. **Date selected bugs by testing (Method B).** For a few bugs, a regression test reproduces the bug. It must fail just before the fix and pass at the fix, or it is rejected. The test is then run on older commits, in Docker without network, to find where the failure first appears.
|
|
135
|
+
|
|
136
|
+
What this means for the numbers:
|
|
137
|
+
|
|
138
|
+
- Bugs that were never fixed, or were fixed in a commit whose subject does not say so, are not counted.
|
|
139
|
+
- A commit that says "fix" but is not a bug fix is counted unless an exclusion rule catches it. This is one reason Method A is reported as PARTIAL.
|
|
140
|
+
- Method A covers every catalogued fix but is a nomination. Method B is evidence, but only for the bugs that have a regression test.
|
|
141
|
+
|
|
142
|
+
## How long it takes
|
|
143
|
+
|
|
144
|
+
<!-- timings:begin -->
|
|
145
|
+
|
|
146
|
+
Measured on Darwin arm64, Docker 28.3.3 arm64. Figures come from committed run records via `scripts/timings.py`; full tables in [`results/TIMINGS.md`](results/TIMINGS.md). Step time excludes container staging and toolchain installs, so expect wall time to be longer.
|
|
147
|
+
|
|
148
|
+
| Stage | What was measured | Time |
|
|
149
|
+
|---|---|---|
|
|
150
|
+
| catalog | last pipeline run, 2 target(s) | 3-4 s |
|
|
151
|
+
| Method A | last pipeline run, 2 target(s) | 27-53 s |
|
|
152
|
+
| bundle | last pipeline run, 2 target(s) | 5-6 s |
|
|
153
|
+
| Feasibility (one run, 20 sampled commits) | 4 runs | 4 min to 7 min per run |
|
|
154
|
+
| Method B sweep (one bug, current search) | 6 sweeps, 84 commits | 101 s to 6 min of step time per bug; median per commit 7 s to 22 s |
|
|
155
|
+
| Method B, all 6 tokenizers sweeps end to end | one unattended run, wall clock | 20 min |
|
|
156
|
+
|
|
157
|
+
Method B dominates: each bug needs a regression test, and the current search built and tested 5 to 23 commits per bug in Docker. Catalog, Method A and bundle together took 36 to 62 s per repository.
|
|
158
|
+
|
|
159
|
+
<!-- timings:end -->
|
|
160
|
+
|
|
161
|
+
## What is deterministic and what uses AI
|
|
162
|
+
|
|
163
|
+
**Deterministic.** These parts make no model calls. Given the same repository, pinned commit and config, they produce the same output, and `make pipeline` needs no API key.
|
|
164
|
+
|
|
165
|
+
| Part | How it works |
|
|
166
|
+
|---|---|
|
|
167
|
+
| Fix catalog | Regular-expression rules over commit subjects, plus exclusion rules (docs, lint, release, ...). Locked by `classify_rules`. |
|
|
168
|
+
| Method A | `git blame` on the lines each fix changed; days, commits, releases and churn computed from git. |
|
|
169
|
+
| Buildability and Method B sweeps | Each commit built and tested in Docker with no network and a date-matched toolchain. PASS or FAIL comes from the test's exit status, and decisive commits are re-run. |
|
|
170
|
+
| Bundle and summary | Medians, 90th percentiles, counts and coverage computed by the CLI; the blameless gate rejects names and emails. |
|
|
171
|
+
| Explorer (viewer) | Displays the bundles it was built from as they are, recomputes the headline numbers in the browser, and ranks search results with fixed rules. |
|
|
172
|
+
| MCP server | Returns values copied from the bundle; unknown input is an error, never a guess. |
|
|
173
|
+
| Article numbers | Inserted from the bundle by `scripts/article.py`, which fails on any number typed into the prose. |
|
|
174
|
+
| History scan | Fixed identity and email patterns over every committed file and message. |
|
|
175
|
+
|
|
176
|
+
**Uses AI.** In each case the AI's output is checked mechanically, and none of it becomes a number.
|
|
177
|
+
|
|
178
|
+
| Part | What AI did | What checks it |
|
|
179
|
+
|---|---|---|
|
|
180
|
+
| Method B regression tests (`results/<target>/repros/*/overlay`) | Drafted by an AI coding agent from each fix's diff | The sweep counts a test only if it fails before the fix and passes after it; otherwise the verdict is `repro-invalid`. Whether a test captures the real bug is a human judgement, so read the overlay. |
|
|
181
|
+
| Article prose (`article/ARTICLE.tmpl.md`) | Drafted with AI | Every number is a bundle reference; the check fails on anything else. |
|
|
182
|
+
| "Ask Claude" in the explorer (optional) | Claude answers a question on the viewer's own account, from the rows the search found | The answer is shown only if every cited SHA and every number appears in those rows. It runs only when the viewer clicks, and is hidden where unavailable. |
|
|
183
|
+
| The code | Written with an AI coding agent (Claude Code) and reviewed by the maintainer | Each change merged through a pull request after `cargo test`; commits carry a `Co-Authored-By` line. |
|
|
184
|
+
|
|
185
|
+
## CLI
|
|
186
|
+
|
|
187
|
+
```
|
|
188
|
+
bug-age catalog --repo DIR --path CRATE [--head REV] [--out FILE]
|
|
189
|
+
bug-age szz --repo DIR --catalog FILE [--window-start YYYY-MM-DD] [--out FILE]
|
|
190
|
+
bug-age sweep --repo DIR --path CRATE --fix SHA --overlay DIR --test CMD
|
|
191
|
+
[--build CMD] [--start SHA] [--runner docker|local] [--repeats N] [--image IMG]
|
|
192
|
+
[--time-travel-script FILE|none] [--scratch DIR] [--logs DIR] [--out FILE]
|
|
193
|
+
[--namespace NAME] [--bucket N]
|
|
194
|
+
bug-age probe --repo DIR --path CRATE --commits SHA,SHA,... --overlay DIR --test CMD [same options]
|
|
195
|
+
bug-age bundle --repo DIR --results DIR --repo-url URL --out DIR [--spike DIR]
|
|
196
|
+
bug-age explorer [--page FILE] --targets NAME=DIR,... --out DIR
|
|
197
|
+
bug-age mcp --bundles NAME=FILE,...
|
|
198
|
+
bug-age pipeline TARGET.toml [--fresh] [--skip-feasibility]
|
|
199
|
+
bug-age dockerfile
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
| Command | What it does |
|
|
203
|
+
|---|---|
|
|
204
|
+
| `catalog` | Lists the fix commits that touch a crate and records why each other commit was excluded (docs, lint, release, and so on) |
|
|
205
|
+
| `szz` | Method A: names a likely introducing commit for each fix in the catalog |
|
|
206
|
+
| `sweep` | Method B: binary-searches for the first commit where the test compiles, then bisects PASS/FAIL above it. Decisive commits are re-run `--repeats` times and flagged UNSTABLE if the result changes |
|
|
207
|
+
| `probe` | Runs the same test at a fixed list of commits, with no search |
|
|
208
|
+
| `bundle` | Builds `bundle.json` (validated against [`bug_age/data/bundle.schema.json`](bug_age/data/bundle.schema.json)), `bundle.js`, and `receipt.json` (sha256 hashes and sizes) |
|
|
209
|
+
| `explorer` | Builds an explorer: a static viewer page for the listed bundles, with one data file per repository; fails over 8 MB. It reads reports and adds no results. |
|
|
210
|
+
| `mcp` | Read-only MCP server on stdio answering bundle queries; see [`dist/agent/AGENTS.md`](dist/agent/AGENTS.md) |
|
|
211
|
+
| `pipeline` | Runs every stage for one target config; paths in the config are relative to the current directory |
|
|
212
|
+
| `dockerfile` | Prints the Dockerfile of the Method B image |
|
|
213
|
+
|
|
214
|
+
## Pipeline reference
|
|
215
|
+
|
|
216
|
+
Each target repository has one config file in [`targets/`](targets/), and nothing else is target-specific:
|
|
217
|
+
|
|
218
|
+
```toml
|
|
219
|
+
name = "rust-sdk"
|
|
220
|
+
repo_url = "https://github.com/modelcontextprotocol/rust-sdk"
|
|
221
|
+
head = "5596a8a0bba2a1ec5519cc855cdadf23e4dcecdc" # pinned subject
|
|
222
|
+
crate_path = "crates/rmcp"
|
|
223
|
+
work_dir = "work/rust-sdk"
|
|
224
|
+
results_dir = "results/rust-sdk"
|
|
225
|
+
spike_dir = "results/rust-sdk/spike"
|
|
226
|
+
dist_dir = "dist/rust-sdk"
|
|
227
|
+
samples = 20
|
|
228
|
+
# optional: szz_window_start = "YYYY-MM-DD", out_of_scope = "STAGE|EVIDENCE|DOES_NOT_PROVE"
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
One command runs every stage:
|
|
232
|
+
|
|
233
|
+
```sh
|
|
234
|
+
bug-age pipeline targets/rust-sdk.toml # or, in a clone: make pipeline TARGET=targets/rust-sdk.toml
|
|
235
|
+
```
|
|
236
|
+
|
|
237
|
+
The stages are clone at the pinned head, feasibility (baseline and time-travel dependencies), catalog, Method A, Method B for every repro under `<results_dir>/repros/<fix>/overlay`, and bundle.
|
|
238
|
+
|
|
239
|
+
- A stage whose output already exists is REUSED, so the expensive Docker stages are never redone silently. Pass `--fresh` to rerun them.
|
|
240
|
+
- A failing stage is recorded with its reason, and the stages that don't depend on it still run.
|
|
241
|
+
- The run writes `<results_dir>/PIPELINE.md` and `pipeline.json`, listing each stage's status, reason, time taken and output sha256.
|
|
242
|
+
- A stage that did not run appears in the bundle's coverage as NOT TESTED.
|
|
243
|
+
|
|
244
|
+
Method B needs a regression test per bug, written as a test overlay. Those tests are drafted per target, so a target without any shows Method B as NOT TESTED.
|
|
245
|
+
|
|
246
|
+
### Unattended sweeps
|
|
247
|
+
|
|
248
|
+
```sh
|
|
249
|
+
make sweep ARGS="results/bench work/tokenizers tokenizers"
|
|
250
|
+
```
|
|
251
|
+
|
|
252
|
+
The sweep runs detached, with no agent involved. It writes `<results>/SUMMARY.md` and sends one macOS notification when it finishes. To check a run against committed results:
|
|
253
|
+
|
|
254
|
+
```sh
|
|
255
|
+
python3 results/compare_runs.py results/tokenizers results/bench # prints IDENTICAL or the differences
|
|
256
|
+
```
|
|
257
|
+
|
|
258
|
+
## Rigor rules
|
|
259
|
+
|
|
260
|
+
- The subject is pinned: repo URL, SHA range, toolchain, and tool versions.
|
|
261
|
+
- Sweeps run with `--network none`, a date-matched toolchain, and dependencies resolved as they were at that date.
|
|
262
|
+
- A result is PASS only when its assertions actually ran. Anything else is reported as NOT TESTED.
|
|
263
|
+
- The bundle is blameless. It contains SHAs, dates, and short commit subjects only. Author names, emails, and usernames are redacted, and a gate fails the build if any remain.
|
|
264
|
+
|
|
265
|
+
## Tests
|
|
266
|
+
|
|
267
|
+
```sh
|
|
268
|
+
python3 -m unittest discover -s tests # unit, CLI, oracle and sweep tests
|
|
269
|
+
BUG_AGE_DOCKER_TESTS=1 python3 -m unittest discover -s tests # adds the Docker runner test (needs the image)
|
|
270
|
+
```
|
|
271
|
+
|
|
272
|
+
The oracle test regenerates both example runs from clones in `work/` and compares every output
|
|
273
|
+
with the committed files byte for byte. The sweep tests build a toy crate, so they need cargo.
|
|
274
|
+
|
|
275
|
+
## Contributing
|
|
276
|
+
|
|
277
|
+
Changes reach `main` only through pull requests.
|
|
278
|
+
|
|
279
|
+
1. Fork the repository and create a branch.
|
|
280
|
+
2. Make the change, then run `python3 -m unittest discover -s tests`. If you changed bundles, results or the article, also run `make article timings`.
|
|
281
|
+
3. Open a pull request against `main`. CI runs the tests and the article and timing checks on Linux and macOS; all must pass before merging. Pull requests are merged with a merge commit.
|
|
282
|
+
|
|
283
|
+
Keep the rules in [Rigor rules](#rigor-rules): numbers come from scripts, unrun stages are NOT TESTED, and bundles stay blameless.
|
|
284
|
+
|
|
285
|
+
## Versions
|
|
286
|
+
|
|
287
|
+
1.0.2 is the last version written in Rust. It measures Rust crates: Method A needs git and a
|
|
288
|
+
Rust 1.86 toolchain to build the CLI (`cargo build`), and Method B needs Docker. It is the version
|
|
289
|
+
that produced the tokenizers and rust-sdk example runs. To use it, check out its tag:
|
|
290
|
+
|
|
291
|
+
```sh
|
|
292
|
+
git clone --branch v1.0.2 https://github.com/jacob-adm/bug-age
|
|
293
|
+
```
|
|
294
|
+
|
|
295
|
+
2.0.0 and later are a Python port of the same tool, installed with `pip install bug-age`. On both
|
|
296
|
+
example runs it reproduces every output of 1.0.2 byte for byte; the only difference in a bundle is
|
|
297
|
+
the `generator` string that names the version.
|
|
298
|
+
|
|
299
|
+
## License
|
|
300
|
+
|
|
301
|
+
Licensed under either of [Apache License, Version 2.0](LICENSE-APACHE) or [MIT License](LICENSE-MIT), at your option, as declared in `pyproject.toml`.
|