bug-age 2.0.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (37) hide show
  1. bug_age-2.0.0/LICENSE-APACHE +202 -0
  2. bug_age-2.0.0/LICENSE-MIT +21 -0
  3. bug_age-2.0.0/PKG-INFO +301 -0
  4. bug_age-2.0.0/README.md +284 -0
  5. bug_age-2.0.0/bug_age/__init__.py +13 -0
  6. bug_age-2.0.0/bug_age/__main__.py +7 -0
  7. bug_age-2.0.0/bug_age/_compat.py +155 -0
  8. bug_age-2.0.0/bug_age/bundle.py +381 -0
  9. bug_age-2.0.0/bug_age/catalog.py +68 -0
  10. bug_age-2.0.0/bug_age/cli.py +265 -0
  11. bug_age-2.0.0/bug_age/data/Dockerfile +14 -0
  12. bug_age-2.0.0/bug_age/data/bundle.schema.json +89 -0
  13. bug_age-2.0.0/bug_age/data/explorer.html +851 -0
  14. bug_age-2.0.0/bug_age/explorer.py +42 -0
  15. bug_age-2.0.0/bug_age/feasibility.py +339 -0
  16. bug_age-2.0.0/bug_age/git.py +71 -0
  17. bug_age-2.0.0/bug_age/mcp.py +211 -0
  18. bug_age-2.0.0/bug_age/pipeline.py +151 -0
  19. bug_age-2.0.0/bug_age/repros.py +178 -0
  20. bug_age-2.0.0/bug_age/runner.py +294 -0
  21. bug_age-2.0.0/bug_age/schema.py +102 -0
  22. bug_age-2.0.0/bug_age/sweep.py +280 -0
  23. bug_age-2.0.0/bug_age/szz.py +135 -0
  24. bug_age-2.0.0/bug_age/timetravel.py +108 -0
  25. bug_age-2.0.0/bug_age.egg-info/PKG-INFO +301 -0
  26. bug_age-2.0.0/bug_age.egg-info/SOURCES.txt +35 -0
  27. bug_age-2.0.0/bug_age.egg-info/dependency_links.txt +1 -0
  28. bug_age-2.0.0/bug_age.egg-info/entry_points.txt +2 -0
  29. bug_age-2.0.0/bug_age.egg-info/top_level.txt +1 -0
  30. bug_age-2.0.0/pyproject.toml +32 -0
  31. bug_age-2.0.0/setup.cfg +4 -0
  32. bug_age-2.0.0/tests/test_bundle.py +128 -0
  33. bug_age-2.0.0/tests/test_explorer_cli_mcp.py +132 -0
  34. bug_age-2.0.0/tests/test_method_a.py +96 -0
  35. bug_age-2.0.0/tests/test_oracle.py +90 -0
  36. bug_age-2.0.0/tests/test_sweep.py +156 -0
  37. bug_age-2.0.0/tests/test_sweep_scenarios.py +80 -0
@@ -0,0 +1,202 @@
1
+
2
+ Apache License
3
+ Version 2.0, January 2004
4
+ http://www.apache.org/licenses/
5
+
6
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
7
+
8
+ 1. Definitions.
9
+
10
+ "License" shall mean the terms and conditions for use, reproduction,
11
+ and distribution as defined by Sections 1 through 9 of this document.
12
+
13
+ "Licensor" shall mean the copyright owner or entity authorized by
14
+ the copyright owner that is granting the License.
15
+
16
+ "Legal Entity" shall mean the union of the acting entity and all
17
+ other entities that control, are controlled by, or are under common
18
+ control with that entity. For the purposes of this definition,
19
+ "control" means (i) the power, direct or indirect, to cause the
20
+ direction or management of such entity, whether by contract or
21
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
22
+ outstanding shares, or (iii) beneficial ownership of such entity.
23
+
24
+ "You" (or "Your") shall mean an individual or Legal Entity
25
+ exercising permissions granted by this License.
26
+
27
+ "Source" form shall mean the preferred form for making modifications,
28
+ including but not limited to software source code, documentation
29
+ source, and configuration files.
30
+
31
+ "Object" form shall mean any form resulting from mechanical
32
+ transformation or translation of a Source form, including but
33
+ not limited to compiled object code, generated documentation,
34
+ and conversions to other media types.
35
+
36
+ "Work" shall mean the work of authorship, whether in Source or
37
+ Object form, made available under the License, as indicated by a
38
+ copyright notice that is included in or attached to the work
39
+ (an example is provided in the Appendix below).
40
+
41
+ "Derivative Works" shall mean any work, whether in Source or Object
42
+ form, that is based on (or derived from) the Work and for which the
43
+ editorial revisions, annotations, elaborations, or other modifications
44
+ represent, as a whole, an original work of authorship. For the purposes
45
+ of this License, Derivative Works shall not include works that remain
46
+ separable from, or merely link (or bind by name) to the interfaces of,
47
+ the Work and Derivative Works thereof.
48
+
49
+ "Contribution" shall mean any work of authorship, including
50
+ the original version of the Work and any modifications or additions
51
+ to that Work or Derivative Works thereof, that is intentionally
52
+ submitted to Licensor for inclusion in the Work by the copyright owner
53
+ or by an individual or Legal Entity authorized to submit on behalf of
54
+ the copyright owner. For the purposes of this definition, "submitted"
55
+ means any form of electronic, verbal, or written communication sent
56
+ to the Licensor or its representatives, including but not limited to
57
+ communication on electronic mailing lists, source code control systems,
58
+ and issue tracking systems that are managed by, or on behalf of, the
59
+ Licensor for the purpose of discussing and improving the Work, but
60
+ excluding communication that is conspicuously marked or otherwise
61
+ designated in writing by the copyright owner as "Not a Contribution."
62
+
63
+ "Contributor" shall mean Licensor and any individual or Legal Entity
64
+ on behalf of whom a Contribution has been received by Licensor and
65
+ subsequently incorporated within the Work.
66
+
67
+ 2. Grant of Copyright License. Subject to the terms and conditions of
68
+ this License, each Contributor hereby grants to You a perpetual,
69
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
70
+ copyright license to reproduce, prepare Derivative Works of,
71
+ publicly display, publicly perform, sublicense, and distribute the
72
+ Work and such Derivative Works in Source or Object form.
73
+
74
+ 3. Grant of Patent License. Subject to the terms and conditions of
75
+ this License, each Contributor hereby grants to You a perpetual,
76
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
77
+ (except as stated in this section) patent license to make, have made,
78
+ use, offer to sell, sell, import, and otherwise transfer the Work,
79
+ where such license applies only to those patent claims licensable
80
+ by such Contributor that are necessarily infringed by their
81
+ Contribution(s) alone or by combination of their Contribution(s)
82
+ with the Work to which such Contribution(s) was submitted. If You
83
+ institute patent litigation against any entity (including a
84
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
85
+ or a Contribution incorporated within the Work constitutes direct
86
+ or contributory patent infringement, then any patent licenses
87
+ granted to You under this License for that Work shall terminate
88
+ as of the date such litigation is filed.
89
+
90
+ 4. Redistribution. You may reproduce and distribute copies of the
91
+ Work or Derivative Works thereof in any medium, with or without
92
+ modifications, and in Source or Object form, provided that You
93
+ meet the following conditions:
94
+
95
+ (a) You must give any other recipients of the Work or
96
+ Derivative Works a copy of this License; and
97
+
98
+ (b) You must cause any modified files to carry prominent notices
99
+ stating that You changed the files; and
100
+
101
+ (c) You must retain, in the Source form of any Derivative Works
102
+ that You distribute, all copyright, patent, trademark, and
103
+ attribution notices from the Source form of the Work,
104
+ excluding those notices that do not pertain to any part of
105
+ the Derivative Works; and
106
+
107
+ (d) If the Work includes a "NOTICE" text file as part of its
108
+ distribution, then any Derivative Works that You distribute must
109
+ include a readable copy of the attribution notices contained
110
+ within such NOTICE file, excluding those notices that do not
111
+ pertain to any part of the Derivative Works, in at least one
112
+ of the following places: within a NOTICE text file distributed
113
+ as part of the Derivative Works; within the Source form or
114
+ documentation, if provided along with the Derivative Works; or,
115
+ within a display generated by the Derivative Works, if and
116
+ wherever such third-party notices normally appear. The contents
117
+ of the NOTICE file are for informational purposes only and
118
+ do not modify the License. You may add Your own attribution
119
+ notices within Derivative Works that You distribute, alongside
120
+ or as an addendum to the NOTICE text from the Work, provided
121
+ that such additional attribution notices cannot be construed
122
+ as modifying the License.
123
+
124
+ You may add Your own copyright statement to Your modifications and
125
+ may provide additional or different license terms and conditions
126
+ for use, reproduction, or distribution of Your modifications, or
127
+ for any such Derivative Works as a whole, provided Your use,
128
+ reproduction, and distribution of the Work otherwise complies with
129
+ the conditions stated in this License.
130
+
131
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
132
+ any Contribution intentionally submitted for inclusion in the Work
133
+ by You to the Licensor shall be under the terms and conditions of
134
+ this License, without any additional terms or conditions.
135
+ Notwithstanding the above, nothing herein shall supersede or modify
136
+ the terms of any separate license agreement you may have executed
137
+ with Licensor regarding such Contributions.
138
+
139
+ 6. Trademarks. This License does not grant permission to use the trade
140
+ names, trademarks, service marks, or product names of the Licensor,
141
+ except as required for reasonable and customary use in describing the
142
+ origin of the Work and reproducing the content of the NOTICE file.
143
+
144
+ 7. Disclaimer of Warranty. Unless required by applicable law or
145
+ agreed to in writing, Licensor provides the Work (and each
146
+ Contributor provides its Contributions) on an "AS IS" BASIS,
147
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
148
+ implied, including, without limitation, any warranties or conditions
149
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
150
+ PARTICULAR PURPOSE. You are solely responsible for determining the
151
+ appropriateness of using or redistributing the Work and assume any
152
+ risks associated with Your exercise of permissions under this License.
153
+
154
+ 8. Limitation of Liability. In no event and under no legal theory,
155
+ whether in tort (including negligence), contract, or otherwise,
156
+ unless required by applicable law (such as deliberate and grossly
157
+ negligent acts) or agreed to in writing, shall any Contributor be
158
+ liable to You for damages, including any direct, indirect, special,
159
+ incidental, or consequential damages of any character arising as a
160
+ result of this License or out of the use or inability to use the
161
+ Work (including but not limited to damages for loss of goodwill,
162
+ work stoppage, computer failure or malfunction, or any and all
163
+ other commercial damages or losses), even if such Contributor
164
+ has been advised of the possibility of such damages.
165
+
166
+ 9. Accepting Warranty or Additional Liability. While redistributing
167
+ the Work or Derivative Works thereof, You may choose to offer,
168
+ and charge a fee for, acceptance of support, warranty, indemnity,
169
+ or other liability obligations and/or rights consistent with this
170
+ License. However, in accepting such obligations, You may act only
171
+ on Your own behalf and on Your sole responsibility, not on behalf
172
+ of any other Contributor, and only if You agree to indemnify,
173
+ defend, and hold each Contributor harmless for any liability
174
+ incurred by, or claims asserted against, such Contributor by reason
175
+ of your accepting any such warranty or additional liability.
176
+
177
+ END OF TERMS AND CONDITIONS
178
+
179
+ APPENDIX: How to apply the Apache License to your work.
180
+
181
+ To apply the Apache License to your work, attach the following
182
+ boilerplate notice, with the fields enclosed by brackets "[]"
183
+ replaced with your own identifying information. (Don't include
184
+ the brackets!) The text should be enclosed in the appropriate
185
+ comment syntax for the file format. We also recommend that a
186
+ file or class name and description of purpose be included on the
187
+ same "printed page" as the copyright notice for easier
188
+ identification within third-party archives.
189
+
190
+ Copyright [yyyy] [name of copyright owner]
191
+
192
+ Licensed under the Apache License, Version 2.0 (the "License");
193
+ you may not use this file except in compliance with the License.
194
+ You may obtain a copy of the License at
195
+
196
+ http://www.apache.org/licenses/LICENSE-2.0
197
+
198
+ Unless required by applicable law or agreed to in writing, software
199
+ distributed under the License is distributed on an "AS IS" BASIS,
200
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
201
+ See the License for the specific language governing permissions and
202
+ limitations under the License.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 jacob-adm
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
bug_age-2.0.0/PKG-INFO ADDED
@@ -0,0 +1,301 @@
1
+ Metadata-Version: 2.4
2
+ Name: bug-age
3
+ Version: 2.0.0
4
+ Summary: Measure how long fixed bugs lived in a git repository, from repository data only.
5
+ License: MIT OR Apache-2.0
6
+ Project-URL: Source, https://github.com/jacob-adm/bug-age
7
+ Classifier: Programming Language :: Python :: 3
8
+ Classifier: Environment :: Console
9
+ Classifier: Topic :: Software Development :: Quality Assurance
10
+ Classifier: License :: OSI Approved :: MIT License
11
+ Classifier: License :: OSI Approved :: Apache Software License
12
+ Requires-Python: >=3.11
13
+ Description-Content-Type: text/markdown
14
+ License-File: LICENSE-APACHE
15
+ License-File: LICENSE-MIT
16
+ Dynamic: license-file
17
+
18
+ # bug-age
19
+
20
+ Measures how long bugs lived in a git repository, using real repository data. For each fixed bug it reports commits elapsed, calendar days, releases shipped, and churn between the commit that introduced it and the fix.
21
+
22
+ It uses two independent methods and compares them:
23
+
24
+ - **Method A, blame-based (SZZ-style).** Runs `git blame` on the lines a fix changed and names the commit that last touched them as the likely introducer. This is fast and needs only git, but it can be wrong. Known failure cases: ghost commits, formatting-only changes, and introducers that were not the latest toucher.
25
+ - **Method B, behavioral sweep.** Takes a regression test that fails before the fix and passes after it, then replays that test across history in a hermetic per-commit Docker environment. The result is the first failing commit, the last passing commit, and every untestable commit with a reason.
26
+
27
+ Every number comes from scripts. AI helped write the code, the Method B regression tests and the article prose, but no model produces or changes a number. See [What is deterministic and what uses AI](#what-is-deterministic-and-what-uses-ai). bug-age supports Rust crates today.
28
+
29
+ What you get, and the three names this README uses for it:
30
+
31
+ - **Report.** What a run produces: the result files in `results/<name>/` and one bundle, `dist/<name>/bundle.json`, which holds every number. Each repository you run gets its own report.
32
+ - **Explorer.** A viewer. It is a static HTML page you build from one or more bundles to browse their reports. It holds no results of its own and runs no analysis.
33
+ - **Live example.** One explorer, built from the reports of the two [example runs](#example-runs-tokenizers-and-rust-sdk) and published online. It is a snapshot: running bug-age on other repositories does not change it.
34
+
35
+ To see the output first, [open the live example](https://claude.ai/artifact/UDw6jyeA4zMDGhnzL7Typb).
36
+
37
+ ## Get started
38
+
39
+ Point bug-age at your own repository in six steps. A first run with Method A only needs git and Python; Docker is needed only for the feasibility check and Method B.
40
+
41
+ **1. Install.**
42
+
43
+ ```sh
44
+ pip install bug-age # Python 3.11 or later, no dependencies
45
+ bug-age dockerfile | docker build -t bug-age-spike:4 - # only for feasibility and Method B
46
+ ```
47
+
48
+ You also need git. The Docker image name `bug-age-spike:4` is the CLI default; pass `--image` to use another.
49
+
50
+ **2. Describe your repository in a config file.** Copy an existing one and edit it:
51
+
52
+ ```sh
53
+ curl -sO https://raw.githubusercontent.com/jacob-adm/bug-age/main/targets/rust-sdk.toml
54
+ mkdir -p targets && mv rust-sdk.toml targets/myrepo.toml
55
+ ```
56
+
57
+ | Key | Required | What to set |
58
+ |---|---|---|
59
+ | `name` | yes | Short name: lowercase letters, digits and `-`. Used in file names and the explorer. |
60
+ | `repo_url` | yes | Clone URL of the repository. |
61
+ | `head` | yes | Full SHA of the commit to analyse up to. Pinning it makes the run reproducible. |
62
+ | `crate_path` | yes | Directory of the crate inside the repository (`.` for the root). Only history under this path is analysed. |
63
+ | `work_dir`, `results_dir`, `spike_dir`, `dist_dir` | yes | Where the clone, results, feasibility reports and bundle go, relative to this repository. Follow the pattern `work/<name>`, `results/<name>`, `results/<name>/spike`, `dist/<name>`. |
64
+ | `release_tag_glob` | no | Pattern matching your release tags, for release counts. Default `v[0-9]*`. Set it if your tags look different, e.g. `mycrate-v[0-9]*`, or every release count will be 0. |
65
+ | `package` | no | Cargo package to build and test in a workspace. Default: the whole workspace. |
66
+ | `samples` | no | Commits sampled by the feasibility check. Default 20. |
67
+ | `szz_window_start` | no | `YYYY-MM-DD`. Flags fixes whose introducer falls on or after this date. |
68
+ | `out_of_scope` | no | `STAGE\|EVIDENCE\|DOES_NOT_PROVE`: a part of the repository you deliberately leave out, listed as NOT TESTED. |
69
+
70
+ **3. Run the pipeline.**
71
+
72
+ ```sh
73
+ bug-age pipeline targets/myrepo.toml --skip-feasibility # Method A only, no Docker
74
+ bug-age pipeline targets/myrepo.toml # adds the Docker feasibility check
75
+ ```
76
+
77
+ **4. Read the report.** `results/myrepo/PIPELINE.md` lists every stage with its status (OK, REUSED, FAILED or NOT TESTED), the reason, and the time taken. The bundle, `dist/myrepo/bundle.json`, is the report of record: every number for this repository is in it. Its sha256 and the exact command that built it are in `dist/myrepo/receipt.json`.
78
+
79
+ **5. Browse the report in your own explorer.** This builds a viewer for your bundle on your machine. The published live example is not affected and keeps showing only the two example runs. To browse several of your repositories together, list them all: `--targets a=dist/a,b=dist/b`.
80
+
81
+ ```sh
82
+ bug-age explorer --targets myrepo=dist/myrepo --out dist/explorer-myrepo
83
+ open dist/explorer-myrepo/index.html # or open the file in any browser
84
+ ```
85
+
86
+ **6. Optional: date selected bugs by testing (Method B).** Pick a fix from `results/myrepo/catalog.json` and add a regression test at `results/myrepo/repros/<first 10 chars of the fix SHA>/overlay/tests/bugage_repro.rs`. The overlay is copied into the crate at every commit tested, so the test can only use the crate's public API. It must fail at the commit just before the fix and pass at the fix; otherwise the sweep rejects it as `repro-invalid`. Run step 3 again: Method B runs whenever a regression test has no sweep yet. See `results/tokenizers/repros/` for six examples.
87
+
88
+ ## Example runs: tokenizers and rust-sdk
89
+
90
+ The same pipeline was run on two Rust codebases. The first, the core crate of [huggingface/tokenizers](https://github.com/huggingface/tokenizers), got every stage, including Method B sweeps for six bugs. The second, the `rmcp` crate of [modelcontextprotocol/rust-sdk](https://github.com/modelcontextprotocol/rust-sdk), was added with a config file alone, to show the pipeline carries over unchanged.
91
+
92
+ **[▶ Open the live example](https://claude.ai/artifact/UDw6jyeA4zMDGhnzL7Typb)**: an explorer built from these two reports and published online. Switch between the two repositories at the top. It is a snapshot of the committed bundles and changes only when it is rebuilt and republished. Each number opens to its evidence: the SHAs, the test runs, the receipt hash and the command that reproduces it. The page also recomputes its headline numbers in your browser and flags any mismatch.
93
+
94
+ [![The bug-age explorer: headline bug ages, self-check and stage coverage for tokenizers. Click to open.](article/figures/overview-tokenizers.png)](https://claude.ai/artifact/UDw6jyeA4zMDGhnzL7Typb)
95
+
96
+ | | tokenizers | rust-sdk |
97
+ |---|---|---|
98
+ | Config | [`targets/tokenizers.toml`](targets/tokenizers.toml) | [`targets/rust-sdk.toml`](targets/rust-sdk.toml) |
99
+ | Method A (every catalogued fix) | [`results/tokenizers/METHOD_A.md`](results/tokenizers/METHOD_A.md) | [`results/rust-sdk/szz.json`](results/rust-sdk/szz.json) |
100
+ | Method B (behavioural sweeps) | Six bugs: [`results/tokenizers/AB.md`](results/tokenizers/AB.md) | NOT TESTED: no regression tests written yet |
101
+ | Stage report | [`results/tokenizers/PIPELINE.md`](results/tokenizers/PIPELINE.md) | [`results/rust-sdk/PIPELINE.md`](results/rust-sdk/PIPELINE.md) |
102
+ | Report of record (bundle) | [`dist/tokenizers/bundle.json`](dist/tokenizers/bundle.json) | [`dist/rust-sdk/bundle.json`](dist/rust-sdk/bundle.json) |
103
+
104
+ ### What the explorer shows
105
+
106
+ These screenshots are captured by `make article` from the explorer built for the two example runs, so they always match the committed bundles.
107
+
108
+ **Lifespans.** Each bar runs from the nominated introducing commit to the fix. A date scrubber shows which bugs were alive on any day.
109
+
110
+ ![Lifespan of every traced tokenizers bug](article/figures/timeline-tokenizers.png)
111
+
112
+ **Commit × test results.** Every commit the pipeline built and tested in Docker without network: green passes, red fails, grey does not build there. Each square opens the run log.
113
+
114
+ ![Commit-by-test results for tokenizers feasibility runs and regression sweeps](article/figures/heatmap-tokenizers.png)
115
+
116
+ ### Reading Method B verdicts
117
+
118
+ Verdicts are honest about what could not be tested:
119
+
120
+ | Verdict | Meaning |
121
+ |---|---|
122
+ | `located` | The exact first failing commit was found |
123
+ | `censored-by-testability` | The bug is present at the first commit where the test compiles, so its age is a lower bound (`≥`) |
124
+ | `left-censored` | The bug is already present at the start of the sweep range (3 years before the fix), so its age is a lower bound |
125
+
126
+ The feasibility check, which measures toolchain and dependency drift across history, is in `spike/results/` for tokenizers and `results/rust-sdk/spike/` for rust-sdk.
127
+
128
+ ## How bugs are found
129
+
130
+ bug-age does not discover new bugs. It works from bugs the project has already fixed, and uses testing only to date some of them.
131
+
132
+ 1. **Find the fixes (no testing).** Read the first-parent commit history of the crate and keep commits whose subject says `fix`, `bug` or `hotfix`. Fixed rules exclude typo, docs, lint, CI, release and similar commits. Each remaining commit counts as one bug. Every decision is recorded with the rule that made it.
133
+ 2. **Date every fix from history (Method A, no testing).** Blame the lines each fix changed; the commit that last touched them is the nominated introducer. Age is the time, commits and releases between that commit and the fix.
134
+ 3. **Date selected bugs by testing (Method B).** For a few bugs, a regression test reproduces the bug. It must fail just before the fix and pass at the fix, or it is rejected. The test is then run on older commits, in Docker without network, to find where the failure first appears.
135
+
136
+ What this means for the numbers:
137
+
138
+ - Bugs that were never fixed, or were fixed in a commit whose subject does not say so, are not counted.
139
+ - A commit that says "fix" but is not a bug fix is counted unless an exclusion rule catches it. This is one reason Method A is reported as PARTIAL.
140
+ - Method A covers every catalogued fix but is a nomination. Method B is evidence, but only for the bugs that have a regression test.
141
+
142
+ ## How long it takes
143
+
144
+ <!-- timings:begin -->
145
+
146
+ Measured on Darwin arm64, Docker 28.3.3 arm64. Figures come from committed run records via `scripts/timings.py`; full tables in [`results/TIMINGS.md`](results/TIMINGS.md). Step time excludes container staging and toolchain installs, so expect wall time to be longer.
147
+
148
+ | Stage | What was measured | Time |
149
+ |---|---|---|
150
+ | catalog | last pipeline run, 2 target(s) | 3-4 s |
151
+ | Method A | last pipeline run, 2 target(s) | 27-53 s |
152
+ | bundle | last pipeline run, 2 target(s) | 5-6 s |
153
+ | Feasibility (one run, 20 sampled commits) | 4 runs | 4 min to 7 min per run |
154
+ | Method B sweep (one bug, current search) | 6 sweeps, 84 commits | 101 s to 6 min of step time per bug; median per commit 7 s to 22 s |
155
+ | Method B, all 6 tokenizers sweeps end to end | one unattended run, wall clock | 20 min |
156
+
157
+ Method B dominates: each bug needs a regression test, and the current search built and tested 5 to 23 commits per bug in Docker. Catalog, Method A and bundle together took 36 to 62 s per repository.
158
+
159
+ <!-- timings:end -->
160
+
161
+ ## What is deterministic and what uses AI
162
+
163
+ **Deterministic.** These parts make no model calls. Given the same repository, pinned commit and config, they produce the same output, and `make pipeline` needs no API key.
164
+
165
+ | Part | How it works |
166
+ |---|---|
167
+ | Fix catalog | Regular-expression rules over commit subjects, plus exclusion rules (docs, lint, release, ...). Locked by `classify_rules`. |
168
+ | Method A | `git blame` on the lines each fix changed; days, commits, releases and churn computed from git. |
169
+ | Buildability and Method B sweeps | Each commit built and tested in Docker with no network and a date-matched toolchain. PASS or FAIL comes from the test's exit status, and decisive commits are re-run. |
170
+ | Bundle and summary | Medians, 90th percentiles, counts and coverage computed by the CLI; the blameless gate rejects names and emails. |
171
+ | Explorer (viewer) | Displays the bundles it was built from as they are, recomputes the headline numbers in the browser, and ranks search results with fixed rules. |
172
+ | MCP server | Returns values copied from the bundle; unknown input is an error, never a guess. |
173
+ | Article numbers | Inserted from the bundle by `scripts/article.py`, which fails on any number typed into the prose. |
174
+ | History scan | Fixed identity and email patterns over every committed file and message. |
175
+
176
+ **Uses AI.** In each case the AI's output is checked mechanically, and none of it becomes a number.
177
+
178
+ | Part | What AI did | What checks it |
179
+ |---|---|---|
180
+ | Method B regression tests (`results/<target>/repros/*/overlay`) | Drafted by an AI coding agent from each fix's diff | The sweep counts a test only if it fails before the fix and passes after it; otherwise the verdict is `repro-invalid`. Whether a test captures the real bug is a human judgement, so read the overlay. |
181
+ | Article prose (`article/ARTICLE.tmpl.md`) | Drafted with AI | Every number is a bundle reference; the check fails on anything else. |
182
+ | "Ask Claude" in the explorer (optional) | Claude answers a question on the viewer's own account, from the rows the search found | The answer is shown only if every cited SHA and every number appears in those rows. It runs only when the viewer clicks, and is hidden where unavailable. |
183
+ | The code | Written with an AI coding agent (Claude Code) and reviewed by the maintainer | Each change merged through a pull request after `cargo test`; commits carry a `Co-Authored-By` line. |
184
+
185
+ ## CLI
186
+
187
+ ```
188
+ bug-age catalog --repo DIR --path CRATE [--head REV] [--out FILE]
189
+ bug-age szz --repo DIR --catalog FILE [--window-start YYYY-MM-DD] [--out FILE]
190
+ bug-age sweep --repo DIR --path CRATE --fix SHA --overlay DIR --test CMD
191
+ [--build CMD] [--start SHA] [--runner docker|local] [--repeats N] [--image IMG]
192
+ [--time-travel-script FILE|none] [--scratch DIR] [--logs DIR] [--out FILE]
193
+ [--namespace NAME] [--bucket N]
194
+ bug-age probe --repo DIR --path CRATE --commits SHA,SHA,... --overlay DIR --test CMD [same options]
195
+ bug-age bundle --repo DIR --results DIR --repo-url URL --out DIR [--spike DIR]
196
+ bug-age explorer [--page FILE] --targets NAME=DIR,... --out DIR
197
+ bug-age mcp --bundles NAME=FILE,...
198
+ bug-age pipeline TARGET.toml [--fresh] [--skip-feasibility]
199
+ bug-age dockerfile
200
+ ```
201
+
202
+ | Command | What it does |
203
+ |---|---|
204
+ | `catalog` | Lists the fix commits that touch a crate and records why each other commit was excluded (docs, lint, release, and so on) |
205
+ | `szz` | Method A: names a likely introducing commit for each fix in the catalog |
206
+ | `sweep` | Method B: binary-searches for the first commit where the test compiles, then bisects PASS/FAIL above it. Decisive commits are re-run `--repeats` times and flagged UNSTABLE if the result changes |
207
+ | `probe` | Runs the same test at a fixed list of commits, with no search |
208
+ | `bundle` | Builds `bundle.json` (validated against [`bug_age/data/bundle.schema.json`](bug_age/data/bundle.schema.json)), `bundle.js`, and `receipt.json` (sha256 hashes and sizes) |
209
+ | `explorer` | Builds an explorer: a static viewer page for the listed bundles, with one data file per repository; fails over 8 MB. It reads reports and adds no results. |
210
+ | `mcp` | Read-only MCP server on stdio answering bundle queries; see [`dist/agent/AGENTS.md`](dist/agent/AGENTS.md) |
211
+ | `pipeline` | Runs every stage for one target config; paths in the config are relative to the current directory |
212
+ | `dockerfile` | Prints the Dockerfile of the Method B image |
213
+
214
+ ## Pipeline reference
215
+
216
+ Each target repository has one config file in [`targets/`](targets/), and nothing else is target-specific:
217
+
218
+ ```toml
219
+ name = "rust-sdk"
220
+ repo_url = "https://github.com/modelcontextprotocol/rust-sdk"
221
+ head = "5596a8a0bba2a1ec5519cc855cdadf23e4dcecdc" # pinned subject
222
+ crate_path = "crates/rmcp"
223
+ work_dir = "work/rust-sdk"
224
+ results_dir = "results/rust-sdk"
225
+ spike_dir = "results/rust-sdk/spike"
226
+ dist_dir = "dist/rust-sdk"
227
+ samples = 20
228
+ # optional: szz_window_start = "YYYY-MM-DD", out_of_scope = "STAGE|EVIDENCE|DOES_NOT_PROVE"
229
+ ```
230
+
231
+ One command runs every stage:
232
+
233
+ ```sh
234
+ bug-age pipeline targets/rust-sdk.toml # or, in a clone: make pipeline TARGET=targets/rust-sdk.toml
235
+ ```
236
+
237
+ The stages are clone at the pinned head, feasibility (baseline and time-travel dependencies), catalog, Method A, Method B for every repro under `<results_dir>/repros/<fix>/overlay`, and bundle.
238
+
239
+ - A stage whose output already exists is REUSED, so the expensive Docker stages are never redone silently. Pass `--fresh` to rerun them.
240
+ - A failing stage is recorded with its reason, and the stages that don't depend on it still run.
241
+ - The run writes `<results_dir>/PIPELINE.md` and `pipeline.json`, listing each stage's status, reason, time taken and output sha256.
242
+ - A stage that did not run appears in the bundle's coverage as NOT TESTED.
243
+
244
+ Method B needs a regression test per bug, written as a test overlay. Those tests are drafted per target, so a target without any shows Method B as NOT TESTED.
245
+
246
+ ### Unattended sweeps
247
+
248
+ ```sh
249
+ make sweep ARGS="results/bench work/tokenizers tokenizers"
250
+ ```
251
+
252
+ The sweep runs detached, with no agent involved. It writes `<results>/SUMMARY.md` and sends one macOS notification when it finishes. To check a run against committed results:
253
+
254
+ ```sh
255
+ python3 results/compare_runs.py results/tokenizers results/bench # prints IDENTICAL or the differences
256
+ ```
257
+
258
+ ## Rigor rules
259
+
260
+ - The subject is pinned: repo URL, SHA range, toolchain, and tool versions.
261
+ - Sweeps run with `--network none`, a date-matched toolchain, and dependencies resolved as they were at that date.
262
+ - A result is PASS only when its assertions actually ran. Anything else is reported as NOT TESTED.
263
+ - The bundle is blameless. It contains SHAs, dates, and short commit subjects only. Author names, emails, and usernames are redacted, and a gate fails the build if any remain.
264
+
265
+ ## Tests
266
+
267
+ ```sh
268
+ python3 -m unittest discover -s tests # unit, CLI, oracle and sweep tests
269
+ BUG_AGE_DOCKER_TESTS=1 python3 -m unittest discover -s tests # adds the Docker runner test (needs the image)
270
+ ```
271
+
272
+ The oracle test regenerates both example runs from clones in `work/` and compares every output
273
+ with the committed files byte for byte. The sweep tests build a toy crate, so they need cargo.
274
+
275
+ ## Contributing
276
+
277
+ Changes reach `main` only through pull requests.
278
+
279
+ 1. Fork the repository and create a branch.
280
+ 2. Make the change, then run `python3 -m unittest discover -s tests`. If you changed bundles, results or the article, also run `make article timings`.
281
+ 3. Open a pull request against `main`. CI runs the tests and the article and timing checks on Linux and macOS; all must pass before merging. Pull requests are merged with a merge commit.
282
+
283
+ Keep the rules in [Rigor rules](#rigor-rules): numbers come from scripts, unrun stages are NOT TESTED, and bundles stay blameless.
284
+
285
+ ## Versions
286
+
287
+ 1.0.2 is the last version written in Rust. It measures Rust crates: Method A needs git and a
288
+ Rust 1.86 toolchain to build the CLI (`cargo build`), and Method B needs Docker. It is the version
289
+ that produced the tokenizers and rust-sdk example runs. To use it, check out its tag:
290
+
291
+ ```sh
292
+ git clone --branch v1.0.2 https://github.com/jacob-adm/bug-age
293
+ ```
294
+
295
+ 2.0.0 and later are a Python port of the same tool, installed with `pip install bug-age`. On both
296
+ example runs it reproduces every output of 1.0.2 byte for byte; the only difference in a bundle is
297
+ the `generator` string that names the version.
298
+
299
+ ## License
300
+
301
+ Licensed under either of [Apache License, Version 2.0](LICENSE-APACHE) or [MIT License](LICENSE-MIT), at your option, as declared in `pyproject.toml`.