bumpcheck-py 0.1.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- bumpcheck_py-0.1.2/PKG-INFO +469 -0
- bumpcheck_py-0.1.2/README.md +455 -0
- bumpcheck_py-0.1.2/pyproject.toml +57 -0
- bumpcheck_py-0.1.2/pyproject.toml.orig +49 -0
- bumpcheck_py-0.1.2/src/bumpcheck/__init__.py +35 -0
- bumpcheck_py-0.1.2/src/bumpcheck/apidiff.py +771 -0
- bumpcheck_py-0.1.2/src/bumpcheck/archive.py +322 -0
- bumpcheck_py-0.1.2/src/bumpcheck/baselines.py +40 -0
- bumpcheck_py-0.1.2/src/bumpcheck/bench/__init__.py +1 -0
- bumpcheck_py-0.1.2/src/bumpcheck/bench/eval.py +349 -0
- bumpcheck_py-0.1.2/src/bumpcheck/bench/eval_worker.py +428 -0
- bumpcheck_py-0.1.2/src/bumpcheck/bench/github.py +676 -0
- bumpcheck_py-0.1.2/src/bumpcheck/bench/inspect_case.py +1162 -0
- bumpcheck_py-0.1.2/src/bumpcheck/bench/labels.py +770 -0
- bumpcheck_py-0.1.2/src/bumpcheck/bench/lockfiles.py +28 -0
- bumpcheck_py-0.1.2/src/bumpcheck/bench/mine.py +1261 -0
- bumpcheck_py-0.1.2/src/bumpcheck/bench/models.py +133 -0
- bumpcheck_py-0.1.2/src/bumpcheck/bench/report.py +495 -0
- bumpcheck_py-0.1.2/src/bumpcheck/bench/snapshot.py +259 -0
- bumpcheck_py-0.1.2/src/bumpcheck/bench/titles.py +156 -0
- bumpcheck_py-0.1.2/src/bumpcheck/check.py +244 -0
- bumpcheck_py-0.1.2/src/bumpcheck/cli.py +916 -0
- bumpcheck_py-0.1.2/src/bumpcheck/envfile.py +114 -0
- bumpcheck_py-0.1.2/src/bumpcheck/lockdiff.py +244 -0
- bumpcheck_py-0.1.2/src/bumpcheck/models.py +80 -0
- bumpcheck_py-0.1.2/src/bumpcheck/projectpython.py +489 -0
- bumpcheck_py-0.1.2/src/bumpcheck/pycompat.py +260 -0
- bumpcheck_py-0.1.2/src/bumpcheck/rules.py +401 -0
- bumpcheck_py-0.1.2/src/bumpcheck/usage.py +754 -0
- bumpcheck_py-0.1.2/src/bumpcheck/verdict.py +117 -0
|
@@ -0,0 +1,469 @@
|
|
|
1
|
+
Metadata-Version: 2.3
|
|
2
|
+
Name: bumpcheck-py
|
|
3
|
+
Version: 0.1.2
|
|
4
|
+
Summary: Analyze whether a dependency bump will break your code.
|
|
5
|
+
Author: Sushit-prog
|
|
6
|
+
Author-email: Sushit-prog <pakrashys@gmail.com>
|
|
7
|
+
Requires-Dist: griffe>=2.4.1
|
|
8
|
+
Requires-Dist: packaging
|
|
9
|
+
Requires-Dist: pydantic>=2
|
|
10
|
+
Requires-Dist: pyyaml>=6
|
|
11
|
+
Requires-Dist: typer
|
|
12
|
+
Requires-Python: >=3.11
|
|
13
|
+
Description-Content-Type: text/markdown
|
|
14
|
+
|
|
15
|
+
BumpCheck analyzes whether a dependency bump in a Dependabot or Renovate pull request will break your code.
|
|
16
|
+
|
|
17
|
+
Status: early development (v0.1.2)
|
|
18
|
+
|
|
19
|
+
## Install
|
|
20
|
+
|
|
21
|
+
BumpCheck is published as the `bumpcheck-py` distribution and installs a
|
|
22
|
+
`bumpcheck` command:
|
|
23
|
+
|
|
24
|
+
```
|
|
25
|
+
uv tool install bumpcheck-py
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
## Real-world validation
|
|
29
|
+
|
|
30
|
+
Validate the lockfile parser against real GitHub pull requests. List the PR
|
|
31
|
+
URLs (one per line, `#` comments allowed) in `realworld/prs.txt`, then run:
|
|
32
|
+
|
|
33
|
+
```
|
|
34
|
+
uv run python scripts/realworld.py fetch
|
|
35
|
+
uv run python scripts/realworld.py report
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
`fetch` downloads the before/after lockfiles into `realworld/data/` and
|
|
39
|
+
`report` writes `realworld/report.md` and `realworld/report.json`, including
|
|
40
|
+
any places where the parser fails or disagrees with the PR title.
|
|
41
|
+
|
|
42
|
+
## API diff
|
|
43
|
+
|
|
44
|
+
Compare the public API of two versions of a PyPI package:
|
|
45
|
+
|
|
46
|
+
```
|
|
47
|
+
uv run bumpcheck diff PACKAGE OLD_VERSION NEW_VERSION [--cache-dir PATH]
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
BumpCheck downloads wheels only (never source distributions, and it never
|
|
51
|
+
installs or imports the package), verifies their hashes, and unpacks them into
|
|
52
|
+
a cache directory (`--cache-dir`, else `$BUMPCHECK_CACHE`, else
|
|
53
|
+
`~/.cache/bumpcheck`). Packages without a usable wheel produce a
|
|
54
|
+
`no_api_diff` result whose `reason` explains why.
|
|
55
|
+
|
|
56
|
+
## Usage tracer
|
|
57
|
+
|
|
58
|
+
Find where a repository's own Python code touches symbols from a package:
|
|
59
|
+
|
|
60
|
+
```
|
|
61
|
+
uv run bumpcheck usage REPO_PATH PACKAGE --symbols SYM [--symbols SYM ...]
|
|
62
|
+
uv run bumpcheck usage REPO_PATH PACKAGE --api-diff API_DIFF.json
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
`--api-diff` accepts an `ApiDiffResult` JSON file and traces every changed
|
|
66
|
+
symbol, including parameter names and change kinds in the `parameters` field
|
|
67
|
+
of the result. `--import-name` overrides the import name(s) used for matching;
|
|
68
|
+
it may be repeated. Without `--api-diff`, the default import name is the
|
|
69
|
+
package name with `-` replaced by `_`. With `--api-diff`, its `top_level`
|
|
70
|
+
list supplies the default import names.
|
|
71
|
+
|
|
72
|
+
The tracer is static: it parses Python files with `ast` and never imports or
|
|
73
|
+
executes repository code. It reports `call`, `attribute`, `import`, and
|
|
74
|
+
`reference` hits with a source snippet, keyword argument names, positional
|
|
75
|
+
argument counts, star-argument flags, and an `exact` or `possible` confidence.
|
|
76
|
+
It also reports risks that make a clean result unsafe to treat as proof:
|
|
77
|
+
|
|
78
|
+
- `star_import`: `from package import *` makes names unresolvable.
|
|
79
|
+
- `dynamic_getattr`: non-literal `getattr` on a package module.
|
|
80
|
+
- `dynamic_import`: `importlib.import_module` / `__import__` with a package
|
|
81
|
+
name.
|
|
82
|
+
- `dynamic_import_unknown_target`: a non-literal dynamic import.
|
|
83
|
+
- `unparsable_file`: at least one file could not be parsed.
|
|
84
|
+
- `scan_truncated`: the 5000-file cap was reached.
|
|
85
|
+
- `method_target_untraced`: a method-like target had no related hits.
|
|
86
|
+
|
|
87
|
+
Same-scope assignments of the form `x = PackageClass(...)` are tracked so
|
|
88
|
+
`x.method()` and `x.attribute` resolve to the target symbol with `possible`
|
|
89
|
+
confidence. Files are parsed as bytes with PEP 263 encoding detection; files
|
|
90
|
+
larger than 2 MB and symlinks are skipped, and excluded directories include
|
|
91
|
+
`.git`, `.venv`, `venv`, `node_modules`, `__pycache__`, `build`, `dist`, and
|
|
92
|
+
`site-packages`.
|
|
93
|
+
|
|
94
|
+
### Usage tracer limitations
|
|
95
|
+
|
|
96
|
+
- Method calls on instances are only tracked for direct same-scope assignments
|
|
97
|
+
(`x = Target(...)`); cross-function flow and return values are not followed.
|
|
98
|
+
- Re-exports through the user's own modules are not followed. A package's own
|
|
99
|
+
`__init__` re-exports are approximated by a final-segment fallback under the
|
|
100
|
+
import root, with `possible` confidence.
|
|
101
|
+
- Transitive dependencies are not traced; only imports that resolve to the
|
|
102
|
+
requested package are considered.
|
|
103
|
+
- Star imports, `getattr`, and dynamic imports are flagged as risks but their
|
|
104
|
+
concrete targets are not resolved.
|
|
105
|
+
- The import table is file-level, not scope-aware, so shadowed or locally
|
|
106
|
+
imported names can produce conservative matches.
|
|
107
|
+
- Package source is skipped only when the repository's `pyproject.toml`
|
|
108
|
+
`[project].name` canonicalizes to the requested package.
|
|
109
|
+
|
|
110
|
+
## Verdicts and rules
|
|
111
|
+
|
|
112
|
+
Classify every bump between two lockfiles against a repository:
|
|
113
|
+
|
|
114
|
+
```
|
|
115
|
+
uv run bumpcheck check REPO_PATH OLD NEW --kind uv.lock [--format json|text|markdown]
|
|
116
|
+
uv run bumpcheck check REPO_PATH OLD NEW --kind uv.lock --fail-on REVIEW
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
The pipeline parses the lockfiles, runs the API diff for each bumped package,
|
|
120
|
+
traces usages in the repository, and applies the first matching rule:
|
|
121
|
+
|
|
122
|
+
| Rule | Condition | Verdict |
|
|
123
|
+
| ---- | --------- | ------- |
|
|
124
|
+
| R0 | A package has multiple versions or no comparable version | `UNKNOWN` |
|
|
125
|
+
| R1 | The API diff returned `no_api_diff` | `UNKNOWN` |
|
|
126
|
+
| R2 | The API diff is `ok` and has no changes | `SAFE` |
|
|
127
|
+
| R3 | Changes exist but usage was not traced | `UNKNOWN` |
|
|
128
|
+
| R4 | An exact-confidence hit matches breaking or high-severity changes | `LIKELY_BREAKING` / `REVIEW` |
|
|
129
|
+
| R5 | Any other usage hit matches a change | `REVIEW` |
|
|
130
|
+
| R6 | No matching hits, but a blocking risk exists | `REVIEW` |
|
|
131
|
+
| R7 | No matching hits and no blocking risks | `SAFE` |
|
|
132
|
+
| R8 | `requires_python` tightens support for a project Python version, or cannot be evaluated | `LIKELY_BREAKING` / `REVIEW` / `UNKNOWN` |
|
|
133
|
+
| R9 | R7 would return `SAFE` but no Python files were scanned | `UNKNOWN` |
|
|
134
|
+
| E1 | One bump raised an exception; other bumps still run | `UNKNOWN` |
|
|
135
|
+
|
|
136
|
+
R4 marks `OBJECT_REMOVED` and `OBJECT_CHANGED_KIND` as `LIKELY_BREAKING`.
|
|
137
|
+
`PARAMETER_REMOVED` is breaking when the call passes that keyword, and
|
|
138
|
+
`PARAMETER_ADDED_REQUIRED` is breaking when the call omits it without star
|
|
139
|
+
arguments or keyword splats. Other high-severity parameter changes are
|
|
140
|
+
`REVIEW`. R6 blocking risks are `star_import`, `dynamic_getattr`,
|
|
141
|
+
`dynamic_import`, `scan_truncated`, `method_target_untraced`, and
|
|
142
|
+
`unparsable_file`; `dynamic_import_unknown_target` alone does not block `SAFE`.
|
|
143
|
+
Overall severity is `LIKELY_BREAKING` > `REVIEW` > `UNKNOWN` > `SAFE`, so any
|
|
144
|
+
`UNKNOWN` bump prevents an overall `SAFE`.
|
|
145
|
+
|
|
146
|
+
R8 compares old and new `requires_python` over CPython 3.8–3.14. A tightened
|
|
147
|
+
specifier that excludes a concrete CI or `.python-version` version is
|
|
148
|
+
`LIKELY_BREAKING`; one that only narrows a declared `pyproject.toml`/`setup`
|
|
149
|
+
range is `REVIEW`; an unevaluable specifier or unknown project set is `UNKNOWN`.
|
|
150
|
+
Project Python resolution uses CI matrix first, then declared config, then
|
|
151
|
+
`.python-version`. R9 prevents R7 from returning `SAFE` when zero Python files
|
|
152
|
+
were scanned.
|
|
153
|
+
|
|
154
|
+
### What SAFE does and does not mean
|
|
155
|
+
|
|
156
|
+
`SAFE` means the API diff found no relevant changes and the static tracer found
|
|
157
|
+
no direct usages that match them. It covers API signatures and direct static
|
|
158
|
+
usage only. It does not prove behavioral compatibility, transitive breakage,
|
|
159
|
+
runtime-only imports, or usage reached through untraced instance flow.
|
|
160
|
+
|
|
161
|
+
## GitHub Action
|
|
162
|
+
|
|
163
|
+
The root `action.yml` is a reusable composite action that compares the base
|
|
164
|
+
and head versions of one lockfile, uploads a Markdown report, and posts a
|
|
165
|
+
sticky pull request comment. This repository also uses it in
|
|
166
|
+
`.github/workflows/bumpcheck-pr.yml`:
|
|
167
|
+
|
|
168
|
+
```
|
|
169
|
+
uses: Sushit-prog/Bumpcheck@main
|
|
170
|
+
with:
|
|
171
|
+
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
172
|
+
lockfile: uv.lock
|
|
173
|
+
fail-on: REVIEW
|
|
174
|
+
```
|
|
175
|
+
|
|
176
|
+
Inputs are `github-token`, `lockfile` (default `uv.lock`), `lock-kind`
|
|
177
|
+
(auto-detected when empty), `fail-on` (default `REVIEW`), `comment` (default
|
|
178
|
+
`true`), and `artifact-name`. Outputs are `overall`, `report-path`, and
|
|
179
|
+
`exit-code`. The example above points at `Sushit-prog/Bumpcheck@main`, which
|
|
180
|
+
matches this repository's `origin` and current `main` branch; this repository's
|
|
181
|
+
own workflow uses `uses: ./` to test the local action. The action checks out the
|
|
182
|
+
base ref for tooling and the PR head into a temporary path for static analysis,
|
|
183
|
+
so the PR's code is never executed. A workflow using the action needs
|
|
184
|
+
`contents: read` and `pull-requests: write`.
|
|
185
|
+
|
|
186
|
+
## Benchmark
|
|
187
|
+
|
|
188
|
+
Mine public Dependabot and Renovate pull requests into hand-labellable
|
|
189
|
+
candidates:
|
|
190
|
+
|
|
191
|
+
```
|
|
192
|
+
uv run bumpcheck bench mine --out bench/v0-candidates --bot both \
|
|
193
|
+
--since 2026-01-01 --limit 60 --logs
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
The miner searches GitHub for closed single-package bump PRs in Python
|
|
197
|
+
projects, keeps only PRs whose before/after lockfile parser output matches a
|
|
198
|
+
single bumped package named in the title, and records the CI outcome for the
|
|
199
|
+
head commit. It writes three files to `--out`:
|
|
200
|
+
|
|
201
|
+
- `candidates.jsonl`: one candidate per line, sorted by `case_id`. Every
|
|
202
|
+
candidate has a null `label` field, ready for hand labelling.
|
|
203
|
+
- `excluded.jsonl`: every inspected PR that was rejected, with a reason:
|
|
204
|
+
`group_bump`, `not_single_package`, `non_python_package`,
|
|
205
|
+
`no_supported_lockfile`, `parse_mismatch`, `no_ci`,
|
|
206
|
+
`ci_not_attributable`, `closed_without_ci_signal`, `noise_repo`,
|
|
207
|
+
`repo_cap`, or `other`. Each record keeps the PR title so rejections can be
|
|
208
|
+
audited, and a short `detail` (at most 200 characters) explains
|
|
209
|
+
`parse_mismatch` (title vs. lockfile bumps), pending/failing check names for
|
|
210
|
+
`other`/`ci_not_attributable`, and exception type/message for `other`
|
|
211
|
+
(URL query strings stripped).
|
|
212
|
+
- `summary.json`: counts per label suggestion, bot, split and exclusion
|
|
213
|
+
reason, plus distinct repos, API calls (`api_calls`), search requests
|
|
214
|
+
(`search_calls`), and the `retried`/`recovered` counters from
|
|
215
|
+
`--retry-excluded`.
|
|
216
|
+
|
|
217
|
+
Renovate single-package bumps carry `autoclosed: true` when the title was
|
|
218
|
+
decorated with `- autoclosed`; Dependabot titles do not set this flag. Renovate
|
|
219
|
+
titles are stripped of trailing `- autoclosed`, `[SECURITY]`-style tags and
|
|
220
|
+
`(major)`-style parenthesised decorations (repeatedly, never down to an empty
|
|
221
|
+
title) before parsing. Parenthesised text inside the title is not stripped by
|
|
222
|
+
that step, but text such as the `(fork)` in `Update dependency foo (fork) to v1`
|
|
223
|
+
prevents the Renovate title from parsing, so the PR is excluded as
|
|
224
|
+
`not_single_package`. Dependabot's
|
|
225
|
+
`Bump X from A to B in the <group> group across N directories` form is treated
|
|
226
|
+
as a single-package bump, not a group bump.
|
|
227
|
+
|
|
228
|
+
Discovery walks fixed `--window-days` windows (default 7) from `--since` to
|
|
229
|
+
`--until`, ordered deterministically by the SHA-256 of each window's start
|
|
230
|
+
date so neighbouring requests are not adjacent dates. Windows are visited
|
|
231
|
+
lazily and sequentially: for each window, each selected bot is searched
|
|
232
|
+
(Dependabot first) for at most `--pages-per-window` pages (default 1, max 10),
|
|
233
|
+
and items from each page are yielded sorted by `(created_at, number)` before
|
|
234
|
+
the next bot or window is fetched. `--ci-status {any,failure,success}` (default
|
|
235
|
+
`any`) adds GitHub's `status:` qualifier to the search and is copied into each
|
|
236
|
+
candidate's `sampled_by`. `--limit` (default 60) counts candidates already
|
|
237
|
+
stored in `candidates.jsonl` from earlier runs; discovery stops when that kept
|
|
238
|
+
count is reached, while `--retry-excluded` recoveries may push it above the
|
|
239
|
+
limit. `--max-per-repo` (default 3) caps kept candidates per repository,
|
|
240
|
+
including earlier runs, and records over-cap hits as `repo_cap` before any PR
|
|
241
|
+
request. There is no window splitting: pages are read only up to
|
|
242
|
+
`--pages-per-window`, and GitHub's 1000-result search cap is not compensated
|
|
243
|
+
for. The next window is fetched only when another item is needed, and reaching
|
|
244
|
+
`--limit` stops all further search requests.
|
|
245
|
+
|
|
246
|
+
Titles classified as single-package bumps whose package name contains `/` or
|
|
247
|
+
starts with `@` (GitHub Actions/Docker references and npm scopes) are excluded
|
|
248
|
+
as `non_python_package` before any per-PR request. `--dependabot-label TEXT`
|
|
249
|
+
(default `python`; pass `""` to disable) appends `label:TEXT` to Dependabot
|
|
250
|
+
search queries only; Renovate queries are unchanged.
|
|
251
|
+
|
|
252
|
+
Repos matching a built-in noise list (for example repo names starting with
|
|
253
|
+
`skills-`, plus `secure-code-game`, `codeql`, `copilot-codespaces`, `opensafely`)
|
|
254
|
+
are recorded as `noise_repo` from search results alone, and a repo that already
|
|
255
|
+
contributed `--max-per-repo` candidates (default 3) is recorded as `repo_cap`.
|
|
256
|
+
Both checks run before any PR request, so they cost no API budget. Passing one
|
|
257
|
+
or more `--skip-repo-pattern` regexes (case-insensitive) replaces the built-in
|
|
258
|
+
list. A pattern that starts with a literal `^` and contains no `/` is matched
|
|
259
|
+
against the bare repo name, so `^skills-` skips `someuser/skills-introduction`
|
|
260
|
+
but keeps `skills-org/real-project`; `^acme` likewise excludes `x/acme-tools`
|
|
261
|
+
but not `acme/widget`. Every other pattern is matched against the full
|
|
262
|
+
`owner/name`, so `opensafely` still hits the owner and `^acme/` targets an
|
|
263
|
+
owner explicitly. Only a leading literal `^` counts: an inline-flag form such
|
|
264
|
+
as `(?i)^skills-` is treated as unanchored and matched against `owner/name`.
|
|
265
|
+
`--ci-status {any,failure,success}` adds GitHub's `status:` qualifier to the
|
|
266
|
+
search and is copied into each candidate's `sampled_by` field.
|
|
267
|
+
|
|
268
|
+
`--retry-excluded REASON` (repeatable) re-processes stored `excluded.jsonl`
|
|
269
|
+
entries with that reason in a separate pass before discovery, even when
|
|
270
|
+
`--limit` is already met. Retried recoveries can therefore push the kept count
|
|
271
|
+
above `--limit`; discovery only runs when the kept count is still below it.
|
|
272
|
+
Each selected entry is handled in sorted `case_id` order directly from its
|
|
273
|
+
stored `repo` and `pr_number`: a stored title is re-classified without a
|
|
274
|
+
request and the PR is fetched only if it now looks like a single-package bump,
|
|
275
|
+
while a legacy entry without a title first fetches `/pulls/{n}` once to read
|
|
276
|
+
the title and author login (the fetched title is stored even if the entry stays
|
|
277
|
+
excluded). Recovered entries go through the normal per-PR pipeline and move to
|
|
278
|
+
`candidates.jsonl`; anything that fails again is re-excluded with its new
|
|
279
|
+
reason and `detail`. The same noise-repo patterns and `--max-per-repo` cap
|
|
280
|
+
apply, and progress lines are prefixed `[retry]`. Entries not selected by
|
|
281
|
+
`--retry-excluded` are untouched.
|
|
282
|
+
|
|
283
|
+
CI outcomes are noisy suggestions, not labels. `OK_SUSPECT` means every
|
|
284
|
+
signal-bearing check passed on a merged PR, `BROKE_SUSPECT` means at least one
|
|
285
|
+
non-lint check failed with a `regression` or `no_base_run` comparison, and
|
|
286
|
+
`UNCLEAR` covers everything else (for example a
|
|
287
|
+
passing PR that was closed without merging). For failing checks the same check
|
|
288
|
+
is compared against the base commit (`base_ci_checks`, `failure_vs_base`):
|
|
289
|
+
a `regression` (passed at base) or a check with `no_base_run` keeps
|
|
290
|
+
`BROKE_SUSPECT`, while failures that were already `preexisting` at base are
|
|
291
|
+
downgraded to `UNCLEAR`. Cancelled, skipped, neutral and
|
|
292
|
+
stale conclusions are ignored; PRs with pending checks are excluded. Every
|
|
293
|
+
candidate still needs hand verification.
|
|
294
|
+
|
|
295
|
+
With `--logs`, each failing check yields a `failure_excerpts` entry whose
|
|
296
|
+
`log_status` is `matched` (package-specific exception lines were found),
|
|
297
|
+
`fetched_no_match`, or `unavailable` (no job id or the log could not be
|
|
298
|
+
fetched; `--logs` itself requires `GITHUB_TOKEN`). For `BROKE_SUSPECT` merged
|
|
299
|
+
PRs, `later_fix_commits`
|
|
300
|
+
lists commits after the merge that touch the lockfile or mention the package;
|
|
301
|
+
the PR's own merge commit, messages that merge this PR (`#<number>`), and
|
|
302
|
+
commits dated at or before the merge/close time are filtered out.
|
|
303
|
+
|
|
304
|
+
Responses are cached under `.bumpcheck-cache/github/` (override with
|
|
305
|
+
`--cache-dir`), and `--offline` replays a warm cache without touching the
|
|
306
|
+
network. `GITHUB_TOKEN` is optional but required for `--logs` and for usable
|
|
307
|
+
rate limits (see [Credentials](#credentials)); it is never written to the cache
|
|
308
|
+
or to any output file. Requests are
|
|
309
|
+
spaced at least 1 second apart (search at least 3), transient network errors
|
|
310
|
+
are retried up to four times with jittered exponential backoff, and
|
|
311
|
+
rate-limit 403/429 responses are retried with the API's own retry hint (or a
|
|
312
|
+
doubling backoff, up to four retries) before failing with the API's message.
|
|
313
|
+
A computed rate-limit wait longer than `--max-rate-limit-wait` (default 900
|
|
314
|
+
seconds) is not attempted: the command exits 3 with the reset time instead.
|
|
315
|
+
Rerunning
|
|
316
|
+
into the same `--out` skips cases that are already recorded; `--limit` counts
|
|
317
|
+
those earlier candidates too. Each processed PR prints one progress line to
|
|
318
|
+
stderr, and both JSONL files are rewritten atomically (temp file + replace)
|
|
319
|
+
after every case, so pressing Ctrl-C saves what has been mined, writes
|
|
320
|
+
`summary.json`, and exits with code 130; a rerun resumes from that state.
|
|
321
|
+
|
|
322
|
+
The `dev`/`test` split is assigned deterministically per repository
|
|
323
|
+
(`sha256(repo)` first byte below 77 means `test`, about 30%), so reruns and
|
|
324
|
+
resumed runs keep the same split and no PR leaks between splits. Discovery uses
|
|
325
|
+
the same fixed `--window-days` windows and SHA-256 ordering described above for
|
|
326
|
+
every selected bot: all items from one window are yielded, sorted by
|
|
327
|
+
`created_at` and PR number, before later windows are visited. Windows are never
|
|
328
|
+
split in half; `--pages-per-window` is the only page bound, and a day with more
|
|
329
|
+
than 1000 matching PRs is still truncated by GitHub's own search results.
|
|
330
|
+
|
|
331
|
+
### Labeling rules
|
|
332
|
+
|
|
333
|
+
Hand-label mined candidates in a store separate from `candidates.jsonl`:
|
|
334
|
+
|
|
335
|
+
```
|
|
336
|
+
uv run bumpcheck bench label --dir bench/v0-candidates
|
|
337
|
+
uv run bumpcheck bench label-stats --dir bench/v0-candidates
|
|
338
|
+
uv run bumpcheck bench agreement --dir bench/v0-candidates
|
|
339
|
+
```
|
|
340
|
+
|
|
341
|
+
`labels.jsonl` holds one JSON object per `(case_id, round)`, rewritten
|
|
342
|
+
atomically after every label. `BROKE` requires a `break_category`, an
|
|
343
|
+
`evidence_kind`, and non-empty evidence tying the failure to the **bumped
|
|
344
|
+
package**: an error naming its symbols (`log_names_package`), an install error
|
|
345
|
+
for it (`install_error`), a maintainer fix in a later commit
|
|
346
|
+
(`maintainer_fix`), a local reproduction (`reproduced_locally`), or
|
|
347
|
+
`Requires-Python`/dependency metadata from the new release (`pypi_metadata`).
|
|
348
|
+
A failing check alone is never enough. A failure already present on the base commit is
|
|
349
|
+
`EXCLUDE` with `preexisting_failure`, not `BROKE`. `OK` is only allowed for a
|
|
350
|
+
merged PR whose `ci_summary` is `all_passed`; closed-unmerged and `autoclosed`
|
|
351
|
+
PRs can never be `OK` and are normally `EXCLUDE` (`superseded`) or `BROKE` with
|
|
352
|
+
evidence. `EXCLUDE` always needs a reason and evidence.
|
|
353
|
+
|
|
354
|
+
The miner's `label_suggestion` is hidden by default so the reviewer is not
|
|
355
|
+
anchored on its judgement (`--show-suggestion` reveals it). Hiding it mitigates
|
|
356
|
+
anchoring but does not remove it: check conclusions are still displayed and can
|
|
357
|
+
influence the decision. Use `bench label --round 2` at least a week later to
|
|
358
|
+
re-label a random 20% of the round-1 `BROKE`/`OK` cases (`--sample 0.2 --seed
|
|
359
|
+
42`), optionally restricted with `--round1-label BROKE`, then report raw
|
|
360
|
+
agreement and Cohen's kappa from `bench agreement`. `--only` filters by miner
|
|
361
|
+
suggestion, and repeatable `--also-dir DIR` skips case_ids already labeled for
|
|
362
|
+
that round in another store. Labels are never saved for a case already labeled
|
|
363
|
+
in the selected round; re-labeling the same `(case_id, round)` replaces it.
|
|
364
|
+
|
|
365
|
+
### Inspecting candidates
|
|
366
|
+
|
|
367
|
+
Print a read-only report for one mined case. No candidate or label files are
|
|
368
|
+
written; only the response cache may be updated:
|
|
369
|
+
|
|
370
|
+
```
|
|
371
|
+
uv run bumpcheck bench inspect acme/widgets#123 --dir bench/v0-candidates
|
|
372
|
+
```
|
|
373
|
+
|
|
374
|
+
The report includes the PR header, PyPI `requires_python` and dependency changes
|
|
375
|
+
for both versions, workflow triggers/jobs/python-version matrices and relevant
|
|
376
|
+
`run:` lines, a package-filtered lockfile diff, Python-version verdict hints
|
|
377
|
+
against the new release's `requires_python`, and inferred workflow/job locations
|
|
378
|
+
for failing checks. `--also-dir DIR` (repeatable) falls back to other candidate
|
|
379
|
+
stores when the case is missing from `--dir`. `--offline` replays the response
|
|
380
|
+
cache only; `--cache-dir` selects the cache directory (default
|
|
381
|
+
`.bumpcheck-cache/github`). Batch inspection is available as
|
|
382
|
+
`bumpcheck bench inspect-batch --dir DIR --out PATH`.
|
|
383
|
+
|
|
384
|
+
### Snapshot, eval and report
|
|
385
|
+
|
|
386
|
+
Fetch offline snapshots, evaluate labeled cases against them, then write the
|
|
387
|
+
Markdown report:
|
|
388
|
+
|
|
389
|
+
```
|
|
390
|
+
uv run bumpcheck bench snapshot --dir bench/v0-candidates
|
|
391
|
+
uv run bumpcheck bench eval --dir bench/v0-candidates [--split dev|test|all]
|
|
392
|
+
uv run bumpcheck bench report --dir bench/v0-candidates [--split dev|test|all]
|
|
393
|
+
```
|
|
394
|
+
|
|
395
|
+
`--split` defaults to `dev` for both `eval` and `report`; `all` includes both
|
|
396
|
+
splits. The test split is evaluated once at the end; pass `--split test` only
|
|
397
|
+
for that.
|
|
398
|
+
|
|
399
|
+
### v0.1 validation
|
|
400
|
+
|
|
401
|
+
The v0.1 rules were frozen after dev labeling at commit `953b0bc`; the held-out
|
|
402
|
+
test split was labeled and evaluated once afterward. This is a small,
|
|
403
|
+
failure-enriched, single-annotator benchmark, so these raw counts are not an
|
|
404
|
+
accuracy estimate for real-world traffic. The test split contained no
|
|
405
|
+
hand-labeled `BROKE` cases, so recall and false-safe rate are not measurable
|
|
406
|
+
there.
|
|
407
|
+
|
|
408
|
+
| split | BROKE/OK evaluated | false-safe | breakage recall | SAFE precision | UNKNOWN |
|
|
409
|
+
| ----- | ------------------ | ---------- | --------------- | -------------- | ------- |
|
|
410
|
+
| dev | 5/23 | 0/5 | 5/5 | 15/15 | 4/28 |
|
|
411
|
+
| test | 0/21 | n/a | n/a | 13/13 | 3/21 |
|
|
412
|
+
| all | 5/44 | 0/5 | 5/5 | 28/28 | 7/49 |
|
|
413
|
+
|
|
414
|
+
All five `BROKE` cases are `python_version_support` failures detected from
|
|
415
|
+
`requires_python` metadata. Two dev cases failed snapshot ingestion and are
|
|
416
|
+
reported separately, never counted as `SAFE`; the held-out split had no
|
|
417
|
+
evaluation errors.
|
|
418
|
+
|
|
419
|
+
## Credentials
|
|
420
|
+
|
|
421
|
+
GitHub commands read `GITHUB_TOKEN` from a `.env`-style file at startup. Files
|
|
422
|
+
are checked in this order, and the first non-empty token wins:
|
|
423
|
+
|
|
424
|
+
1. the file named by `BUMPCHECK_ENV_FILE` (it must exist, or the command exits 2)
|
|
425
|
+
2. `~/.bumpcheck/.env`
|
|
426
|
+
3. `./.env` in the current directory (last, so a cloned repo cannot override
|
|
427
|
+
your own token)
|
|
428
|
+
|
|
429
|
+
A non-empty `GITHUB_TOKEN` already in the real environment always wins over
|
|
430
|
+
every file; an empty or whitespace-only value counts as unset. Only
|
|
431
|
+
`GITHUB_TOKEN` is ever read from these files: every other key (`PATH`,
|
|
432
|
+
`PYTHONPATH`, proxy variables, ...) is ignored, so a hostile repository cannot
|
|
433
|
+
change how the command runs. Values are used verbatim: no variable expansion
|
|
434
|
+
and no multi-line values. On POSIX, a file readable by group or others prints a
|
|
435
|
+
one-line warning; restrict it with `chmod 600`.
|
|
436
|
+
|
|
437
|
+
Without a token, requests are unauthenticated (60 per hour). `bench mine` and
|
|
438
|
+
`bench inspect` print a warning and exit 2 unless you pass
|
|
439
|
+
`--allow-unauthenticated`; `--offline` skips the check entirely. A token that
|
|
440
|
+
GitHub rejects (401) exits 2 with a clear message. Use
|
|
441
|
+
`--max-rate-limit-wait SECONDS` (default 900, `0` fails immediately) to cap
|
|
442
|
+
rate-limit waits; exceeding it exits 3 and prints the reset time. Run
|
|
443
|
+
`bumpcheck --verbose ...` to see which file supplied the token (the value is
|
|
444
|
+
never printed).
|
|
445
|
+
|
|
446
|
+
```
|
|
447
|
+
# ~/.bumpcheck/.env, outside any repository you commit
|
|
448
|
+
GITHUB_TOKEN=ghp_your_token_here
|
|
449
|
+
```
|
|
450
|
+
|
|
451
|
+
Never commit the file. This project's `.gitignore` already ignores `.env` and
|
|
452
|
+
`.env.*`; keep the token in `~/.bumpcheck/.env` or point
|
|
453
|
+
`BUMPCHECK_ENV_FILE` at a file outside the repository.
|
|
454
|
+
|
|
455
|
+
## Acceptance run
|
|
456
|
+
|
|
457
|
+
Check the API diff against known breaking releases listed in
|
|
458
|
+
`acceptance/pairs.json` (each entry names the package, both versions, the
|
|
459
|
+
expected breakages, and a source link). This fetches wheels from PyPI:
|
|
460
|
+
|
|
461
|
+
```
|
|
462
|
+
uv run python scripts/apidiff_acceptance.py run [--pairs PATH] [--cache-dir PATH] [--only PACKAGE]
|
|
463
|
+
```
|
|
464
|
+
|
|
465
|
+
Each expectation is reported as `HIT` or `MISS`; a pair whose comparison is
|
|
466
|
+
impossible is reported as `NO_API_DIFF` with its reason, and any other failure
|
|
467
|
+
as `ERROR` without aborting the run. Results are written to
|
|
468
|
+
`acceptance/report.md` and `acceptance/report.json`.
|
|
469
|
+
|