bumpcheck-py 0.1.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (30) hide show
  1. bumpcheck_py-0.1.2/PKG-INFO +469 -0
  2. bumpcheck_py-0.1.2/README.md +455 -0
  3. bumpcheck_py-0.1.2/pyproject.toml +57 -0
  4. bumpcheck_py-0.1.2/pyproject.toml.orig +49 -0
  5. bumpcheck_py-0.1.2/src/bumpcheck/__init__.py +35 -0
  6. bumpcheck_py-0.1.2/src/bumpcheck/apidiff.py +771 -0
  7. bumpcheck_py-0.1.2/src/bumpcheck/archive.py +322 -0
  8. bumpcheck_py-0.1.2/src/bumpcheck/baselines.py +40 -0
  9. bumpcheck_py-0.1.2/src/bumpcheck/bench/__init__.py +1 -0
  10. bumpcheck_py-0.1.2/src/bumpcheck/bench/eval.py +349 -0
  11. bumpcheck_py-0.1.2/src/bumpcheck/bench/eval_worker.py +428 -0
  12. bumpcheck_py-0.1.2/src/bumpcheck/bench/github.py +676 -0
  13. bumpcheck_py-0.1.2/src/bumpcheck/bench/inspect_case.py +1162 -0
  14. bumpcheck_py-0.1.2/src/bumpcheck/bench/labels.py +770 -0
  15. bumpcheck_py-0.1.2/src/bumpcheck/bench/lockfiles.py +28 -0
  16. bumpcheck_py-0.1.2/src/bumpcheck/bench/mine.py +1261 -0
  17. bumpcheck_py-0.1.2/src/bumpcheck/bench/models.py +133 -0
  18. bumpcheck_py-0.1.2/src/bumpcheck/bench/report.py +495 -0
  19. bumpcheck_py-0.1.2/src/bumpcheck/bench/snapshot.py +259 -0
  20. bumpcheck_py-0.1.2/src/bumpcheck/bench/titles.py +156 -0
  21. bumpcheck_py-0.1.2/src/bumpcheck/check.py +244 -0
  22. bumpcheck_py-0.1.2/src/bumpcheck/cli.py +916 -0
  23. bumpcheck_py-0.1.2/src/bumpcheck/envfile.py +114 -0
  24. bumpcheck_py-0.1.2/src/bumpcheck/lockdiff.py +244 -0
  25. bumpcheck_py-0.1.2/src/bumpcheck/models.py +80 -0
  26. bumpcheck_py-0.1.2/src/bumpcheck/projectpython.py +489 -0
  27. bumpcheck_py-0.1.2/src/bumpcheck/pycompat.py +260 -0
  28. bumpcheck_py-0.1.2/src/bumpcheck/rules.py +401 -0
  29. bumpcheck_py-0.1.2/src/bumpcheck/usage.py +754 -0
  30. bumpcheck_py-0.1.2/src/bumpcheck/verdict.py +117 -0
@@ -0,0 +1,469 @@
1
+ Metadata-Version: 2.3
2
+ Name: bumpcheck-py
3
+ Version: 0.1.2
4
+ Summary: Analyze whether a dependency bump will break your code.
5
+ Author: Sushit-prog
6
+ Author-email: Sushit-prog <pakrashys@gmail.com>
7
+ Requires-Dist: griffe>=2.4.1
8
+ Requires-Dist: packaging
9
+ Requires-Dist: pydantic>=2
10
+ Requires-Dist: pyyaml>=6
11
+ Requires-Dist: typer
12
+ Requires-Python: >=3.11
13
+ Description-Content-Type: text/markdown
14
+
15
+ BumpCheck analyzes whether a dependency bump in a Dependabot or Renovate pull request will break your code.
16
+
17
+ Status: early development (v0.1.2)
18
+
19
+ ## Install
20
+
21
+ BumpCheck is published as the `bumpcheck-py` distribution and installs a
22
+ `bumpcheck` command:
23
+
24
+ ```
25
+ uv tool install bumpcheck-py
26
+ ```
27
+
28
+ ## Real-world validation
29
+
30
+ Validate the lockfile parser against real GitHub pull requests. List the PR
31
+ URLs (one per line, `#` comments allowed) in `realworld/prs.txt`, then run:
32
+
33
+ ```
34
+ uv run python scripts/realworld.py fetch
35
+ uv run python scripts/realworld.py report
36
+ ```
37
+
38
+ `fetch` downloads the before/after lockfiles into `realworld/data/` and
39
+ `report` writes `realworld/report.md` and `realworld/report.json`, including
40
+ any places where the parser fails or disagrees with the PR title.
41
+
42
+ ## API diff
43
+
44
+ Compare the public API of two versions of a PyPI package:
45
+
46
+ ```
47
+ uv run bumpcheck diff PACKAGE OLD_VERSION NEW_VERSION [--cache-dir PATH]
48
+ ```
49
+
50
+ BumpCheck downloads wheels only (never source distributions, and it never
51
+ installs or imports the package), verifies their hashes, and unpacks them into
52
+ a cache directory (`--cache-dir`, else `$BUMPCHECK_CACHE`, else
53
+ `~/.cache/bumpcheck`). Packages without a usable wheel produce a
54
+ `no_api_diff` result whose `reason` explains why.
55
+
56
+ ## Usage tracer
57
+
58
+ Find where a repository's own Python code touches symbols from a package:
59
+
60
+ ```
61
+ uv run bumpcheck usage REPO_PATH PACKAGE --symbols SYM [--symbols SYM ...]
62
+ uv run bumpcheck usage REPO_PATH PACKAGE --api-diff API_DIFF.json
63
+ ```
64
+
65
+ `--api-diff` accepts an `ApiDiffResult` JSON file and traces every changed
66
+ symbol, including parameter names and change kinds in the `parameters` field
67
+ of the result. `--import-name` overrides the import name(s) used for matching;
68
+ it may be repeated. Without `--api-diff`, the default import name is the
69
+ package name with `-` replaced by `_`. With `--api-diff`, its `top_level`
70
+ list supplies the default import names.
71
+
72
+ The tracer is static: it parses Python files with `ast` and never imports or
73
+ executes repository code. It reports `call`, `attribute`, `import`, and
74
+ `reference` hits with a source snippet, keyword argument names, positional
75
+ argument counts, star-argument flags, and an `exact` or `possible` confidence.
76
+ It also reports risks that make a clean result unsafe to treat as proof:
77
+
78
+ - `star_import`: `from package import *` makes names unresolvable.
79
+ - `dynamic_getattr`: non-literal `getattr` on a package module.
80
+ - `dynamic_import`: `importlib.import_module` / `__import__` with a package
81
+ name.
82
+ - `dynamic_import_unknown_target`: a non-literal dynamic import.
83
+ - `unparsable_file`: at least one file could not be parsed.
84
+ - `scan_truncated`: the 5000-file cap was reached.
85
+ - `method_target_untraced`: a method-like target had no related hits.
86
+
87
+ Same-scope assignments of the form `x = PackageClass(...)` are tracked so
88
+ `x.method()` and `x.attribute` resolve to the target symbol with `possible`
89
+ confidence. Files are parsed as bytes with PEP 263 encoding detection; files
90
+ larger than 2 MB and symlinks are skipped, and excluded directories include
91
+ `.git`, `.venv`, `venv`, `node_modules`, `__pycache__`, `build`, `dist`, and
92
+ `site-packages`.
93
+
94
+ ### Usage tracer limitations
95
+
96
+ - Method calls on instances are only tracked for direct same-scope assignments
97
+ (`x = Target(...)`); cross-function flow and return values are not followed.
98
+ - Re-exports through the user's own modules are not followed. A package's own
99
+ `__init__` re-exports are approximated by a final-segment fallback under the
100
+ import root, with `possible` confidence.
101
+ - Transitive dependencies are not traced; only imports that resolve to the
102
+ requested package are considered.
103
+ - Star imports, `getattr`, and dynamic imports are flagged as risks but their
104
+ concrete targets are not resolved.
105
+ - The import table is file-level, not scope-aware, so shadowed or locally
106
+ imported names can produce conservative matches.
107
+ - Package source is skipped only when the repository's `pyproject.toml`
108
+ `[project].name` canonicalizes to the requested package.
109
+
110
+ ## Verdicts and rules
111
+
112
+ Classify every bump between two lockfiles against a repository:
113
+
114
+ ```
115
+ uv run bumpcheck check REPO_PATH OLD NEW --kind uv.lock [--format json|text|markdown]
116
+ uv run bumpcheck check REPO_PATH OLD NEW --kind uv.lock --fail-on REVIEW
117
+ ```
118
+
119
+ The pipeline parses the lockfiles, runs the API diff for each bumped package,
120
+ traces usages in the repository, and applies the first matching rule:
121
+
122
+ | Rule | Condition | Verdict |
123
+ | ---- | --------- | ------- |
124
+ | R0 | A package has multiple versions or no comparable version | `UNKNOWN` |
125
+ | R1 | The API diff returned `no_api_diff` | `UNKNOWN` |
126
+ | R2 | The API diff is `ok` and has no changes | `SAFE` |
127
+ | R3 | Changes exist but usage was not traced | `UNKNOWN` |
128
+ | R4 | An exact-confidence hit matches breaking or high-severity changes | `LIKELY_BREAKING` / `REVIEW` |
129
+ | R5 | Any other usage hit matches a change | `REVIEW` |
130
+ | R6 | No matching hits, but a blocking risk exists | `REVIEW` |
131
+ | R7 | No matching hits and no blocking risks | `SAFE` |
132
+ | R8 | `requires_python` tightens support for a project Python version, or cannot be evaluated | `LIKELY_BREAKING` / `REVIEW` / `UNKNOWN` |
133
+ | R9 | R7 would return `SAFE` but no Python files were scanned | `UNKNOWN` |
134
+ | E1 | One bump raised an exception; other bumps still run | `UNKNOWN` |
135
+
136
+ R4 marks `OBJECT_REMOVED` and `OBJECT_CHANGED_KIND` as `LIKELY_BREAKING`.
137
+ `PARAMETER_REMOVED` is breaking when the call passes that keyword, and
138
+ `PARAMETER_ADDED_REQUIRED` is breaking when the call omits it without star
139
+ arguments or keyword splats. Other high-severity parameter changes are
140
+ `REVIEW`. R6 blocking risks are `star_import`, `dynamic_getattr`,
141
+ `dynamic_import`, `scan_truncated`, `method_target_untraced`, and
142
+ `unparsable_file`; `dynamic_import_unknown_target` alone does not block `SAFE`.
143
+ Overall severity is `LIKELY_BREAKING` > `REVIEW` > `UNKNOWN` > `SAFE`, so any
144
+ `UNKNOWN` bump prevents an overall `SAFE`.
145
+
146
+ R8 compares old and new `requires_python` over CPython 3.8–3.14. A tightened
147
+ specifier that excludes a concrete CI or `.python-version` version is
148
+ `LIKELY_BREAKING`; one that only narrows a declared `pyproject.toml`/`setup`
149
+ range is `REVIEW`; an unevaluable specifier or unknown project set is `UNKNOWN`.
150
+ Project Python resolution uses CI matrix first, then declared config, then
151
+ `.python-version`. R9 prevents R7 from returning `SAFE` when zero Python files
152
+ were scanned.
153
+
154
+ ### What SAFE does and does not mean
155
+
156
+ `SAFE` means the API diff found no relevant changes and the static tracer found
157
+ no direct usages that match them. It covers API signatures and direct static
158
+ usage only. It does not prove behavioral compatibility, transitive breakage,
159
+ runtime-only imports, or usage reached through untraced instance flow.
160
+
161
+ ## GitHub Action
162
+
163
+ The root `action.yml` is a reusable composite action that compares the base
164
+ and head versions of one lockfile, uploads a Markdown report, and posts a
165
+ sticky pull request comment. This repository also uses it in
166
+ `.github/workflows/bumpcheck-pr.yml`:
167
+
168
+ ```
169
+ uses: Sushit-prog/Bumpcheck@main
170
+ with:
171
+ github-token: ${{ secrets.GITHUB_TOKEN }}
172
+ lockfile: uv.lock
173
+ fail-on: REVIEW
174
+ ```
175
+
176
+ Inputs are `github-token`, `lockfile` (default `uv.lock`), `lock-kind`
177
+ (auto-detected when empty), `fail-on` (default `REVIEW`), `comment` (default
178
+ `true`), and `artifact-name`. Outputs are `overall`, `report-path`, and
179
+ `exit-code`. The example above points at `Sushit-prog/Bumpcheck@main`, which
180
+ matches this repository's `origin` and current `main` branch; this repository's
181
+ own workflow uses `uses: ./` to test the local action. The action checks out the
182
+ base ref for tooling and the PR head into a temporary path for static analysis,
183
+ so the PR's code is never executed. A workflow using the action needs
184
+ `contents: read` and `pull-requests: write`.
185
+
186
+ ## Benchmark
187
+
188
+ Mine public Dependabot and Renovate pull requests into hand-labellable
189
+ candidates:
190
+
191
+ ```
192
+ uv run bumpcheck bench mine --out bench/v0-candidates --bot both \
193
+ --since 2026-01-01 --limit 60 --logs
194
+ ```
195
+
196
+ The miner searches GitHub for closed single-package bump PRs in Python
197
+ projects, keeps only PRs whose before/after lockfile parser output matches a
198
+ single bumped package named in the title, and records the CI outcome for the
199
+ head commit. It writes three files to `--out`:
200
+
201
+ - `candidates.jsonl`: one candidate per line, sorted by `case_id`. Every
202
+ candidate has a null `label` field, ready for hand labelling.
203
+ - `excluded.jsonl`: every inspected PR that was rejected, with a reason:
204
+ `group_bump`, `not_single_package`, `non_python_package`,
205
+ `no_supported_lockfile`, `parse_mismatch`, `no_ci`,
206
+ `ci_not_attributable`, `closed_without_ci_signal`, `noise_repo`,
207
+ `repo_cap`, or `other`. Each record keeps the PR title so rejections can be
208
+ audited, and a short `detail` (at most 200 characters) explains
209
+ `parse_mismatch` (title vs. lockfile bumps), pending/failing check names for
210
+ `other`/`ci_not_attributable`, and exception type/message for `other`
211
+ (URL query strings stripped).
212
+ - `summary.json`: counts per label suggestion, bot, split and exclusion
213
+ reason, plus distinct repos, API calls (`api_calls`), search requests
214
+ (`search_calls`), and the `retried`/`recovered` counters from
215
+ `--retry-excluded`.
216
+
217
+ Renovate single-package bumps carry `autoclosed: true` when the title was
218
+ decorated with `- autoclosed`; Dependabot titles do not set this flag. Renovate
219
+ titles are stripped of trailing `- autoclosed`, `[SECURITY]`-style tags and
220
+ `(major)`-style parenthesised decorations (repeatedly, never down to an empty
221
+ title) before parsing. Parenthesised text inside the title is not stripped by
222
+ that step, but text such as the `(fork)` in `Update dependency foo (fork) to v1`
223
+ prevents the Renovate title from parsing, so the PR is excluded as
224
+ `not_single_package`. Dependabot's
225
+ `Bump X from A to B in the <group> group across N directories` form is treated
226
+ as a single-package bump, not a group bump.
227
+
228
+ Discovery walks fixed `--window-days` windows (default 7) from `--since` to
229
+ `--until`, ordered deterministically by the SHA-256 of each window's start
230
+ date so neighbouring requests are not adjacent dates. Windows are visited
231
+ lazily and sequentially: for each window, each selected bot is searched
232
+ (Dependabot first) for at most `--pages-per-window` pages (default 1, max 10),
233
+ and items from each page are yielded sorted by `(created_at, number)` before
234
+ the next bot or window is fetched. `--ci-status {any,failure,success}` (default
235
+ `any`) adds GitHub's `status:` qualifier to the search and is copied into each
236
+ candidate's `sampled_by`. `--limit` (default 60) counts candidates already
237
+ stored in `candidates.jsonl` from earlier runs; discovery stops when that kept
238
+ count is reached, while `--retry-excluded` recoveries may push it above the
239
+ limit. `--max-per-repo` (default 3) caps kept candidates per repository,
240
+ including earlier runs, and records over-cap hits as `repo_cap` before any PR
241
+ request. There is no window splitting: pages are read only up to
242
+ `--pages-per-window`, and GitHub's 1000-result search cap is not compensated
243
+ for. The next window is fetched only when another item is needed, and reaching
244
+ `--limit` stops all further search requests.
245
+
246
+ Titles classified as single-package bumps whose package name contains `/` or
247
+ starts with `@` (GitHub Actions/Docker references and npm scopes) are excluded
248
+ as `non_python_package` before any per-PR request. `--dependabot-label TEXT`
249
+ (default `python`; pass `""` to disable) appends `label:TEXT` to Dependabot
250
+ search queries only; Renovate queries are unchanged.
251
+
252
+ Repos matching a built-in noise list (for example repo names starting with
253
+ `skills-`, plus `secure-code-game`, `codeql`, `copilot-codespaces`, `opensafely`)
254
+ are recorded as `noise_repo` from search results alone, and a repo that already
255
+ contributed `--max-per-repo` candidates (default 3) is recorded as `repo_cap`.
256
+ Both checks run before any PR request, so they cost no API budget. Passing one
257
+ or more `--skip-repo-pattern` regexes (case-insensitive) replaces the built-in
258
+ list. A pattern that starts with a literal `^` and contains no `/` is matched
259
+ against the bare repo name, so `^skills-` skips `someuser/skills-introduction`
260
+ but keeps `skills-org/real-project`; `^acme` likewise excludes `x/acme-tools`
261
+ but not `acme/widget`. Every other pattern is matched against the full
262
+ `owner/name`, so `opensafely` still hits the owner and `^acme/` targets an
263
+ owner explicitly. Only a leading literal `^` counts: an inline-flag form such
264
+ as `(?i)^skills-` is treated as unanchored and matched against `owner/name`.
265
+ `--ci-status {any,failure,success}` adds GitHub's `status:` qualifier to the
266
+ search and is copied into each candidate's `sampled_by` field.
267
+
268
+ `--retry-excluded REASON` (repeatable) re-processes stored `excluded.jsonl`
269
+ entries with that reason in a separate pass before discovery, even when
270
+ `--limit` is already met. Retried recoveries can therefore push the kept count
271
+ above `--limit`; discovery only runs when the kept count is still below it.
272
+ Each selected entry is handled in sorted `case_id` order directly from its
273
+ stored `repo` and `pr_number`: a stored title is re-classified without a
274
+ request and the PR is fetched only if it now looks like a single-package bump,
275
+ while a legacy entry without a title first fetches `/pulls/{n}` once to read
276
+ the title and author login (the fetched title is stored even if the entry stays
277
+ excluded). Recovered entries go through the normal per-PR pipeline and move to
278
+ `candidates.jsonl`; anything that fails again is re-excluded with its new
279
+ reason and `detail`. The same noise-repo patterns and `--max-per-repo` cap
280
+ apply, and progress lines are prefixed `[retry]`. Entries not selected by
281
+ `--retry-excluded` are untouched.
282
+
283
+ CI outcomes are noisy suggestions, not labels. `OK_SUSPECT` means every
284
+ signal-bearing check passed on a merged PR, `BROKE_SUSPECT` means at least one
285
+ non-lint check failed with a `regression` or `no_base_run` comparison, and
286
+ `UNCLEAR` covers everything else (for example a
287
+ passing PR that was closed without merging). For failing checks the same check
288
+ is compared against the base commit (`base_ci_checks`, `failure_vs_base`):
289
+ a `regression` (passed at base) or a check with `no_base_run` keeps
290
+ `BROKE_SUSPECT`, while failures that were already `preexisting` at base are
291
+ downgraded to `UNCLEAR`. Cancelled, skipped, neutral and
292
+ stale conclusions are ignored; PRs with pending checks are excluded. Every
293
+ candidate still needs hand verification.
294
+
295
+ With `--logs`, each failing check yields a `failure_excerpts` entry whose
296
+ `log_status` is `matched` (package-specific exception lines were found),
297
+ `fetched_no_match`, or `unavailable` (no job id or the log could not be
298
+ fetched; `--logs` itself requires `GITHUB_TOKEN`). For `BROKE_SUSPECT` merged
299
+ PRs, `later_fix_commits`
300
+ lists commits after the merge that touch the lockfile or mention the package;
301
+ the PR's own merge commit, messages that merge this PR (`#<number>`), and
302
+ commits dated at or before the merge/close time are filtered out.
303
+
304
+ Responses are cached under `.bumpcheck-cache/github/` (override with
305
+ `--cache-dir`), and `--offline` replays a warm cache without touching the
306
+ network. `GITHUB_TOKEN` is optional but required for `--logs` and for usable
307
+ rate limits (see [Credentials](#credentials)); it is never written to the cache
308
+ or to any output file. Requests are
309
+ spaced at least 1 second apart (search at least 3), transient network errors
310
+ are retried up to four times with jittered exponential backoff, and
311
+ rate-limit 403/429 responses are retried with the API's own retry hint (or a
312
+ doubling backoff, up to four retries) before failing with the API's message.
313
+ A computed rate-limit wait longer than `--max-rate-limit-wait` (default 900
314
+ seconds) is not attempted: the command exits 3 with the reset time instead.
315
+ Rerunning
316
+ into the same `--out` skips cases that are already recorded; `--limit` counts
317
+ those earlier candidates too. Each processed PR prints one progress line to
318
+ stderr, and both JSONL files are rewritten atomically (temp file + replace)
319
+ after every case, so pressing Ctrl-C saves what has been mined, writes
320
+ `summary.json`, and exits with code 130; a rerun resumes from that state.
321
+
322
+ The `dev`/`test` split is assigned deterministically per repository
323
+ (`sha256(repo)` first byte below 77 means `test`, about 30%), so reruns and
324
+ resumed runs keep the same split and no PR leaks between splits. Discovery uses
325
+ the same fixed `--window-days` windows and SHA-256 ordering described above for
326
+ every selected bot: all items from one window are yielded, sorted by
327
+ `created_at` and PR number, before later windows are visited. Windows are never
328
+ split in half; `--pages-per-window` is the only page bound, and a day with more
329
+ than 1000 matching PRs is still truncated by GitHub's own search results.
330
+
331
+ ### Labeling rules
332
+
333
+ Hand-label mined candidates in a store separate from `candidates.jsonl`:
334
+
335
+ ```
336
+ uv run bumpcheck bench label --dir bench/v0-candidates
337
+ uv run bumpcheck bench label-stats --dir bench/v0-candidates
338
+ uv run bumpcheck bench agreement --dir bench/v0-candidates
339
+ ```
340
+
341
+ `labels.jsonl` holds one JSON object per `(case_id, round)`, rewritten
342
+ atomically after every label. `BROKE` requires a `break_category`, an
343
+ `evidence_kind`, and non-empty evidence tying the failure to the **bumped
344
+ package**: an error naming its symbols (`log_names_package`), an install error
345
+ for it (`install_error`), a maintainer fix in a later commit
346
+ (`maintainer_fix`), a local reproduction (`reproduced_locally`), or
347
+ `Requires-Python`/dependency metadata from the new release (`pypi_metadata`).
348
+ A failing check alone is never enough. A failure already present on the base commit is
349
+ `EXCLUDE` with `preexisting_failure`, not `BROKE`. `OK` is only allowed for a
350
+ merged PR whose `ci_summary` is `all_passed`; closed-unmerged and `autoclosed`
351
+ PRs can never be `OK` and are normally `EXCLUDE` (`superseded`) or `BROKE` with
352
+ evidence. `EXCLUDE` always needs a reason and evidence.
353
+
354
+ The miner's `label_suggestion` is hidden by default so the reviewer is not
355
+ anchored on its judgement (`--show-suggestion` reveals it). Hiding it mitigates
356
+ anchoring but does not remove it: check conclusions are still displayed and can
357
+ influence the decision. Use `bench label --round 2` at least a week later to
358
+ re-label a random 20% of the round-1 `BROKE`/`OK` cases (`--sample 0.2 --seed
359
+ 42`), optionally restricted with `--round1-label BROKE`, then report raw
360
+ agreement and Cohen's kappa from `bench agreement`. `--only` filters by miner
361
+ suggestion, and repeatable `--also-dir DIR` skips case_ids already labeled for
362
+ that round in another store. Labels are never saved for a case already labeled
363
+ in the selected round; re-labeling the same `(case_id, round)` replaces it.
364
+
365
+ ### Inspecting candidates
366
+
367
+ Print a read-only report for one mined case. No candidate or label files are
368
+ written; only the response cache may be updated:
369
+
370
+ ```
371
+ uv run bumpcheck bench inspect acme/widgets#123 --dir bench/v0-candidates
372
+ ```
373
+
374
+ The report includes the PR header, PyPI `requires_python` and dependency changes
375
+ for both versions, workflow triggers/jobs/python-version matrices and relevant
376
+ `run:` lines, a package-filtered lockfile diff, Python-version verdict hints
377
+ against the new release's `requires_python`, and inferred workflow/job locations
378
+ for failing checks. `--also-dir DIR` (repeatable) falls back to other candidate
379
+ stores when the case is missing from `--dir`. `--offline` replays the response
380
+ cache only; `--cache-dir` selects the cache directory (default
381
+ `.bumpcheck-cache/github`). Batch inspection is available as
382
+ `bumpcheck bench inspect-batch --dir DIR --out PATH`.
383
+
384
+ ### Snapshot, eval and report
385
+
386
+ Fetch offline snapshots, evaluate labeled cases against them, then write the
387
+ Markdown report:
388
+
389
+ ```
390
+ uv run bumpcheck bench snapshot --dir bench/v0-candidates
391
+ uv run bumpcheck bench eval --dir bench/v0-candidates [--split dev|test|all]
392
+ uv run bumpcheck bench report --dir bench/v0-candidates [--split dev|test|all]
393
+ ```
394
+
395
+ `--split` defaults to `dev` for both `eval` and `report`; `all` includes both
396
+ splits. The test split is evaluated once at the end; pass `--split test` only
397
+ for that.
398
+
399
+ ### v0.1 validation
400
+
401
+ The v0.1 rules were frozen after dev labeling at commit `953b0bc`; the held-out
402
+ test split was labeled and evaluated once afterward. This is a small,
403
+ failure-enriched, single-annotator benchmark, so these raw counts are not an
404
+ accuracy estimate for real-world traffic. The test split contained no
405
+ hand-labeled `BROKE` cases, so recall and false-safe rate are not measurable
406
+ there.
407
+
408
+ | split | BROKE/OK evaluated | false-safe | breakage recall | SAFE precision | UNKNOWN |
409
+ | ----- | ------------------ | ---------- | --------------- | -------------- | ------- |
410
+ | dev | 5/23 | 0/5 | 5/5 | 15/15 | 4/28 |
411
+ | test | 0/21 | n/a | n/a | 13/13 | 3/21 |
412
+ | all | 5/44 | 0/5 | 5/5 | 28/28 | 7/49 |
413
+
414
+ All five `BROKE` cases are `python_version_support` failures detected from
415
+ `requires_python` metadata. Two dev cases failed snapshot ingestion and are
416
+ reported separately, never counted as `SAFE`; the held-out split had no
417
+ evaluation errors.
418
+
419
+ ## Credentials
420
+
421
+ GitHub commands read `GITHUB_TOKEN` from a `.env`-style file at startup. Files
422
+ are checked in this order, and the first non-empty token wins:
423
+
424
+ 1. the file named by `BUMPCHECK_ENV_FILE` (it must exist, or the command exits 2)
425
+ 2. `~/.bumpcheck/.env`
426
+ 3. `./.env` in the current directory (last, so a cloned repo cannot override
427
+ your own token)
428
+
429
+ A non-empty `GITHUB_TOKEN` already in the real environment always wins over
430
+ every file; an empty or whitespace-only value counts as unset. Only
431
+ `GITHUB_TOKEN` is ever read from these files: every other key (`PATH`,
432
+ `PYTHONPATH`, proxy variables, ...) is ignored, so a hostile repository cannot
433
+ change how the command runs. Values are used verbatim: no variable expansion
434
+ and no multi-line values. On POSIX, a file readable by group or others prints a
435
+ one-line warning; restrict it with `chmod 600`.
436
+
437
+ Without a token, requests are unauthenticated (60 per hour). `bench mine` and
438
+ `bench inspect` print a warning and exit 2 unless you pass
439
+ `--allow-unauthenticated`; `--offline` skips the check entirely. A token that
440
+ GitHub rejects (401) exits 2 with a clear message. Use
441
+ `--max-rate-limit-wait SECONDS` (default 900, `0` fails immediately) to cap
442
+ rate-limit waits; exceeding it exits 3 and prints the reset time. Run
443
+ `bumpcheck --verbose ...` to see which file supplied the token (the value is
444
+ never printed).
445
+
446
+ ```
447
+ # ~/.bumpcheck/.env, outside any repository you commit
448
+ GITHUB_TOKEN=ghp_your_token_here
449
+ ```
450
+
451
+ Never commit the file. This project's `.gitignore` already ignores `.env` and
452
+ `.env.*`; keep the token in `~/.bumpcheck/.env` or point
453
+ `BUMPCHECK_ENV_FILE` at a file outside the repository.
454
+
455
+ ## Acceptance run
456
+
457
+ Check the API diff against known breaking releases listed in
458
+ `acceptance/pairs.json` (each entry names the package, both versions, the
459
+ expected breakages, and a source link). This fetches wheels from PyPI:
460
+
461
+ ```
462
+ uv run python scripts/apidiff_acceptance.py run [--pairs PATH] [--cache-dir PATH] [--only PACKAGE]
463
+ ```
464
+
465
+ Each expectation is reported as `HIT` or `MISS`; a pair whose comparison is
466
+ impossible is reported as `NO_API_DIFF` with its reason, and any other failure
467
+ as `ERROR` without aborting the run. Results are written to
468
+ `acceptance/report.md` and `acceptance/report.json`.
469
+