blot-git 0.1.0__tar.gz → 0.3.0__tar.gz

{blot_git-0.1.0 → blot_git-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: blot-git
-Version: 0.1.0
+Version: 0.3.0
 Summary: A pre-commit check that has read your project's history: warns when a change removes a fix, re-adds a fixed bug, or forgets a file that always changes too.
 Project-URL: Homepage, https://github.com/Sym-jay/Blot
 Project-URL: Issues, https://github.com/Sym-jay/Blot/issues
@@ -42,8 +42,10 @@ pip install blot-git
 cd your-repo
 blot index           # one-time, builds .git/blot.db
 blot install-hook    # adds Blot to your pre-commit hook
+blot why src/app.py:42   # which commit explains this line, skipping refactors
+blot index --github  # optional: issue/PR titles and labels via the gh CLI
 ```
 
-> **Status:** pre-alpha. All three checks work; expect rough edges and please report false alarms.
+> **Status:** pre-alpha. All three checks work. Replayed on 500 recent commits each of Flask, requests, Express, Django and React, Blot warns on 0.6–10% of commits; see the bench results on GitHub. Please report false alarms.
 
 Meet Sora the slime, read the docs, or contribute on GitHub: https://github.com/Sym-jay/Blot

{blot_git-0.1.0 → blot_git-0.3.0}/README.md

@@ -22,7 +22,7 @@ It runs locally, needs no account or AI, and stays quiet unless the history give
 
 *A real run on Flask. Plain `git blame` points to a 2023 commit that only moved this line; Blot traces past it to the 2020 fix that explains it. (Regenerate with `python assets/demo/make_demo.py`.)*
 
-> **Status:** pre-alpha. Milestones M1 and M2 are done: `blot index` builds the history index (Flask's 3,800 commits in ~2 s), `blot install-hook` wires up git, and `blot check` runs the three checks. On Flask it traces an edit to `self_ref = weakref.ref(self)` past the 2023 file split to the 2020 fix, in under 500 ms. See the [roadmap in the PRD](docs/PRD.md#12-milestones-4-weeks-part-time).
+> **Status:** pre-alpha. Milestones M1 and M2 are done: `blot index` builds the history index (Flask's 3,800 commits in ~2 s), `blot install-hook` wires up git, and `blot check` runs the three checks. On Flask it traces an edit to `self_ref = weakref.ref(self)` past the 2023 file split to the 2020 fix, in under 500 ms. Replayed on the last 500 commits of Flask, requests, Express, Django and React, it warns on 0.6–10% of commits; see [bench/RESULTS.md](bench/RESULTS.md). Roadmap: [PRD](docs/PRD.md#12-milestones-4-weeks-part-time).
 
 ## Meet Sora
 
@@ -59,6 +59,34 @@ Remove it any time with `blot install-hook --uninstall`. Settings go in `[tool.b
 
 Or with the [pre-commit](https://pre-commit.com) framework, see `.pre-commit-hooks.yaml`.
 
+## Ask why a line exists
+
+```
+$ blot why src/flask/app.py:355
+src/flask/app.py:355
+  | self_ref = weakref.ref(self)
+  explained by  8efea0cc  2020-10-03  Break reference cycle created by default in Flask instances.  [bug fix]
+  traced past   0ec7f713  2023-06-11  Split the App and Blueprint into Sansio and IO parts  [refactor]
+```
+
+`blot why FILE:LINE` (or `FILE:START-END`) finds the commit that explains a line, skipping commits that only moved it. Plain `git blame` stops at the refactor.
+
+## Issue and PR titles (optional)
+
+```bash
+blot index --github
+```
+
+With the [GitHub CLI](https://cli.github.com) logged in, Blot fetches the titles and labels of the issues and PRs your fix commits mention, so warnings read `fixes #2935 "copy_current_request_context copies old session object"` (a real Flask issue) instead of a bare number. A `bug` label makes a commit count as a load-bearing fix; a fix-looking commit whose issues are only labelled `enhancement` or `documentation` is demoted. Refreshes only fetch what's new. Repos with GitHub Issues turned off (they track bugs elsewhere) are skipped.
+
+## Measure it on your repo
+
+```bash
+blot bench --replay 500
+```
+
+Re-checks your last 500 commits as if each were being committed, and lists every warning Blot would have shown. See [bench/README.md](bench/README.md).
+
 ## Docs
 
 - [What is this? (simple explanation)](docs/EXPLAINER.md)

blot_git-0.3.0/bench/README.md

@@ -0,0 +1,42 @@
+# Bench
+
+How we keep Blot's warnings accurate and rare. Two tools, both in `blot bench`.
+
+## 1. Cases: known right answers
+
+`cases.json` lists real edits on public repos, each labeled with what Blot **should** say.
+
+```bash
+blot bench                      # clones case repos into ~/.cache/blot-bench, prints precision/recall
+blot bench --workdir D:\bench   # put the clones somewhere else (short paths help on Windows)
+```
+
+Each case:
+
+| Field | Meaning |
+|---|---|
+| `id` | short unique name |
+| `repo`, `at` | public repo URL and the commit to check out |
+| `file` | file to edit |
+| `replace` | `["old text", "new text"]`; the old text must appear **exactly once** in the file |
+| `expect.check` | which check is being tested: `removed-fix`, `readded-bug` or `forgot-related-file` |
+| `expect.warn` | `true` if it should warn, `false` if it must stay quiet |
+| `expect.commit` | (warn cases) the commit the warning must cite; a prefix is fine |
+| `notes` | why: link the PR or issue that explains the code |
+
+Add **quiet cases** (`"warn": false`) as often as warning cases. They protect against false alarms.
+
+## 2. Replay: how noisy is Blot on real history?
+
+```bash
+cd some-big-repo
+blot bench --replay 500 --json replay.json
+```
+
+Replay re-checks the last 500 non-merge commits as if each were being committed at that moment,
+using only the history that existed before it. It prints how many commits would have shown a warning
+and lists every warning, so you can label each one as useful or noise. Noise patterns become fixes in
+`classify.py` or the checks, plus a quiet case here so they never come back.
+
+Target: **at most 1 warning per 10 commits**, and most of those should be worth reading.
+See [RESULTS.md](RESULTS.md) for the latest numbers.

blot_git-0.3.0/bench/RESULTS.md

@@ -0,0 +1,73 @@
+# Bench results
+
+Measured 2026-10-22 with `blot bench`, on Windows 11, Python 3.12.
+
+## Cases: 9/9 correct
+
+| Check | Cases | Precision | Recall |
+|---|---|---|---|
+| removed-fix | 6 (3 warn, 3 quiet) | 100% | 100% |
+| readded-bug | 2 (2 warn) | 100% | 100% |
+| forgot-related-file | 1 (1 quiet) | n/a | n/a |
+
+Nine cases is a start, not a verdict. Each new case is a good first issue (see [README.md](README.md)).
+
+## Replay: last 500 non-merge commits of 5 repos
+
+Each commit is checked as if it were being committed at that moment, using only the history before
+it. "Before" is Blot 0.1.0's rules; "after" is this release.
+
+| Repo | Commits | Before: warnings / 10 commits | After | Commits with a warning (after) | Avg check time |
+|---|---|---|---|---|---|
+| pallets/flask | 500 | 1.96 | **0.10** | 5 (1.0%) | 106 ms |
+| psf/requests | 500 | 1.12 | **0.16** | 3 (0.6%) | 52 ms |
+| expressjs/express | 498 | 1.00 | **0.18** | 7 (1.4%) | 49 ms |
+| django/django | 497 | 26.30 | **1.29** | 50 (10.1%) | 479 ms |
+| facebook/react | 483 | 10.00 | **1.30** | 39 (8.1%) | 821 ms |
+
+Target from the PRD: at most 1 warning per 10 commits. Three repos are well under; Django and React are
+slightly over. Check times are in-process (no Python startup) and grow with the size of the change.
+The last rule change (type hints that insert `None`) was re-measured only on requests; it can only
+remove warnings, so the other four numbers are upper bounds.
+
+## Were the warnings worth reading?
+
+Labeled by hand: every warning for Flask, requests and Express; a sample for Django and React.
+
+- **Flask, 5 warnings, 3 useful.** The best: commit `cabda59` changed
+  `bp_subdomain is None` to `is not None` on a line from fix `d7b6c1f` (#4834); a later commit had to
+  fix `subdomain_matching=False` behavior. Noise: a line from a feature commit, a docstring.
+- **Express, 9 warnings, about 7 useful.** The best: commit `697547cd` **reverted the CVE-2024-51999
+  security patch**. Both `removed-fix` and `readded-bug` fired. Others: changes to code from fixes for
+  proxy handling (#2252), redirect encoding and charset handling.
+- **requests, 8 warnings, 1 useful** (dropping `chardet` from code added by fix #4102). The rest come
+  from one mass reformat and one typing commit, which still slip through in a few lines.
+- **Django, 64 warnings.** Mostly fixes and CVE follow-ups that touch code from earlier fixes, e.g.
+  `Fixed #37403 -- Fixed JSONField __in crash on SQLite` flagged for changing code from
+  `#37377 -- Fixed JSONField __in crash`. Useful context for a reviewer, though Django's maintainers
+  know their history well.
+- **React, 63 warnings.** About half are co-change warnings. Useful when a fix lands in one bundler
+  variant (`react-server-dom-webpack`) but not its mirrors (turbopack, parcel); noise for feature-flag
+  fork files, which are often edited separately on purpose.
+
+## What the replays changed
+
+Every noise pattern became a rule and, where possible, a bench case:
+
+| Noise | Fix |
+|---|---|
+| "This fixes #N" on features, refactors, docs (old Flask) | feature/refactor/docs subjects count as routine |
+| `Fixed #N -- Added ...` (Django's ticket-first style) | judge the text after `--` |
+| tests, docs, lock files, translations, CI config | skipped by default (`check-tests` / `check-docs` to opt in) |
+| code moved between files | not a removal or a re-add |
+| reformatting and type hints | a line whose tokens survive in order, with nothing logic-changing inserted, is kept |
+| imports, bare keywords, comments, docstrings | ignored |
+| co-change with hub files | the link must hold both ways (≥ 80% each direction) |
+| re-added lines matching fixes in other files | only fixes to the same file count |
+
+## Reproduce
+
+```bash
+git clone https://github.com/pallets/flask && cd flask
+blot bench --replay 500 --json replay.json
+```

blot_git-0.3.0/bench/cases.json

@@ -0,0 +1,83 @@
+[
+  {
+    "id": "flask-weakref-static-route",
+    "repo": "https://github.com/pallets/flask",
+    "at": "d086db85",
+    "file": "src/flask/app.py",
+    "replace": ["self_ref = weakref.ref(self)", "self_ref = self"],
+    "expect": {"check": "removed-fix", "warn": true, "commit": "8efea0cc"},
+    "notes": "The weakref breaks an app <-> view function reference cycle (#3761). Every git blame variant stops at the refactor 0ec7f713 (Split the App and Blueprint into Sansio and IO parts); the cited commit must be the 2020 fix."
+  },
+  {
+    "id": "flask-readd-bound-send-static-file",
+    "repo": "https://github.com/pallets/flask",
+    "at": "d086db85",
+    "file": "src/flask/app.py",
+    "replace": ["view_func=lambda **kw: self_ref().send_static_file(**kw),  # type: ignore", "view_func=self.send_static_file,"],
+    "expect": {"check": "readded-bug", "warn": true, "commit": "8efea0cc"},
+    "notes": "Restores the exact line 8efea0cc removed; the bound method is what created the reference cycle."
+  },
+  {
+    "id": "flask-quiet-feature-line",
+    "repo": "https://github.com/pallets/flask",
+    "at": "d086db85",
+    "file": "src/flask/app.py",
+    "replace": ["setattr(Flask, method.__name__, add_ctx(base_method))", "setattr(Flask, method.__name__, base_method)"],
+    "expect": {"check": "removed-fix", "warn": false},
+    "notes": "Line comes from 6a649690 (pass context through dispatch methods), a feature commit. Must stay quiet."
+  },
+  {
+    "id": "flask-quiet-cochange-app",
+    "repo": "https://github.com/pallets/flask",
+    "at": "d086db85",
+    "file": "src/flask/app.py",
+    "replace": ["self_ref = weakref.ref(self)", "self_ref = self"],
+    "expect": {"check": "forgot-related-file", "warn": false},
+    "notes": "app.py's strongest co-change partner is CHANGES.rst at ~45%, below the 80% threshold. Must stay quiet."
+  },
+  {
+    "id": "flask-subdomain-logic-edit",
+    "repo": "https://github.com/pallets/flask",
+    "at": "d7b6c1f6703df405c69da45e7e0ba3d1aed512ce",
+    "file": "src/flask/blueprints.py",
+    "replace": ["if state.subdomain is not None and bp_subdomain is None:", "if state.subdomain is not None and bp_subdomain is not None:"],
+    "expect": {"check": "removed-fix", "warn": true, "commit": "d7b6c1f"},
+    "notes": "Real history: cabda59 changed this line right after fix d7b6c1f (#4834); a later commit had to fix subdomain_matching=False behavior. Same tokens plus `not` is a logic edit, not a reformat."
+  },
+  {
+    "id": "express-cve-2024-51999-revert-removed",
+    "repo": "https://github.com/expressjs/express",
+    "at": "2f64f68c37c64ae333e41ff38032d21860f22255",
+    "file": "lib/utils.js",
+    "replace": ["plainObjects: true", "allowPrototypes: true"],
+    "expect": {"check": "removed-fix", "warn": true, "commit": "2f64f68c"},
+    "notes": "Real history: 697547cd reverted the CVE-2024-51999 patch the same day. Found by blot bench --replay."
+  },
+  {
+    "id": "express-cve-2024-51999-revert-readded",
+    "repo": "https://github.com/expressjs/express",
+    "at": "2f64f68c37c64ae333e41ff38032d21860f22255",
+    "file": "lib/utils.js",
+    "replace": ["plainObjects: true", "allowPrototypes: true"],
+    "expect": {"check": "readded-bug", "warn": true, "commit": "2f64f68c"},
+    "notes": "Same revert: `allowPrototypes: true` is the exact line the security patch removed."
+  },
+  {
+    "id": "requests-quiet-black-quotes",
+    "repo": "https://github.com/psf/requests",
+    "at": "2d5517682b3b38547634d153cea43d48fbc8cdb5",
+    "file": "requests/auth.py",
+    "replace": ["if _algorithm == 'MD5' or _algorithm == 'MD5-SESS':", "if _algorithm == \"MD5\" or _algorithm == \"MD5-SESS\":"],
+    "expect": {"check": "removed-fix", "warn": false},
+    "notes": "Black's quote change from 2a6f290b on a line added by fix 6931bd6. Formatting only: must stay quiet."
+  },
+  {
+    "id": "requests-quiet-type-hint",
+    "repo": "https://github.com/psf/requests",
+    "at": "8f6cda9969f4a98c45ea2922f6bcbefc02256202",
+    "file": "src/requests/models.py",
+    "replace": ["    def links(self):", "    def links(self) -> dict[str, dict[str, str]]:"],
+    "expect": {"check": "removed-fix", "warn": false},
+    "notes": "Adding a type hint (561e4b68) to a method added by fix d8dcd35 (#711). Must stay quiet."
+  }
+]

{blot_git-0.1.0 → blot_git-0.3.0}/docs/PRD.md

@@ -190,8 +190,8 @@ The user either reads the linked issue and adjusts the change, or commits anyway
 |---|---|---|
 | 1 | **M1: index + config + hook installer** | `blot index` builds the DB on Flask in under 10 s; `install-hook` is safe with existing hooks |
 | 2 | **M2: three checks** | All unit tests pass, including the "trace past refactor" test (currently skipped) |
-| 3 | **M3: `why` + `--github` + output polish** | The Flask case prints the 2020 fix with the #3761 link |
-| 4 | **M4: bench + launch** | 30 bench cases, precision ≥ 80%, PyPI release, README GIF, Show HN / r/Python post |
+| 3 | **M3: `why` + `--github` + output polish** ✅ | The Flask case prints the 2020 fix, traced past the 2023 refactor; `--github` adds issue titles and labels |
+| 4 | **M4: bench + launch** (bench ✅, launch pending) | 30 bench cases, precision ≥ 80%, PyPI release, README GIF, Show HN / r/Python post |
 
 ## 13. Open questions
 

{blot_git-0.1.0 → blot_git-0.3.0}/docs/PYPI.md

@@ -17,8 +17,10 @@ pip install blot-git
 cd your-repo
 blot index           # one-time, builds .git/blot.db
 blot install-hook    # adds Blot to your pre-commit hook
+blot why src/app.py:42   # which commit explains this line, skipping refactors
+blot index --github  # optional: issue/PR titles and labels via the gh CLI
 ```
 
-> **Status:** pre-alpha. All three checks work; expect rough edges and please report false alarms.
+> **Status:** pre-alpha. All three checks work. Replayed on 500 recent commits each of Flask, requests, Express, Django and React, Blot warns on 0.6–10% of commits; see the bench results on GitHub. Please report false alarms.
 
 Meet Sora the slime, read the docs, or contribute on GitHub: https://github.com/Sym-jay/Blot

{blot_git-0.1.0 → blot_git-0.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "blot-git"
-version = "0.1.0"
+version = "0.3.0"
 description = "A pre-commit check that has read your project's history: warns when a change removes a fix, re-adds a fixed bug, or forgets a file that always changes too."
 readme = "docs/PYPI.md"  # text-only; the GitHub README's relative Sora images break on PyPI
 license = { text = "MIT" }

{blot_git-0.1.0 → blot_git-0.3.0}/src/blot/__init__.py

@@ -1,3 +1,3 @@
 """blot: history-aware pre-commit checks."""
 
-__version__ = "0.1.0"
+__version__ = "0.3.0"

blot_git-0.3.0/src/blot/bench.py

@@ -0,0 +1,253 @@
+"""`blot bench`: measure how accurate Blot is.
+
+Two modes:
+    cases   Run bench/cases.json: scripted edits on public repos with a known right answer
+            ("this edit must warn and cite commit X" / "this edit must stay quiet"), and report
+            precision and recall per check.
+    replay  Re-check the last N commits of a repo as if each one were being committed right then,
+            to count how often Blot would have spoken up and to collect warnings to label by hand.
+
+Replay never lets a commit see its own future. Commit facts (is it a fix? a refactor?) come from a
+full index, which is safe because checks only ever ask about the commit's ancestors. Lines removed
+by fixes only count if that fix is an ancestor of the commit being replayed, and co-change counts
+come from a second index built just before the replay window.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import re
+import shutil
+import subprocess
+import tempfile
+import time
+from dataclasses import asdict, dataclass, field
+
+from blot.checks import REGISTRY, all_checks
+from blot.checks.base import Finding
+from blot.config import Config
+from blot.diff import parse_unified_diff
+from blot.git import run_git
+from blot.index import build_index
+from blot.store import CommitRow, Store
+
+DEFAULT_WORKDIR = os.path.join(os.path.expanduser("~"), ".cache", "blot-bench")
+
+
+# --- replay -----------------------------------------------------------------------------
+
+
+class _ReplayStore:
+    """Looks like a Store to the checks, but hides anything newer than the commit being replayed."""
+
+    def __init__(self, repo: str, full: Store, early: Store):
+        self.repo = repo
+        self.rev = "HEAD"
+        self._full, self._early = full, early
+        self._ancestor: dict[tuple[str, str], bool] = {}
+
+    def get_commit(self, sha: str) -> CommitRow | None:
+        return self._full.get_commit(sha)
+
+    def describe_issue(self, ref: str) -> str:
+        return self._full.describe_issue(ref)
+
+    def fixes_that_removed(self, line_hash: str) -> list[tuple[str, str]]:
+        return [(p, s) for p, s in self._full.fixes_that_removed(line_hash) if self._is_ancestor(s)]
+
+    def partners(self, path: str, min_support: int, min_confidence: float) -> list[tuple[str, int, float]]:
+        return self._early.partners(path, min_support, min_confidence)
+
+    def _is_ancestor(self, sha: str) -> bool:
+        key = (sha, self.rev)
+        if key not in self._ancestor:
+            done = subprocess.run(["git", "-C", self.repo, "merge-base", "--is-ancestor", sha, self.rev],
+                                  capture_output=True)
+            self._ancestor[key] = done.returncode == 0
+        return self._ancestor[key]
+
+
+@dataclass
+class ReplayCommit:
+    sha: str
+    subject: str
+    findings: list[dict]
+    ms: int
+
+
+@dataclass
+class ReplayResult:
+    repo: str
+    head: str
+    checked: int
+    skipped_large: int
+    seconds: float
+    commits: list[ReplayCommit] = field(default_factory=list)
+
+    @property
+    def warned(self) -> list[ReplayCommit]:
+        return [c for c in self.commits if c.findings]
+
+    def by_check(self) -> dict[str, int]:
+        counts = {cls.id: 0 for cls in REGISTRY}
+        for c in self.commits:
+            for f in c.findings:
+                counts[f["check"]] = counts.get(f["check"], 0) + 1
+        return counts
+
+
+def replay(repo: str, count: int, config: Config, max_files: int = 50, progress=None) -> ReplayResult:
+    start = time.perf_counter()
+    head = run_git(repo, "rev-parse", "HEAD").strip()
+    commits = run_git(repo, "rev-list", "--no-merges", f"--max-count={count}", "HEAD").split()[::-1]
+    tmp = tempfile.mkdtemp(prefix="blot-replay-")
+    try:
+        build_index(repo, full=True, config=config, db_path=os.path.join(tmp, "full.db"))
+        oldest_parent = _parent(repo, commits[0]) if commits else None
+        early_db = os.path.join(tmp, "early.db")
+        build_index(repo, full=True, config=config, upto=oldest_parent or commits[0], db_path=early_db)
+        full, early = Store.open(repo, os.path.join(tmp, "full.db")), Store.open(repo, early_db)
+        store = _ReplayStore(repo, full, early)
+        checks = all_checks(config)
+        result = ReplayResult(repo, head, 0, 0, 0.0)
+        for i, sha in enumerate(commits, 1):
+            parent = _parent(repo, sha)
+            if parent is None:
+                continue
+            diff = run_git(repo, "diff", "-U0", "--no-color", "--no-ext-diff", "-M", parent, sha)
+            changes = parse_unified_diff(diff)
+            if len(changes) > max_files:
+                result.skipped_large += 1
+                continue
+            store.rev = parent
+            t = time.perf_counter()
+            findings: list[Finding] = [f for check in checks for f in check.run(changes, store, config)]
+            ms = int((time.perf_counter() - t) * 1000)
+            subject = run_git(repo, "log", "-1", "--format=%s", sha).strip()
+            result.commits.append(ReplayCommit(sha, subject, [asdict(f) for f in findings], ms))
+            result.checked += 1
+            if progress:
+                progress(i, len(commits))
+        full.close()
+        early.close()
+        result.seconds = time.perf_counter() - start
+        return result
+    finally:
+        shutil.rmtree(tmp, ignore_errors=True)
+
+
+def _parent(repo: str, sha: str) -> str | None:
+    parts = run_git(repo, "rev-list", "--parents", "-n", "1", sha).split()
+    return parts[1] if len(parts) > 1 else None
+
+
+def format_replay(result: ReplayResult) -> str:
+    per10 = 10 * sum(len(c.findings) for c in result.commits) / max(result.checked, 1)
+    lines = [
+        f"blot bench --replay: {result.checked} commits checked in {result.seconds:.0f}s "
+        f"({result.skipped_large} skipped as too large)",
+        f"  commits with a warning: {len(result.warned)} ({100 * len(result.warned) / max(result.checked, 1):.1f}%)",
+        f"  warnings per 10 commits: {per10:.2f}",
+        "  by check: " + ", ".join(f"{k} {v}" for k, v in result.by_check().items()),
+    ]
+    for c in result.warned:
+        lines.append(f"\n  {c.sha[:9]} {c.subject}")
+        for f in c.findings:
+            where = f"{f['path']}:{f['line']}" if f["line"] else f["path"]
+            lines.append(f"    ! {f['check']}  {where}")
+            lines.extend(f"      {e}" for e in f["evidence"])
+    return "\n".join(lines) + "\n"
+
+
+# --- cases ------------------------------------------------------------------------------
+
+
+@dataclass
+class CaseResult:
+    id: str
+    check: str
+    expected_warn: bool
+    warned: bool
+    cited_right_commit: bool
+    detail: str = ""
+
+    @property
+    def correct(self) -> bool:
+        return self.cited_right_commit if self.expected_warn else not self.warned
+
+
+def _repo_dir(url: str, workdir: str) -> str:
+    # always a fresh clone under workdir: the bench checks out old commits and edits files,
+    # so it must never run inside a repo the user is working in (even if `repo` is a local path)
+    name = re.split(r"[\\/]", url.rstrip("/\\"))[-1].removesuffix(".git") or "repo"
+    path = os.path.join(workdir, name)
+    os.makedirs(workdir, exist_ok=True)
+    if not os.path.isdir(os.path.join(path, ".git")):
+        subprocess.run(["git", "clone", "-q", url, path], check=True)
+    return path
+
+
+def run_case(case: dict, workdir: str, config: Config) -> CaseResult:
+    expect = case["expect"]
+    repo = _repo_dir(case["repo"], workdir)
+    at = case["at"]
+    if subprocess.run(["git", "-C", repo, "cat-file", "-e", f"{at}^{{commit}}"], capture_output=True).returncode:
+        subprocess.run(["git", "-C", repo, "fetch", "-q", "origin"], check=True)
+    run_git(repo, "checkout", "-q", "--detach", "--force", at)
+    db = os.path.join(workdir, f"{os.path.basename(repo)}-{at[:12]}.db")
+    build_index(repo, config=config, upto=at, db_path=db)
+
+    path, (old, new) = case["file"], case["replace"]
+    file = os.path.join(repo, path)
+    with open(file, encoding="utf-8", newline="") as f:
+        text = f.read()
+    if text.count(old) != 1:
+        return CaseResult(case["id"], expect["check"], expect["warn"], False, False,
+                          f"edit text found {text.count(old)} times in {path} (needs exactly 1)")
+    try:
+        with open(file, "w", encoding="utf-8", newline="") as f:
+            f.write(text.replace(old, new))
+        changes = parse_unified_diff(run_git(repo, "diff", "-U0", "--no-color", "--no-ext-diff", "--", path))
+        store = Store.open(repo, db)
+        store.rev = at
+        check = next(cls() for cls in REGISTRY if cls.id == expect["check"])
+        findings = check.run(changes, store, config)
+        store.close()
+    finally:
+        run_git(repo, "checkout", "-q", "--", path)
+
+    want = expect.get("commit", "")
+    cited = bool(findings) and any(e.startswith(want[:7]) for f in findings for e in f.evidence) if want else bool(findings)
+    detail = "; ".join(e for f in findings for e in f.evidence[:1]) or "no warning"
+    return CaseResult(case["id"], expect["check"], expect["warn"], bool(findings), cited, detail)
+
+
+def run_cases(cases_path: str, workdir: str, config: Config) -> list[CaseResult]:
+    with open(cases_path, encoding="utf-8") as f:
+        cases = json.load(f)
+    return [run_case(case, workdir, config) for case in cases]
+
+
+def format_cases(results: list[CaseResult]) -> str:
+    lines = []
+    for r in results:
+        mark = "ok  " if r.correct else "FAIL"
+        want = "warn" if r.expected_warn else "quiet"
+        lines.append(f"  {mark} {r.id}  [{r.check}, expect {want}]  {r.detail}")
+    lines.append("")
+    for cls in REGISTRY:
+        rs = [r for r in results if r.check == cls.id]
+        if not rs:
+            continue
+        tp = sum(r.expected_warn and r.cited_right_commit for r in rs)
+        fn = sum(r.expected_warn and not r.cited_right_commit for r in rs)
+        fp = sum(not r.expected_warn and r.warned for r in rs)
+        precision = tp / (tp + fp) if tp + fp else None
+        recall = tp / (tp + fn) if tp + fn else None
+        fmt = lambda v: "n/a" if v is None else f"{v:.0%}"  # noqa: E731
+        lines.append(f"  {cls.id}: {len(rs)} cases, precision {fmt(precision)}, recall {fmt(recall)} "
+                     f"(tp {tp}, fp {fp}, fn {fn})")
+    correct = sum(r.correct for r in results)
+    lines.append(f"blot bench: {correct}/{len(results)} cases correct")
+    return "\n".join(lines) + "\n"

blot_git-0.3.0/src/blot/checks/base.py

@@ -0,0 +1,65 @@
+"""The interface every check implements. Adding a check = one new file + one registry line."""
+
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass, field
+from fnmatch import fnmatch
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from blot.config import Config
+    from blot.diff import FileChange
+    from blot.store import CommitRow, Store
+
+
+@dataclass
+class Finding:
+    check: str  # check id, e.g. "removed-fix"
+    path: str
+    line: int | None  # None for file-level findings (forgot-related-file)
+    message: str  # one or two plain-English sentences
+    evidence: list[str] = field(default_factory=list)  # 'abc1234 "subject" (date, fixes #12)'
+    severity: int = 2
+
+
+class Check(ABC):
+    id: str = ""
+    description: str = ""
+
+    @abstractmethod
+    def run(self, changes: list[FileChange], store: Store, config: Config) -> list[Finding]:
+        """Return findings for the staged changes. Must not call the network; target < 100 ms."""
+
+
+TEST_PATTERNS = ("test/*", "tests/*", "*/test/*", "*/tests/*", "*/__tests__/*", "__tests__/*",
+                 "test_*", "*/test_*", "*_test.*", "*.test.*", "*.spec.*", "*/testing/*", "conftest.py", "*/conftest.py")
+DOC_PATTERNS = ("doc/*", "docs/*", "*/docs/*", "*.md", "*.rst", "*.txt", "*.adoc", "*.markdown")
+
+
+# generated files: their lines come from tools, not from the fix's author
+GENERATED_PATTERNS = ("*.lock", "*package-lock.json", "*pnpm-lock.yaml", "*npm-shrinkwrap.json", "*go.sum",
+                      "*.min.js", "*.min.css", "*.map", "*.mo", "*.po", "*/locale/*",  # translations
+                      ".github/*", ".circleci/*", ".gitlab-ci.yml")  # CI configuration
+
+
+def is_skipped(path: str, config: Config) -> bool:
+    """Paths the checks ignore: generated files, config.ignore_paths, and tests and docs unless opted in."""
+    patterns = [*GENERATED_PATTERNS, *config.ignore_paths]
+    if not config.check_tests:
+        patterns += TEST_PATTERNS
+    if not config.check_docs:
+        patterns += DOC_PATTERNS
+    return any(fnmatch(path, p) for p in patterns)
+
+
+def describe_commit(row: CommitRow, store: Store | None = None) -> str:
+    """Evidence line: '8efea0c "subject" (2020-10-03, fixes #3761 "Memory leak in app")'.
+
+    Issue titles appear once `blot index --github` has fetched them.
+    """
+    details = [row.date[:10]]
+    if row.issues:
+        refs = [store.describe_issue(ref) for ref in row.issues] if store else row.issues
+        details.append("fixes " + ", ".join(refs))
+    return f'{row.sha[:7]} "{row.subject}" ({", ".join(details)})'

{blot_git-0.1.0 → blot_git-0.3.0}/src/blot/checks/cochange.py

@@ -12,9 +12,7 @@ Commits touching more than config.cochange_max_files_per_commit files are ignore
 
 from __future__ import annotations
 
-from fnmatch import fnmatch
-
-from blot.checks.base import Check, Finding
+from blot.checks.base import Check, Finding, is_skipped
 from blot.git import tracked_files
 
 
@@ -23,7 +21,7 @@ class ForgotRelatedFile(Check):
     description = "Changed a file without the file that usually changes with it"
 
     def run(self, changes, store, config) -> list[Finding]:
-        ignored = lambda p: any(fnmatch(p, pattern) for pattern in config.ignore_paths)  # noqa: E731
+        ignored = lambda p: is_skipped(p, config)  # noqa: E731
         staged = {p for c in changes for p in (c.path, c.old_path) if p}
         head_files: set[str] | None = None
         findings: list[Finding] = []
@@ -34,7 +32,7 @@ class ForgotRelatedFile(Check):
             if not partners and change.old_path and change.old_path != change.path:
                 partners = store.partners(change.old_path, config.cochange_min_support, config.cochange_min_confidence)
             if partners and head_files is None:
-                head_files = tracked_files(store.repo)
+                head_files = tracked_files(store.repo, store.rev)
             for partner, support, confidence in partners:  # strongest first
                 if partner in staged or partner not in head_files or ignored(partner):
                     continue