gitmole 0.9.0__tar.gz → 0.10.0__tar.gz

{gitmole-0.9.0 → gitmole-0.10.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: gitmole
-Version: 0.9.0
+Version: 0.10.0
 Summary: Offline git repository analysis with a terminal report: hotspots, coupling, ownership, code age, secrets, repo health.
 License: MIT
 Project-URL: Homepage, https://github.com/antvinni/gitmole
@@ -32,7 +32,6 @@ Free. Any Stack. Local. Offline. Deterministic. Fast.
 - **Any stack.** It reads what every repository has: the git log, git blame and the files themselves.
 - **Local & Offline.** Everything runs against a clone on your machine. Nothing is uploaded, nothing phones home; the vulnerability database is a copy you download once.
 - **Deterministic.** No AI at runtime. Every finding is a plain rule over counts you can recompute by hand. The JSON export carries each finding's rule and the numbers it fired on. The same clone gives the same report every time. 
-- **Fast.** A 4,400-commit repository takes under thirty seconds.
 
 ## Install
 
@@ -71,59 +70,15 @@ blocks on secrets in source files and still posts the report. Every option:
 
 ## What you get
 
-The opening of the report for [react](https://github.com/facebook/react), 21,703 commits
-since 2013, at a pinned commit:
-
-```text
-╭─ react ──────────────────────────────────────────────────────────────────────────────────────────╮
-│ 21703 commits  ·  2013-05-28 → 2026-09-16  ·  1843 identities  ·  branch main                    │
-│ 681,078 lines in 4781 files  ·  JavaScript, TypeScript, Rust, CSS                                │
-│ most commits on Wed at 15:00  ·  14% of commits are fixes  ·  1% of commits are reverts  ·  19%  │
-│ of surviving code from 2026                                                                      │
-│ 1 critical, 6 warnings, 9 notes                                                                  │
-╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
-
-◎ Watch list
-  file                                              why                                             
-  ──────────────────────────────────────────────────────────────────────────────────────────────────
-  packages/react-server/src/ReactFlightServer.js    changed 319 times · fixed once in six months ·  
-                                                    visitAsyncNodeImpl() complexity 46              
-  packages/react-server/src/ReactFizzServer.js      changed 297 times · fixed twice in six months · 
-                                                    retryNode() complexity 41                       
-  packages/react-reconciler/src/ReactFiberWorkLoo   changed 312 times · fixed 4 times in six months 
-  p.js                                              · flushSpawnedWork() complexity 48              
-  packages/react-reconciler/src/ReactFiberCommitW   changed 284 times · fixed 28 times ·            
-  ork.js                                            commitLayoutEffectOnFiber() complexity 72       
-  packages/react-reconciler/src/ReactFiberBeginWo   changed 361 times · fixed once in six months ·  
-  rk.js                                             beginWork() complexity 52                       
-  ranked by revisions × lines of code; the reasons say what else counts against each file           
-  6 months ago this list would have named 11 of the 46 files fixed since (a random 15 of the 1802   
-  files that had changed more than once would name 0.3; the 15 most changed would name 7)           
-```
+Reports on repositories you know, each at a pinned commit, published as gitmole wrote them:
+
+| Repository | Commit | Commits | Lines | gitmole run |
+|---|---|---:|---:|---:|
+| [curl](https://github.com/antvinni/gitmole/blob/main/docs/examples/curl.md) | [`540ee5b5`](https://github.com/curl/curl/commit/540ee5b560cc6e775e11317048a13cc7e355bf91) | 39,758 | 247,179 | 61 s |
+| [django](https://github.com/antvinni/gitmole/blob/main/docs/examples/django.md) | [`8cbdd4a8`](https://github.com/django/django/commit/8cbdd4a814397f81adf0129288f32b615bd1f94f) | 34,933 | 431,749 | 153 s |
+| [react](https://github.com/antvinni/gitmole/blob/main/docs/examples/react.md) | [`2b19aecd`](https://github.com/facebook/react/commit/2b19aecd0e9111b774fad0fad9862e50bcb5bc8a) | 21,703 | 681,078 | 138 s |
 
-The watch list is the point: the five source files most likely to need a fix
-next, the reasons in words, and a backtest that says how the same list, drawn
-six months earlier, would have done against the fixes that followed. The list
-ranks by revisions × lines of code: measured at six cut-offs on three
-repositories
-([validation](https://github.com/antvinni/gitmole/blob/main/docs/validation.md)),
-that named more of the files fixed next than churn alone, size alone or a
-weighted product of fixes, complexity and ownership. Between the header and that
-list the full report puts its findings, 16 for react (1 critical, 6 warnings, 9
-notes); below it, tables for people, the knowledge map, the timeline, change
-coupling, complex functions and repo health; `--full` adds the hotspots table
-behind the list, size, activity and code age. Every section is explained in
-[docs/output.md](https://github.com/antvinni/gitmole/blob/main/docs/output.md).
-
-Reports on repositories you know, each at a pinned commit with a fixed
-reference date, published as gitmole wrote them; the repo-health numbers
-come from git-sizer over the whole clone, so a fresh clone can differ there:
-
-| Repository | Commits | Lines | Watch list backtest |
-|---|---:|---:|---|
-| [curl](https://github.com/antvinni/gitmole/blob/main/docs/examples/curl.md) | 39,758 | 247,179 | named 15 of the 238 files fixed in the next six months; a random pick would name 4.9, the 15 most changed 15 |
-| [django](https://github.com/antvinni/gitmole/blob/main/docs/examples/django.md) | 34,933 | 431,749 | named 15 of the 213 files fixed in the next six months; a random pick would name 3.1, the 15 most changed 13 |
-| [react](https://github.com/antvinni/gitmole/blob/main/docs/examples/react.md) | 21,703 | 681,078 | named 11 of the 46 files fixed in the next six months; a random pick would name 0.3, the 15 most changed 7 |
+Run times are one `gitmole CLONE` with every default step, on a MacBook Pro (M4, 16 GB).
 
 ## The tool set
 

{gitmole-0.9.0 → gitmole-0.10.0}/README.md

@@ -12,7 +12,6 @@ Free. Any Stack. Local. Offline. Deterministic. Fast.
 - **Any stack.** It reads what every repository has: the git log, git blame and the files themselves.
 - **Local & Offline.** Everything runs against a clone on your machine. Nothing is uploaded, nothing phones home; the vulnerability database is a copy you download once.
 - **Deterministic.** No AI at runtime. Every finding is a plain rule over counts you can recompute by hand. The JSON export carries each finding's rule and the numbers it fired on. The same clone gives the same report every time. 
-- **Fast.** A 4,400-commit repository takes under thirty seconds.
 
 ## Install
 
@@ -51,59 +50,15 @@ blocks on secrets in source files and still posts the report. Every option:
 
 ## What you get
 
-The opening of the report for [react](https://github.com/facebook/react), 21,703 commits
-since 2013, at a pinned commit:
-
-```text
-╭─ react ──────────────────────────────────────────────────────────────────────────────────────────╮
-│ 21703 commits  ·  2013-05-28 → 2026-09-16  ·  1843 identities  ·  branch main                    │
-│ 681,078 lines in 4781 files  ·  JavaScript, TypeScript, Rust, CSS                                │
-│ most commits on Wed at 15:00  ·  14% of commits are fixes  ·  1% of commits are reverts  ·  19%  │
-│ of surviving code from 2026                                                                      │
-│ 1 critical, 6 warnings, 9 notes                                                                  │
-╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
-
-◎ Watch list
-  file                                              why                                             
-  ──────────────────────────────────────────────────────────────────────────────────────────────────
-  packages/react-server/src/ReactFlightServer.js    changed 319 times · fixed once in six months ·  
-                                                    visitAsyncNodeImpl() complexity 46              
-  packages/react-server/src/ReactFizzServer.js      changed 297 times · fixed twice in six months · 
-                                                    retryNode() complexity 41                       
-  packages/react-reconciler/src/ReactFiberWorkLoo   changed 312 times · fixed 4 times in six months 
-  p.js                                              · flushSpawnedWork() complexity 48              
-  packages/react-reconciler/src/ReactFiberCommitW   changed 284 times · fixed 28 times ·            
-  ork.js                                            commitLayoutEffectOnFiber() complexity 72       
-  packages/react-reconciler/src/ReactFiberBeginWo   changed 361 times · fixed once in six months ·  
-  rk.js                                             beginWork() complexity 52                       
-  ranked by revisions × lines of code; the reasons say what else counts against each file           
-  6 months ago this list would have named 11 of the 46 files fixed since (a random 15 of the 1802   
-  files that had changed more than once would name 0.3; the 15 most changed would name 7)           
-```
+Reports on repositories you know, each at a pinned commit, published as gitmole wrote them:
+
+| Repository | Commit | Commits | Lines | gitmole run |
+|---|---|---:|---:|---:|
+| [curl](https://github.com/antvinni/gitmole/blob/main/docs/examples/curl.md) | [`540ee5b5`](https://github.com/curl/curl/commit/540ee5b560cc6e775e11317048a13cc7e355bf91) | 39,758 | 247,179 | 61 s |
+| [django](https://github.com/antvinni/gitmole/blob/main/docs/examples/django.md) | [`8cbdd4a8`](https://github.com/django/django/commit/8cbdd4a814397f81adf0129288f32b615bd1f94f) | 34,933 | 431,749 | 153 s |
+| [react](https://github.com/antvinni/gitmole/blob/main/docs/examples/react.md) | [`2b19aecd`](https://github.com/facebook/react/commit/2b19aecd0e9111b774fad0fad9862e50bcb5bc8a) | 21,703 | 681,078 | 138 s |
 
-The watch list is the point: the five source files most likely to need a fix
-next, the reasons in words, and a backtest that says how the same list, drawn
-six months earlier, would have done against the fixes that followed. The list
-ranks by revisions × lines of code: measured at six cut-offs on three
-repositories
-([validation](https://github.com/antvinni/gitmole/blob/main/docs/validation.md)),
-that named more of the files fixed next than churn alone, size alone or a
-weighted product of fixes, complexity and ownership. Between the header and that
-list the full report puts its findings, 16 for react (1 critical, 6 warnings, 9
-notes); below it, tables for people, the knowledge map, the timeline, change
-coupling, complex functions and repo health; `--full` adds the hotspots table
-behind the list, size, activity and code age. Every section is explained in
-[docs/output.md](https://github.com/antvinni/gitmole/blob/main/docs/output.md).
-
-Reports on repositories you know, each at a pinned commit with a fixed
-reference date, published as gitmole wrote them; the repo-health numbers
-come from git-sizer over the whole clone, so a fresh clone can differ there:
-
-| Repository | Commits | Lines | Watch list backtest |
-|---|---:|---:|---|
-| [curl](https://github.com/antvinni/gitmole/blob/main/docs/examples/curl.md) | 39,758 | 247,179 | named 15 of the 238 files fixed in the next six months; a random pick would name 4.9, the 15 most changed 15 |
-| [django](https://github.com/antvinni/gitmole/blob/main/docs/examples/django.md) | 34,933 | 431,749 | named 15 of the 213 files fixed in the next six months; a random pick would name 3.1, the 15 most changed 13 |
-| [react](https://github.com/antvinni/gitmole/blob/main/docs/examples/react.md) | 21,703 | 681,078 | named 11 of the 46 files fixed in the next six months; a random pick would name 0.3, the 15 most changed 7 |
+Run times are one `gitmole CLONE` with every default step, on a MacBook Pro (M4, 16 GB).
 
 ## The tool set
 

{gitmole-0.9.0 → gitmole-0.10.0}/gitmole/__init__.py

@@ -1,3 +1,3 @@
 """gitmole: offline git repository analysis with a terminal report."""
 
-__version__ = "0.9.0"
+__version__ = "0.10.0"

{gitmole-0.9.0 → gitmole-0.10.0}/gitmole/backtest.py

@@ -22,9 +22,11 @@ import tempfile
 from . import filetypes, load, maat, trend
 
 
-def size_at(repo: str, rev: str, out_dir: str) -> str:
-    """scc's --by-file JSON over the tree at rev, exported through a temporary index so no
-    archive is held in memory and export-ignore attributes do not thin the tree.
+def snapshot_at(repo: str, rev: str, out_dir: str) -> tuple:
+    """The tree at rev, measured and classified as it was then: scc's --by-file JSON, the generated files
+    and the vendored paths. Exported through a temporary index so no archive is held in memory and
+    export-ignore attributes do not thin the tree; the same index lets git check-attr read that tree's
+    .gitattributes, so the classification is the cut-off's, not HEAD's.
 
     The tree is written under `out_dir`, not the system temp directory: a SIGKILL cannot run the
     cleanup, and a checkout left next to the report is one the next run clears away."""
@@ -34,7 +36,15 @@ def size_at(repo: str, rev: str, out_dir: str) -> str:
         env = dict(os.environ, GIT_INDEX_FILE=os.path.join(tmp, "index"))
         subprocess.run(["git", "read-tree", rev], cwd=repo, env=env, check=True, capture_output=True, text=True)
         subprocess.run(["git", "checkout-index", "-a", f"--prefix={tree}/"], cwd=repo, env=env, check=True, capture_output=True, text=True)
-        return subprocess.run(["scc", "--by-file", "--format", "json"], cwd=tree, capture_output=True, text=True, check=True).stdout
+        size = subprocess.run(["scc", "--by-file", "--format", "json"], cwd=tree, capture_output=True, text=True, check=True).stdout
+        # the text files of that tree, as blame.text_files lists HEAD's: git grep prints "rev:path"
+        proc = subprocess.run([*filetypes.GIT, "grep", "-I", "--name-only", "-z", "-e", "", rev], cwd=repo, capture_output=True)
+        if proc.returncode not in (0, 1):   # 1 is grep's "no match" (an empty tree), not a failure
+            raise subprocess.CalledProcessError(proc.returncode, proc.args, proc.stdout.decode("utf-8", "replace"),
+                                                 proc.stderr.decode("utf-8", "replace"))
+        paths = sorted(p.decode("utf-8", "surrogateescape").split(":", 1)[1] for p in proc.stdout.split(b"\0") if p)
+        attrs = filetypes.attributes(repo, paths, cached=True, env=env)
+        return size, filetypes.generated_files(tree, paths, attrs), filetypes.vendored_paths(tree, paths, attrs)
 
 
 def main(argv=None) -> int:
@@ -66,7 +76,7 @@ def main(argv=None) -> int:
     types = filetypes.parse(meta.get("file_types"))
     maat.write_all(log_path, sub, os.path.join(args.out, "meta.json") if "aliases" in meta else None, types, now=until, until=until)
     try:
-        size = size_at(args.repo, rev, args.out)
+        size, generated, vendored = snapshot_at(args.repo, rev, args.out)
     except subprocess.CalledProcessError as e:
         first = ((e.stderr or "").strip().splitlines() or [f"{' '.join(e.cmd)} exited {e.returncode}"])[0]
         print(f"backtest: {first}", file=sys.stderr)
@@ -74,7 +84,8 @@ def main(argv=None) -> int:
     with open(os.path.join(sub, "size.json"), "w", encoding="utf-8") as fh:
         fh.write(size)
     with open(os.path.join(sub, "meta.json"), "w", encoding="utf-8") as fh:
-        json.dump({"now": until, "last_date": until, "file_types": meta.get("file_types"), "aliases": meta.get("aliases", {})}, fh)
+        json.dump({"now": until, "last_date": until, "file_types": meta.get("file_types"), "aliases": meta.get("aliases", {}),
+                   "generated": generated, "vendored": vendored}, fh)
     return 0
 
 

gitmole-0.10.0/gitmole/classify.py

@@ -0,0 +1,88 @@
+"""Why a file is out of the scored pool: one answer for every table, the watch list and --risk.
+
+Built once per report, so the type filter, scc's rows, the generated and vendored lists, the amalgamations
+and the plumbing are read once and every section asks the same object. Not in filetypes, which blame.py
+and maat.py import as scripts and which must stay free of package imports."""
+from __future__ import annotations
+
+from collections import Counter
+
+from . import filetypes, hotspots
+
+# In descriptive order: what a reader would call the file first. package.json is a release file before
+# it is not a source type; vendor/x_test.go is vendored before it is a test file; a generated test is
+# generated. The order is part of the contract: reason() is the first that applies.
+REASONS = ("generated", "vendored", "test file", "example code", "release file", "amalgamation", "not a source type", "not in the tree")
+
+# The coverage line's nouns, by count.
+_NOUNS = {"scored": ("scored", "scored"), "generated": ("generated", "generated"), "vendored": ("vendored", "vendored"),
+          "test file": ("test file", "test files"), "example code": ("example code", "example code"),
+          "release file": ("release file", "release files"), "amalgamation": ("amalgamation", "amalgamations"),
+          "not a source type": ("not a source type", "not a source type"), "not counted by scc": ("not counted by scc", "not counted by scc")}
+
+
+class Classifier:
+    def __init__(self, report: dict):
+        meta = report.get("meta") or {}
+        # a run records its --file-types spec (None for the default list); a run from before that record
+        # was measured unfiltered and is classified unfiltered, as load.load_report filters scc
+        self.types = filetypes.parse(meta["file_types"]) if "file_types" in meta else None
+        self.tree = (report.get("size") or {}).get("files") or {}
+        self.generated = set(meta.get("generated") or [])
+        self.vendored = filetypes.vendor_dirs(report)
+        self.amalgamations = hotspots.amalgamations(report)
+        self.plumbing = filetypes.plumbing_paths(report)
+        self._reasons = {}   # memoised per instance: every hide pass in a report asks the same paths again
+
+    def reasons(self, path: str) -> tuple:
+        """Every reason that applies, in REASONS order. `not in the tree` needs a tree to judge by: a run
+        whose scc step was killed classifies nothing as gone, so it cannot empty every table. Cached per
+        path and returned as a tuple so callers cannot mutate the cached result."""
+        if path in self._reasons:
+            return self._reasons[path]
+        out = []
+        if path in self.generated:
+            out.append("generated")
+        if filetypes.is_vendored(path, self.vendored):
+            out.append("vendored")
+        if filetypes.is_test_path(path):
+            out.append("test file")
+        if filetypes.is_sample_path(path):
+            out.append("example code")
+        if filetypes.is_release(path, self.plumbing):
+            out.append("release file")
+        if path in self.amalgamations:
+            out.append("amalgamation")
+        if not filetypes.matches(path, self.types):
+            out.append("not a source type")
+        if self.tree and path not in self.tree:
+            out.append("not in the tree")
+        self._reasons[path] = tuple(out)
+        return self._reasons[path]
+
+    def reason(self, path: str):
+        """The first reason, or None for a file in the scored pool."""
+        found = self.reasons(path)
+        return found[0] if found else None
+
+    def excluded(self, path: str, reasons) -> bool:
+        """Whether a table that hides `reasons` hides this file: any of its reasons is enough, so a table
+        that hides tests still hides a vendored test."""
+        return any(r in reasons for r in self.reasons(path))
+
+
+def coverage(classifier: Classifier, tracked: list) -> dict:
+    """How many tracked files each reason claims, `scored` for none. A tracked file with no scc row is a
+    type scc does not classify rather than a deleted one, so it counts as `not counted by scc`."""
+    counts = Counter()
+    for path in tracked:
+        reason = classifier.reason(path)
+        counts["scored" if reason is None else "not counted by scc" if reason == "not in the tree" else reason] += 1
+    return dict(counts)
+
+
+def coverage_line(cov: dict) -> str:
+    """'4,781 files: 3,900 scored · 610 test files · …', the buckets in REASONS order."""
+    order = ["scored", *REASONS[:-1], "not counted by scc"]
+    parts = [f"{cov[k]:,} {_NOUNS[k][0 if cov[k] == 1 else 1]}" for k in order if cov.get(k)]
+    return f"{sum(cov.values()):,} files: " + " · ".join(parts)

{gitmole-0.9.0 → gitmole-0.10.0}/gitmole/cli.py

@@ -44,6 +44,7 @@ def parse_args(argv):
     p.add_argument("--fail-on", choices=findings.SEVERITIES, help="exit 3 if any finding is at this severity or worse")
     p.add_argument("--risk", metavar="BASE", help="score the files changed since BASE (merge base with HEAD) by their share of the repository's revisions × lines of code; needs a local path")
     p.add_argument("--risk-threshold", type=float, metavar="N", help="with --risk: exit 3 when the changed files hold more than N percent of the repository's revisions × lines of code")
+    p.add_argument("--compare", metavar="BEFORE_JSON", help="add a 'Since last report' section against an earlier --json export of the same clone")
     p.add_argument("--version", action="version", version=f"gitmole {__version__}")
     return p.parse_args(argv)
 
@@ -127,11 +128,13 @@ def _check_args(args, err, kind=None) -> int | None:
     if kind is None:
         bad = ("--yes needs --clean" if args.yes and not args.clean else
                "target required" if args.target is None and not args.clean else
-               "--risk-threshold needs --risk" if args.risk_threshold is not None and not args.risk else None)
+               "--risk-threshold needs --risk" if args.risk_threshold is not None and not args.risk else
+               "--compare: no such file: " + args.compare if args.compare and not os.path.isfile(args.compare) else None)
     elif kind == "path":
         bad = None
     else:
-        bad = ("--risk needs a local path" if args.risk else
+        bad = ("--compare needs one repository, not owner/*" if args.compare and kind == "org" else
+               "--risk needs a local path" if args.risk else
                "--list-file-types needs a local path" if args.list_file_types else None)
     if bad:
         err.print(f"[red]{bad}[/red]")
@@ -297,12 +300,14 @@ def _meta_for_run(repo_dir: str, args, estimate, age_ok: bool, plots_ok: bool, p
     types_spec = _types_spec(args.file_types)
 
     meta = run.collect_meta(repo_dir, since=args.since_date)
+    meta["run"] = run.manifest(repo_dir, args)   # what produced this report: commit, gitmole and tool versions, the options
     meta["file_types"] = types_spec   # the loader filters scc's size data the way every other step was filtered
     meta["gone_months"] = args.gone
-    ignore = list(run.DATA_IGNORES if args.ignore_data else []) + list(args.ignore)
-    tracked = blame.text_files(repo_dir, ignore)
-    meta["generated"] = filetypes.generated_files(repo_dir, tracked)   # hidden from the tables, out of the findings
-    meta["vendored"] = filetypes.vendored_dirs(repo_dir, tracked)     # somebody else's code, by the licence it carries
+    tracked = blame.text_files(repo_dir)   # every tracked text file: --ignore shapes blame, functions and duplicates, never what a file is
+    attrs = filetypes.attributes(repo_dir, tracked)   # one git check-attr pass, shared by the two lists below
+    meta["generated"] = filetypes.generated_files(repo_dir, tracked, attrs=attrs)   # hidden from the tables, out of the findings
+    meta["vendored"] = filetypes.vendored_paths(repo_dir, tracked, attrs=attrs)    # somebody else's code, by the licence it carries or the attribute it declares
+    meta["credential_files"] = filetypes.credential_files(filetypes.git_paths(repo_dir, "ls-files"))   # by name, over every tracked file
     if args.since_date and meta["commits"] == 0:
         raise NoCommits(f"no commits since {args.since_date}; widen --since")
     if args.now:
@@ -348,6 +353,17 @@ def _record_statuses(meta, results, age_ok: bool, plots_ok: bool, lizard_ok: boo
     meta["steps"] = {name: "run" if rc == 0 else (rc if isinstance(rc, str) else "failed") for name, rc in results.items()}
 
 
+def _coverage(repo_dir: str, out_dir: str) -> dict:
+    """How many tracked text files each reason claims, from the report as the steps left it. An
+    unreadable output directory (a killed run) records nothing rather than failing the run."""
+    from . import classify
+    try:
+        report = load.load_report(out_dir, nested=False)
+    except load.Unreadable:
+        return {}
+    return classify.coverage(classify.Classifier(report), blame.text_files(repo_dir))
+
+
 def _analyse(repo_dir: str, out_dir: str, args, ui: Console, planner, estimator) -> None:
     """Run the whole pipeline for one repository into out_dir."""
     os.makedirs(os.path.join(out_dir, "theseus"), exist_ok=True)
@@ -372,6 +388,7 @@ def _analyse(repo_dir: str, out_dir: str, args, ui: Console, planner, estimator)
         raise Interrupted()
 
     _record_statuses(meta, results, age_ok, plots_ok, lizard_ok, cut, duplicates_ok)
+    meta["coverage"] = _coverage(repo_dir, out_dir)
     run.save_meta(meta, out_dir)
 
     failed = [n for n, rc in results.items() if rc != 0]
@@ -505,12 +522,30 @@ def _render(out_dir: str, console: Console, ui: Console, args, err: Console) ->
             return 2
         from . import watch
         risk = {"base": args.risk, **watch.change_risk(report, files)}
+    comparison = None
+    if args.compare:
+        from . import compare as _compare
+        try:
+            with open(args.compare, encoding="utf-8") as fh:
+                before = json.load(fh)
+        except (OSError, ValueError) as e:
+            err.print(f"[red]--compare {args.compare}:[/red] {e}", soft_wrap=True)
+            return 2
+        if not _compare.is_export(before):
+            err.print(f"[red]--compare {args.compare}:[/red] not a gitmole --json export (it needs meta, findings with rule ids, and watch; "
+                      "exports from before 0.8.0 have no rule ids)", soft_wrap=True)
+            return 2
+        if before["meta"].get("name") != report["meta"].get("name"):
+            err.print(f"[red]--compare {args.compare}:[/red] it describes {before['meta'].get('name')}, this run describes {report['meta'].get('name')}; "
+                      "the two exports must be of the same clone", soft_wrap=True)
+            return 2
+        comparison = _compare.compare(before, report, found)
     if args.json:
-        _write(json.dumps(render.to_json(report, found, risk=risk), indent=2) + "\n", args.json, console)
+        _write(json.dumps(render.to_json(report, found, risk=risk, compare=comparison), indent=2) + "\n", args.json, console)
     if args.markdown:
-        _write(render.markdown(report, found, full=args.full, risk=risk, base=args.risk), args.markdown, console)
+        _write(render.markdown(report, found, full=args.full, risk=risk, base=args.risk, compare=comparison), args.markdown, console)
     if "-" not in (args.json, args.markdown):
-        render.report(report, found, console, full=args.full, risk=risk, base=args.risk)
+        render.report(report, found, console, full=args.full, risk=risk, base=args.risk, compare=comparison)
     if args.fail_on and any(findings.SEVERITIES.index(f["severity"]) <= findings.SEVERITIES.index(args.fail_on) for f in found):
         return 3
     if risk is not None and args.risk_threshold is not None and risk["total"] > args.risk_threshold:

gitmole-0.10.0/gitmole/compare.py

@@ -0,0 +1,68 @@
+"""What changed since an earlier --json export: findings that are new, resolved or persisting, files that
+entered or left the watch list, the tally before and after. Pure over two report dicts."""
+from __future__ import annotations
+
+from . import findings, watch
+
+# A rule emits one finding per report, except these two, which emit one per row; the evidence field
+# that tells the rows apart joins the rule id in the key.
+KEY_FIELDS = {"repo_health": "metric", "placeholder_identity": "email"}
+
+
+def key(finding: dict) -> tuple:
+    """A finding's identity across two reports: the rule id alone, unless the rule is one of KEY_FIELDS,
+    where one report can hold several findings for the same rule and the evidence field is what tells
+    them apart (a metric name, an email)."""
+    rid = finding["rule"]["id"]
+    field = KEY_FIELDS.get(rid)
+    return (rid, (finding.get("evidence") or {}).get(field)) if field else (rid,)
+
+
+def is_export(data) -> bool:
+    """A gitmole --json export: a report with its findings and watch list. Every finding must carry a
+    rule id and a severity, the shape key() and _tally() read without a default — exports from before
+    0.8.0 have findings without a `rule`, and would otherwise pass this check and crash later on a bare
+    KeyError instead of being refused here."""
+    if not (isinstance(data, dict) and isinstance(data.get("meta"), dict) and "findings" in data and "watch" in data):
+        return False
+    found, watch_rows = data.get("findings"), data.get("watch")
+    return (isinstance(found, list) and isinstance(watch_rows, list) and all(isinstance(r, dict) for r in watch_rows) and
+            all(isinstance(f, dict) and isinstance(f.get("rule"), dict) and "id" in f["rule"] and "severity" in f for f in found))
+
+
+def _tally(found: list) -> dict:
+    counts = {s: 0 for s in findings.SEVERITIES}
+    for f in found:
+        counts[f["severity"]] += 1
+    return counts
+
+
+def _ordered(found: list) -> list:
+    return sorted(found, key=lambda f: (findings.SEVERITIES.index(f["severity"]), f["title"], str(key(f))))
+
+
+def _options_differ(before_meta: dict, after_meta: dict) -> list:
+    """The options that change what a run sees: --since and --file-types from meta's top level, --ignore and
+    --ignore-data from the manifest when both exports have one. --deep is recorded there too but only
+    decides whether code age, plots and duplicates ran, none of which reach the findings or the watch list."""
+    out = [name for name in ("since", "file_types") if before_meta.get(name) != after_meta.get(name)]
+    b, a = (before_meta.get("run") or {}).get("options"), (after_meta.get("run") or {}).get("options")
+    if b is not None and a is not None:
+        out += [name for name in ("ignore", "ignore_data") if b.get(name) != a.get(name)]
+    return out
+
+
+def compare(before: dict, report: dict, found: list, top: int = watch.WATCH_TOP) -> dict:
+    """before: an earlier export; report and found: this run's loaded report and its findings."""
+    b = {key(f): f for f in before.get("findings") or []}
+    a = {key(f): f for f in found}
+    persisting = [{**a[k], "was": b[k]["severity"]} for k in a if k in b]
+    before_watch = [f for f in (r.get("file") for r in (before.get("watch") or [])[:top]) if f]
+    after_watch = [r["file"] for r in watch.risks(report)[:top]]
+    meta_b, meta_a = before.get("meta") or {}, report.get("meta") or {}
+    return {"new": _ordered([a[k] for k in a if k not in b]), "resolved": _ordered([b[k] for k in b if k not in a]),
+            "persisting": _ordered(persisting),
+            "watch_entered": [f for f in after_watch if f not in before_watch], "watch_left": [f for f in before_watch if f not in after_watch],
+            "tally": {"before": _tally(before.get("findings") or []), "after": _tally(found)},
+            "before": {"commit": (meta_b.get("run") or {}).get("commit"), "date": meta_b.get("last_date"),
+                       "options_differ": _options_differ(meta_b, meta_a)}}

{gitmole-0.9.0 → gitmole-0.10.0}/gitmole/evaluate.py

@@ -42,13 +42,14 @@ def fixed_between(commits: list, start: str, end: str) -> set:
             for p, _, _ in c["files"] if not filetypes.is_test_path(p)}
 
 
-def report_at(commits: list, t: str, size: dict, meta: dict) -> dict:
-    """The report watch.risks reads, from the commits before `t` and scc's listing of the tree at `t`.
+def report_at(commits: list, t: str, size: dict, meta: dict, generated: list, vendored: list) -> dict:
+    """The report watch.risks reads, from the commits before `t`, scc's listing of the tree at `t`, and
+    that tree's own generated and vendored files (snapshot_at classified the cut-off, not HEAD).
     No coupling and no functions: neither enters the score, and the pipeline's own backtest has no functions either."""
     past = maat.in_window(commits, until=t)
     bots = {b["name"] for b in meta.get("bots") or []}
     ownership = [r for r in maat.entity_ownership(past) if r["author"] not in bots and not identity.is_bot(r["author"])]
-    return {"meta": {"now": t, "generated": meta.get("generated") or []}, "size": size, "revisions": maat.revisions(past),
+    return {"meta": {"now": t, "generated": generated, "vendored": vendored}, "size": size, "revisions": maat.revisions(past),
             "plumbing": maat.plumbing(past), "authors": maat.authors(past), "ownership": ownership,
             "fixes": maat.fixes(past, now=t), "coupling": [], "functions": []}
 
@@ -163,8 +164,9 @@ def main(argv=None) -> int:
         rev = trend.rev_before(args.repo, t, end_of_day=False)
         if not rev:
             continue                      # the history does not reach back this far
-        size = load.parse_scc(backtest.size_at(args.repo, rev, args.out), types)
-        report = report_at(commits, t, size, meta)
+        size_json, generated, vendored = backtest.snapshot_at(args.repo, rev, args.out)
+        size = load.parse_scc(size_json, types)
+        report = report_at(commits, t, size, meta, generated, vendored)
         fixed = fixed_between(commits, t, months_after(t, args.horizon))
         pool = set(variants(report)["churn"])
         results.append((t, len(fixed & pool), len(pool), score(report, fixed, args.top)))