gitlogstats 1.4.1__tar.gz → 1.5.0__tar.gz

{gitlogstats-1.4.1 → gitlogstats-1.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: gitlogstats
-Version: 1.4.1
+Version: 1.5.0
 Summary: Track developers' contributions to one or more git repositories within a particular time range.
 Project-URL: Homepage, https://github.com/bloombar/gitlogstats
 Project-URL: Repository, https://github.com/bloombar/gitlogstats.git
@@ -726,7 +726,7 @@ The command `gitlogstats --help` shows the usage instructions:
 ```
 usage: gitlogstats [-h] (-r REPOSITORY | -rf REPOFILE) [-u USER] [-s START]
                    [-e END] [-x EXCLUSIONS] [-f {csv,json,markdown}]
-                   [-b BRANCH] [-v] [-c] [--github]
+                   [-b BRANCH] [-v] [-c] [--github] [--clobber]
 
 options:
   -h, --help            show this help message and exit
@@ -757,6 +757,9 @@ options:
                         and their commits made on the GitHub website, and
                         match GitHub users to git authors. Needs the GitHub
                         CLI (gh), signed in
+  --clobber             Delete the repository clones this run used from the
+                        repos/ directory after printing the results (default:
+                        keep them for faster later runs)
 ```
 
 In case `gitlogstats` is not found as a command, even after installing, try `python -m gitlogstats` or `python3 -m gitlogstats` instead.
@@ -837,6 +840,16 @@ By default, every contributor in a repository's history is listed, with zeros fo
 gitlogstats -rf repos.txt -c
 ```
 
+### Cloned repositories
+
+Each repository is cloned into a `repos/` directory under the directory you run the command from, and later runs update (`git pull`) the existing clone instead of cloning again, which makes them much faster. Add `--clobber` to delete the clones the run used, once the results have been printed; `repos/` itself is removed too if nothing else is left in it. Clones from other runs are kept, and nothing is deleted if the run fails before printing, so you can inspect it. With `-v`, each deleted path is listed on standard error.
+
+```
+gitlogstats -rf repos.txt --clobber
+```
+
+`--clobber` is unrelated to `-c/--clean`, which only hides contributors without activity.
+
 ### GitHub statistics
 
 Add `--github` to also report each contributor's pull requests, pushes, website commits and email addresses. Nine columns are appended to every output format:

{gitlogstats-1.4.1 → gitlogstats-1.5.0}/README.md

@@ -29,7 +29,7 @@ The command `gitlogstats --help` shows the usage instructions:
 ```
 usage: gitlogstats [-h] (-r REPOSITORY | -rf REPOFILE) [-u USER] [-s START]
                    [-e END] [-x EXCLUSIONS] [-f {csv,json,markdown}]
-                   [-b BRANCH] [-v] [-c] [--github]
+                   [-b BRANCH] [-v] [-c] [--github] [--clobber]
 
 options:
   -h, --help            show this help message and exit
@@ -60,6 +60,9 @@ options:
                         and their commits made on the GitHub website, and
                         match GitHub users to git authors. Needs the GitHub
                         CLI (gh), signed in
+  --clobber             Delete the repository clones this run used from the
+                        repos/ directory after printing the results (default:
+                        keep them for faster later runs)
 ```
 
 In case `gitlogstats` is not found as a command, even after installing, try `python -m gitlogstats` or `python3 -m gitlogstats` instead.
@@ -140,6 +143,16 @@ By default, every contributor in a repository's history is listed, with zeros fo
 gitlogstats -rf repos.txt -c
 ```
 
+### Cloned repositories
+
+Each repository is cloned into a `repos/` directory under the directory you run the command from, and later runs update (`git pull`) the existing clone instead of cloning again, which makes them much faster. Add `--clobber` to delete the clones the run used, once the results have been printed; `repos/` itself is removed too if nothing else is left in it. Clones from other runs are kept, and nothing is deleted if the run fails before printing, so you can inspect it. With `-v`, each deleted path is listed on standard error.
+
+```
+gitlogstats -rf repos.txt --clobber
+```
+
+`--clobber` is unrelated to `-c/--clean`, which only hides contributors without activity.
+
 ### GitHub statistics
 
 Add `--github` to also report each contributor's pull requests, pushes, website commits and email addresses. Nine columns are appended to every output format:

{gitlogstats-1.4.1 → gitlogstats-1.5.0}/docs/SPEC.md

@@ -29,7 +29,7 @@ This document describes what the tool does today and what is planned. Each requi
 
 #### REPO-3 Local clone cache
 
-Repositories are cloned into a `repos/` directory under the current working directory, in a folder named the way `git clone` names it (the last path segment, minus a trailing `.git`; dots in the name are kept). On later runs the existing clone is updated with `git pull` instead of being cloned again.
+Repositories are cloned into a `repos/` directory under the current working directory, in a folder named the way `git clone` names it (the last path segment, minus a trailing `.git`; dots in the name are kept). On later runs the existing clone is updated with `git pull` instead of being cloned again. `--clobber` removes the clones after the run (REPO-5).
 
 **Status:** Done
 
@@ -39,6 +39,12 @@ Repositories are cloned into a `repos/` directory under the current working dire
 
 **Status:** Done
 
+#### REPO-5 Remove clones after the run
+
+`--clobber` deletes the clones this run used from the `repos/` directory (REPO-3) once the results have been printed, and then removes `repos/` itself if nothing else is left in it. Clones in `repos/` that this run did not use are kept. It is off by default, so clones are kept and reused by later runs. If the run fails before the results are printed, nothing is deleted, so the clones can be inspected. Only directories inside `repos/` are deleted; a symbolic link in place of a clone is removed without following it. If `repos/` is itself a symbolic link to another directory, the clones are deleted from that directory and the link is kept. A clone that cannot be fully deleted (e.g. it contains a folder the user cannot open) is reported on standard error and the remaining clones are still deleted; the exit status stays 0, since the results were printed. Read-only files that git writes inside a clone (as on Windows) are deleted too. With `-v`, each deleted path is printed to standard error. The name follows build tools such as Rake, where `clean` removes intermediate files and `clobber` also removes caches; it is unrelated to `-c/--clean` (FILT-4).
+
+**Status:** Done
+
 ### 2. Contribution Statistics
 
 #### STAT-1 Commit count

{gitlogstats-1.4.1 → gitlogstats-1.5.0}/pyproject.toml

@@ -5,7 +5,7 @@ build-backend = "hatchling.build"
 [project]
 name = "gitlogstats"
 description = "Track developers' contributions to one or more git repositories within a particular time range."
-version = "1.4.1"
+version = "1.5.0"
 authors = [
   { name="Tobody Tothing", email="nobody.nothing@nowhere.com" },
 ]

{gitlogstats-1.4.1 → gitlogstats-1.5.0}/scripts/board/manifest.yaml

@@ -194,6 +194,7 @@
     directory, in a folder named the way git clone names it (the last path
     segment, minus a trailing .git; dots in the name are kept). On later runs
     the existing clone is updated with git pull instead of being cloned again.
+    --clobber removes the clones after the run (REPO-5).
   section: §1 Repository Selection
   family: REPO
   phase: 1
@@ -209,6 +210,22 @@
   family: REPO
   phase: 1
   status: Done
+- id: REPO-5
+  title: Remove clones after the run
+  full: --clobber deletes the clones this run used from the repos/ directory
+    (REPO-3) once the results have been printed, and then removes repos/ itself
+    if nothing else is left in it. Clones in repos/ that this run did not use
+    are kept. It is off by default, so clones are kept and reused by later runs.
+    If the run fails before the results are printed, nothing is deleted, so the
+    clones can be inspected. Only directories inside repos/ are deleted; a
+    symbolic link in place of a clone is removed without following it. If repos/
+    is itself a symbolic link to another directory, the clones are deleted from
+    that directory and the link is kept. A clone that cannot be fully deleted
+    (e.g. it contains a folder…
+  section: §1 Repository Selection
+  family: REPO
+  phase: 1
+  status: Done
 - id: STAT-1
   title: Commit count
   full: "For each contributor, the number of non-merge commits they authored in

{gitlogstats-1.4.1 → gitlogstats-1.5.0}/src/gitlogstats/__main__.py

@@ -5,6 +5,7 @@ import datetime
 import re
 import sys
 from . import GitLogsParser
+from .clones import remove_clones
 from .git_logs_parser import FIELDNAMES
 from .github import (
     ACTIVITY_FIELDNAMES,
@@ -155,6 +156,14 @@ def main():
         default=False,
         action="store_true",
     )
+    parser.add_argument(
+        "--clobber",
+        help="Delete the repository clones this run used from the repos/ "
+        "directory after printing the results (default: keep them for faster "
+        "later runs)",
+        default=False,
+        action="store_true",
+    )
     args = parser.parse_args()
 
     if datetime.datetime.strptime(args.start, "%m/%d/%Y") > datetime.datetime.strptime(
@@ -179,6 +188,7 @@ def main():
         with open(args.repofile, "r", encoding="utf8") as f:
             repository_urls = GitLogsParser.read_repository_urls(f.read())
 
+    start_dir = os.getcwd()  # REPO-5: where the command was started from
     # navigate into the directory where repos will be stored
     repos_dir = os.path.join(
         os.getcwd(), "repos"
@@ -190,6 +200,7 @@ def main():
     # loop through each git repository url, accumulating all results
     all_results = []
     last_parser = None
+    used_clones = []  # REPO-5: the clone directories this run cloned or pulled
     for repo_url in repository_urls:
         os.chdir(repos_dir)  # start from the parent directory of all repos
 
@@ -221,6 +232,8 @@ def main():
                 ["git", "pull"], capture_output=True, check=True
             )  # get the latest code from github
 
+        used_clones.append(repo_dir)
+
         # with --github, -u and -c are applied after GitHub users are combined
         last_parser = GitLogsParser(
             repo=repo_dir,
@@ -250,6 +263,13 @@ def main():
         # end with exactly one newline: CSV and Markdown output already has one
         print(output, end="" if output.endswith("\n") else "\n")
 
+    # REPO-5: only now that the results are out, delete the clones. Leave the
+    # clone we were last inside first, so the deletion never depends on it.
+    if args.clobber:
+        sys.stdout.flush()
+        os.chdir(start_dir)
+        remove_clones(repos_dir, used_clones, args.verbose)
+
 
 # if this script is being run directly...
 if __name__ == "__main__":

gitlogstats-1.5.0/src/gitlogstats/clones.py

@@ -0,0 +1,85 @@
+"""Deleting the repository clones a run used (REPO-5, --clobber)."""
+
+import os
+import shutil
+import stat
+import sys
+
+
+def _make_writable_and_retry(function, path, exc_info):
+    """
+    Error handler for shutil.rmtree: git marks some files read-only (notably
+    pack files on Windows), which stops them being deleted. Clear the read-only
+    bit on the file and its folder, then try again. Any other error is raised,
+    as is a failure to open or list a folder: retrying os.open or os.scandir
+    with only a path would itself fail with a TypeError.
+    """
+    if not isinstance(exc_info[1], PermissionError) or function not in (
+        os.unlink,
+        os.remove,
+        os.rmdir,
+    ):
+        raise exc_info[1]
+    for target in (os.path.dirname(path), path):
+        try:
+            os.chmod(target, os.stat(target).st_mode | stat.S_IRWXU)
+        except OSError:
+            pass
+    function(path)
+
+
+def _delete(path):
+    """
+    Delete one clone. A symbolic link is unlinked, never followed.
+    @param path: the clone's path
+    """
+    if os.path.islink(path) or not os.path.isdir(path):
+        os.remove(path)
+    else:
+        shutil.rmtree(path, onerror=_make_writable_and_retry)
+
+
+def remove_clones(repos_dir, clone_dirs, verbose=False):
+    """
+    Delete the given clones, then remove repos_dir if nothing else is left in it.
+    Only direct children of repos_dir are deleted; any other path is skipped.
+    A failed deletion is reported on standard error and does not stop the rest.
+    @param repos_dir: the directory holding the clones
+    @param clone_dirs: the paths of the clones to delete (duplicates are fine)
+    @param verbose: whether to report each deleted or skipped path on stderr
+    """
+    real_repos = os.path.realpath(repos_dir)
+    seen = set()
+    for clone_dir in clone_dirs:
+        path = os.path.normpath(os.path.abspath(clone_dir))
+        if path in seen:
+            continue
+        seen.add(path)
+        # the folder that holds the clone must be repos_dir itself; checking the
+        # parent (not the path) means a symbolic link is judged by where it sits
+        if os.path.realpath(os.path.dirname(path)) != real_repos:
+            if verbose:
+                print(
+                    f"gitlogstats: not deleting {path}: not inside {repos_dir}",
+                    file=sys.stderr,
+                )
+            continue
+        if not os.path.lexists(path):
+            continue
+        try:
+            _delete(path)
+        except OSError as exc:
+            print(
+                f"gitlogstats: could not delete {path}: {exc.strerror or exc}",
+                file=sys.stderr,
+            )
+            continue
+        if verbose:
+            print(f"gitlogstats: deleted {path}", file=sys.stderr)
+
+    try:
+        os.rmdir(repos_dir)  # only succeeds when empty
+    except OSError:
+        return
+    if verbose:
+        print(f"gitlogstats: deleted {repos_dir}", file=sys.stderr)

gitlogstats-1.5.0/tests/test_clobber.py

@@ -0,0 +1,252 @@
+"""Tests for --clobber (REPO-5): deleting the clones a run used."""
+
+import json
+import os
+import stat
+import sys
+
+import pytest
+
+from gitlogstats import clones
+from gitlogstats.__main__ import main
+from gitlogstats.clones import remove_clones
+from test_integration import (
+    DATES,
+    GITHUB_URL,
+    Repo,
+    github_world,
+    lines,
+    pr,
+    run_github,
+)
+
+
+@pytest.fixture
+def remotes(tmp_path):
+    """Two 'remote' repositories that the command clones by path."""
+    one = Repo(tmp_path / "remotes" / "one")
+    one.write("a.txt", lines(3))
+    one.commit("Alice", "one: add")
+    two = Repo(tmp_path / "remotes" / "two.repo")
+    two.write("c.txt", lines(2))
+    two.commit("Alice", "two: add")
+    return one, two
+
+
+@pytest.fixture
+def work(tmp_path):
+    path = tmp_path / "work"
+    path.mkdir()
+    return path
+
+
+def run(monkeypatch, capsys, work, *argv):
+    """Run the command in *work*; returns (stdout, stderr)."""
+    monkeypatch.chdir(work)
+    monkeypatch.setattr(
+        sys, "argv", ["gitlogstats", *argv, "-s", "01/01/2024"] + ["-e", "12/31/2024"]
+    )
+    main()
+    captured = capsys.readouterr()
+    return captured.out, captured.err
+
+
+class TestCommand:
+    def test_clones_are_kept_without_the_flag(self, remotes, work, monkeypatch, capsys):
+        run(monkeypatch, capsys, work, "-r", remotes[0].path)
+        assert (work / "repos" / "one" / ".git").is_dir()
+
+    def test_clobber_prints_the_same_results_and_removes_everything(
+        self, remotes, work, monkeypatch, capsys
+    ):
+        kept, _ = run(monkeypatch, capsys, work, "-r", remotes[0].path, "-f", "json")
+        assert (work / "repos").exists()
+        # a fresh directory so that the second run clones again
+        other = work.parent / "work2"
+        other.mkdir()
+        out, err = run(
+            monkeypatch, capsys, other, "-r", remotes[0].path, "-f", "json", "--clobber"
+        )
+        assert out == kept and json.loads(out)[0]["commits"] == 1
+        assert err == ""
+        assert os.listdir(other) == []  # repos/ itself is gone
+
+    def test_other_clones_and_repos_dir_are_kept(
+        self, remotes, work, monkeypatch, capsys
+    ):
+        run(monkeypatch, capsys, work, "-r", remotes[1].path)
+        out, _ = run(monkeypatch, capsys, work, "-r", remotes[0].path, "--clobber")
+        assert "Alice,one," in out
+        assert sorted(os.listdir(work / "repos")) == ["two.repo"]
+
+    def test_repofile_with_two_repos_and_a_duplicate(
+        self, remotes, work, monkeypatch, capsys
+    ):
+        repofile = work.parent / "list.txt"
+        repofile.write_text(
+            f"{remotes[0].path}\n{remotes[1].path}\n{remotes[0].path}\n"
+        )
+        out, _ = run(monkeypatch, capsys, work, "-rf", str(repofile), "--clobber")
+        assert "Alice,one," in out and "Alice,two.repo," in out
+        assert os.listdir(work) == []
+
+    def test_failure_before_output_deletes_nothing(
+        self, remotes, work, monkeypatch, capsys
+    ):
+        repofile = work.parent / "list.txt"
+        repofile.write_text(f"{remotes[0].path}\n{work.parent / 'missing'}\n")
+        with pytest.raises(Exception):
+            run(monkeypatch, capsys, work, "-rf", str(repofile), "--clobber")
+        assert (work / "repos" / "one" / ".git").is_dir()
+        assert capsys.readouterr().out == ""
+
+    def test_verbose_lists_deleted_paths_on_stderr_only(
+        self, remotes, work, monkeypatch, capsys
+    ):
+        quiet, _ = run(monkeypatch, capsys, work, "-r", remotes[0].path, "--clobber")
+        out, err = run(
+            monkeypatch, capsys, work, "-r", remotes[0].path, "--clobber", "-v"
+        )
+        assert out == quiet
+        repos = os.path.realpath(work / "repos")
+        assert f"gitlogstats: deleted {os.path.join(repos, 'one')}" in err
+        assert f"gitlogstats: deleted {repos}\n" in err
+        assert "deleted" not in out
+
+    def test_clobber_with_github(self, remotes, tmp_path, monkeypatch, capsys):
+        github_world(tmp_path, monkeypatch, remotes[0], prs=[], contributors=[])
+        out, _ = run_github(
+            monkeypatch,
+            capsys,
+            tmp_path,
+            "-r",
+            GITHUB_URL,
+            "--github",
+            *DATES,
+            "--clobber",
+        )
+        assert "Alice" in out
+        assert not (tmp_path / "work" / "repos").exists()
+
+    def test_nothing_to_delete_when_no_repositories(self, work, monkeypatch, capsys):
+        repofile = work.parent / "empty.txt"
+        repofile.write_text("# nothing\n")
+        out, _ = run(monkeypatch, capsys, work, "-rf", str(repofile), "--clobber")
+        assert out == "" and os.listdir(work) == []
+
+
+class TestRemoveClones:
+    def test_read_only_files_are_deleted(self, tmp_path):
+        repos = tmp_path / "repos"
+        clone = repos / "x"
+        (clone / "objects").mkdir(parents=True)
+        for name in ("a", "objects/pack"):
+            (clone / name).write_text("data")
+            os.chmod(clone / name, stat.S_IREAD)
+        os.chmod(clone / "objects", stat.S_IREAD | stat.S_IEXEC)
+        remove_clones(str(repos), [str(clone)])
+        assert not repos.exists()
+
+    def test_symlink_is_unlinked_and_target_untouched(self, tmp_path):
+        target = tmp_path / "precious"
+        target.mkdir()
+        (target / "keep.txt").write_text("keep")
+        repos = tmp_path / "repos"
+        repos.mkdir()
+        os.symlink(target, repos / "link")
+        remove_clones(str(repos), [str(repos / "link")])
+        assert not repos.exists()
+        assert (target / "keep.txt").read_text() == "keep"
+
+    def test_paths_outside_repos_are_skipped(self, tmp_path, capsys):
+        repos = tmp_path / "repos"
+        repos.mkdir()
+        outside = tmp_path / "outside"
+        outside.mkdir()
+        nested = repos / "a" / "b"
+        nested.mkdir(parents=True)
+        for path in (outside, repos / ".." / "outside", nested, repos):
+            remove_clones(str(repos), [str(path)], verbose=True)
+        assert outside.is_dir() and nested.is_dir() and repos.is_dir()
+        assert capsys.readouterr().err.count("not deleting") == 4
+
+    def test_skipped_paths_are_silent_without_verbose(self, tmp_path, capsys):
+        repos = tmp_path / "repos"
+        repos.mkdir()
+        outside = tmp_path / "outside"
+        outside.mkdir()
+        remove_clones(str(repos), [str(outside)])
+        assert outside.is_dir() and capsys.readouterr().err == ""
+
+    def test_a_file_in_place_of_a_clone_is_removed(self, tmp_path):
+        repos = tmp_path / "repos"
+        repos.mkdir()
+        (repos / "f").write_text("x")
+        remove_clones(str(repos), [str(repos / "f")])
+        assert not repos.exists()
+
+    def test_deletion_error_is_reported_and_the_rest_continues(
+        self, tmp_path, monkeypatch, capsys
+    ):
+        repos = tmp_path / "repos"
+        for name in ("bad", "good"):
+            (repos / name).mkdir(parents=True)
+
+        real = clones._delete
+
+        def flaky(path):
+            if path.endswith("bad"):
+                raise PermissionError(13, "Permission denied")
+            real(path)
+
+        monkeypatch.setattr(clones, "_delete", flaky)
+        remove_clones(str(repos), [str(repos / "bad"), str(repos / "good")])
+        err = capsys.readouterr().err
+        assert (
+            f"gitlogstats: could not delete {repos / 'bad'}: Permission denied" in err
+        )
+        assert os.listdir(repos) == ["bad"]
+
+    @pytest.mark.skipif(
+        sys.platform == "win32" or os.geteuid() == 0,
+        reason="needs POSIX permissions that apply to the current user",
+    )
+    def test_unreadable_folder_is_reported_and_the_rest_continues(
+        self, tmp_path, capsys
+    ):
+        # a folder that cannot be opened makes rmtree call the handler with
+        # os.open; retrying that must not crash after the results are printed
+        repos = tmp_path / "repos"
+        locked = repos / "one" / "notes" / "private"
+        locked.mkdir(parents=True)
+        (locked / "a.txt").write_text("x")
+        (repos / "two").mkdir()
+        locked.chmod(0)
+        try:
+            remove_clones(str(repos), [str(repos / "one"), str(repos / "two")])
+        finally:
+            locked.chmod(stat.S_IRWXU)
+        err = capsys.readouterr().err
+        assert f"gitlogstats: could not delete {repos / 'one'}" in err
+        assert not (repos / "two").exists()
+
+    def test_other_errors_while_deleting_are_not_retried(self, tmp_path):
+        with pytest.raises(FileNotFoundError):
+            clones._make_writable_and_retry(
+                os.remove, str(tmp_path / "gone"), (None, FileNotFoundError(), None)
+            )
+
+    def test_retry_gives_up_when_chmod_is_impossible(self, tmp_path):
+        # the paths do not exist, so chmod fails quietly and the retry raises
+        with pytest.raises(FileNotFoundError):
+            clones._make_writable_and_retry(
+                os.remove,
+                str(tmp_path / "no" / "gone"),
+                (None, PermissionError(), None),
+            )
+
+    def test_a_clone_that_is_already_gone_is_not_an_error(self, tmp_path, capsys):
+        repos = tmp_path / "repos"
+        repos.mkdir()
+        remove_clones(str(repos), [str(repos / "gone")], verbose=True)
+        assert capsys.readouterr().err == f"gitlogstats: deleted {repos}\n"