gitlogstats 1.6.0__tar.gz → 1.7.0__tar.gz

{gitlogstats-1.6.0 → gitlogstats-1.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: gitlogstats
-Version: 1.6.0
+Version: 1.7.0
 Summary: Track developers' contributions to one or more git repositories within a particular time range.
 Project-URL: Homepage, https://github.com/bloombar/gitlogstats
 Project-URL: Repository, https://github.com/bloombar/gitlogstats.git
@@ -726,7 +726,7 @@ The command `gitlogstats --help` shows the usage instructions:
 ```
 usage: gitlogstats [-h] (-r REPOSITORY | -rf REPOFILE) [-u USER] [-s START]
                    [-e END] [-x EXCLUSIONS] [-f {csv,json,markdown}]
-                   [-b BRANCH] [-v] [-c] [-a {developer,repository}]
+                   [-b BRANCH] [-v] [-c] [-a {developer,repository,day}]
                    [--github] [--clobber]
 
 options:
@@ -752,10 +752,11 @@ options:
                         The branch to parse, if not the default branch
   -v, --verbose         Whether to output debugging info
   -c, --clean           Remove contributors without any contribuition
-  -a {developer,repository}, --aggregate {developer,repository}
+  -a {developer,repository,day}, --aggregate {developer,repository,day}
                         Combine the results into one row per developer, summed
-                        over all repositories, or one row per repository,
-                        summed over its contributors
+                        over all repositories, one row per repository, summed
+                        over its contributors, or one row per day with
+                        activity, summed over everyone
   --github              Also report GitHub pull requests opened, merged,
                         closed and self-merged per contributor, with their git
                         and GitHub emails, their pushes to the analyzed branch
@@ -856,6 +857,16 @@ gitlogstats -rf repos.txt -a repository
 
 Developer rows have no `repository` column; instead a `repositories` column counts the repositories the developer was active in during the date range. Repository rows have no username or email columns; a `contributors` column counts the contributors who were active in the repository during the date range. `-u` and `-c` are applied first, so `-u alice -a repository` shows Alice's share of each repository. With `--github`, a developer's rows are combined by GitHub username, so different git names linked to one GitHub account become one row, and their email addresses are listed together. A developer matched to a GitHub account in one repository but not in another (for example a repository not on GitHub) keeps a separate row for each.
 
+### Activity per day
+
+Add `-a day` to get one row per day on which anything happened, summed over every contributor and repository, in date order:
+
+```
+gitlogstats -rf repos.txt -a day
+```
+
+The columns are `date`, the usual counts, `repositories` (how many repositories had activity that day) and `contributors` (how many developers were active that day; someone active in several repositories counts once, but, as with `-a developer`, someone matched to a GitHub account in one repository and not in another counts twice). Every event is counted on its own day in local time: commits and merges on their commit date and, with `--github`, pull requests on the day they were opened, merged or closed and pushes on the day they were pushed. The days therefore add up to the totals printed without `-a`. Days without activity are left out. Combine with `-u` to see one contributor's days.
+
 ### Cloned repositories
 
 Each repository is cloned into a `repos/` directory under the directory you run the command from, and later runs update (`git pull`) the existing clone instead of cloning again, which makes them much faster. Add `--clobber` to delete the clones the run used, once the results have been printed; `repos/` itself is removed too if nothing else is left in it. Clones from other runs are kept, and nothing is deleted if the run fails before printing, so you can inspect it. With `-v`, each deleted path is listed on standard error.

{gitlogstats-1.6.0 → gitlogstats-1.7.0}/README.md

@@ -29,7 +29,7 @@ The command `gitlogstats --help` shows the usage instructions:
 ```
 usage: gitlogstats [-h] (-r REPOSITORY | -rf REPOFILE) [-u USER] [-s START]
                    [-e END] [-x EXCLUSIONS] [-f {csv,json,markdown}]
-                   [-b BRANCH] [-v] [-c] [-a {developer,repository}]
+                   [-b BRANCH] [-v] [-c] [-a {developer,repository,day}]
                    [--github] [--clobber]
 
 options:
@@ -55,10 +55,11 @@ options:
                         The branch to parse, if not the default branch
   -v, --verbose         Whether to output debugging info
   -c, --clean           Remove contributors without any contribuition
-  -a {developer,repository}, --aggregate {developer,repository}
+  -a {developer,repository,day}, --aggregate {developer,repository,day}
                         Combine the results into one row per developer, summed
-                        over all repositories, or one row per repository,
-                        summed over its contributors
+                        over all repositories, one row per repository, summed
+                        over its contributors, or one row per day with
+                        activity, summed over everyone
   --github              Also report GitHub pull requests opened, merged,
                         closed and self-merged per contributor, with their git
                         and GitHub emails, their pushes to the analyzed branch
@@ -159,6 +160,16 @@ gitlogstats -rf repos.txt -a repository
 
 Developer rows have no `repository` column; instead a `repositories` column counts the repositories the developer was active in during the date range. Repository rows have no username or email columns; a `contributors` column counts the contributors who were active in the repository during the date range. `-u` and `-c` are applied first, so `-u alice -a repository` shows Alice's share of each repository. With `--github`, a developer's rows are combined by GitHub username, so different git names linked to one GitHub account become one row, and their email addresses are listed together. A developer matched to a GitHub account in one repository but not in another (for example a repository not on GitHub) keeps a separate row for each.
 
+### Activity per day
+
+Add `-a day` to get one row per day on which anything happened, summed over every contributor and repository, in date order:
+
+```
+gitlogstats -rf repos.txt -a day
+```
+
+The columns are `date`, the usual counts, `repositories` (how many repositories had activity that day) and `contributors` (how many developers were active that day; someone active in several repositories counts once, but, as with `-a developer`, someone matched to a GitHub account in one repository and not in another counts twice). Every event is counted on its own day in local time: commits and merges on their commit date and, with `--github`, pull requests on the day they were opened, merged or closed and pushes on the day they were pushed. The days therefore add up to the totals printed without `-a`. Days without activity are left out. Combine with `-u` to see one contributor's days.
+
 ### Cloned repositories
 
 Each repository is cloned into a `repos/` directory under the directory you run the command from, and later runs update (`git pull`) the existing clone instead of cloning again, which makes them much faster. Add `--clobber` to delete the clones the run used, once the results have been printed; `repos/` itself is removed too if nothing else is left in it. Clones from other runs are kept, and nothing is deleted if the run fails before printing, so you can inspect it. With `-v`, each deleted path is listed on standard error.

{gitlogstats-1.6.0 → gitlogstats-1.7.0}/docs/SPEC.md

@@ -212,6 +212,14 @@ In both modes every count is the sum of the combined rows, so each total equals
 
 **Status:** Done
 
+#### OUT-7 Results by day
+
+`-a/--aggregate day` prints one row per day on which anything was counted, summed over every contributor and repository analyzed, in date order. Its columns are `date, commits, insertions, deletions, files, merges, repositories, contributors`, where `date` is the day in `mm/dd/yyyy` format, `repositories` is the number of repositories with any activity that day and `contributors` the number of developers with any activity that day (a developer is identified as in OUT-6, so one person active in several repositories counts once, while one matched to a GitHub user in one repository but not in another counts twice). With `--github`, the columns `prs_opened, prs_merged, prs_closed_unmerged, prs_self_merged, pushes_to_default, web_commits` follow. There are no `username`, `repository`, `start_date`, `end_date`, `github_username`, `git_email` or `github_email` columns.
+
+Each event is counted on the day it happened, in local time, the same way FILT-1 limits the date range: a commit, merge or website commit on its commit date; a pull request as opened on the day it was created, as merged (and self-merged) on the day it was merged, and as closed without merging on the day it was closed; a push on the day it was pushed. So the day totals add up to the totals printed without `-a`. Days without any activity are left out. `-u` limits the counts to that contributor (as in CLI-3 with `--github`); `-c` changes nothing, since inactive contributors add nothing to a day. A GitHub count is empty (`null` in JSON) only when no repository with activity that day could be asked about (CLI-3). The columns are printed in the CSV header even when there are no rows.
+
+**Status:** Done
+
 ### 5. Command-Line Interface
 
 #### CLI-1 Help and usage

{gitlogstats-1.6.0 → gitlogstats-1.7.0}/pyproject.toml

@@ -5,7 +5,7 @@ build-backend = "hatchling.build"
 [project]
 name = "gitlogstats"
 description = "Track developers' contributions to one or more git repositories within a particular time range."
-version = "1.6.0"
+version = "1.7.0"
 authors = [
   { name="Tobody Tothing", email="nobody.nothing@nowhere.com" },
 ]

{gitlogstats-1.6.0 → gitlogstats-1.7.0}/scripts/board/manifest.yaml

@@ -148,6 +148,22 @@
   family: OUT
   phase: 1
   status: Done
+- id: OUT-7
+  title: Results by day
+  full: -a/--aggregate day prints one row per day on which anything was counted,
+    summed over every contributor and repository analyzed, in date order. Its
+    columns are date, commits, insertions, deletions, files, merges,
+    repositories, contributors, where date is the day in mm/dd/yyyy format,
+    repositories is the number of repositories with any activity that day and
+    contributors the number of developers with any activity that day (a
+    developer is identified as in OUT-6, so one person active in several
+    repositories counts once, while one matched to a GitHub user in one
+    repository but not in another counts twice). With --github, the columns
+    prs_opened, prs_merged, prs_closed_unmerged, prs_self_merged…
+  section: §4 Output
+  family: OUT
+  phase: 1
+  status: Done
 - id: PKG-1
   title: Installable package
   full: The tool installs with pip install gitlogstats, which provides a

{gitlogstats-1.6.0 → gitlogstats-1.7.0}/src/gitlogstats/__main__.py

@@ -15,7 +15,7 @@ from .github import (
     GitHubUnavailable,
     check_gh,
 )
-from .github_stats import github_rows
+from .github_stats import github_day_rows, github_rows
 
 
 def valid_date(value):
@@ -151,7 +151,8 @@ def main():
         "-a",
         "--aggregate",
         help="Combine the results into one row per developer, summed over all "
-        "repositories, or one row per repository, summed over its contributors",
+        "repositories, one row per repository, summed over its contributors, "
+        "or one row per day with activity, summed over everyone",
         default=None,
         choices=aggregate.MODES,
     )
@@ -253,18 +254,29 @@ def main():
             verbose=args.verbose,
             clean=False if args.github else args.clean,
         )
-        rows = last_parser.parse()
-        if args.github:
-            rows = github_rows(
-                last_parser, rows, repo_url, last_parser.get_identities(), args
-            )
+        if args.aggregate == aggregate.DAY:
+            # OUT-7: rows for single days, each with its date
+            if args.github:
+                rows = github_day_rows(
+                    last_parser, repo_url, last_parser.get_identities(), args
+                )
+            else:
+                rows = last_parser.day_rows()
+            # an idle contributor adds nothing to a day; keep memory small
+            rows = [row for row in rows if aggregate.is_active(row)]
+        else:
+            rows = last_parser.parse()
+            if args.github:
+                rows = github_rows(
+                    last_parser, rows, repo_url, last_parser.get_identities(), args
+                )
         all_results.extend(rows)  # collect across repos
 
     # emit a single table (or JSON array) covering all repositories
     if last_parser is not None:
         # an empty CSV table still gets the columns it would have had
         if args.aggregate:
-            # OUT-6: one row per developer or per repository
+            # OUT-6, OUT-7: one row per developer, repository or day
             all_results = aggregate.aggregate(all_results, args.aggregate, args.github)
             fieldnames = aggregate.fieldnames(args.aggregate, args.github)
         elif args.github:

{gitlogstats-1.6.0 → gitlogstats-1.7.0}/src/gitlogstats/aggregate.py

@@ -1,9 +1,11 @@
-"""Combine result rows by developer or by repository (OUT-6).
+"""Combine result rows by developer or by repository (OUT-6), or by day (OUT-7).
 
 Everything here is pure logic over the rows the rest of the tool produces: no
 git, no network. Each aggregated row sums the counts of the rows it combines.
 """
 
+import datetime
+
 from gitlogstats.git_logs_parser import FIELDNAMES
 from gitlogstats.github import (
     ACTIVITY_FIELDNAMES,
@@ -14,12 +16,13 @@ from gitlogstats.identity import join_emails
 
 DEVELOPER = "developer"
 REPOSITORY = "repository"
-MODES = [DEVELOPER, REPOSITORY]
+DAY = "day"
+MODES = [DEVELOPER, REPOSITORY, DAY]
 
-# the git counts (commits to merges), summed in both modes
+# the git counts (commits to merges), summed in every mode
 GIT_COUNTS = FIELDNAMES[4:]
 
-# the GitHub pull-request counts (CLI-3), summed in both modes
+# the GitHub pull-request counts (CLI-3), summed in every mode
 PR_COUNTS = GITHUB_FIELDNAMES[1:]
 
 # every count; a row with any of them non-zero is active, as for -c (FILT-4)
@@ -29,7 +32,7 @@ ALL_COUNTS = GIT_COUNTS + PR_COUNTS + ACTIVITY_FIELDNAMES
 def fieldnames(mode, github):
     """
     The output columns of an aggregated table, in order.
-    @param mode: DEVELOPER or REPOSITORY
+    @param mode: DEVELOPER, REPOSITORY or DAY
     @param github: whether the GitHub columns are reported (--github)
     @returns: a list of column names
     """
@@ -39,6 +42,10 @@ def fieldnames(mode, github):
         if github:
             columns += ["github_username"] + PR_COUNTS + EMAIL_FIELDNAMES
             columns += ACTIVITY_FIELDNAMES
+    elif mode == DAY:
+        columns = ["date"] + GIT_COUNTS + ["repositories", "contributors"]
+        if github:
+            columns += PR_COUNTS + ACTIVITY_FIELDNAMES
     else:
         columns = ["repository"] + dates + GIT_COUNTS + ["contributors"]
         if github:
@@ -126,7 +133,7 @@ def merge_emails(values):
 def combine(rows, columns):
     """
     Build the aggregated row of one group.
-    @param rows: the rows of one developer or one repository
+    @param rows: the rows of one developer, repository or day
     @param columns: the output columns, from fieldnames()
     @returns: one row with those columns
     """
@@ -135,12 +142,12 @@ def combine(rows, columns):
     for field in columns:
         if field == "username":
             out[field] = main_username(rows)
-        elif field in ("repository", "start_date", "end_date"):
+        elif field in ("repository", "start_date", "end_date", "date"):
             out[field] = rows[0][field]
         elif field == "repositories":
             out[field] = len({row["repository"] for row in active})
         elif field == "contributors":
-            out[field] = len(active)
+            out[field] = len({developer_key(row) for row in active})
         elif field == "github_username":
             out[field] = first_text(row[field] for row in rows)
         elif field in EMAIL_FIELDNAMES:
@@ -152,22 +159,33 @@ def combine(rows, columns):
 
 def aggregate(rows, mode, github):
     """
-    Combine result rows into one row per developer or one row per repository,
-    sorted by username (or GitHub username, for a developer with no git
-    username) or by repository, ignoring case.
-    @param rows: result rows of every repository, after -u and -c
-    @param mode: DEVELOPER or REPOSITORY
+    Combine result rows into one row per developer, repository or day. Rows
+    are sorted by username (or GitHub username, for a developer with no git
+    username) or by repository, ignoring case, or by date. Days only count
+    active rows, so a day without activity is left out (OUT-7).
+    @param rows: result rows of every repository, after -u and -c; in DAY
+        mode, rows for single days, each with a "date" key
+    @param mode: DEVELOPER, REPOSITORY or DAY
     @param github: whether the rows carry the GitHub columns (--github)
     @returns: the aggregated rows, sorted
     """
     groups = {}
     for row in rows:
-        key = developer_key(row) if mode == DEVELOPER else row["repository"]
+        if mode == DEVELOPER:
+            key = developer_key(row)
+        elif mode == REPOSITORY:
+            key = row["repository"]
+        elif is_active(row):
+            key = row["date"]
+        else:
+            continue
         groups.setdefault(key, []).append(row)
     columns = fieldnames(mode, github)
     combined = [combine(group, columns) for group in groups.values()]
 
     def sort_key(row):
+        if mode == DAY:
+            return (datetime.datetime.strptime(row["date"], "%m/%d/%Y"), "")
         if mode == DEVELOPER:
             name = row["username"] or row.get("github_username") or ""
         else:

{gitlogstats-1.6.0 → gitlogstats-1.7.0}/src/gitlogstats/git_logs_parser.py

@@ -17,6 +17,10 @@ RECORD_SEP = "\x1e"
 FIELD_SEP = "\x1f"
 LOG_FORMAT = f"--format={RECORD_SEP}%H{FIELD_SEP}%aN{FIELD_SEP}%aE"
 
+# OUT-7: the same header with the commit date, as a local mm/dd/yyyy day
+DAY_LOG_FORMAT = f"{LOG_FORMAT}{FIELD_SEP}%cd"
+DAY_DATE_ARG = "--date=format-local:%m/%d/%Y"
+
 FILES_RE = re.compile(r"(\d+) files? changed")
 INSERTIONS_RE = re.compile(r"(\d+) insertions?\(\+\)")
 DELETIONS_RE = re.compile(r"(\d+) deletions?\(-\)")
@@ -151,6 +155,7 @@ class GitLogsParser:
             "--no-merges",
             *(["--shortstat"] if shortstat else []),
             log_format,
+            *([DAY_DATE_ARG] if log_format == DAY_LOG_FORMAT else []),
             *self.date_range_args(),
             "--",
             ".",
@@ -173,7 +178,7 @@ class GitLogsParser:
                 pathspecs.append(f":(exclude,glob)**/{pattern}/**")
         return pathspecs
 
-    def merge_log_command(self):
+    def merge_log_command(self, log_format=LOG_FORMAT):
         """
         Build the git log command that lists the merge commits made directly on
         the analyzed branch in the date range, with their authors.
@@ -181,6 +186,8 @@ class GitLogsParser:
         branch to update it) are not on the branch's first-parent line, so they
         are left out. File exclusions do not apply: a merge brings in a branch,
         whichever files it touches.
+        @param log_format: the --format argument; DAY_LOG_FORMAT adds each
+            merge's day (OUT-7)
         @returns: the command as a list of arguments, ready for subprocess
         """
         return [
@@ -188,7 +195,8 @@ class GitLogsParser:
             "log",
             "--first-parent",
             "--merges",
-            LOG_FORMAT,
+            log_format,
+            *([DAY_DATE_ARG] if log_format == DAY_LOG_FORMAT else []),
             *self.date_range_args(),
         ]
 
@@ -245,7 +253,7 @@ class GitLogsParser:
         """
         Split raw git log output (produced by git_log_command) into per-commit records.
         @param logs: the text output of git log
-        @returns: a list of dictionaries, one per commit, with the author's name and email and the commit's files, insertions and deletions
+        @returns: a list of dictionaries, one per commit, with the author's name and email and the commit's files, insertions and deletions, and its day (OUT-7) when the log has one
         """
         commits = []
         for record in logs.split(RECORD_SEP)[1:]:
@@ -256,16 +264,17 @@ class GitLogsParser:
             files = FILES_RE.search(body)
             insertions = INSERTIONS_RE.search(body)
             deletions = DELETIONS_RE.search(body)
-            commits.append(
-                {
-                    "hash": fields[0],
-                    "name": fields[1],
-                    "email": fields[2],
-                    "files": int(files.group(1)) if files else 0,
-                    "insertions": int(insertions.group(1)) if insertions else 0,
-                    "deletions": int(deletions.group(1)) if deletions else 0,
-                }
-            )
+            commit = {
+                "hash": fields[0],
+                "name": fields[1],
+                "email": fields[2],
+                "files": int(files.group(1)) if files else 0,
+                "insertions": int(insertions.group(1)) if insertions else 0,
+                "deletions": int(deletions.group(1)) if deletions else 0,
+            }
+            if len(fields) > 3:
+                commit["day"] = fields[3].strip()
+            commits.append(commit)
         return commits
 
     def parse(self):
@@ -275,12 +284,59 @@ class GitLogsParser:
         """
         commits = self.run_git_log(self.git_log_command())
         merges = self.run_git_log(self.merge_log_command())
+        return self.tally(commits, merges)
 
-        # limit to the specified username, if any; otherwise all contributors
+    def parse_days(self, extra_days=()):
+        """
+        OUT-7: parse the git logs one day at a time.
+        @param extra_days: days (mm/dd/yyyy) to report even without git
+            activity, e.g. days with GitHub activity
+        @returns: {day: contribution stats of all users on that day, as parse()
+            returns them}, for every day with a commit or merge and every extra
+            day, in date order
+        """
+        commits = self.run_git_log(self.git_log_command(DAY_LOG_FORMAT))
+        merges = self.run_git_log(self.merge_log_command(DAY_LOG_FORMAT))
+        days = {day: ([], []) for day in extra_days}
+        for commit in commits:
+            days.setdefault(commit["day"], ([], []))[0].append(commit)
+        for merge in merges:
+            days.setdefault(merge["day"], ([], []))[1].append(merge)
+        contributors = self.contributors()  # once, not once per day
+        order = sorted(days, key=lambda d: datetime.datetime.strptime(d, "%m/%d/%Y"))
+        return {day: self.tally(*days[day], contributors) for day in order}
+
+    def day_rows(self):
+        """
+        OUT-7: the rows of parse_days as one list, each with its "date".
+        @returns: a list of row dictionaries
+        """
+        return [
+            dict(row, date=day)
+            for day, rows in self.parse_days().items()
+            for row in rows
+        ]
+
+    def contributors(self):
+        """
+        The contributors to report: the one given with -u, or everyone.
+        @returns: a list of usernames
+        """
         if self.username:
-            contributors = [self.username]
-        else:
-            contributors = self.get_contributors()
+            return [self.username]
+        return self.get_contributors()
+
+    def tally(self, commits, merges, contributors=None):
+        """
+        Credit commits and merges to each contributing user.
+        @param commits: non-merge commits, from parse_log
+        @param merges: merge commits, from parse_log
+        @param contributors: the usernames to report; defaults to contributors()
+        @returns: contribution stats of all users, as a list of dictionaries, one per user
+        """
+        # limit to the specified username, if any; otherwise all contributors
+        if contributors is None:
+            contributors = self.contributors()
 
         # set up stats for each contributor in dictionary form
         entries = {}

{gitlogstats-1.6.0 → gitlogstats-1.7.0}/src/gitlogstats/github.py

@@ -302,6 +302,51 @@ def _range_test(start, end):
     return in_range
 
 
+def local_day(moment):
+    """
+    OUT-7: the local day of a moment, as git reports a commit's day.
+    @param moment: a GitHub UTC timestamp string, or seconds since the epoch
+    @returns: the day as mm/dd/yyyy
+    """
+    if isinstance(moment, str):
+        moment = _local_time(moment)
+    else:
+        moment = datetime.datetime.fromtimestamp(moment)
+    return moment.strftime("%m/%d/%Y")
+
+
+def events_by_day(prs, pushes, start, end, branch=None):
+    """
+    OUT-7: group pull requests and pushes by the local days in a date range on
+    which something happened to them, so each day is counted from its own
+    events only.
+    @param prs: the list from fetch_pull_requests
+    @param pushes: the list from fetch_pushes
+    @param start: first day, mm/dd/yyyy
+    @param end: last day, mm/dd/yyyy
+    @param branch: keep only pull requests with this base branch, if given
+    @returns: {day (mm/dd/yyyy): {"prs": [...], "pushes": [...], "commits":
+        []}}; a pull request is listed on each day it was opened, merged or
+        closed, and "commits" is left for the caller to fill
+    """
+    in_range = _range_test(start, end)
+    days = {}
+
+    def bucket(day):
+        return days.setdefault(day, {"prs": [], "pushes": [], "commits": []})
+
+    for pr in prs:
+        if branch is not None and pr["base"] != branch:
+            continue
+        moments = (pr["created_at"], pr["merged_at"], pr["closed_at"])
+        for day in {local_day(m) for m in moments if in_range(m)}:
+            bucket(day)["prs"].append(pr)
+    for push in pushes:
+        if in_range(push["timestamp"]):
+            bucket(local_day(push["timestamp"]))["pushes"].append(push)
+    return days
+
+
 def count_pull_requests(prs, start, end, branch=None):
     """
     STAT-8, STAT-9, STAT-10, STAT-12: count each user's pull requests in a date range.

{gitlogstats-1.6.0 → gitlogstats-1.7.0}/src/gitlogstats/github_stats.py

@@ -19,9 +19,12 @@ from .github import (
     count_pull_requests,
     count_pushes,
     count_web_commits,
+    GITHUB_COMMITTER_EMAIL,
+    events_by_day,
     fetch_pull_requests,
     fetch_pushes,
     github_repo_from_url,
+    local_day,
     user_profile,
 )
 from .identity import consolidate, match_identities, unavailable_row
@@ -41,39 +44,29 @@ def warn(repo_url, reason):
     )
 
 
-def combine(parser, rows, identities, github_repo, args):
+def fetch(parser, identities, github_repo):
     """
-    STAT-8 to STAT-11, STAT-14, STAT-15: fetch the repository's pull requests
-    and pushes, match GitHub users to git authors and merge their rows.
+    STAT-8 to STAT-11, STAT-14, STAT-15: fetch the repository's pull requests,
+    pushes and website-commit candidates, and match GitHub users to git authors.
+    Nothing is counted yet, so the result serves any date range (OUT-7).
     @param parser: the GitLogsParser of the repository (for verbose output)
-    @param rows: git rows from parser.parse(), unfiltered
     @param identities: {author_name: {email: commit hash}} from get_identities
     @param github_repo: "owner/repo"
-    @param args: the parsed command-line arguments (start, end, branch)
-    @returns: (combined rows, {author_name: (login, tier)})
+    @returns: a dict with prs, pushes (to the checked-out branch), commits (committed_commits), matches
+        ({author_name: (login, tier)}) and github_emails ({login: email})
     @raises GitHubError: if GitHub cannot be read for this repository
     """
     api_calls = 3  # the pull request, contributor and activity lists
     prs = fetch_pull_requests(github_repo)
-    pr_counts = count_pull_requests(prs, args.start, args.end, args.branch)
 
     # STAT-14: pushes to the branch that was actually analyzed; a detached
     # HEAD is no branch, so nothing was pushed to it
     branch = parser.current_branch()
     if branch is None:
         api_calls -= 1
-        pushes = {}
+        pushes = []
     else:
-        pushes = count_pushes(fetch_pushes(github_repo, branch), args.start, args.end)
-    known = {login.casefold(): login for login in pr_counts}
-    for login, number in pushes.items():
-        key = known.setdefault(login.casefold(), login)
-        found = pr_counts.setdefault(key, {})
-        found["pushes_to_default"] = found.get("pushes_to_default", 0) + number
-
-    # STAT-15: commits GitHub created itself, apart from pull request merges
-    web_commits = count_web_commits(parser.committed_commits(), prs)
-    rows = [dict(row, web_commits=web_commits.get(row["username"], 0)) for row in rows]
+        pushes = fetch_pushes(github_repo, branch)
 
     logins = {pr["login"] for pr in prs} | contributor_logins(github_repo)
     candidates = {}
@@ -127,18 +120,71 @@ def combine(parser, rows, identities, github_repo, args):
             if "(HTTP 404)" not in str(exc):
                 raise
     parser.verboseprint(f"GitHub API calls: {api_calls}")
+    return {
+        "prs": prs,
+        "pushes": pushes,
+        "commits": parser.committed_commits(),
+        "matches": matches,
+        "github_emails": github_emails,
+    }
+
+
+def tally(parser, rows, data, identities, start, end, branch, events=None):
+    """
+    Count the fetched GitHub activity in a date range and merge it into the
+    git rows of the same range.
+    @param parser: the GitLogsParser of the repository
+    @param rows: git rows for the range, unfiltered
+    @param data: the result of fetch
+    @param identities: {author_name: {email: commit hash}} from get_identities
+    @param start: first day, mm/dd/yyyy
+    @param end: last day, mm/dd/yyyy
+    @param branch: count only pull requests with this base branch, if given
+    @param events: {"prs", "pushes", "commits"} limited to the range (OUT-7);
+        defaults to everything in data (the whole -s/-e range)
+    @returns: the combined rows
+    """
+    events = events or data
+    pr_counts = count_pull_requests(events["prs"], start, end, branch)
+    pushes = count_pushes(events["pushes"], start, end)
+    known = {login.casefold(): login for login in pr_counts}
+    for login, number in pushes.items():
+        key = known.setdefault(login.casefold(), login)
+        found = pr_counts.setdefault(key, {})
+        found["pushes_to_default"] = found.get("pushes_to_default", 0) + number
+
+    # STAT-15: commits GitHub created itself, apart from pull request merges
+    web_commits = count_web_commits(events["commits"], data["prs"])
+    rows = [dict(row, web_commits=web_commits.get(row["username"], 0)) for row in rows]
+
     repository = GitLogsParser.repo_name_from_url(parser.repository)
-    combined = consolidate(
+    return consolidate(
         rows,
-        matches,
+        data["matches"],
         pr_counts,
         repository,
-        args.start,
-        args.end,
+        start,
+        end,
         identities,
-        github_emails,
+        data["github_emails"],
     )
-    return combined, matches
+
+
+def combine(parser, rows, identities, github_repo, args):
+    """
+    STAT-8 to STAT-11, STAT-14, STAT-15: fetch the repository's pull requests
+    and pushes, match GitHub users to git authors and merge their rows.
+    @param parser: the GitLogsParser of the repository (for verbose output)
+    @param rows: git rows from parser.parse(), unfiltered
+    @param identities: {author_name: {email: commit hash}} from get_identities
+    @param github_repo: "owner/repo"
+    @param args: the parsed command-line arguments (start, end, branch)
+    @returns: (combined rows, {author_name: (login, tier)})
+    @raises GitHubError: if GitHub cannot be read for this repository
+    """
+    data = fetch(parser, identities, github_repo)
+    combined = tally(parser, rows, data, identities, args.start, args.end, args.branch)
+    return combined, data["matches"]
 
 
 def member_names(row, matches):
@@ -190,12 +236,79 @@ def github_rows(parser, rows, repo_url, identities, args):
         result, matches = combine(parser, rows, identities, github_repo, args)
     except GitHubError as exc:
         warn(repo_url, exc)
-        result = [
-            unavailable_row(row, identities.get(row["username"], ())) for row in rows
-        ]
+        result = unavailable_rows(rows, identities)
+
+    return select(result, args, matches, identities)
+
+
+def unavailable_rows(rows, identities):
+    """
+    Mark the GitHub columns of git rows unavailable (CLI-3), keeping any
+    "date" (OUT-7).
+    @param rows: git rows
+    @param identities: {author_name: {email: commit hash}} from get_identities
+    @returns: new rows from unavailable_row
+    """
+    result = []
+    for row in rows:
+        out = unavailable_row(row, identities.get(row["username"], ()))
+        if "date" in row:
+            out["date"] = row["date"]
+        result.append(out)
+    return result
+
+
+def github_day_rows(parser, repo_url, identities, args):
+    """
+    OUT-7: like github_rows, one day at a time. GitHub is read once; each
+    event is counted on its own local day.
+    @param parser: the GitLogsParser of the repository
+    @param repo_url: the repository as given on the command line
+    @param identities: {author_name: {email: commit hash}} from get_identities
+    @param args: the parsed command-line arguments
+    @returns: the final rows for this repository, each with a "date" key
+    """
+    matches = {}
+    github_repo = github_repo_from_url(repo_url)
+    result = []
+    try:
+        if github_repo is None:
+            raise GitHubError("not a github.com repository")
+        data = fetch(parser, identities, github_repo)
+    except GitHubError as exc:
+        warn(repo_url, exc)
+        result = unavailable_rows(parser.day_rows(), identities)
+    else:
+        matches = data["matches"]
+        days = events_by_day(
+            data["prs"], data["pushes"], args.start, args.end, args.branch
+        )
+        # STAT-15: only commits GitHub committed can be website commits
+        none = {"prs": [], "pushes": [], "commits": []}
+        for commit in data["commits"]:
+            if commit["committer_email"].strip().casefold() == GITHUB_COMMITTER_EMAIL:
+                day = local_day(commit["committed"])
+                days.setdefault(day, dict(none, commits=[]))["commits"].append(commit)
+        for day, rows in parser.parse_days(days).items():
+            events = days.get(day, none)
+            for row in tally(
+                parser, rows, data, identities, day, day, args.branch, events
+            ):
+                result.append(dict(row, date=day))
+    return select(result, args, matches, identities)
 
+
+def select(rows, args, matches, identities):
+    """
+    Apply -u and -c to combined rows (CLI-3).
+    @param rows: rows from tally or unavailable_row
+    @param args: the parsed command-line arguments (user, clean)
+    @param matches: {author_name: (login, tier)}
+    @param identities: {author_name: {email: commit hash}} from get_identities
+    @returns: the rows that -u and -c keep
+    """
     if args.user:
-        result = [r for r in result if keep_user(r, args.user, matches, identities)]
+        rows = [r for r in rows if keep_user(r, args.user, matches, identities)]
     if args.clean:
-        result = [r for r in result if any(r[f] for f in ACTIVITY_FIELDS)]
-    return result
+        rows = [r for r in rows if any(r[f] for f in ACTIVITY_FIELDS)]
+    return rows

{gitlogstats-1.6.0 → gitlogstats-1.7.0}/tests/test_aggregate.py

@@ -6,6 +6,7 @@ All names, emails and logins here are invented.
 import pytest
 
 from gitlogstats.aggregate import (
+    DAY,
     DEVELOPER,
     REPOSITORY,
     aggregate,
@@ -363,6 +364,92 @@ class TestByRepository:
         assert all(two[f] is None for f in PR_COUNTS + ACTIVITY_FIELDNAMES)
 
 
-@pytest.mark.parametrize("mode", [DEVELOPER, REPOSITORY])
+def day_row(row, date):
+    """A row for a single day (OUT-7)."""
+    return dict(row, date=date)
+
+
+class TestByDay:
+    def test_fieldnames(self):
+        assert fieldnames(DAY, False) == [
+            "date",
+            *GIT_COUNTS,
+            "repositories",
+            "contributors",
+        ]
+        assert fieldnames(DAY, True) == fieldnames(DAY, False) + [
+            *PR_COUNTS,
+            *ACTIVITY_FIELDNAMES,
+        ]
+
+    def test_sums_each_day_in_date_order(self):
+        rows = [
+            day_row(git_row("Ada", "one", commits=2, insertions=3), "01/02/2025"),
+            day_row(git_row("Ada", "one", commits=1, files=1), "12/31/2024"),
+            day_row(git_row("Bob", "two", commits=1, deletions=4), "12/31/2024"),
+            day_row(git_row("Ada", "two", commits=1, insertions=5), "12/31/2024"),
+        ]
+        out = aggregate(rows, DAY, False)
+        assert out == [
+            {
+                "date": "12/31/2024",
+                "commits": 3,
+                "insertions": 5,
+                "deletions": 4,
+                "files": 1,
+                "merges": 0,
+                "repositories": 2,
+                "contributors": 2,  # Ada counts once over two repositories
+            },
+            {
+                "date": "01/02/2025",
+                "commits": 2,
+                "insertions": 3,
+                "deletions": 0,
+                "files": 0,
+                "merges": 0,
+                "repositories": 1,
+                "contributors": 1,
+            },
+        ]
+
+    def test_inactive_rows_and_days_are_left_out(self):
+        rows = [
+            day_row(git_row("Ada", "one"), "03/01/2024"),
+            day_row(git_row("Bob", "two", commits=0), "03/01/2024"),
+            day_row(git_row("Bob", "two", commits=0), "03/02/2024"),
+        ]
+        (row,) = aggregate(rows, DAY, False)
+        assert (row["date"], row["repositories"], row["contributors"]) == (
+            "03/01/2024",
+            1,
+            1,
+        )
+
+    def test_github_counts_and_one_person_under_two_logins_cases(self):
+        rows = [
+            day_row(github_row("ada", "one", login="AdaL", prs_opened=1), "03/01/2024"),
+            day_row(
+                github_row("", "two", login="adal", commits=0, pushes_to_default=2),
+                "03/01/2024",
+            ),
+            day_row(
+                github_row("", "two", login="mia", commits=0, prs_merged=1),
+                "03/01/2024",
+            ),
+            day_row(unavailable("bob", "three"), "03/01/2024"),
+            day_row(unavailable("bob", "three"), "03/02/2024"),
+        ]
+        first, second = aggregate(rows, DAY, True)
+        assert list(first) == fieldnames(DAY, True)
+        assert (first["repositories"], first["contributors"]) == (3, 3)
+        assert (first["prs_opened"], first["prs_merged"]) == (1, 1)
+        assert (first["pushes_to_default"], first["commits"]) == (2, 2)
+        # only a repository GitHub could not be asked about was active that day
+        assert second["commits"] == 1
+        assert all(second[f] is None for f in PR_COUNTS + ACTIVITY_FIELDNAMES)
+
+
+@pytest.mark.parametrize("mode", [DEVELOPER, REPOSITORY, DAY])
 def test_no_rows(mode):
     assert aggregate([], mode, True) == []

{gitlogstats-1.6.0 → gitlogstats-1.7.0}/tests/test_github.py

@@ -19,9 +19,11 @@ from gitlogstats.github import (
     commit_login,
     contributor_logins,
     count_pull_requests,
+    events_by_day,
     fetch_pull_requests,
     gh_api,
     github_repo_from_url,
+    local_day,
     user_profile,
 )
 
@@ -800,3 +802,39 @@ def test_count_web_commits_skips_other_committers_and_merge_results():
     ]
     # x1 and x2 are 3 and 4 seconds away; "m" is 1 second before the merge
     assert github.count_web_commits(commits, prs) == {"A": 3}
+
+
+# ─── days (OUT-7) ────────────────────────────────────────────────────────────
+
+
+def test_local_day_of_timestamp_and_epoch_seconds():
+    late = datetime.datetime(2024, 3, 10, 23, 59, 59)
+    early = datetime.datetime(2024, 3, 11, 0, 0, 0)
+    assert local_day(utc_z(late)) == "03/10/2024"
+    assert local_day(utc_z(early)) == "03/11/2024"
+    assert local_day(time.mktime(late.timetuple())) == "03/10/2024"
+    assert local_day(int(time.mktime(early.timetuple()))) == "03/11/2024"
+
+
+def test_events_by_day_groups_pull_requests_and_pushes_by_local_day():
+    day = lambda d, h=12: utc_z(datetime.datetime(2024, 3, d, h))
+    merged = make_pr(created=day(1), merged=day(2), closed=day(2))
+    closed = make_pr(created=day(3), closed=day(4))
+    same_day = make_pr(created=day(5, 9), merged=day(5, 17), closed=day(5, 17))
+    other_base = make_pr(created=day(7), base="dev")
+    too_early = make_pr(created=utc_z(datetime.datetime(2023, 12, 31, 23, 59, 59)))
+    early = {"login": "u", "timestamp": day(6)}
+    late = {"login": "u", "timestamp": day(31)}
+    prs = [merged, closed, same_day, other_base, too_early]
+    days = events_by_day(prs, [early, late], "01/01/2024", "03/30/2024", "main")
+    assert {d: (e["prs"], e["pushes"], e["commits"]) for d, e in days.items()} == {
+        "03/01/2024": ([merged], [], []),
+        "03/02/2024": ([merged], [], []),
+        "03/03/2024": ([closed], [], []),
+        "03/04/2024": ([closed], [], []),
+        "03/05/2024": ([same_day], [], []),  # listed once for its two events
+        "03/06/2024": ([], [early], []),
+    }
+    # without a branch, every base counts
+    assert "03/07/2024" in events_by_day(prs, [], "01/01/2024", "03/30/2024")
+    assert events_by_day([], [], "01/01/2024", "03/30/2024") == {}

{gitlogstats-1.6.0 → gitlogstats-1.7.0}/tests/test_integration.py

@@ -2486,3 +2486,188 @@ class TestAggregate:
         assert not {"username", "github_username", "git_email", "github_email"} & set(
             rows[0]
         )
+
+
+# ─── Results by day (OUT-7) ──────────────────────────────────────────────────
+
+DAY_HEADER = "date,commits,insertions,deletions,files,merges,repositories,contributors"
+
+
+class TestAggregateByDay:
+    @pytest.fixture
+    def repofile(self, tmp_path):
+        """
+        Two repositories. Alice works just before and after midnight on
+        03/10-03/11, in both repositories on 03/11; Bob merges on 03/12.
+        """
+        one = Repo(tmp_path / "remotes" / "one")
+        one.write("a.txt", lines(3))
+        one.commit("Alice", "late", date="2024-03-10 23:59:59")
+        one.write("b.txt", lines(1))
+        one.commit("Bob", "early", date="2024-03-11 00:00:00")
+        one.branch("feature")
+        one.write("c.txt", lines(2))
+        one.commit("Alice", "feature", date="2024-03-11 09:00:00")
+        one.switch("main")
+        one.merge("Bob", "feature", date="2024-03-12 10:00:00")
+        two = Repo(tmp_path / "remotes" / "two")
+        two.write("old.txt", lines(9))
+        two.commit("Carol", "old work", date="2020-01-01 12:00:00")
+        two.write("d.txt", lines(4))
+        two.commit("Alice", "two", date="2024-03-11 15:00:00")
+        path = tmp_path / "repos.txt"
+        path.write_text(f"{one.path}\n{two.path}\n")
+        return str(path)
+
+    def run(self, monkeypatch, capsys, tmp_path, repofile, *argv, dates=DATES):
+        out, _ = run_github(
+            monkeypatch, capsys, tmp_path, "-rf", repofile, *dates, *argv
+        )
+        return out
+
+    def test_one_row_per_active_day(self, repofile, tmp_path, monkeypatch, capsys):
+        out = self.run(monkeypatch, capsys, tmp_path, repofile, "-a", "day")
+        assert out.splitlines() == [
+            DAY_HEADER,
+            "03/10/2024,1,3,0,1,0,1,1",
+            # Bob and Alice in one, Alice again in two: she counts once
+            "03/11/2024,3,7,0,3,0,2,2",
+            "03/12/2024,0,0,0,0,1,1,1",
+        ]
+
+    def test_days_add_up_to_the_totals(self, repofile, tmp_path, monkeypatch, capsys):
+        days = json.loads(
+            self.run(monkeypatch, capsys, tmp_path, repofile, "-a", "day", "-f", "json")
+        )
+        plain = json.loads(
+            self.run(monkeypatch, capsys, tmp_path, repofile, "-f", "json")
+        )
+        for column in COUNT_COLUMNS:
+            assert sum(d[column] for d in days) == sum(r[column] for r in plain)
+
+    def test_user_and_range(self, repofile, tmp_path, monkeypatch, capsys):
+        out = self.run(
+            monkeypatch, capsys, tmp_path, repofile, "-a", "day", "-u", "Alice"
+        )
+        assert out.splitlines()[1:] == [
+            "03/10/2024,1,3,0,1,0,1,1",
+            "03/11/2024,2,6,0,2,0,2,1",
+        ]
+        out = self.run(
+            monkeypatch, capsys, tmp_path, repofile, "-a", "day",
+            dates=["-s", "03/11/2024", "-e", "03/11/2024"],
+        )  # fmt: skip
+        assert out.splitlines()[1:] == ["03/11/2024,3,7,0,3,0,2,2"]
+
+    def test_clean_changes_nothing(self, repofile, tmp_path, monkeypatch, capsys):
+        argv = ["-a", "day", "-f", "markdown"]
+        out = self.run(monkeypatch, capsys, tmp_path, repofile, *argv)
+        assert self.run(monkeypatch, capsys, tmp_path, repofile, *argv, "-c") == out
+        assert out.splitlines()[0] == "| " + " | ".join(DAY_HEADER.split(",")) + " |"
+
+    def test_no_activity_prints_only_the_header(
+        self, repofile, tmp_path, monkeypatch, capsys
+    ):
+        argv = ["-a", "day"]
+        dates = ["-s", "01/01/2023", "-e", "12/31/2023"]
+        out = self.run(monkeypatch, capsys, tmp_path, repofile, *argv, dates=dates)
+        assert out == DAY_HEADER + "\n"
+        out = self.run(
+            monkeypatch, capsys, tmp_path, repofile, *argv, "-f", "json", dates=dates
+        )
+        assert json.loads(out) == []
+
+    def test_github_events_are_counted_on_their_own_days(
+        self, tmp_path, monkeypatch, capsys
+    ):
+        widgets = Repo(tmp_path / "remotes" / "widgets")
+        widgets.write("a.txt", lines(2))
+        widgets.commit(
+            "ada", "work", date="2024-03-10 12:00:00", email="ada@laptop.example"
+        )
+        first = head(widgets)
+        widgets.write("w.txt", lines(1))
+        widgets.commit(
+            "ada", "Update w.txt", date="2024-03-12 12:00:00",
+            email="ada@laptop.example", committer=("GitHub", "noreply@github.com"),
+        )  # fmt: skip
+        web = head(widgets)
+        local = Repo(tmp_path / "remotes" / "local")
+        local.write("b.txt", lines(5))
+        local.commit("Bob", "work", date="2024-03-11 08:00:00", email="bob@example.com")
+        github_world(
+            tmp_path,
+            monkeypatch,
+            widgets,
+            prs=[
+                # opened on a day without commits, merged by its author later
+                pr(
+                    "adal99", at(2024, 3, 9), merged=at(2024, 3, 11), merged_by="adal99"
+                ),
+                pr("mia", at(2024, 3, 11)),
+                pr("mia", at(2023, 12, 31)),  # before the range
+            ],
+            users={"adal99": {"name": "Ada", "email": None}},
+            contributors=["adal99"],
+            linked={first: "adal99", web: "adal99"},
+            pushes=[push("adal99", at(2024, 3, 13))],
+        )
+        repofile = tmp_path / "repos.txt"
+        repofile.write_text(f"{GITHUB_URL}\n{local.path}\n")
+        argv = ["-rf", str(repofile), "--github", *DATES, "-f", "json"]
+        out, _ = run_github(monkeypatch, capsys, tmp_path, *argv, "-a", "day")
+        days = {d["date"]: d for d in json.loads(out)}
+        assert list(days) == [
+            "03/09/2024", "03/10/2024", "03/11/2024", "03/12/2024", "03/13/2024",
+        ]  # fmt: skip
+        assert set(days["03/09/2024"]) == set(
+            DAY_HEADER.split(",") + PR_COLUMNS + ACTIVITY_COLUMNS
+        )
+        summary = {
+            day: (
+                d["commits"], d["repositories"], d["contributors"],
+                *[d[c] for c in PR_COLUMNS], *[d[c] for c in ACTIVITY_COLUMNS],
+            )
+            for day, d in days.items()
+        }  # fmt: skip
+        assert summary == {
+            "03/09/2024": (0, 1, 1, 1, 0, 0, 0, 0, 0),
+            "03/10/2024": (1, 1, 1, 0, 0, 0, 0, 0, 0),
+            # ada's self-merge, mia's new pull request and Bob's local commit
+            "03/11/2024": (1, 2, 3, 1, 1, 0, 1, 0, 0),
+            "03/12/2024": (1, 1, 1, 0, 0, 0, 0, 0, 1),
+            "03/13/2024": (0, 1, 1, 0, 0, 0, 0, 1, 0),
+        }
+        # the days add up to the totals printed without -a
+        plain = json.loads(run_github(monkeypatch, capsys, tmp_path, *argv)[0])
+        for column in COUNT_COLUMNS + PR_COLUMNS + ACTIVITY_COLUMNS:
+            assert sum(d[column] for d in days.values()) == sum(
+                r[column] or 0 for r in plain
+            ), column
+
+        # -u selects one developer's activity on each day, by GitHub username
+        out, _ = run_github(
+            monkeypatch, capsys, tmp_path, *argv, "-a", "day", "-u", "adal99"
+        )
+        assert [(d["date"], d["contributors"]) for d in json.loads(out)] == [
+            ("03/09/2024", 1),
+            ("03/10/2024", 1),
+            ("03/11/2024", 1),
+            ("03/12/2024", 1),
+            ("03/13/2024", 1),
+        ]
+
+    def test_unavailable_repository_has_empty_github_counts(
+        self, tmp_path, monkeypatch, capsys
+    ):
+        local = Repo(tmp_path / "remotes" / "local")
+        local.write("b.txt", lines(5))
+        local.commit("Bob", "work", date="2024-03-11 08:00:00", email="bob@example.com")
+        github_world(tmp_path, monkeypatch, local)
+        argv = ["-r", local.path, "--github", *DATES, "-a", "day"]
+        out, err = run_github(monkeypatch, capsys, tmp_path, *argv)
+        assert "not a github.com repository" in err
+        assert out.splitlines() == [
+            DAY_HEADER + "," + ",".join(PR_COLUMNS + ACTIVITY_COLUMNS),
+            "03/11/2024,1,5,0,1,0,1,1,,,,,,",
+        ]