github-backup 0.64.2__tar.gz → 0.65.0__tar.gz

{github_backup-0.64.2 → github_backup-0.65.0}/CHANGES.rst

@@ -1,9 +1,80 @@
 Changelog
 =========
 
-0.64.2 (2026-07-18)
+0.65.0 (2026-07-27)
 -------------------
 ------------------------
+- Refresh issue timelines when cross-references change (#168) [Rodos]
+
+  Cross-references do not change an issue's updated_at, so an incremental
+  run never lists the issue and never re-fetches its timeline. An issue
+  can be referenced from another repository years after its last activity
+  and the backup would never see it.
+
+  On incremental runs with --issue-timeline, ask GraphQL for each item's
+  cross-reference count and newest timestamp, compare both against what is
+  already saved, and re-fetch the timeline of anything that differs. Items
+  the since listing did not return are fetched individually. Both values
+  are needed: the count alone would miss a reference being added and
+  another deleted between runs, and the timestamp alone would miss a
+  deletion.
+
+  Pull requests are swept too when they are being stored as issues, which
+  is the case unless --pulls is used. They need their own query because
+  the issues connection excludes them, and because totalCount ignores the
+  itemTypes filter on a pull request although it honours it on an issue,
+  so their cross-references are counted from the nodes instead. Where that
+  count may have been truncated, the comparison falls back to the
+  timestamp alone rather than re-fetching on every run.
+
+  This costs one rate-limit point per 100 items against the GraphQL quota,
+  measured at 6 requests for a repository with 528 issues and pull
+  requests, and stores no new state: the comparison is against the
+  cross-references already in each item's timeline_data. A GraphQL failure
+  warns and leaves the rest of the backup unaffected.
+
+  Closes #168
+- Add --issue-timeline flag to capture cross-reference events (#168)
+  [Rodos]
+
+  The issue events endpoint never returns cross-references, so an issue
+  mentioned from another issue or pull request leaves no trace in the
+  backup. GitHub only exposes those through the timeline endpoint. Full
+  backups miss them as well as incremental ones.
+
+  Adds --issue-timeline, which stores the timeline under a separate
+  timeline_data key alongside the unmodified event_data. The two
+  endpoints are complementary rather than nested: the timeline adds
+  cross-references, commits and reviews, while events carries
+  referenced entries the timeline drops, so neither replaces the other.
+
+  Timeline entries for comments are filtered out, since --issue-comments
+  already saves the bodies. Everything else is stored verbatim. The flag
+  is included in --all.
+
+  This makes the data obtainable, but does not change incremental
+  selection: a cross-reference does not bump the referenced issue's
+  updated_at, so an incremental run still never lists it. That half of
+  the issue remains open.
+
+  References #168
+- Chore(deps): bump actions/setup-python from 6 to 7. [dependabot[bot]]
+
+  Bumps [actions/setup-python](https://github.com/actions/setup-python) from 6 to 7.
+  - [Release notes](https://github.com/actions/setup-python/releases)
+  - [Commits](https://github.com/actions/setup-python/compare/v6...v7)
+
+  ---
+  updated-dependencies:
+  - dependency-name: actions/setup-python
+    dependency-version: '7'
+    dependency-type: direct:production
+    update-type: version-update:semver-major
+  ...
+
+
+0.64.2 (2026-07-18)
+-------------------
 - Add tests. [Duncan Ogilvie]
 - Skip fetching gists that have not been updated. [Duncan Ogilvie]
 - Skip retrieving repositories when not necessary. [Duncan Ogilvie]

{github_backup-0.64.2/github_backup.egg-info → github_backup-0.65.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: github-backup
-Version: 0.64.2
+Version: 0.65.0
 Summary: backup a github user or organization
 Home-page: http://github.com/josegonzalez/python-github-backup
 Author: Jose Diaz-Gonzalez
@@ -71,7 +71,8 @@ CLI Help output::
                   [--incremental-by-files]
                   [--starred] [--all-starred] [--starred-skip-size-over MB]
                   [--watched] [--followers] [--following] [--all]
-                  [--issues] [--issue-comments] [--issue-events] [--pulls]
+                  [--issues] [--issue-comments] [--issue-events]
+                  [--issue-timeline] [--pulls]
                   [--pull-comments] [--pull-reviews] [--pull-commits]
                   [--pull-details]
                   [--labels] [--hooks] [--milestones] [--security-advisories]
@@ -126,6 +127,8 @@ CLI Help output::
       --issues              include issues in backup
       --issue-comments      include issue comments in backup
       --issue-events        include issue events in backup
+      --issue-timeline      include issue timeline in backup (cross-references,
+                            commits and reviews; complements --issue-events)
       --pulls               include pull requests in backup
       --pull-comments       include pull request review comments in backup
       --pull-reviews        include pull request reviews in backup
@@ -377,6 +380,27 @@ For finer control, avoid using ``--assets`` with starred repos, or use ``--skip-
 
 Alternatively, consider just storing links to starred repos in JSON format with ``--starred``.
 
+
+About issue events and the issue timeline
+-----------------------------------------
+
+Use ``--issue-events`` and ``--issue-timeline`` with ``--issues``; on their own they back up nothing.
+
+``--issue-events`` backs up GitHub's issue events endpoint under each issue's ``event_data`` key. That endpoint returns a restricted set of event types and **never** includes cross-references, so an issue that was mentioned from another issue or pull request shows no sign of it. GitHub only exposes those through the timeline.
+
+``--issue-timeline`` backs up the timeline endpoint under a separate ``timeline_data`` key. The two flags are independent: ``event_data`` keeps its existing contents and shape whether or not the timeline is enabled, so nothing that already reads it is affected.
+
+Use both flags together for a complete picture. Neither endpoint is a superset of the other, so they are complementary rather than alternatives. Only the timeline reports ``cross-referenced``, ``committed`` and ``reviewed`` entries. Only the events endpoint reliably reports ``referenced`` entries, where a commit elsewhere in the repository mentions the issue; the timeline omits many of these, substituting the issue's own ``committed`` entries instead.
+
+Note that timeline entries are not all the same shape as events. ``cross-referenced`` and ``committed`` entries have no ``id`` field, so anything consuming ``timeline_data`` should not assume one is present. ``commented`` entries are excluded from the backup because they embed the full comment body, which ``--issue-comments`` already saves.
+
+``--issue-timeline`` is included in ``--all``. On issues with many cross-references it can be noticeably larger and slower than ``--issue-events`` alone, because each ``cross-referenced`` entry embeds the referencing issue in full. On a heavily cross-referenced issue this can mean several times the API requests and many times the JSON on disk. Ordinary repositories see little difference.
+
+Incremental backups need extra help here, because being referenced from elsewhere does not change an issue's ``updated_at``. An issue can be mentioned from another repository years after its last activity, and the usual ``since`` listing will never report it as changed. On an incremental run with ``--issue-timeline``, github-backup therefore asks GitHub's GraphQL API for each item's cross-reference count and newest timestamp, compares those against what is already saved, and re-fetches the timeline of anything that differs. Pull requests are included in that check when they are being stored as issues, which is the case unless ``--pulls`` is used. The check needs no extra state on disk and also notices references that were later deleted. It costs one rate-limit point per 100 items checked, against GitHub's GraphQL quota, which is separate from the REST one. That is cheap in quota terms even on very large repositories, but it is not instant: a repository with 18,000 issues and pull requests took around 200 points and five minutes for the check alone.
+
+If you enable ``--issue-timeline`` on an existing incremental backup, cross-references on older issues are picked up automatically by that check. Timeline entries of other kinds, such as commits and reviews, are only fetched when the issue itself is updated, so run once without ``--incremental`` if you want those filled in for historical issues too.
+
+
 About pull request reviews
 --------------------------
 
@@ -480,14 +504,14 @@ Quietly and incrementally backup useful Github user data (public and private rep
     FINE_ACCESS_TOKEN=SOME-GITHUB-TOKEN
     GH_USER=YOUR-GITHUB-USER
 
-    github-backup -f $FINE_ACCESS_TOKEN --prefer-ssh -o ~/github-backup/ -l error -P -i --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --pulls --pull-comments --pull-reviews --pull-commits --labels --milestones --security-advisories --discussions --repositories --wikis --releases --assets --attachments --pull-details --gists --starred-gists $GH_USER
+    github-backup -f $FINE_ACCESS_TOKEN --prefer-ssh -o ~/github-backup/ -l error -P -i --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --issue-timeline --pulls --pull-comments --pull-reviews --pull-commits --labels --milestones --security-advisories --discussions --repositories --wikis --releases --assets --attachments --pull-details --gists --starred-gists $GH_USER
 
 Debug an error/block or incomplete backup into a temporary directory. Omit "incremental" to fill a previous incomplete backup. ::
 
     FINE_ACCESS_TOKEN=SOME-GITHUB-TOKEN
     GH_USER=YOUR-GITHUB-USER
 
-    github-backup -f $FINE_ACCESS_TOKEN -o /tmp/github-backup/ -l debug -P --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --pulls --pull-comments --pull-reviews --pull-commits --labels --milestones --discussions --repositories --wikis --releases --assets --pull-details --gists --starred-gists $GH_USER
+    github-backup -f $FINE_ACCESS_TOKEN -o /tmp/github-backup/ -l debug -P --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --issue-timeline --pulls --pull-comments --pull-reviews --pull-commits --labels --milestones --discussions --repositories --wikis --releases --assets --pull-details --gists --starred-gists $GH_USER
 
 Pipe a token from stdin to avoid storing it in environment variables or command history (Unix-like systems only)::
 

{github_backup-0.64.2 → github_backup-0.65.0}/README

@@ -41,7 +41,8 @@ CLI Help output::
                   [--incremental-by-files]
                   [--starred] [--all-starred] [--starred-skip-size-over MB]
                   [--watched] [--followers] [--following] [--all]
-                  [--issues] [--issue-comments] [--issue-events] [--pulls]
+                  [--issues] [--issue-comments] [--issue-events]
+                  [--issue-timeline] [--pulls]
                   [--pull-comments] [--pull-reviews] [--pull-commits]
                   [--pull-details]
                   [--labels] [--hooks] [--milestones] [--security-advisories]
@@ -96,6 +97,8 @@ CLI Help output::
       --issues              include issues in backup
       --issue-comments      include issue comments in backup
       --issue-events        include issue events in backup
+      --issue-timeline      include issue timeline in backup (cross-references,
+                            commits and reviews; complements --issue-events)
       --pulls               include pull requests in backup
       --pull-comments       include pull request review comments in backup
       --pull-reviews        include pull request reviews in backup
@@ -347,6 +350,27 @@ For finer control, avoid using ``--assets`` with starred repos, or use ``--skip-
 
 Alternatively, consider just storing links to starred repos in JSON format with ``--starred``.
 
+
+About issue events and the issue timeline
+-----------------------------------------
+
+Use ``--issue-events`` and ``--issue-timeline`` with ``--issues``; on their own they back up nothing.
+
+``--issue-events`` backs up GitHub's issue events endpoint under each issue's ``event_data`` key. That endpoint returns a restricted set of event types and **never** includes cross-references, so an issue that was mentioned from another issue or pull request shows no sign of it. GitHub only exposes those through the timeline.
+
+``--issue-timeline`` backs up the timeline endpoint under a separate ``timeline_data`` key. The two flags are independent: ``event_data`` keeps its existing contents and shape whether or not the timeline is enabled, so nothing that already reads it is affected.
+
+Use both flags together for a complete picture. Neither endpoint is a superset of the other, so they are complementary rather than alternatives. Only the timeline reports ``cross-referenced``, ``committed`` and ``reviewed`` entries. Only the events endpoint reliably reports ``referenced`` entries, where a commit elsewhere in the repository mentions the issue; the timeline omits many of these, substituting the issue's own ``committed`` entries instead.
+
+Note that timeline entries are not all the same shape as events. ``cross-referenced`` and ``committed`` entries have no ``id`` field, so anything consuming ``timeline_data`` should not assume one is present. ``commented`` entries are excluded from the backup because they embed the full comment body, which ``--issue-comments`` already saves.
+
+``--issue-timeline`` is included in ``--all``. On issues with many cross-references it can be noticeably larger and slower than ``--issue-events`` alone, because each ``cross-referenced`` entry embeds the referencing issue in full. On a heavily cross-referenced issue this can mean several times the API requests and many times the JSON on disk. Ordinary repositories see little difference.
+
+Incremental backups need extra help here, because being referenced from elsewhere does not change an issue's ``updated_at``. An issue can be mentioned from another repository years after its last activity, and the usual ``since`` listing will never report it as changed. On an incremental run with ``--issue-timeline``, github-backup therefore asks GitHub's GraphQL API for each item's cross-reference count and newest timestamp, compares those against what is already saved, and re-fetches the timeline of anything that differs. Pull requests are included in that check when they are being stored as issues, which is the case unless ``--pulls`` is used. The check needs no extra state on disk and also notices references that were later deleted. It costs one rate-limit point per 100 items checked, against GitHub's GraphQL quota, which is separate from the REST one. That is cheap in quota terms even on very large repositories, but it is not instant: a repository with 18,000 issues and pull requests took around 200 points and five minutes for the check alone.
+
+If you enable ``--issue-timeline`` on an existing incremental backup, cross-references on older issues are picked up automatically by that check. Timeline entries of other kinds, such as commits and reviews, are only fetched when the issue itself is updated, so run once without ``--incremental`` if you want those filled in for historical issues too.
+
+
 About pull request reviews
 --------------------------
 
@@ -450,14 +474,14 @@ Quietly and incrementally backup useful Github user data (public and private rep
     FINE_ACCESS_TOKEN=SOME-GITHUB-TOKEN
     GH_USER=YOUR-GITHUB-USER
 
-    github-backup -f $FINE_ACCESS_TOKEN --prefer-ssh -o ~/github-backup/ -l error -P -i --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --pulls --pull-comments --pull-reviews --pull-commits --labels --milestones --security-advisories --discussions --repositories --wikis --releases --assets --attachments --pull-details --gists --starred-gists $GH_USER
+    github-backup -f $FINE_ACCESS_TOKEN --prefer-ssh -o ~/github-backup/ -l error -P -i --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --issue-timeline --pulls --pull-comments --pull-reviews --pull-commits --labels --milestones --security-advisories --discussions --repositories --wikis --releases --assets --attachments --pull-details --gists --starred-gists $GH_USER
 
 Debug an error/block or incomplete backup into a temporary directory. Omit "incremental" to fill a previous incomplete backup. ::
 
     FINE_ACCESS_TOKEN=SOME-GITHUB-TOKEN
     GH_USER=YOUR-GITHUB-USER
 
-    github-backup -f $FINE_ACCESS_TOKEN -o /tmp/github-backup/ -l debug -P --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --pulls --pull-comments --pull-reviews --pull-commits --labels --milestones --discussions --repositories --wikis --releases --assets --pull-details --gists --starred-gists $GH_USER
+    github-backup -f $FINE_ACCESS_TOKEN -o /tmp/github-backup/ -l debug -P --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --issue-timeline --pulls --pull-comments --pull-reviews --pull-commits --labels --milestones --discussions --repositories --wikis --releases --assets --pull-details --gists --starred-gists $GH_USER
 
 Pipe a token from stdin to avoid storing it in environment variables or command history (Unix-like systems only)::
 

{github_backup-0.64.2 → github_backup-0.65.0}/README.rst

@@ -41,7 +41,8 @@ CLI Help output::
                   [--incremental-by-files]
                   [--starred] [--all-starred] [--starred-skip-size-over MB]
                   [--watched] [--followers] [--following] [--all]
-                  [--issues] [--issue-comments] [--issue-events] [--pulls]
+                  [--issues] [--issue-comments] [--issue-events]
+                  [--issue-timeline] [--pulls]
                   [--pull-comments] [--pull-reviews] [--pull-commits]
                   [--pull-details]
                   [--labels] [--hooks] [--milestones] [--security-advisories]
@@ -96,6 +97,8 @@ CLI Help output::
       --issues              include issues in backup
       --issue-comments      include issue comments in backup
       --issue-events        include issue events in backup
+      --issue-timeline      include issue timeline in backup (cross-references,
+                            commits and reviews; complements --issue-events)
       --pulls               include pull requests in backup
       --pull-comments       include pull request review comments in backup
       --pull-reviews        include pull request reviews in backup
@@ -347,6 +350,27 @@ For finer control, avoid using ``--assets`` with starred repos, or use ``--skip-
 
 Alternatively, consider just storing links to starred repos in JSON format with ``--starred``.
 
+
+About issue events and the issue timeline
+-----------------------------------------
+
+Use ``--issue-events`` and ``--issue-timeline`` with ``--issues``; on their own they back up nothing.
+
+``--issue-events`` backs up GitHub's issue events endpoint under each issue's ``event_data`` key. That endpoint returns a restricted set of event types and **never** includes cross-references, so an issue that was mentioned from another issue or pull request shows no sign of it. GitHub only exposes those through the timeline.
+
+``--issue-timeline`` backs up the timeline endpoint under a separate ``timeline_data`` key. The two flags are independent: ``event_data`` keeps its existing contents and shape whether or not the timeline is enabled, so nothing that already reads it is affected.
+
+Use both flags together for a complete picture. Neither endpoint is a superset of the other, so they are complementary rather than alternatives. Only the timeline reports ``cross-referenced``, ``committed`` and ``reviewed`` entries. Only the events endpoint reliably reports ``referenced`` entries, where a commit elsewhere in the repository mentions the issue; the timeline omits many of these, substituting the issue's own ``committed`` entries instead.
+
+Note that timeline entries are not all the same shape as events. ``cross-referenced`` and ``committed`` entries have no ``id`` field, so anything consuming ``timeline_data`` should not assume one is present. ``commented`` entries are excluded from the backup because they embed the full comment body, which ``--issue-comments`` already saves.
+
+``--issue-timeline`` is included in ``--all``. On issues with many cross-references it can be noticeably larger and slower than ``--issue-events`` alone, because each ``cross-referenced`` entry embeds the referencing issue in full. On a heavily cross-referenced issue this can mean several times the API requests and many times the JSON on disk. Ordinary repositories see little difference.
+
+Incremental backups need extra help here, because being referenced from elsewhere does not change an issue's ``updated_at``. An issue can be mentioned from another repository years after its last activity, and the usual ``since`` listing will never report it as changed. On an incremental run with ``--issue-timeline``, github-backup therefore asks GitHub's GraphQL API for each item's cross-reference count and newest timestamp, compares those against what is already saved, and re-fetches the timeline of anything that differs. Pull requests are included in that check when they are being stored as issues, which is the case unless ``--pulls`` is used. The check needs no extra state on disk and also notices references that were later deleted. It costs one rate-limit point per 100 items checked, against GitHub's GraphQL quota, which is separate from the REST one. That is cheap in quota terms even on very large repositories, but it is not instant: a repository with 18,000 issues and pull requests took around 200 points and five minutes for the check alone.
+
+If you enable ``--issue-timeline`` on an existing incremental backup, cross-references on older issues are picked up automatically by that check. Timeline entries of other kinds, such as commits and reviews, are only fetched when the issue itself is updated, so run once without ``--incremental`` if you want those filled in for historical issues too.
+
+
 About pull request reviews
 --------------------------
 
@@ -450,14 +474,14 @@ Quietly and incrementally backup useful Github user data (public and private rep
     FINE_ACCESS_TOKEN=SOME-GITHUB-TOKEN
     GH_USER=YOUR-GITHUB-USER
 
-    github-backup -f $FINE_ACCESS_TOKEN --prefer-ssh -o ~/github-backup/ -l error -P -i --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --pulls --pull-comments --pull-reviews --pull-commits --labels --milestones --security-advisories --discussions --repositories --wikis --releases --assets --attachments --pull-details --gists --starred-gists $GH_USER
+    github-backup -f $FINE_ACCESS_TOKEN --prefer-ssh -o ~/github-backup/ -l error -P -i --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --issue-timeline --pulls --pull-comments --pull-reviews --pull-commits --labels --milestones --security-advisories --discussions --repositories --wikis --releases --assets --attachments --pull-details --gists --starred-gists $GH_USER
 
 Debug an error/block or incomplete backup into a temporary directory. Omit "incremental" to fill a previous incomplete backup. ::
 
     FINE_ACCESS_TOKEN=SOME-GITHUB-TOKEN
     GH_USER=YOUR-GITHUB-USER
 
-    github-backup -f $FINE_ACCESS_TOKEN -o /tmp/github-backup/ -l debug -P --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --pulls --pull-comments --pull-reviews --pull-commits --labels --milestones --discussions --repositories --wikis --releases --assets --pull-details --gists --starred-gists $GH_USER
+    github-backup -f $FINE_ACCESS_TOKEN -o /tmp/github-backup/ -l debug -P --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --issue-timeline --pulls --pull-comments --pull-reviews --pull-commits --labels --milestones --discussions --repositories --wikis --releases --assets --pull-details --gists --starred-gists $GH_USER
 
 Pipe a token from stdin to avoid storing it in environment variables or command history (Unix-like systems only)::
 

github_backup-0.65.0/github_backup/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.65.0"

{github_backup-0.64.2 → github_backup-0.65.0}/github_backup/github_backup.py

@@ -18,6 +18,7 @@ import subprocess
 import sys
 import threading
 import time
+import traceback
 from collections.abc import Generator
 from datetime import datetime
 from http.client import IncompleteRead
@@ -33,10 +34,14 @@ except ImportError:
     VERSION = "unknown"
 
 from .graphql_queries import (
+    CROSS_REFERENCE_NODE_LIMIT,
+    CROSS_REFERENCE_PAGE_SIZE,
     DISCUSSION_DETAIL_QUERY,
     DISCUSSION_LIST_QUERY,
     DISCUSSION_PAGE_SIZE,
     DISCUSSION_REPLIES_QUERY,
+    ISSUE_CROSS_REFERENCE_COUNT_QUERY,
+    PULL_CROSS_REFERENCE_COUNT_QUERY,
 )
 
 FILE_URI_PREFIX = "file://"
@@ -278,6 +283,12 @@ def parse_args(args=None):
         dest="include_issue_events",
         help="include issue events in backup",
     )
+    parser.add_argument(
+        "--issue-timeline",
+        action="store_true",
+        dest="include_issue_timeline",
+        help="include issue timeline in backup (cross-references, commits and reviews; complements --issue-events)",
+    )
     parser.add_argument(
         "--pulls",
         action="store_true",
@@ -2569,6 +2580,141 @@ def backup_discussions(args, repo_cwd, repository):
         )
 
 
+def cross_reference_state(timeline_data):
+    """Return (count, newest timestamp) of the cross-references in a timeline."""
+    created = [
+        item.get("created_at")
+        for item in timeline_data or []
+        if item.get("event") == "cross-referenced"
+    ]
+    return len(created), max((c for c in created if c), default=None)
+
+
+def _cross_reference_page(args, repository, query, connection, use_total_count):
+    """Page one GraphQL connection, yielding {number: (count, newest)}.
+
+    totalCount honours the itemTypes filter on issues but not on pull
+    requests, so pull request cross-references are counted from the nodes.
+    """
+    owner, name = _repository_owner_name(repository)
+    state = {}
+    after = None
+    page = 1
+
+    while True:
+        variables = {
+            "owner": owner,
+            "name": name,
+            "after": after,
+            "pageSize": CROSS_REFERENCE_PAGE_SIZE,
+        }
+        if not use_total_count:
+            # Only the pull request query declares this.
+            variables["nodeLimit"] = CROSS_REFERENCE_NODE_LIMIT
+
+        data = retrieve_graphql_data(
+            args,
+            query,
+            variables,
+            log_context="cross-references {0} {1} page {2}".format(
+                repository["full_name"], connection, page
+            ),
+        )
+        repository_data = data.get("repository")
+        if repository_data is None:
+            raise Exception(
+                "Repository {0} not found in GraphQL response".format(
+                    repository["full_name"]
+                )
+            )
+
+        items = repository_data.get(connection) or {}
+        for node in _connection_nodes(items):
+            timeline = node.get("timelineItems") or {}
+            created = [n.get("createdAt") for n in timeline.get("nodes") or [] if n]
+            if use_total_count:
+                count = timeline.get("totalCount", 0)
+            elif len(created) >= CROSS_REFERENCE_NODE_LIMIT:
+                # Counted from the nodes, so it may be truncated. Compare on
+                # the timestamp alone rather than on a count we cannot trust.
+                count = None
+            else:
+                count = len(created)
+            state[node["number"]] = (count, created[-1] if created else None)
+
+        page_info = items.get("pageInfo") or {}
+        if not page_info.get("hasNextPage"):
+            break
+
+        after = page_info.get("endCursor")
+        page += 1
+
+    return state
+
+
+def retrieve_cross_reference_state(args, repository, include_pulls=False):
+    """Ask GitHub for each item's cross-reference count and newest timestamp."""
+    state = _cross_reference_page(
+        args,
+        repository,
+        ISSUE_CROSS_REFERENCE_COUNT_QUERY,
+        "issues",
+        use_total_count=True,
+    )
+    if include_pulls:
+        state.update(
+            _cross_reference_page(
+                args,
+                repository,
+                PULL_CROSS_REFERENCE_COUNT_QUERY,
+                "pullRequests",
+                use_total_count=False,
+            )
+        )
+    return state
+
+
+def find_stale_timeline_issues(args, issue_cwd, repository, include_pulls=False):
+    """Issue numbers whose stored cross-references no longer match GitHub's.
+
+    Incremental runs select issues by updated_at, which a cross-reference does
+    not change, so those issues would otherwise never be revisited.
+    """
+    if not get_graphql_auth(args):
+        # The check needs the GraphQL API, which always requires a token.
+        logger.debug("Skipping cross-reference check, no authentication")
+        return set()
+
+    try:
+        live = retrieve_cross_reference_state(args, repository, include_pulls)
+    except Exception as e:
+        logger.warning(
+            "Unable to check {0} for new cross-references, "
+            "issues referenced since the last backup may be missed: {1}".format(
+                repository["full_name"], e
+            )
+        )
+        logger.debug(traceback.format_exc())
+        return set()
+
+    stale = set()
+    for number, (count, newest) in live.items():
+        existing = read_json_file_if_exists("{0}/{1}.json".format(issue_cwd, number))
+        if existing is None:
+            continue
+        stored = cross_reference_state(existing.get("timeline_data"))
+        changed = stored[1] != newest if count is None else stored != (count, newest)
+        if changed:
+            stale.add(number)
+
+    if stale:
+        logger.info(
+            "Refreshing {0} issues with new cross-references".format(len(stale))
+        )
+        logger.debug("Stale timelines: {0}".format(sorted(stale)))
+    return stale
+
+
 def backup_issues(args, repo_cwd, repository, repos_template):
     has_issues_dir = os.path.isdir("{0}/issues/.git".format(repo_cwd))
     if args.skip_existing and has_issues_dir:
@@ -2584,6 +2730,7 @@ def backup_issues(args, repo_cwd, repository, repos_template):
     _issue_template = "{0}/{1}/issues".format(repos_template, repository["full_name"])
 
     should_include_pulls = args.include_pulls or args.include_everything
+    include_timeline = args.include_issue_timeline or args.include_everything
     issue_states = ["open", "closed"]
     for issue_state in issue_states:
         query_args = {"filter": "all", "state": issue_state}
@@ -2600,6 +2747,30 @@ def backup_issues(args, repo_cwd, repository, repos_template):
 
             issues[issue["number"]] = issue
 
+    stale_timelines = set()
+    incremental_run = args.since or args.incremental_by_files
+    if (
+        include_timeline
+        and incremental_run
+        and any(f.endswith(".json") for f in os.listdir(issue_cwd))
+    ):
+        stale_timelines = find_stale_timeline_issues(
+            args, issue_cwd, repository, include_pulls=not should_include_pulls
+        )
+        for number in sorted(stale_timelines - set(issues)):
+            try:
+                issues[number] = retrieve_data(
+                    args, "{0}/{1}".format(_issue_template, number), paginated=False
+                )[0]
+            except (HTTPError, RepositoryUnavailableError, IndexError) as e:
+                # Deleted or transferred between the sweep and this fetch. The
+                # refresh is opportunistic, so skip rather than fail the run.
+                logger.warning(
+                    "Unable to refresh cross-references for issue {0}: {1}".format(
+                        number, e
+                    )
+                )
+
     if issues_skipped:
         issues_skipped_message = " (skipped {0} pull requests)".format(issues_skipped)
 
@@ -2610,9 +2781,14 @@ def backup_issues(args, repo_cwd, repository, repos_template):
     )
     comments_template = _issue_template + "/{0}/comments"
     events_template = _issue_template + "/{0}/events"
+    timeline_template = _issue_template + "/{0}/timeline"
     for number, issue in list(issues.items()):
         issue_file = "{0}/{1}.json".format(issue_cwd, number)
-        if args.incremental_by_files and os.path.isfile(issue_file):
+        if (
+            args.incremental_by_files
+            and os.path.isfile(issue_file)
+            and number not in stale_timelines
+        ):
             modified = os.path.getmtime(issue_file)
             modified = datetime.fromtimestamp(modified).strftime("%Y-%m-%dT%H:%M:%SZ")
             if modified > issue["updated_at"]:
@@ -2629,6 +2805,16 @@ def backup_issues(args, repo_cwd, repository, repos_template):
         if args.include_issue_events or args.include_everything:
             template = events_template.format(number)
             issues[number]["event_data"] = retrieve_data(args, template)
+        if include_timeline:
+            template = timeline_template.format(number)
+            # "commented" timeline entries embed the full comment body, which
+            # --issue-comments already backs up; storing them twice invites the
+            # two copies to diverge.
+            issues[number]["timeline_data"] = [
+                item
+                for item in retrieve_data(args, template)
+                if item.get("event") != "commented"
+            ]
         if args.include_attachments:
             download_attachments(
                 args, issue_cwd, issues[number], number, repository, item_type="issue"

{github_backup-0.64.2 → github_backup-0.65.0}/github_backup/graphql_queries.py

@@ -2,6 +2,66 @@
 
 DISCUSSION_PAGE_SIZE = 100
 
+CROSS_REFERENCE_PAGE_SIZE = 100
+
+# Cross-references do not change an issue's updatedAt, so the REST listing
+# cannot report them as changed. Count and newest timestamp are compared
+# against what is on disk: the count alone would miss an add paired with a
+# delete, the timestamp alone would miss a delete.
+ISSUE_CROSS_REFERENCE_COUNT_QUERY = """
+query($owner: String!, $name: String!, $after: String, $pageSize: Int!) {
+  repository(owner: $owner, name: $name) {
+    issues(first: $pageSize, after: $after) {
+      nodes {
+        number
+        timelineItems(last: 1, itemTypes: [CROSS_REFERENCED_EVENT]) {
+          totalCount
+          nodes {
+            ... on CrossReferencedEvent {
+              createdAt
+            }
+          }
+        }
+      }
+      pageInfo {
+        hasNextPage
+        endCursor
+      }
+    }
+  }
+}
+"""
+
+# Pull requests are stored as issues when --pulls is not used, and the issues
+# connection above excludes them. This query is shaped differently because
+# totalCount ignores itemTypes on a pull request, unlike on an issue, so the
+# cross-references have to be counted from the nodes themselves.
+CROSS_REFERENCE_NODE_LIMIT = 100
+
+PULL_CROSS_REFERENCE_COUNT_QUERY = """
+query($owner: String!, $name: String!, $after: String, $pageSize: Int!,
+      $nodeLimit: Int!) {
+  repository(owner: $owner, name: $name) {
+    pullRequests(first: $pageSize, after: $after) {
+      nodes {
+        number
+        timelineItems(last: $nodeLimit, itemTypes: [CROSS_REFERENCED_EVENT]) {
+          nodes {
+            ... on CrossReferencedEvent {
+              createdAt
+            }
+          }
+        }
+      }
+      pageInfo {
+        hasNextPage
+        endCursor
+      }
+    }
+  }
+}
+"""
+
 DISCUSSION_LIST_QUERY = """
 query($owner: String!, $name: String!, $after: String, $pageSize: Int!) {
   repository(owner: $owner, name: $name) {

{github_backup-0.64.2 → github_backup-0.65.0/github_backup.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: github-backup
-Version: 0.64.2
+Version: 0.65.0
 Summary: backup a github user or organization
 Home-page: http://github.com/josegonzalez/python-github-backup
 Author: Jose Diaz-Gonzalez
@@ -71,7 +71,8 @@ CLI Help output::
                   [--incremental-by-files]
                   [--starred] [--all-starred] [--starred-skip-size-over MB]
                   [--watched] [--followers] [--following] [--all]
-                  [--issues] [--issue-comments] [--issue-events] [--pulls]
+                  [--issues] [--issue-comments] [--issue-events]
+                  [--issue-timeline] [--pulls]
                   [--pull-comments] [--pull-reviews] [--pull-commits]
                   [--pull-details]
                   [--labels] [--hooks] [--milestones] [--security-advisories]
@@ -126,6 +127,8 @@ CLI Help output::
       --issues              include issues in backup
       --issue-comments      include issue comments in backup
       --issue-events        include issue events in backup
+      --issue-timeline      include issue timeline in backup (cross-references,
+                            commits and reviews; complements --issue-events)
       --pulls               include pull requests in backup
       --pull-comments       include pull request review comments in backup
       --pull-reviews        include pull request reviews in backup
@@ -377,6 +380,27 @@ For finer control, avoid using ``--assets`` with starred repos, or use ``--skip-
 
 Alternatively, consider just storing links to starred repos in JSON format with ``--starred``.
 
+
+About issue events and the issue timeline
+-----------------------------------------
+
+Use ``--issue-events`` and ``--issue-timeline`` with ``--issues``; on their own they back up nothing.
+
+``--issue-events`` backs up GitHub's issue events endpoint under each issue's ``event_data`` key. That endpoint returns a restricted set of event types and **never** includes cross-references, so an issue that was mentioned from another issue or pull request shows no sign of it. GitHub only exposes those through the timeline.
+
+``--issue-timeline`` backs up the timeline endpoint under a separate ``timeline_data`` key. The two flags are independent: ``event_data`` keeps its existing contents and shape whether or not the timeline is enabled, so nothing that already reads it is affected.
+
+Use both flags together for a complete picture. Neither endpoint is a superset of the other, so they are complementary rather than alternatives. Only the timeline reports ``cross-referenced``, ``committed`` and ``reviewed`` entries. Only the events endpoint reliably reports ``referenced`` entries, where a commit elsewhere in the repository mentions the issue; the timeline omits many of these, substituting the issue's own ``committed`` entries instead.
+
+Note that timeline entries are not all the same shape as events. ``cross-referenced`` and ``committed`` entries have no ``id`` field, so anything consuming ``timeline_data`` should not assume one is present. ``commented`` entries are excluded from the backup because they embed the full comment body, which ``--issue-comments`` already saves.
+
+``--issue-timeline`` is included in ``--all``. On issues with many cross-references it can be noticeably larger and slower than ``--issue-events`` alone, because each ``cross-referenced`` entry embeds the referencing issue in full. On a heavily cross-referenced issue this can mean several times the API requests and many times the JSON on disk. Ordinary repositories see little difference.
+
+Incremental backups need extra help here, because being referenced from elsewhere does not change an issue's ``updated_at``. An issue can be mentioned from another repository years after its last activity, and the usual ``since`` listing will never report it as changed. On an incremental run with ``--issue-timeline``, github-backup therefore asks GitHub's GraphQL API for each item's cross-reference count and newest timestamp, compares those against what is already saved, and re-fetches the timeline of anything that differs. Pull requests are included in that check when they are being stored as issues, which is the case unless ``--pulls`` is used. The check needs no extra state on disk and also notices references that were later deleted. It costs one rate-limit point per 100 items checked, against GitHub's GraphQL quota, which is separate from the REST one. That is cheap in quota terms even on very large repositories, but it is not instant: a repository with 18,000 issues and pull requests took around 200 points and five minutes for the check alone.
+
+If you enable ``--issue-timeline`` on an existing incremental backup, cross-references on older issues are picked up automatically by that check. Timeline entries of other kinds, such as commits and reviews, are only fetched when the issue itself is updated, so run once without ``--incremental`` if you want those filled in for historical issues too.
+
+
 About pull request reviews
 --------------------------
 
@@ -480,14 +504,14 @@ Quietly and incrementally backup useful Github user data (public and private rep
     FINE_ACCESS_TOKEN=SOME-GITHUB-TOKEN
     GH_USER=YOUR-GITHUB-USER
 
-    github-backup -f $FINE_ACCESS_TOKEN --prefer-ssh -o ~/github-backup/ -l error -P -i --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --pulls --pull-comments --pull-reviews --pull-commits --labels --milestones --security-advisories --discussions --repositories --wikis --releases --assets --attachments --pull-details --gists --starred-gists $GH_USER
+    github-backup -f $FINE_ACCESS_TOKEN --prefer-ssh -o ~/github-backup/ -l error -P -i --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --issue-timeline --pulls --pull-comments --pull-reviews --pull-commits --labels --milestones --security-advisories --discussions --repositories --wikis --releases --assets --attachments --pull-details --gists --starred-gists $GH_USER
 
 Debug an error/block or incomplete backup into a temporary directory. Omit "incremental" to fill a previous incomplete backup. ::
 
     FINE_ACCESS_TOKEN=SOME-GITHUB-TOKEN
     GH_USER=YOUR-GITHUB-USER
 
-    github-backup -f $FINE_ACCESS_TOKEN -o /tmp/github-backup/ -l debug -P --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --pulls --pull-comments --pull-reviews --pull-commits --labels --milestones --discussions --repositories --wikis --releases --assets --pull-details --gists --starred-gists $GH_USER
+    github-backup -f $FINE_ACCESS_TOKEN -o /tmp/github-backup/ -l debug -P --all-starred --starred --watched --followers --following --issues --issue-comments --issue-events --issue-timeline --pulls --pull-comments --pull-reviews --pull-commits --labels --milestones --discussions --repositories --wikis --releases --assets --pull-details --gists --starred-gists $GH_USER
 
 Pipe a token from stdin to avoid storing it in environment variables or command history (Unix-like systems only)::
 

{github_backup-0.64.2 → github_backup-0.65.0}/github_backup.egg-info/SOURCES.txt

@@ -26,6 +26,7 @@ tests/test_discussions.py
 tests/test_gists.py
 tests/test_http_451.py
 tests/test_incremental_per_repository.py
+tests/test_issue_timeline.py
 tests/test_json_dump_if_changed.py
 tests/test_logging_subprocess.py
 tests/test_pagination.py

github_backup-0.65.0/tests/test_issue_timeline.py

@@ -0,0 +1,434 @@
+"""Tests for issue timeline backup (GitHub issue #168).
+
+The ``/issues/{number}/events`` endpoint never returns cross-reference events
+("mentioned in #22"); GitHub only exposes those via ``/issues/{number}/timeline``.
+``--issue-timeline`` backs up the timeline alongside the events, so those
+references are captured.
+"""
+
+import json
+import os
+import time
+
+from github_backup import github_backup
+
+
+def _issue(number=1, **extra):
+    issue = {
+        "number": number,
+        "title": "an issue",
+        "updated_at": "2026-07-01T00:00:00Z",
+    }
+    issue.update(extra)
+    return issue
+
+
+def _fake_retrieve(responses, calls):
+    """Return a retrieve_data stub serving ``responses`` keyed by URL suffix."""
+
+    def _retrieve(args, template, query_args=None, **kwargs):
+        calls.append(template)
+        for suffix, payload in responses.items():
+            if template.endswith(suffix):
+                return payload
+        return []
+
+    return _retrieve
+
+
+def _run(create_args, tmp_path, monkeypatch, responses, **arg_overrides):
+    args = create_args(include_issues=True, since=None, **arg_overrides)
+    calls = []
+    monkeypatch.setattr(
+        github_backup, "retrieve_data", _fake_retrieve(responses, calls)
+    )
+    github_backup.backup_issues(
+        args, str(tmp_path), {"full_name": "owner/repo"}, "https://api.github.com/repos"
+    )
+    issue_file = tmp_path / "issues" / "1.json"
+    assert issue_file.is_file(), "issue was not written to disk"
+    return json.loads(issue_file.read_text()), calls
+
+
+def test_parse_args_issue_timeline_flag():
+    # The create_args fixture sets attributes directly, so only this test
+    # catches the flag being wired to the wrong dest.
+    args = github_backup.parse_args(["--issue-timeline", "testuser"])
+    assert args.include_issue_timeline is True
+
+
+TIMELINE = [
+    {"id": 1, "event": "labeled", "label": {"name": "bug"}},
+    {"event": "cross-referenced", "source": {"type": "issue", "issue": {"number": 22}}},
+    {"id": 2, "event": "commented", "body": "a comment body"},
+    {"event": "committed", "sha": "abc123"},
+]
+
+
+def test_comments_filtered_and_everything_else_stored_verbatim(
+    create_args, tmp_path, monkeypatch
+):
+    saved, calls = _run(
+        create_args,
+        tmp_path,
+        monkeypatch,
+        {"/issues": [_issue()], "/1/timeline": TIMELINE},
+        include_issue_timeline=True,
+    )
+
+    assert any(c.endswith("/1/timeline") for c in calls)
+    # "commented" is dropped because --issue-comments already covers it.
+    # Everything else is stored exactly as GitHub returned it, including the
+    # fat source object on cross-references.
+    assert saved["timeline_data"] == [TIMELINE[0], TIMELINE[1], TIMELINE[3]]
+
+
+def test_timeline_not_fetched_without_flag(create_args, tmp_path, monkeypatch):
+    saved, calls = _run(
+        create_args,
+        tmp_path,
+        monkeypatch,
+        {"/issues": [_issue()], "/1/timeline": TIMELINE},
+    )
+
+    assert not any(c.endswith("/1/timeline") for c in calls)
+    assert "timeline_data" not in saved
+
+
+def test_events_unchanged_by_timeline_flag(create_args, tmp_path, monkeypatch):
+    """--issue-events keeps reading /events, so event_data's shape is untouched."""
+    events = [{"id": 1, "event": "labeled", "label": {"name": "bug"}}]
+    saved, calls = _run(
+        create_args,
+        tmp_path,
+        monkeypatch,
+        {"/issues": [_issue()], "/1/events": events, "/1/timeline": TIMELINE},
+        include_issue_events=True,
+        include_issue_timeline=True,
+    )
+
+    assert any(c.endswith("/1/events") for c in calls)
+    assert saved["event_data"] == events
+    assert len(saved["timeline_data"]) == 3
+
+
+def test_timeline_included_in_all(create_args, tmp_path, monkeypatch):
+    saved, calls = _run(
+        create_args,
+        tmp_path,
+        monkeypatch,
+        {"/issues": [_issue()], "/1/timeline": TIMELINE},
+        include_everything=True,
+    )
+
+    assert any(c.endswith("/1/timeline") for c in calls)
+    assert len(saved["timeline_data"]) == 3
+
+
+def _issue_node(number, count, newest):
+    """An issues-connection node: totalCount honours the itemTypes filter."""
+    return {
+        "number": number,
+        "timelineItems": {
+            "totalCount": count,
+            "nodes": [{"createdAt": newest}] if newest else [],
+        },
+    }
+
+
+def _pull_node(number, stamps):
+    """A pullRequests-connection node.
+
+    GitHub's totalCount ignores itemTypes here, so it is deliberately wrong:
+    anything reading it instead of counting the nodes will fail these tests.
+    """
+    return {
+        "number": number,
+        "timelineItems": {
+            "totalCount": 999,
+            "nodes": [{"createdAt": s} for s in stamps],
+        },
+    }
+
+
+def _graphql_pages(issue_pages, pull_pages=None):
+    """Stub retrieve_graphql_data serving one or more pages per connection."""
+    pages = {"issues": issue_pages, "pullRequests": pull_pages or [[]]}
+    seen = []
+
+    def _retrieve(args, query, variables=None, log_context=None):
+        connection = "pullRequests" if "pullRequests" in query else "issues"
+        cursor = (variables or {}).get("after")
+        index = 0 if cursor is None else int(cursor)
+        seen.append((connection, cursor))
+        nodes = pages[connection][index]
+        has_next = index + 1 < len(pages[connection])
+        return {
+            "repository": {
+                connection: {
+                    "nodes": nodes,
+                    "pageInfo": {
+                        "hasNextPage": has_next,
+                        "endCursor": str(index + 1) if has_next else None,
+                    },
+                }
+            }
+        }
+
+    _retrieve.seen = seen
+    return _retrieve
+
+
+def _graphql_state(counts):
+    """Single-page issues-only stub, keyed {number: (count, newest)}."""
+    return _graphql_pages(
+        [[_issue_node(n, c, newest) for n, (c, newest) in counts.items()]]
+    )
+
+
+def _sweep_args(create_args, **overrides):
+    """Args for the cross-reference sweep, which needs a token to reach GraphQL."""
+    return create_args(token_classic="faketoken", **overrides)
+
+
+def _stored(tmp_path, number, timeline_data):
+    issues = tmp_path / "issues"
+    issues.mkdir(parents=True, exist_ok=True)
+    (issues / "{0}.json".format(number)).write_text(
+        json.dumps({"number": number, "timeline_data": timeline_data})
+    )
+    return str(issues)
+
+
+def _xref(created_at):
+    return {"event": "cross-referenced", "created_at": created_at}
+
+
+def test_stale_when_a_cross_reference_is_added(create_args, tmp_path, monkeypatch):
+    issue_cwd = _stored(tmp_path, 1, [_xref("2026-01-01T00:00:00Z")])
+    monkeypatch.setattr(
+        github_backup,
+        "retrieve_graphql_data",
+        _graphql_state({1: (2, "2026-07-01T00:00:00Z")}),
+    )
+
+    stale = github_backup.find_stale_timeline_issues(
+        _sweep_args(create_args), issue_cwd, {"full_name": "owner/repo"}
+    )
+
+    assert stale == {1}
+
+
+def test_stale_when_one_added_and_one_deleted(create_args, tmp_path, monkeypatch):
+    """The count is unchanged, so only the newest timestamp reveals the change."""
+    issue_cwd = _stored(
+        tmp_path, 1, [_xref("2026-01-01T00:00:00Z"), _xref("2026-02-01T00:00:00Z")]
+    )
+    monkeypatch.setattr(
+        github_backup,
+        "retrieve_graphql_data",
+        _graphql_state({1: (2, "2026-07-01T00:00:00Z")}),
+    )
+
+    stale = github_backup.find_stale_timeline_issues(
+        _sweep_args(create_args), issue_cwd, {"full_name": "owner/repo"}
+    )
+
+    assert stale == {1}
+
+
+def test_not_stale_when_unchanged(create_args, tmp_path, monkeypatch):
+    issue_cwd = _stored(tmp_path, 1, [_xref("2026-01-01T00:00:00Z")])
+    monkeypatch.setattr(
+        github_backup,
+        "retrieve_graphql_data",
+        _graphql_state({1: (1, "2026-01-01T00:00:00Z")}),
+    )
+
+    stale = github_backup.find_stale_timeline_issues(
+        _sweep_args(create_args), issue_cwd, {"full_name": "owner/repo"}
+    )
+
+    assert stale == set()
+
+
+def test_graphql_failure_warns_and_continues(
+    create_args, tmp_path, monkeypatch, caplog
+):
+    """A backup must not abort because the cross-reference check failed."""
+    issue_cwd = _stored(tmp_path, 1, [])
+
+    def _boom(*a, **kw):
+        raise Exception("GraphQL unavailable")
+
+    monkeypatch.setattr(github_backup, "retrieve_graphql_data", _boom)
+
+    stale = github_backup.find_stale_timeline_issues(
+        _sweep_args(create_args), issue_cwd, {"full_name": "owner/repo"}
+    )
+
+    assert stale == set()
+    assert "may be missed" in caplog.text
+
+
+def test_stale_issue_backed_up_though_since_excludes_it(
+    create_args, tmp_path, monkeypatch
+):
+    """The whole point: an issue since cannot see is still refreshed."""
+    _stored(tmp_path, 7, [])
+    args = _sweep_args(
+        create_args,
+        include_issues=True,
+        include_issue_timeline=True,
+        since="2026-07-01T00:00:00Z",
+    )
+    timeline = [_xref("2026-07-20T00:00:00Z")]
+    calls = []
+    monkeypatch.setattr(
+        github_backup,
+        "retrieve_data",
+        _fake_retrieve(
+            {"/issues": [], "/issues/7": [_issue(7)], "/7/timeline": timeline}, calls
+        ),
+    )
+    monkeypatch.setattr(
+        github_backup,
+        "retrieve_graphql_data",
+        _graphql_state({7: (1, "2026-07-20T00:00:00Z")}),
+    )
+
+    github_backup.backup_issues(
+        args, str(tmp_path), {"full_name": "owner/repo"}, "https://api.github.com/repos"
+    )
+
+    saved = json.loads((tmp_path / "issues" / "7.json").read_text())
+    assert saved["timeline_data"] == timeline
+
+
+def test_sweep_follows_pagination_cursors(create_args, tmp_path, monkeypatch):
+    """A cursor bug would silently truncate the sweep and strand later issues."""
+    issue_cwd = _stored(tmp_path, 1, [])
+    _stored(tmp_path, 2, [])
+    _stored(tmp_path, 3, [])
+    fake = _graphql_pages(
+        [
+            [_issue_node(1, 0, None)],
+            [_issue_node(2, 0, None)],
+            [_issue_node(3, 1, "2026-07-01T00:00:00Z")],
+        ]
+    )
+    monkeypatch.setattr(github_backup, "retrieve_graphql_data", fake)
+
+    stale = github_backup.find_stale_timeline_issues(
+        _sweep_args(create_args), issue_cwd, {"full_name": "owner/repo"}
+    )
+
+    # Issue 3 is only reachable by following two cursors.
+    assert stale == {3}
+    assert [c for _, c in fake.seen] == [None, "1", "2"]
+
+
+def test_pull_requests_are_counted_from_nodes_not_total_count(
+    create_args, tmp_path, monkeypatch
+):
+    """totalCount ignores the itemTypes filter on pull requests, so it lies."""
+    issue_cwd = _stored(tmp_path, 5, [_xref("2026-01-01T00:00:00Z")])
+    monkeypatch.setattr(
+        github_backup,
+        "retrieve_graphql_data",
+        _graphql_pages([[]], [[_pull_node(5, ["2026-01-01T00:00:00Z"])]]),
+    )
+
+    stale = github_backup.find_stale_timeline_issues(
+        _sweep_args(create_args),
+        issue_cwd,
+        {"full_name": "owner/repo"},
+        include_pulls=True,
+    )
+
+    # Stored state matches, so nothing to do. Reading totalCount (999) instead
+    # of counting the one node would flag this pull request on every run.
+    assert stale == set()
+
+
+def test_pull_requests_not_swept_when_backed_up_separately(
+    create_args, tmp_path, monkeypatch
+):
+    issue_cwd = _stored(tmp_path, 5, [])
+    fake = _graphql_pages([[]], [[_pull_node(5, ["2026-07-01T00:00:00Z"])]])
+    monkeypatch.setattr(github_backup, "retrieve_graphql_data", fake)
+
+    stale = github_backup.find_stale_timeline_issues(
+        _sweep_args(create_args),
+        issue_cwd,
+        {"full_name": "owner/repo"},
+        include_pulls=False,
+    )
+
+    assert stale == set()
+    assert [connection for connection, _ in fake.seen] == ["issues"]
+
+
+def test_truncated_node_count_compares_on_timestamp_only(
+    create_args, tmp_path, monkeypatch
+):
+    """At the node limit the count is unreliable; comparing it loops forever."""
+    limit = github_backup.CROSS_REFERENCE_NODE_LIMIT
+    stamps = [
+        "2026-01-01T{0:02d}:{1:02d}:00Z".format(i // 60, i % 60) for i in range(limit)
+    ]
+    # Stored has more cross-references than a single page can return.
+    stored = [_xref(s) for s in stamps] + [_xref("2025-12-01T00:00:00Z")]
+    issue_cwd = _stored(tmp_path, 5, stored)
+    monkeypatch.setattr(
+        github_backup,
+        "retrieve_graphql_data",
+        _graphql_pages([[]], [[_pull_node(5, stamps)]]),
+    )
+
+    stale = github_backup.find_stale_timeline_issues(
+        _sweep_args(create_args),
+        issue_cwd,
+        {"full_name": "owner/repo"},
+        include_pulls=True,
+    )
+
+    # Counts differ (101 stored vs 100 returned) but the newest matches, so
+    # this must not be reported stale on every run.
+    assert stale == set()
+
+
+def test_incremental_by_files_still_refreshes_cross_referenced_issues(
+    create_args, tmp_path, monkeypatch
+):
+    """--incremental-by-files sets no since, and skips by mtime, so the
+    cross-reference sweep is the only thing that can catch these."""
+    _stored(tmp_path, 3, [])
+    issue_file = tmp_path / "issues" / "3.json"
+    os.utime(issue_file, (0, time.time()))  # mtime newer than updated_at
+
+    args = _sweep_args(
+        create_args,
+        include_issues=True,
+        include_issue_timeline=True,
+        incremental_by_files=True,
+        since=None,
+    )
+    timeline = [_xref("2026-07-20T00:00:00Z")]
+    monkeypatch.setattr(
+        github_backup,
+        "retrieve_data",
+        _fake_retrieve({"/issues": [_issue(3)], "/3/timeline": timeline}, []),
+    )
+    monkeypatch.setattr(
+        github_backup,
+        "retrieve_graphql_data",
+        _graphql_pages([[_issue_node(3, 1, "2026-07-20T00:00:00Z")]]),
+    )
+
+    github_backup.backup_issues(
+        args, str(tmp_path), {"full_name": "owner/repo"}, "https://api.github.com/repos"
+    )
+
+    saved = json.loads(issue_file.read_text())
+    assert saved["timeline_data"] == timeline

github_backup-0.64.2/github_backup/__init__.py

@@ -1 +0,0 @@
-__version__ = "0.64.2"