pypi-tracker 0.1.0__tar.gz

pypi_tracker-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Yasser Toruño
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pypi_tracker-0.1.0/PKG-INFO

@@ -0,0 +1,107 @@
+Metadata-Version: 2.4
+Name: pypi-tracker
+Version: 0.1.0
+Summary: Permanently preserve PyPI download history and export it as static JSON reports.
+License-Expression: MIT
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Dynamic: license-file
+
+# pypi-tracker
+
+pypi-tracker is a lightweight tool for collecting and preserving PyPI package download statistics. It maintains historical data beyond [PyPIStats](https://pypistats.org)' retention window. It generates static, chart-ready JSON reports that can be used to visualize download trends, track package adoption, and power dashboards or other applications without a backend.
+
+
+## Try it
+
+```bash
+pip install pypi-tracker
+
+
+pypi-tracker sync --package dj-control-room
+```
+
+You now have the full history in `data/dj-control-room.csv` and reports in
+`public/packages/dj-control-room/`, such as `last-30-days.json`:
+
+```json
+{
+  "package": "dj-control-room",
+  "report": "last-30-days",
+  "as_of": "2026-11-01",
+  "downloads": 38071,
+  "complete": true,
+  "observations": [
+    {"date": "2026-10-03", "downloads": 510},
+    {"date": "2026-10-04", "downloads": 757}
+  ]
+}
+```
+
+The `observations` list has one entry per day in the window (shortened here). A day with no data is `null`,
+never `0`, and `complete` tells you whether any day is missing.
+
+## Commands
+
+| Command | What it does |
+|---|---|
+| `init` | Creates a starter config file. |
+| `sync` | Runs `collect`, then `build`. This is the one to schedule. |
+| `collect` | Fetches new data from PyPIStats and merges it into the CSVs. |
+| `build` | Writes the JSON report files to publish. |
+| `validate` | Checks the CSVs for bad data and missing dates. |
+
+## Tracking several packages
+
+You will typically want to keep a config file where you specify the packages you want to track.
+
+```bash
+pypi-tracker init  # generates a new pypi-tracker.toml file in the current directory
+```
+
+A complete config file will have the sections seen below. Note that the `pypi-tracker` also
+allows for cli args that can supersede these config values; though typically it is much
+easier to simply rely on a config file.
+```toml
+# pypi-tracker.toml
+packages = [
+    "dj-control-room",
+    "dj-control-room-base",
+    "dj-redis-panel",
+    "dj-cache-panel",
+    "dj-celery-panel",
+    "dj-urls-panel",
+    "dj-signals-panel",
+]
+
+[reports]
+rolling_days = [7, 30, 90, 180]
+calendar = ["weekly", "monthly", "yearly"]
+historical = ["daily"]
+summary = true
+
+[storage]
+directory = "data/"
+
+[output]
+directory = "public/"
+```
+
+You can update both reports and historical data by running a single `sync` command.
+```bash
+pypi-tracker sync --config pypi-tracker.toml
+```
+
+## Keep your CSVs
+
+Run `sync` on a schedule and commit `data/`, because **the CSV files are your history**. If you lose them, or
+stop collecting for longer than PyPIStats keeps data, those days are gone. History also can't reach back
+further than PyPIStats did on your first run, so totals are "recorded" downloads, not lifetime downloads.
+
+## More
+
+The [guide](docs/guide.md) covers configuration, every report format, the Python API, and GitHub Actions and
+Cloudflare Pages setup.

pypi_tracker-0.1.0/README.md

@@ -0,0 +1,95 @@
+# pypi-tracker
+
+pypi-tracker is a lightweight tool for collecting and preserving PyPI package download statistics. It maintains historical data beyond [PyPIStats](https://pypistats.org)' retention window. It generates static, chart-ready JSON reports that can be used to visualize download trends, track package adoption, and power dashboards or other applications without a backend.
+
+
+## Try it
+
+```bash
+pip install pypi-tracker
+
+
+pypi-tracker sync --package dj-control-room
+```
+
+You now have the full history in `data/dj-control-room.csv` and reports in
+`public/packages/dj-control-room/`, such as `last-30-days.json`:
+
+```json
+{
+  "package": "dj-control-room",
+  "report": "last-30-days",
+  "as_of": "2026-11-01",
+  "downloads": 38071,
+  "complete": true,
+  "observations": [
+    {"date": "2026-10-03", "downloads": 510},
+    {"date": "2026-10-04", "downloads": 757}
+  ]
+}
+```
+
+The `observations` list has one entry per day in the window (shortened here). A day with no data is `null`,
+never `0`, and `complete` tells you whether any day is missing.
+
+## Commands
+
+| Command | What it does |
+|---|---|
+| `init` | Creates a starter config file. |
+| `sync` | Runs `collect`, then `build`. This is the one to schedule. |
+| `collect` | Fetches new data from PyPIStats and merges it into the CSVs. |
+| `build` | Writes the JSON report files to publish. |
+| `validate` | Checks the CSVs for bad data and missing dates. |
+
+## Tracking several packages
+
+You will typically want to keep a config file where you specify the packages you want to track.
+
+```bash
+pypi-tracker init  # generates a new pypi-tracker.toml file in the current directory
+```
+
+A complete config file will have the sections seen below. Note that the `pypi-tracker` also
+allows for cli args that can supersede these config values; though typically it is much
+easier to simply rely on a config file.
+```toml
+# pypi-tracker.toml
+packages = [
+    "dj-control-room",
+    "dj-control-room-base",
+    "dj-redis-panel",
+    "dj-cache-panel",
+    "dj-celery-panel",
+    "dj-urls-panel",
+    "dj-signals-panel",
+]
+
+[reports]
+rolling_days = [7, 30, 90, 180]
+calendar = ["weekly", "monthly", "yearly"]
+historical = ["daily"]
+summary = true
+
+[storage]
+directory = "data/"
+
+[output]
+directory = "public/"
+```
+
+You can update both reports and historical data by running a single `sync` command.
+```bash
+pypi-tracker sync --config pypi-tracker.toml
+```
+
+## Keep your CSVs
+
+Run `sync` on a schedule and commit `data/`, because **the CSV files are your history**. If you lose them, or
+stop collecting for longer than PyPIStats keeps data, those days are gone. History also can't reach back
+further than PyPIStats did on your first run, so totals are "recorded" downloads, not lifetime downloads.
+
+## More
+
+The [guide](docs/guide.md) covers configuration, every report format, the Python API, and GitHub Actions and
+Cloudflare Pages setup.

pypi_tracker-0.1.0/pypi_tracker/__init__.py

@@ -0,0 +1,22 @@
+"""Collect, permanently preserve, aggregate and export PyPI download statistics."""
+
+from .aggregation import find_gaps, monthly, rolling_window, summarize, weekly, yearly
+from .config import ReportSettings
+from .errors import ConfigError, HistoryError, PartialFailure, PyPIStatsError, TrackerError
+from .tracker import Tracker
+
+__all__ = [
+    "ConfigError",
+    "HistoryError",
+    "PartialFailure",
+    "PyPIStatsError",
+    "ReportSettings",
+    "Tracker",
+    "TrackerError",
+    "find_gaps",
+    "monthly",
+    "rolling_window",
+    "summarize",
+    "weekly",
+    "yearly",
+]

pypi_tracker-0.1.0/pypi_tracker/aggregation.py

@@ -0,0 +1,130 @@
+"""Pure calculations over daily observations. Missing days are never treated as zero."""
+
+import calendar
+from collections.abc import Callable, Iterator
+from datetime import date, timedelta
+
+from .storage import History
+
+_ONE_DAY = timedelta(days=1)
+
+
+def tally(history: History, start: date, end: date) -> dict:
+    """Total an inclusive date range. `downloads` is None when no day in it was recorded."""
+    days = (end - start).days + 1
+    values = [history[day] for day in _dates(start, end) if day in history]
+    return {
+        "start": start.isoformat(),
+        "end": end.isoformat(),
+        "days": days,
+        "downloads": sum(values) if values else None,
+        "recorded_days": len(values),
+        "missing_days": days - len(values),
+        "complete": len(values) == days,
+    }
+
+
+def rolling_window(history: History, days: int) -> dict:
+    """Tally the `days` days ending at the latest recorded date, not at today's date.
+
+    `observations` has one entry per day in the window, so charts always get `days` points.
+    A day with no data has `downloads: None`. With no history at all the list is empty.
+    """
+    if days < 1:
+        raise ValueError(f"days must be positive, got {days}")
+    if not history:
+        return {
+            "start": None,
+            "end": None,
+            "days": days,
+            "downloads": None,
+            "recorded_days": 0,
+            "missing_days": days,
+            "complete": False,
+            "observations": [],
+        }
+    end = max(history)
+    start = end - _ONE_DAY * (days - 1)
+    return {
+        **tally(history, start, end),
+        "observations": [
+            {"date": day.isoformat(), "downloads": history.get(day)} for day in _dates(start, end)
+        ],
+    }
+
+
+def weekly(history: History) -> list[dict]:
+    """ISO calendar weeks, Monday through Sunday."""
+    return [
+        {"week": "{}-W{:02d}".format(*start.isocalendar()[:2]), **tally(history, start, end)}
+        for start, end in _spans(history, _week_span)
+    ]
+
+
+def monthly(history: History) -> list[dict]:
+    return [
+        {"month": f"{start:%Y-%m}", **tally(history, start, end)}
+        for start, end in _spans(history, _month_span)
+    ]
+
+
+def yearly(history: History) -> list[dict]:
+    return [
+        {"year": start.year, **tally(history, start, end)}
+        for start, end in _spans(history, _year_span)
+    ]
+
+
+def find_gaps(history: History) -> list[dict]:
+    """Runs of unrecorded days between the earliest and latest observations."""
+    days = sorted(history)
+    return [
+        {
+            "start": (before + _ONE_DAY).isoformat(),
+            "end": (after - _ONE_DAY).isoformat(),
+            "days": (after - before).days - 1,
+        }
+        for before, after in zip(days, days[1:])
+        if after - before > _ONE_DAY
+    ]
+
+
+def summarize(history: History) -> dict:
+    """Overall coverage. Totals are "recorded": history may not reach back to a package's start."""
+    gaps = find_gaps(history)
+    return {
+        "earliest_date": min(history).isoformat() if history else None,
+        "latest_date": max(history).isoformat() if history else None,
+        "recorded_downloads": sum(history.values()),
+        "recorded_days": len(history),
+        "missing_days": sum(gap["days"] for gap in gaps),
+        "gaps": gaps,
+    }
+
+
+def _dates(start: date, end: date) -> list[date]:
+    return [start + _ONE_DAY * offset for offset in range((end - start).days + 1)]
+
+
+def _spans(history: History, span: Callable[[date], tuple[date, date]]) -> Iterator[tuple[date, date]]:
+    """Every period from the earliest to latest observation, including periods with no data."""
+    if not history:
+        return
+    day, last = min(history), max(history)
+    while day <= last:
+        start, end = span(day)
+        yield start, end
+        day = end + _ONE_DAY
+
+
+def _week_span(day: date) -> tuple[date, date]:
+    monday = day - _ONE_DAY * day.weekday()
+    return monday, monday + _ONE_DAY * 6
+
+
+def _month_span(day: date) -> tuple[date, date]:
+    return day.replace(day=1), day.replace(day=calendar.monthrange(day.year, day.month)[1])
+
+
+def _year_span(day: date) -> tuple[date, date]:
+    return date(day.year, 1, 1), date(day.year, 12, 31)

pypi_tracker-0.1.0/pypi_tracker/cli.py

@@ -0,0 +1,117 @@
+"""The `pypi-tracker` command line. All real work happens in Tracker."""
+
+import argparse
+import sys
+from collections.abc import Sequence
+from importlib.metadata import version
+
+from .config import resolve_path
+from .errors import ConfigError, TrackerError
+from .tracker import Tracker
+
+EXAMPLE_CONFIG = """\
+packages = ["example-package"]
+
+[reports]
+rolling_days = [7, 30, 90, 180]
+calendar = ["weekly", "monthly", "yearly"]
+historical = ["daily"]
+summary = true
+
+[storage]
+directory = "data/"
+
+[output]
+directory = "public/"
+"""
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+    args = _parser().parse_args(argv)
+    try:
+        return args.command(args)
+    except (TrackerError, OSError) as error:
+        print(f"error: {error}", file=sys.stderr)
+        return 1
+
+
+def _parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        prog="pypi-tracker", description="Preserve PyPI download history and export static JSON."
+    )
+    # shortcut: needs installed package metadata, so the CLI won't start from a bare PYTHONPATH checkout.
+    parser.add_argument("--version", action="version", version=f"%(prog)s {version('pypi-tracker')}")
+    commands = parser.add_subparsers(required=True)
+
+    source = argparse.ArgumentParser(add_help=False)
+    source.add_argument("--config", help="TOML configuration file")
+    source.add_argument("--data-dir", help="directory of historical CSV files")
+    source.add_argument(
+        "--package", action="append", help="package to process, instead of the config's (repeatable)"
+    )
+    output = argparse.ArgumentParser(add_help=False)
+    output.add_argument("--output-dir", help="directory receiving JSON reports")
+
+    def add(name: str, handler, help: str, parents: list) -> argparse.ArgumentParser:
+        command = commands.add_parser(name, help=help, parents=parents)
+        command.set_defaults(command=handler)
+        return command
+
+    add("collect", _collect, "fetch observations and merge them into CSV storage", [source])
+    add("build", _build, "generate JSON reports from stored history", [source, output])
+    add("sync", _sync, "collect, then build", [source, output])
+    add("validate", _validate, "check the historical dataset", [source])
+
+    init = add("init", _init, "write an example configuration file", [])
+    init.add_argument("--output", default="pypi-tracker.toml", help="where to write it")
+    return parser
+
+
+def _tracker(args: argparse.Namespace) -> Tracker:
+    output_dir = getattr(args, "output_dir", None)
+    if args.config:
+        return Tracker.from_config(
+            args.config, data_dir=args.data_dir, output_dir=output_dir, packages=args.package
+        )
+    paths = {"data_dir": args.data_dir, "output_dir": output_dir}
+    return Tracker(args.package or (), **{key: value for key, value in paths.items() if value})
+
+
+def _collect(args: argparse.Namespace) -> int:
+    tracker = _tracker(args)
+    changed = tracker.collect()
+    print(f"collected {len(tracker.packages)} package(s), {len(changed)} changed")
+    return 0
+
+
+def _build(args: argparse.Namespace) -> int:
+    written = _tracker(args).build()
+    print(f"built reports, {len(written)} file(s) changed")
+    return 0
+
+
+def _sync(args: argparse.Namespace) -> int:
+    changed, written = _tracker(args).sync()
+    print(f"synced: {len(changed)} package(s) changed, {len(written)} report file(s) changed")
+    return 0
+
+
+def _validate(args: argparse.Namespace) -> int:
+    results = _tracker(args).validate()
+    for package, result in results.items():
+        print(f"{package}: {result['recorded_days']} recorded day(s)")
+        for problem in result["problems"]:
+            print(f"  problem: {problem}")
+        for gap in result["gaps"]:
+            print(f"  coverage gap: {gap['start']} to {gap['end']} ({gap['days']} day(s))")
+    return 1 if any(result["problems"] for result in results.values()) else 0
+
+
+def _init(args: argparse.Namespace) -> int:
+    path = resolve_path(args.output)
+    if path.exists():
+        raise ConfigError(f"{path} already exists")
+    path.parent.mkdir(parents=True, exist_ok=True)
+    path.write_text(EXAMPLE_CONFIG, encoding="utf-8")
+    print(f"wrote {path}")
+    return 0

pypi_tracker-0.1.0/pypi_tracker/client.py

@@ -0,0 +1,78 @@
+"""Read daily download counts from the PyPIStats `/overall` endpoint."""
+
+import http.client
+import json
+import time
+from datetime import UTC, date, datetime
+from email.utils import parsedate_to_datetime
+from urllib.error import HTTPError, URLError
+from urllib.request import Request, urlopen
+
+from .errors import PyPIStatsError
+
+URL = "https://pypistats.org/api/packages/{package}/overall?mirrors=false"
+CATEGORY = "without_mirrors"
+TIMEOUT_SECONDS = 30
+MAX_ATTEMPTS = 4
+MAX_WAIT_SECONDS = 120
+
+
+def fetch_overall(package: str) -> dict[date, int]:
+    """Return all daily observations PyPIStats has for a package."""
+    request = Request(URL.format(package=package), headers={"User-Agent": "pypi-tracker"})
+    problem = ""
+    for attempt in range(1, MAX_ATTEMPTS + 1):
+        wait = 2**attempt
+        try:
+            with urlopen(request, timeout=TIMEOUT_SECONDS) as response:
+                body = response.read()
+        except HTTPError as error:
+            if error.code == 404:
+                raise PyPIStatsError("package not found on PyPIStats") from error
+            if error.code != 429 and error.code < 500:
+                raise PyPIStatsError(f"PyPIStats returned HTTP {error.code}") from error
+            problem = f"PyPIStats returned HTTP {error.code}"
+            wait = _retry_after(error.headers.get("Retry-After")) or wait
+        except (URLError, OSError, http.client.HTTPException) as error:
+            problem = f"network error: {error}"
+        else:
+            return parse_overall(body)
+
+        if wait > MAX_WAIT_SECONDS:
+            raise PyPIStatsError(f"{problem}; server asked to wait {wait:.0f} seconds")
+        if attempt < MAX_ATTEMPTS:
+            time.sleep(wait)
+    raise PyPIStatsError(f"{problem} (after {MAX_ATTEMPTS} attempts)")
+
+
+def parse_overall(body: bytes) -> dict[date, int]:
+    """Turn a response body into {date: downloads}, rejecting anything suspicious."""
+    observations: dict[date, int] = {}
+    try:
+        for row in json.loads(body)["data"]:
+            if row["category"] != CATEGORY:
+                continue
+            day = date.fromisoformat(row["date"])
+            downloads = row["downloads"]
+            if type(downloads) is not int or downloads < 0:
+                raise ValueError(f"invalid download count {downloads!r} for {day}")
+            if day in observations:
+                raise ValueError(f"duplicate observation for {day}")
+            observations[day] = downloads
+    except (ValueError, KeyError, TypeError) as error:
+        raise PyPIStatsError(f"malformed PyPIStats response: {error!r}") from error
+    return observations
+
+
+def _retry_after(header: str | None) -> float | None:
+    """Parse a Retry-After header, which is either seconds or an HTTP date."""
+    if not header:
+        return None
+    try:
+        return max(float(header), 0)
+    except ValueError:
+        pass
+    try:
+        return max((parsedate_to_datetime(header) - datetime.now(UTC)).total_seconds(), 0)
+    except (ValueError, TypeError):
+        return None