gtfs-digester 0.1.0__tar.gz

gtfs_digester-0.1.0/PKG-INFO

@@ -0,0 +1,119 @@
+Metadata-Version: 2.4
+Name: gtfs-digester
+Version: 0.1.0
+Summary: Canonicalization, fingerprinting, and change detection for GTFS Schedule feeds
+Author: Chris Alfano
+Author-email: Chris Alfano <chris@jarv.us>
+License-Expression: MIT
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Scientific/Engineering :: GIS
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: blake3>=1.0.8
+Requires-Dist: fsspec>=2026.3.0
+Requires-Dist: gcsfs>=2026.3.0
+Requires-Dist: pyarrow>=23.0.1
+Requires-Python: >=3.11
+Project-URL: Homepage, https://github.com/JarvusInnovations/gtfs-digester
+Project-URL: Repository, https://github.com/JarvusInnovations/gtfs-digester
+Project-URL: Issues, https://github.com/JarvusInnovations/gtfs-digester/issues
+Description-Content-Type: text/markdown
+
+# gtfs-digester
+
+Canonicalization, fingerprinting, and change detection for GTFS Schedule feeds.
+
+## What It Does
+
+Takes a GTFS zip and produces:
+
+- A **content fingerprint** (BLAKE3 merkle hash) — identical for semantically identical feeds regardless of zip metadata, file ordering, CSV whitespace, or time formatting
+- **Canonical Arrow tables** for each file — normalized, sorted by primary key, all values as strings
+- **Per-file hashes** for efficient hierarchical change detection
+- **Archive diffs** — added/removed/modified files and rows by primary key
+- **Exploded parquet storage** — write/read from local or cloud (GCS, S3) via fsspec
+
+All files and columns are preserved, including non-standard ones.
+
+## Install
+
+```bash
+pip install gtfs-digester
+# or
+uv add gtfs-digester
+```
+
+## Quick Start
+
+```python
+from gtfs_digester import GTFSArchive
+
+# Load and fingerprint
+archive = GTFSArchive.from_zip("google_transit.zip")
+print(archive.fingerprint.hex())  # v1:abc123...
+
+# Access canonical tables
+stops = archive.arrow_table("stops.txt")
+print(stops.num_rows)
+
+# Compare two feed versions
+old = GTFSArchive.from_zip("old.zip")
+new = GTFSArchive.from_zip("new.zip")
+diff = old.diff(new)
+print(diff.is_identical)
+for f in diff.modified_files:
+    fd = diff.file_diff(f)
+    print(f"{f}: {fd.summary()}")
+
+# Write as exploded parquet
+from gtfs_digester import write_exploded
+write_exploded(archive, "gs://bucket/schedules/feed-1", schedule_url="https://...")
+```
+
+## How Fingerprinting Works
+
+1. Each `.txt` file is parsed to a PyArrow table (all strings)
+2. Columns reordered per GTFS spec (unknown columns preserved, sorted alphabetically after spec columns)
+3. Values normalized: whitespace stripped, times zero-padded (`9:05:00` → `09:05:00`)
+4. Rows sorted by primary key (numeric columns sorted numerically, not lexicographically)
+5. Canonical CSV serialized and BLAKE3 hashed per file
+6. Archive fingerprint = BLAKE3 of sorted `filename:hash` pairs (merkle tree)
+
+Unknown files (not in the GTFS spec) are preserved with lexicographic row sorting.
+
+## Storage Layout
+
+`write_exploded()` produces a version-first directory:
+
+```
+base_path/
+  _feed_digest={v1:abc...}/
+    agency.parquet
+    stops.parquet
+    routes.parquet
+    trips.parquet
+    stop_times.parquet
+    ...
+    metadata.json       # provenance + digester info, written last (commit marker)
+```
+
+DuckDB reads across versions with hive partitioning:
+
+```sql
+SELECT * FROM read_parquet('base_path/_feed_digest=*/stops.parquet', hive_partitioning=true);
+```
+
+## Development
+
+```bash
+uv sync --group dev
+uv run pytest tests/ -k "not Real"   # unit tests (~0.5s)
+uv run pytest tests/                  # includes real feed integration tests
+```
+
+## License
+
+MIT

gtfs_digester-0.1.0/README.md

@@ -0,0 +1,95 @@
+# gtfs-digester
+
+Canonicalization, fingerprinting, and change detection for GTFS Schedule feeds.
+
+## What It Does
+
+Takes a GTFS zip and produces:
+
+- A **content fingerprint** (BLAKE3 merkle hash) — identical for semantically identical feeds regardless of zip metadata, file ordering, CSV whitespace, or time formatting
+- **Canonical Arrow tables** for each file — normalized, sorted by primary key, all values as strings
+- **Per-file hashes** for efficient hierarchical change detection
+- **Archive diffs** — added/removed/modified files and rows by primary key
+- **Exploded parquet storage** — write/read from local or cloud (GCS, S3) via fsspec
+
+All files and columns are preserved, including non-standard ones.
+
+## Install
+
+```bash
+pip install gtfs-digester
+# or
+uv add gtfs-digester
+```
+
+## Quick Start
+
+```python
+from gtfs_digester import GTFSArchive
+
+# Load and fingerprint
+archive = GTFSArchive.from_zip("google_transit.zip")
+print(archive.fingerprint.hex())  # v1:abc123...
+
+# Access canonical tables
+stops = archive.arrow_table("stops.txt")
+print(stops.num_rows)
+
+# Compare two feed versions
+old = GTFSArchive.from_zip("old.zip")
+new = GTFSArchive.from_zip("new.zip")
+diff = old.diff(new)
+print(diff.is_identical)
+for f in diff.modified_files:
+    fd = diff.file_diff(f)
+    print(f"{f}: {fd.summary()}")
+
+# Write as exploded parquet
+from gtfs_digester import write_exploded
+write_exploded(archive, "gs://bucket/schedules/feed-1", schedule_url="https://...")
+```
+
+## How Fingerprinting Works
+
+1. Each `.txt` file is parsed to a PyArrow table (all strings)
+2. Columns reordered per GTFS spec (unknown columns preserved, sorted alphabetically after spec columns)
+3. Values normalized: whitespace stripped, times zero-padded (`9:05:00` → `09:05:00`)
+4. Rows sorted by primary key (numeric columns sorted numerically, not lexicographically)
+5. Canonical CSV serialized and BLAKE3 hashed per file
+6. Archive fingerprint = BLAKE3 of sorted `filename:hash` pairs (merkle tree)
+
+Unknown files (not in the GTFS spec) are preserved with lexicographic row sorting.
+
+## Storage Layout
+
+`write_exploded()` produces a version-first directory:
+
+```
+base_path/
+  _feed_digest={v1:abc...}/
+    agency.parquet
+    stops.parquet
+    routes.parquet
+    trips.parquet
+    stop_times.parquet
+    ...
+    metadata.json       # provenance + digester info, written last (commit marker)
+```
+
+DuckDB reads across versions with hive partitioning:
+
+```sql
+SELECT * FROM read_parquet('base_path/_feed_digest=*/stops.parquet', hive_partitioning=true);
+```
+
+## Development
+
+```bash
+uv sync --group dev
+uv run pytest tests/ -k "not Real"   # unit tests (~0.5s)
+uv run pytest tests/                  # includes real feed integration tests
+```
+
+## License
+
+MIT

gtfs_digester-0.1.0/pyproject.toml

@@ -0,0 +1,47 @@
+[project]
+name = "gtfs-digester"
+version = "0.1.0"
+description = "Canonicalization, fingerprinting, and change detection for GTFS Schedule feeds"
+readme = "README.md"
+authors = [
+    { name = "Chris Alfano", email = "chris@jarv.us" }
+]
+license = "MIT"
+requires-python = ">=3.11"
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "Topic :: Scientific/Engineering :: GIS",
+"Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+]
+
+dependencies = [
+    "blake3>=1.0.8",
+    "fsspec>=2026.3.0",
+    "gcsfs>=2026.3.0",
+    "pyarrow>=23.0.1",
+]
+
+[project.urls]
+Homepage = "https://github.com/JarvusInnovations/gtfs-digester"
+Repository = "https://github.com/JarvusInnovations/gtfs-digester"
+Issues = "https://github.com/JarvusInnovations/gtfs-digester/issues"
+
+[build-system]
+requires = ["uv_build>=0.11.3,<0.12.0"]
+build-backend = "uv_build"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+markers = [
+    "slow: marks tests that process real GTFS feeds (deselect with '-k not Real')",
+]
+
+[dependency-groups]
+dev = [
+    "pytest>=9.0.2",
+    "pytest-cov>=7.1.0",
+]

gtfs_digester-0.1.0/src/gtfs_digester/__init__.py

@@ -0,0 +1,26 @@
+"""gtfs-digester: Canonicalization, fingerprinting, and change detection for GTFS Schedule feeds."""
+
+from .archive import GTFSArchive
+from .diff import ArchiveDiff, FileDiff
+from .file import GTFSFile
+from .fingerprint import ArchiveFingerprint
+from .metadata import FeedMetadata
+from .schema import GTFS_SCHEMAS, FileSchema, get_schema
+from .storage import list_versions, read_metadata, read_table, version_exists, write_exploded
+
+__all__ = [
+    "GTFSArchive",
+    "GTFSFile",
+    "ArchiveFingerprint",
+    "ArchiveDiff",
+    "FileDiff",
+    "FeedMetadata",
+    "FileSchema",
+    "get_schema",
+    "GTFS_SCHEMAS",
+    "write_exploded",
+    "read_metadata",
+    "read_table",
+    "list_versions",
+    "version_exists",
+]

gtfs_digester-0.1.0/src/gtfs_digester/archive.py

@@ -0,0 +1,181 @@
+"""GTFSArchive: load from zip or directory, access Arrow tables, fingerprint, diff."""
+
+from __future__ import annotations
+
+import io
+import warnings
+import zipfile
+from pathlib import Path
+
+import pyarrow as pa
+
+from .diff import ArchiveDiff, FileDiff, compute_file_diff
+from .file import GTFSFile
+from .fingerprint import ArchiveFingerprint
+from .schema import get_schema
+
+
+class GTFSArchive:
+    """Represents a complete GTFS feed.
+
+    Provides access to individual files, computes fingerprints,
+    and writes normalized output. Archives are immutable.
+
+    All .txt files are preserved — both known (with schema-aware normalization)
+    and unknown (with lexicographic sorting).
+    """
+
+    def __init__(self, files: dict[str, GTFSFile]) -> None:
+        self._files = files
+        self._fingerprint: ArchiveFingerprint | None = None
+
+    @staticmethod
+    def from_zip(
+        source: str | bytes | io.BytesIO | Path,
+    ) -> GTFSArchive:
+        """Load a GTFS archive from a zip file.
+
+        Args:
+            source: Path to zip file, raw bytes, or BytesIO.
+        """
+        if isinstance(source, (str, Path)):
+            with open(source, "rb") as f:
+                zip_bytes = f.read()
+        elif isinstance(source, bytes):
+            zip_bytes = source
+        else:
+            zip_bytes = source.read()
+
+        zf = zipfile.ZipFile(io.BytesIO(zip_bytes))
+        files: dict[str, GTFSFile] = {}
+
+        for name in sorted(zf.namelist()):
+            if name.endswith("/") or name.startswith(".") or name.startswith("__"):
+                continue
+
+            # Strip directory prefix (some zips nest files in a subdirectory)
+            basename = name.split("/")[-1]
+
+            if not basename.endswith(".txt"):
+                continue
+
+            schema = get_schema(basename)
+            data = zf.read(name)
+            gtfs_file = GTFSFile.from_csv_bytes(
+                filename=basename,
+                data=data,
+                schema=schema,
+            )
+            files[basename] = gtfs_file
+
+        return GTFSArchive(files=files)
+
+    @staticmethod
+    def from_directory(
+        path: str | Path,
+    ) -> GTFSArchive:
+        """Load a GTFS archive from a directory of .txt files.
+
+        Args:
+            path: Path to directory containing GTFS .txt files.
+        """
+        dirpath = Path(path)
+        if not dirpath.is_dir():
+            raise ValueError(f"Not a directory: {dirpath}")
+
+        files: dict[str, GTFSFile] = {}
+
+        for txt_file in sorted(dirpath.glob("*.txt")):
+            basename = txt_file.name
+            if basename.startswith(".") or basename.startswith("__"):
+                continue
+
+            schema = get_schema(basename)
+            data = txt_file.read_bytes()
+            gtfs_file = GTFSFile.from_csv_bytes(
+                filename=basename,
+                data=data,
+                schema=schema,
+            )
+            files[basename] = gtfs_file
+
+        return GTFSArchive(files=files)
+
+    @property
+    def filenames(self) -> set[str]:
+        """Set of filenames in this archive."""
+        return set(self._files.keys())
+
+    def __contains__(self, filename: str) -> bool:
+        return filename in self._files
+
+    def __getitem__(self, filename: str) -> GTFSFile:
+        return self._files[filename]
+
+    def arrow_table(self, filename: str) -> pa.Table:
+        """Get the canonical Arrow table for a file."""
+        return self[filename].table
+
+    @property
+    def fingerprint(self) -> ArchiveFingerprint:
+        """Compute (or return cached) archive fingerprint."""
+        if self._fingerprint is None:
+            file_hashes: dict[str, str] = {}
+            for filename, gtfs_file in self._files.items():
+                file_hashes[filename] = gtfs_file.fingerprint_hash()
+            self._fingerprint = ArchiveFingerprint.compute(file_hashes)
+        return self._fingerprint
+
+    def to_normalized_zip(self, path: str | Path) -> None:
+        """Write the archive as a normalized GTFS zip.
+
+        Contains canonical CSV files and a .gtfs_digester.json metadata file.
+        """
+        fp = self.fingerprint
+
+        with zipfile.ZipFile(str(path), "w", compression=zipfile.ZIP_DEFLATED) as zf:
+            for filename in sorted(self._files.keys()):
+                gtfs_file = self._files[filename]
+                csv_bytes = gtfs_file.to_canonical_csv()
+                zf.writestr(filename, csv_bytes)
+
+            zf.writestr(".gtfs_digester.json", fp.to_json())
+
+    def diff(self, other: GTFSArchive) -> ArchiveDiff:
+        """Compare this archive with another, returning an ArchiveDiff.
+
+        This archive is treated as "old", other as "new".
+        """
+        old_files = self.filenames
+        new_files = other.filenames
+
+        added = new_files - old_files
+        removed = old_files - new_files
+        common = old_files & new_files
+
+        modified: set[str] = set()
+        unchanged: set[str] = set()
+        file_diffs: dict[str, FileDiff] = {}
+
+        for filename in common:
+            old_hash = self._files[filename].fingerprint_hash()
+            new_hash = other._files[filename].fingerprint_hash()
+            if old_hash == new_hash:
+                unchanged.add(filename)
+            else:
+                modified.add(filename)
+                old_table = self._files[filename].table
+                new_table = other._files[filename].table
+                old_schema = self._files[filename].schema
+                pk = old_schema.primary_key if old_schema else []
+                file_diffs[filename] = compute_file_diff(
+                    filename, old_table, new_table, pk
+                )
+
+        return ArchiveDiff(
+            added_files=added,
+            removed_files=removed,
+            modified_files=modified,
+            unchanged_files=unchanged,
+            _file_diffs=file_diffs,
+        )

gtfs_digester-0.1.0/src/gtfs_digester/diff.py

@@ -0,0 +1,160 @@
+"""ArchiveDiff and FileDiff: archive and file-level diffing by primary key.
+
+Diffing is hierarchical:
+1. Compare archive fingerprints — if equal, done.
+2. Compare file fingerprints — identify added/removed/modified files.
+3. For modified files, compare rows by primary key.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+import pyarrow as pa
+
+
+@dataclass
+class FileDiff:
+    """Row-level diff for a single modified GTFS file."""
+
+    filename: str
+    added: pa.Table
+    removed: pa.Table
+    modified: pa.Table
+
+    @property
+    def added_count(self) -> int:
+        return self.added.num_rows
+
+    @property
+    def removed_count(self) -> int:
+        return self.removed.num_rows
+
+    @property
+    def modified_count(self) -> int:
+        return self.modified.num_rows
+
+    def summary(self) -> str:
+        return (
+            f"+{self.added_count} added, "
+            f"-{self.removed_count} removed, "
+            f"~{self.modified_count} modified"
+        )
+
+
+@dataclass
+class ArchiveDiff:
+    """Diff between two GTFS archives."""
+
+    added_files: set[str]
+    removed_files: set[str]
+    modified_files: set[str]
+    unchanged_files: set[str]
+    _file_diffs: dict[str, FileDiff]
+
+    def file_diff(self, filename: str) -> FileDiff:
+        """Get the row-level diff for a modified file."""
+        if filename not in self._file_diffs:
+            raise KeyError(
+                f"No file diff for '{filename}'. "
+                f"Modified files: {sorted(self.modified_files)}"
+            )
+        return self._file_diffs[filename]
+
+    @property
+    def is_identical(self) -> bool:
+        return (
+            not self.added_files
+            and not self.removed_files
+            and not self.modified_files
+        )
+
+
+def compute_file_diff(
+    filename: str,
+    old_table: pa.Table,
+    new_table: pa.Table,
+    primary_key: list[str],
+) -> FileDiff:
+    """Compute row-level diff between two tables using primary key."""
+    if not primary_key:
+        return _diff_no_pk(filename, old_table, new_table)
+
+    old_keys: dict[tuple[str, ...], int] = {}
+    for i in range(old_table.num_rows):
+        key = tuple(old_table.column(col)[i].as_py() for col in primary_key)
+        old_keys[key] = i
+
+    new_keys: dict[tuple[str, ...], int] = {}
+    for i in range(new_table.num_rows):
+        key = tuple(new_table.column(col)[i].as_py() for col in primary_key)
+        new_keys[key] = i
+
+    old_key_set = set(old_keys.keys())
+    new_key_set = set(new_keys.keys())
+
+    added_keys = new_key_set - old_key_set
+    removed_keys = old_key_set - new_key_set
+    common_keys = old_key_set & new_key_set
+
+    modified_indices_new = []
+    all_columns = new_table.column_names
+
+    for key in common_keys:
+        old_idx = old_keys[key]
+        new_idx = new_keys[key]
+        for col_name in all_columns:
+            if col_name in old_table.column_names:
+                old_val = old_table.column(col_name)[old_idx].as_py()
+                new_val = new_table.column(col_name)[new_idx].as_py()
+                if old_val != new_val:
+                    modified_indices_new.append(new_idx)
+                    break
+
+    added_indices = sorted(new_keys[k] for k in added_keys)
+    removed_indices = sorted(old_keys[k] for k in removed_keys)
+
+    added_table = new_table.take(added_indices) if added_indices else new_table.slice(0, 0)
+    removed_table = old_table.take(removed_indices) if removed_indices else old_table.slice(0, 0)
+    modified_table = (
+        new_table.take(sorted(modified_indices_new))
+        if modified_indices_new
+        else new_table.slice(0, 0)
+    )
+
+    return FileDiff(
+        filename=filename,
+        added=added_table,
+        removed=removed_table,
+        modified=modified_table,
+    )
+
+
+def _diff_no_pk(filename: str, old_table: pa.Table, new_table: pa.Table) -> FileDiff:
+    """Diff tables with no primary key by comparing full row content."""
+    def _row_tuple(table: pa.Table, idx: int) -> tuple[str, ...]:
+        return tuple(table.column(c)[idx].as_py() for c in table.column_names)
+
+    old_rows = {_row_tuple(old_table, i) for i in range(old_table.num_rows)}
+    new_rows = {_row_tuple(new_table, i) for i in range(new_table.num_rows)}
+
+    added_rows = new_rows - old_rows
+    removed_rows = old_rows - new_rows
+
+    cols = new_table.column_names
+    added_data = {c: [] for c in cols}
+    for row in sorted(added_rows):
+        for c, v in zip(cols, row):
+            added_data[c].append(v)
+
+    removed_data = {c: [] for c in cols}
+    for row in sorted(removed_rows):
+        for c, v in zip(cols, row):
+            removed_data[c].append(v)
+
+    return FileDiff(
+        filename=filename,
+        added=pa.table({c: pa.array(v, type=pa.string()) for c, v in added_data.items()}),
+        removed=pa.table({c: pa.array(v, type=pa.string()) for c, v in removed_data.items()}),
+        modified=new_table.slice(0, 0),
+    )