gharc 0.1.0__tar.gz

gharc-0.1.0/CITATION.cff

@@ -0,0 +1,32 @@
+cff-version: 1.2.0
+message: "If you use this software, please cite it as below."
+type: software
+title: "gharc: A stream-and-filter tool for the GitHub Archive on consumer hardware"
+abstract: >-
+  gharc filters the GitHub Archive on consumer hardware by streaming
+  each hourly archive through memory and writing only matching events
+  to Parquet or JSONL. Peak local storage stays bounded by a single
+  in-flight download regardless of the time range processed.
+authors:
+  - family-names: Panwar
+    given-names: Arav
+    orcid: https://orcid.org/0009-0009-3013-5970
+version: 0.1.0
+date-released: 2026-04-26
+license: MIT
+repository-code: https://github.com/aravpanwar/gharc
+doi: 10.5281/zenodo.19814232
+identifiers:
+  - type: doi
+    value: 10.5281/zenodo.19814232
+    description: Concept DOI for all versions
+  - type: doi
+    value: 10.5281/zenodo.19814233
+    description: DOI for v0.1.0
+keywords:
+  - github
+  - mining-software-repositories
+  - msr
+  - stream-processing
+  - parquet
+  - python

gharc-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Arav Panwar
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

gharc-0.1.0/MANIFEST.in

@@ -0,0 +1,6 @@
+include README.md
+include LICENSE
+include CITATION.cff
+recursive-include paper *
+recursive-include examples *.py
+recursive-include tests *.py

gharc-0.1.0/PKG-INFO

@@ -0,0 +1,261 @@
+Metadata-Version: 2.4
+Name: gharc
+Version: 0.1.0
+Summary: A stream-processing tool for GitHub Archive data filtering.
+Author-email: Arav Panwar <aravpanwar@outlook.com>
+Project-URL: Homepage, https://github.com/aravpanwar/gharc
+Project-URL: Bug Tracker, https://github.com/aravpanwar/gharc/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests>=2.28.0
+Requires-Dist: click>=8.0.0
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: pyarrow>=12.0.0
+Requires-Dist: tqdm>=4.65.0
+Provides-Extra: fast
+Requires-Dist: orjson>=3.9.0; extra == "fast"
+Provides-Extra: test
+Requires-Dist: pytest>=7.0; extra == "test"
+Dynamic: license-file
+
+# gharc: GitHub Archive Stream-Processor
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Tests](https://github.com/aravpanwar/gharc/actions/workflows/test.yml/badge.svg)](https://github.com/aravpanwar/gharc/actions)
+[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![DOI](https://zenodo.org/badge/1112791047.svg)](https://doi.org/10.5281/zenodo.19814232)
+
+**Mine the GitHub Archive on a standard laptop.**
+
+`gharc` is a command-line tool and Python library that filters the [GitHub Archive](https://www.gharchive.org/) dataset on consumer hardware. Each hourly archive is streamed through memory, filtered against your criteria, and written out as Parquet or JSONL. Peak local storage stays bounded by a single in-flight download (about 150 MB) regardless of how long a window you process.
+
+---
+
+## Why gharc?
+
+The full GitHub Archive dataset exceeds petabytes in size. Traditional analysis requires either massive local storage or expensive cloud warehousing (BigQuery).
+
+`gharc` solves this by implementing a **Stream-and-Filter** architecture:
+1.  **Streaming:** Downloads each hourly archive (~60 to 150 MB compressed in 2024) to a temporary file.
+2.  **Filtering:** Extracts only events matching your criteria (e.g., specific repos or event types).
+3.  **Writing:** Streams matching events into a single **Parquet** or **JSONL** file via `pyarrow.ParquetWriter` for true append.
+4.  **Cleanup:** Deletes the temporary download immediately after, so disk usage never accumulates.
+
+**Ideal for:**
+- Academic research on Open Source Software (OSS).
+- Large scale data mining on consumer hardware.
+- Creating custom datasets for specific organizations or ecosystems.
+
+![Architecture: GHArchive HTTPS to thread pool to resumable download to temp file to streaming decode and filter to DataWriter to output file.](paper/figures/architecture.png)
+
+---
+
+## Key Features
+
+* **Zero-Storage Overhead:** Processes terabytes of data with a constant disk footprint of <100MB.
+* **Resumable Downloads:** Smart handling of network interruptions (common with residential internet) using HTTP Range requests.
+* **High Performance:**
+    * Parallel processing with thread pools.
+    * Optimized "Fast String Check" (zero-copy filtering) to skip irrelevant data.
+    * Optional `orjson` support for 3-5x faster parsing.
+* **Parquet Native:** Outputs columnar data ready for Pandas, Spark, or Polars, often reducing file size by 90% compared to JSON.
+
+---
+
+## Performance
+
+Measured on a Windows 11 laptop (12 logical cores, 15 GB RAM) over a typical residential connection. Reproducible scripts in [`benchmarks/`](benchmarks/).
+
+A six-hour window of GHArchive (2024-01-01 00:00 to 06:00 UTC), filtered to `apache/spark`:
+
+| Workers | Wall-clock | Hours/sec | Spark events | Peak RSS |
+|---|---|---|---|---|
+| 1 | 76.0 s | 0.079 | 14 | 94.2 MB |
+| 4 | 58.1 s | 0.103 | 14 | 106.7 MB |
+
+Both runs recovered the same events, so concurrency does not affect output. Peak RSS stays below 110 MB. The bottleneck on residential links is HTTPS download throughput rather than CPU; additional workers help up to a point and then saturate the connection.
+
+The same six-hour window comprises about 1.2 GB of compressed source on the GHArchive side, while the filtered Parquet output is 53 KB. That is a storage saving of roughly 22,000 to 1, and at no point does peak local disk exceed the size of a single in-flight temporary file (about 150 MB).
+
+---
+
+## Installation
+
+### Prerequisites
+- Python 3.8 or higher
+- `pip`
+
+### Install from Source
+```bash
+git clone https://github.com/aravpanwar/gharc.git
+cd gharc
+python3 -m venv venv
+source venv/bin/activate
+pip install -e .
+```
+
+### Optional Performance Boost
+
+For maximum speed, install with the `fast` extra. `gharc` detects and uses `orjson` automatically when available.
+
+```bash
+pip install -e ".[fast]"
+```
+
+---
+
+## Usage
+
+### Basic Command
+
+Download all activity for a specific repository over a one-day window.
+Note that `--end` is exclusive, so this covers all 24 hours of 2024-01-01.
+
+```bash
+gharc download \
+    --start 2024-01-01 \
+    --end 2024-01-02 \
+    --repos "apache/spark" \
+    --output spark_data.parquet
+
+```
+
+### Advanced Filtering
+
+Filter for multiple repositories and specific event types (e.g., only Pull Requests and Pushes).
+This covers all of June 2023 (June 1 inclusive through July 1 exclusive).
+
+```bash
+gharc download \
+    --start 2023-06-01 \
+    --end 2023-07-01 \
+    --repos "apache/spark, pandas-dev/pandas, pytorch/pytorch" \
+    --event-types "PullRequestEvent, PushEvent" \
+    --output oss_summer_2023.parquet \
+    --workers 4
+
+```
+
+### Arguments
+
+| Argument | Description | Example |
+| --- | --- | --- |
+| `--start` | Start date, inclusive (YYYY-MM-DD or YYYY-MM-DD-HH) | `2024-01-01` |
+| `--end` | End date, exclusive (YYYY-MM-DD or YYYY-MM-DD-HH) | `2024-02-01` |
+| `--repos` | Comma-separated list of repositories to keep | `apache/spark,tensorflow/tensorflow` |
+| `--event-types` | Comma-separated list of GHArchive event types | `WatchEvent,ForkEvent` |
+| `--output` | Output filename (`.parquet` or `.jsonl`) | `data.parquet` |
+| `--workers` | Number of parallel download threads (default: 4) | `8` |
+
+---
+
+## Resumable runs
+
+For long jobs, `gharc` keeps a small `<output>.state.json` next to the output file listing which hours it has already processed. If the run crashes, restarting the same command picks up where it left off rather than redoing completed hours. The state file is removed automatically when the run finishes cleanly.
+
+Resume support requires JSONL output. Parquet writers cannot append to a closed file, so for multi-hour runs use `--output run.jsonl` and convert to Parquet at the end:
+
+```bash
+gharc convert run.jsonl run.parquet
+```
+
+---
+
+## Python API
+
+The CLI is a thin wrapper around `gharc.process_range`, which you can call directly:
+
+```python
+from datetime import datetime
+import gharc
+
+gharc.setup_logging()
+gharc.process_range(
+    start=datetime(2024, 1, 1),
+    end=datetime(2024, 1, 2),
+    repos=["apache/spark"],
+    event_types=None,
+    output="spark_one_day.jsonl",
+    workers=4,
+)
+
+gharc.jsonl_to_parquet("spark_one_day.jsonl", "spark_one_day.parquet")
+```
+
+`__all__` in `gharc/__init__.py` lists the public surface (`process_range`, `jsonl_to_parquet`, `DataWriter`, `parse_date`, `date_range`, `get_url_for_time`, `setup_logging`, plus the filter helpers).
+
+---
+
+##  Automating Bulk Downloads
+
+For long date ranges, the included [`examples/orchestrator.py`](examples/orchestrator.py) script runs `gharc` month by month so each year produces one Parquet file per month rather than one giant output:
+
+```bash
+python examples/orchestrator.py \
+    --start 2023-01-01 \
+    --end 2024-01-01 \
+    --repos "apache/spark,pandas-dev/pandas" \
+    --output-dir ./gharc_out \
+    --workers 4
+```
+
+---
+
+## Repository Layout
+
+```
+gharc/
+├── src/gharc/        # Library + CLI entry point
+├── tests/            # pytest test suite
+├── benchmarks/       # Reproducible runs that back the performance claims
+├── examples/         # Driver scripts (e.g. month-by-month orchestrator)
+├── paper/            # paper.md, paper.bib, figures (the JOSS submission)
+└── CITATION.cff      # GitHub-detectable citation metadata
+```
+
+---
+
+## Contributing
+
+Contributions are welcome. Please read [CONTRIBUTING.md](CONTRIBUTING.md) for details on the process for submitting pull requests.
+
+**Running Tests:**
+
+```bash
+pip install -e ".[test]"
+pytest tests/
+```
+
+---
+
+## Citation
+
+The accompanying paper is at [`paper/paper.pdf`](paper/paper.pdf) and is rebuilt automatically on every push by the [Paper CI workflow](.github/workflows/paper.yml).
+
+If you use `gharc` in your research, please cite it using the metadata in `CITATION.cff` or as follows:
+
+```bibtex
+@software{gharc2026,
+  author = {Panwar, Arav},
+  title = {gharc: A stream-and-filter tool for the GitHub Archive on consumer hardware},
+  year = {2026},
+  url = {https://github.com/aravpanwar/gharc}
+}
+
+```
+
+---
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+Created by Arav Panwar
+[aravpanwar.com](https://www.aravpanwar.com)
+

gharc-0.1.0/README.md

@@ -0,0 +1,236 @@
+# gharc: GitHub Archive Stream-Processor
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Tests](https://github.com/aravpanwar/gharc/actions/workflows/test.yml/badge.svg)](https://github.com/aravpanwar/gharc/actions)
+[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![DOI](https://zenodo.org/badge/1112791047.svg)](https://doi.org/10.5281/zenodo.19814232)
+
+**Mine the GitHub Archive on a standard laptop.**
+
+`gharc` is a command-line tool and Python library that filters the [GitHub Archive](https://www.gharchive.org/) dataset on consumer hardware. Each hourly archive is streamed through memory, filtered against your criteria, and written out as Parquet or JSONL. Peak local storage stays bounded by a single in-flight download (about 150 MB) regardless of how long a window you process.
+
+---
+
+## Why gharc?
+
+The full GitHub Archive dataset exceeds petabytes in size. Traditional analysis requires either massive local storage or expensive cloud warehousing (BigQuery).
+
+`gharc` solves this by implementing a **Stream-and-Filter** architecture:
+1.  **Streaming:** Downloads each hourly archive (~60 to 150 MB compressed in 2024) to a temporary file.
+2.  **Filtering:** Extracts only events matching your criteria (e.g., specific repos or event types).
+3.  **Writing:** Streams matching events into a single **Parquet** or **JSONL** file via `pyarrow.ParquetWriter` for true append.
+4.  **Cleanup:** Deletes the temporary download immediately after, so disk usage never accumulates.
+
+**Ideal for:**
+- Academic research on Open Source Software (OSS).
+- Large scale data mining on consumer hardware.
+- Creating custom datasets for specific organizations or ecosystems.
+
+![Architecture: GHArchive HTTPS to thread pool to resumable download to temp file to streaming decode and filter to DataWriter to output file.](paper/figures/architecture.png)
+
+---
+
+## Key Features
+
+* **Zero-Storage Overhead:** Processes terabytes of data with a constant disk footprint of <100MB.
+* **Resumable Downloads:** Smart handling of network interruptions (common with residential internet) using HTTP Range requests.
+* **High Performance:**
+    * Parallel processing with thread pools.
+    * Optimized "Fast String Check" (zero-copy filtering) to skip irrelevant data.
+    * Optional `orjson` support for 3-5x faster parsing.
+* **Parquet Native:** Outputs columnar data ready for Pandas, Spark, or Polars, often reducing file size by 90% compared to JSON.
+
+---
+
+## Performance
+
+Measured on a Windows 11 laptop (12 logical cores, 15 GB RAM) over a typical residential connection. Reproducible scripts in [`benchmarks/`](benchmarks/).
+
+A six-hour window of GHArchive (2024-01-01 00:00 to 06:00 UTC), filtered to `apache/spark`:
+
+| Workers | Wall-clock | Hours/sec | Spark events | Peak RSS |
+|---|---|---|---|---|
+| 1 | 76.0 s | 0.079 | 14 | 94.2 MB |
+| 4 | 58.1 s | 0.103 | 14 | 106.7 MB |
+
+Both runs recovered the same events, so concurrency does not affect output. Peak RSS stays below 110 MB. The bottleneck on residential links is HTTPS download throughput rather than CPU; additional workers help up to a point and then saturate the connection.
+
+The same six-hour window comprises about 1.2 GB of compressed source on the GHArchive side, while the filtered Parquet output is 53 KB. That is a storage saving of roughly 22,000 to 1, and at no point does peak local disk exceed the size of a single in-flight temporary file (about 150 MB).
+
+---
+
+## Installation
+
+### Prerequisites
+- Python 3.8 or higher
+- `pip`
+
+### Install from Source
+```bash
+git clone https://github.com/aravpanwar/gharc.git
+cd gharc
+python3 -m venv venv
+source venv/bin/activate
+pip install -e .
+```
+
+### Optional Performance Boost
+
+For maximum speed, install with the `fast` extra. `gharc` detects and uses `orjson` automatically when available.
+
+```bash
+pip install -e ".[fast]"
+```
+
+---
+
+## Usage
+
+### Basic Command
+
+Download all activity for a specific repository over a one-day window.
+Note that `--end` is exclusive, so this covers all 24 hours of 2024-01-01.
+
+```bash
+gharc download \
+    --start 2024-01-01 \
+    --end 2024-01-02 \
+    --repos "apache/spark" \
+    --output spark_data.parquet
+
+```
+
+### Advanced Filtering
+
+Filter for multiple repositories and specific event types (e.g., only Pull Requests and Pushes).
+This covers all of June 2023 (June 1 inclusive through July 1 exclusive).
+
+```bash
+gharc download \
+    --start 2023-06-01 \
+    --end 2023-07-01 \
+    --repos "apache/spark, pandas-dev/pandas, pytorch/pytorch" \
+    --event-types "PullRequestEvent, PushEvent" \
+    --output oss_summer_2023.parquet \
+    --workers 4
+
+```
+
+### Arguments
+
+| Argument | Description | Example |
+| --- | --- | --- |
+| `--start` | Start date, inclusive (YYYY-MM-DD or YYYY-MM-DD-HH) | `2024-01-01` |
+| `--end` | End date, exclusive (YYYY-MM-DD or YYYY-MM-DD-HH) | `2024-02-01` |
+| `--repos` | Comma-separated list of repositories to keep | `apache/spark,tensorflow/tensorflow` |
+| `--event-types` | Comma-separated list of GHArchive event types | `WatchEvent,ForkEvent` |
+| `--output` | Output filename (`.parquet` or `.jsonl`) | `data.parquet` |
+| `--workers` | Number of parallel download threads (default: 4) | `8` |
+
+---
+
+## Resumable runs
+
+For long jobs, `gharc` keeps a small `<output>.state.json` next to the output file listing which hours it has already processed. If the run crashes, restarting the same command picks up where it left off rather than redoing completed hours. The state file is removed automatically when the run finishes cleanly.
+
+Resume support requires JSONL output. Parquet writers cannot append to a closed file, so for multi-hour runs use `--output run.jsonl` and convert to Parquet at the end:
+
+```bash
+gharc convert run.jsonl run.parquet
+```
+
+---
+
+## Python API
+
+The CLI is a thin wrapper around `gharc.process_range`, which you can call directly:
+
+```python
+from datetime import datetime
+import gharc
+
+gharc.setup_logging()
+gharc.process_range(
+    start=datetime(2024, 1, 1),
+    end=datetime(2024, 1, 2),
+    repos=["apache/spark"],
+    event_types=None,
+    output="spark_one_day.jsonl",
+    workers=4,
+)
+
+gharc.jsonl_to_parquet("spark_one_day.jsonl", "spark_one_day.parquet")
+```
+
+`__all__` in `gharc/__init__.py` lists the public surface (`process_range`, `jsonl_to_parquet`, `DataWriter`, `parse_date`, `date_range`, `get_url_for_time`, `setup_logging`, plus the filter helpers).
+
+---
+
+##  Automating Bulk Downloads
+
+For long date ranges, the included [`examples/orchestrator.py`](examples/orchestrator.py) script runs `gharc` month by month so each year produces one Parquet file per month rather than one giant output:
+
+```bash
+python examples/orchestrator.py \
+    --start 2023-01-01 \
+    --end 2024-01-01 \
+    --repos "apache/spark,pandas-dev/pandas" \
+    --output-dir ./gharc_out \
+    --workers 4
+```
+
+---
+
+## Repository Layout
+
+```
+gharc/
+├── src/gharc/        # Library + CLI entry point
+├── tests/            # pytest test suite
+├── benchmarks/       # Reproducible runs that back the performance claims
+├── examples/         # Driver scripts (e.g. month-by-month orchestrator)
+├── paper/            # paper.md, paper.bib, figures (the JOSS submission)
+└── CITATION.cff      # GitHub-detectable citation metadata
+```
+
+---
+
+## Contributing
+
+Contributions are welcome. Please read [CONTRIBUTING.md](CONTRIBUTING.md) for details on the process for submitting pull requests.
+
+**Running Tests:**
+
+```bash
+pip install -e ".[test]"
+pytest tests/
+```
+
+---
+
+## Citation
+
+The accompanying paper is at [`paper/paper.pdf`](paper/paper.pdf) and is rebuilt automatically on every push by the [Paper CI workflow](.github/workflows/paper.yml).
+
+If you use `gharc` in your research, please cite it using the metadata in `CITATION.cff` or as follows:
+
+```bibtex
+@software{gharc2026,
+  author = {Panwar, Arav},
+  title = {gharc: A stream-and-filter tool for the GitHub Archive on consumer hardware},
+  year = {2026},
+  url = {https://github.com/aravpanwar/gharc}
+}
+
+```
+
+---
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+Created by Arav Panwar
+[aravpanwar.com](https://www.aravpanwar.com)
+

gharc-0.1.0/examples/orchestrator.py

@@ -0,0 +1,69 @@
+"""Run gharc month-by-month over a long date range.
+
+Useful when you're working on a laptop and want one parquet file per month
+rather than one giant file. Edit the configuration at the top to fit your
+machine, then run: python examples/orchestrator.py
+"""
+import argparse
+import os
+import subprocess
+from datetime import datetime, timedelta
+
+
+def get_month_ranges(start, end):
+    current = start
+    while current < end:
+        next_month = (current.replace(day=1) + timedelta(days=32)).replace(day=1)
+        chunk_end = min(next_month, end)
+
+        s_str = current.strftime("%Y-%m-%d-%H")
+        e_str = chunk_end.strftime("%Y-%m-%d-%H")
+
+        yield current, chunk_end, s_str, e_str
+        current = next_month
+
+
+def main():
+    p = argparse.ArgumentParser(description="Run gharc month-by-month.")
+    p.add_argument("--start", required=True, help="Start date YYYY-MM-DD")
+    p.add_argument("--end", required=True, help="End date YYYY-MM-DD (exclusive)")
+    p.add_argument("--repos", required=True, help="Comma-separated repos")
+    p.add_argument("--output-dir", default="./gharc_out", help="Output directory")
+    p.add_argument("--workers", type=int, default=4)
+    args = p.parse_args()
+
+    start = datetime.strptime(args.start, "%Y-%m-%d")
+    end = datetime.strptime(args.end, "%Y-%m-%d")
+
+    os.makedirs(args.output_dir, exist_ok=True)
+    print(f"Batch run: {start.date()} to {end.date()}")
+    print(f"Output: {args.output_dir}")
+
+    for start_dt, _end_dt, s_str, e_str in get_month_ranges(start, end):
+        month_name = start_dt.strftime("%Y-%m")
+        output_file = os.path.join(args.output_dir, f"gharchive_{month_name}.parquet")
+
+        if os.path.exists(output_file):
+            print(f"Skipping {month_name} (file exists)")
+            continue
+
+        print(f"\nProcessing {month_name}...")
+
+        cmd = [
+            "gharc", "download",
+            "--start", s_str,
+            "--end", e_str,
+            "--repos", args.repos,
+            "--output", output_file,
+            "--workers", str(args.workers),
+        ]
+
+        try:
+            subprocess.run(cmd, check=True)
+            print(f"Finished {month_name}")
+        except subprocess.CalledProcessError:
+            print(f"Error processing {month_name}, continuing")
+
+
+if __name__ == "__main__":
+    main()

gharc-0.1.0/paper/figures/architecture.mmd

@@ -0,0 +1,9 @@
+%% gharc stream-and-filter architecture.
+%% Render with mermaid-cli: mmdc -i architecture.mmd -o architecture.png -b transparent
+flowchart LR
+    A["GHArchive<br/>HTTPS (per hour)"] --> B["Thread pool<br/>(N workers)"]
+    B --> C["Resumable<br/>download"]
+    C --> D["Temp .json.gz<br/>(deleted after)"]
+    D --> E["Streaming<br/>decode + filter"]
+    E --> F["DataWriter<br/>(ParquetWriter / JSONL)"]
+    F --> G["Output file<br/>on disk"]