litsync 0.0.2__tar.gz

litsync-0.0.2/PKG-INFO

@@ -0,0 +1,125 @@
+Metadata-Version: 2.4
+Name: litsync
+Version: 0.0.2
+Summary: Incremental mirror for PubMed, PMC, FDA, and ClinicalTrials.gov
+Author: Literature Downloader Contributors
+Author-email: Rahul Brahma <rahul.brahma@uni-greifswald.de>
+License: MIT
+Project-URL: Homepage, https://github.com/takshan/litsync
+Project-URL: Repository, https://github.com/takshan/litsync
+Keywords: pubmed,pmc,fda,clinicaltrials,biomedical,mirror
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: requests>=2.31
+Requires-Dist: rich>=13.0
+
+# litsync — incremental PubMed + PMC + FDA + ClinicalTrials.gov mirror
+
+A modern, daily-runnable CLI for mirroring bulk biomedical datasets. It tracks every
+file in a SQLite state DB so re-runs do the minimum work: already-verified immutable
+files are skipped with no network request beyond the directory/manifest listing.
+
+## Install
+
+```bash
+pip install -e .
+```
+
+Or use the Makefile:
+
+```bash
+make install
+make dev
+```
+
+## Quick start
+
+```bash
+litsync --data-root /data/literature --email you@institute.org
+```
+
+Common options:
+
+```bash
+litsync --data-root /data/literature --email you@institute.org \
+  --sources pubmed pmc fda clinicaltrials \
+  --fda-endpoints drug/event drug/label
+```
+
+```bash
+--sources pubmed pmc fda clinicaltrials   # which corpora (default: all four)
+--fda-endpoints drug/event drug/label     # default: all openFDA endpoints
+--pmc-groups oa_comm oa_noncomm oa_other
+--pmc-formats xml txt                     # default: xml
+--workers 4                               # concurrent downloads (keep modest; be polite)
+--dry-run                                 # plan only, download nothing
+--reverify                                # re-download local files (integrity audit)
+--prune                                   # delete local files no longer on the server
+--count-articles                          # count articles in already-downloaded files (no network)
+--no-rich                                 # disable Rich progress bars / tables
+```
+
+## On-disk layout
+
+```
+/data/literature/
+  pubmed/baseline/                    pubmed26nXXXX.xml.gz (+ .md5 verified)
+  pubmed/updatefiles/                 daily citation deltas
+  pmc/oa_bulk/<group>/<fmt>/          baseline + dated incremental .tar.gz
+  pmc/oa_file_list.csv                PMCID <-> PMID id map
+  fda/<category>/<endpoint>/          openFDA bulk snapshot zips + extracted JSON
+  clinicaltrials/ctg-public-xml.zip   ClinicalTrials.gov full XML dump
+  clinicaltrials/ctg-public-xml/      extracted study XML files
+  _state/state.sqlite                 file ledger (status, size, mtime, md5, etag, attempts)
+  _state/logs/                        dated run logs
+  _state/litsync.lock                 run lock (prevents overlapping cron runs)
+```
+
+## Cron (daily 02:30)
+
+```cron
+30 2 * * *  /path/to/venv/bin/litsync --data-root /data/literature --email you@institute.org >> /data/literature/_state/cron.log 2>&1
+```
+
+## Extract corpus to sharded JSONL
+
+```bash
+litsync-extract --data-root /data/literature --out /data/corpus \
+  --sources pubmed pmc fda clinicaltrials
+```
+
+Or with Make:
+
+```bash
+make extract DATA_ROOT=/data/literature CORPUS_OUT=/data/corpus
+make extract-test DATA_ROOT=/data/literature
+```
+
+## Integrity model
+
+- **PubMed**: every `.xml.gz` is verified against its NCBI `.md5` sidecar.
+- **PMC**: bulk packages have no md5 sidecar, so they are verified by `Content-Length`
+  and an `ETag` is recorded for change detection.
+- **openFDA / ClinicalTrials.gov**: these sources publish full snapshots. The downloader
+  detects changed snapshots via `ETag` / `Last-Modified` / `Content-Length` and only
+  re-downloads when the snapshot changes. When a snapshot changes it is extracted
+  again next to the zip file.
+- Downloads are atomic (`.part` -> rename) and resumable via HTTP Range.
+- Exit code is non-zero if any file failed, so cron/monitoring can alert.
+
+## Notes on sources
+
+- **openFDA** bulk data is zipped JSON. The manifest is fetched from `https://api.fda.gov/download.json`.
+  Each endpoint partition becomes one downloaded/extracted unit.
+- **ClinicalTrials.gov** bulk data is the full public XML dump from
+  `https://clinicaltrials.gov/api/legacy/public-xml?format=zip`. One XML file per study.
+- Both sources are snapshots, not daily deltas. Daily runs are still cheap because unchanged
+  snapshots are skipped; changed snapshots are replaced in full.
+

litsync-0.0.2/README.md

@@ -0,0 +1,103 @@
+# litsync — incremental PubMed + PMC + FDA + ClinicalTrials.gov mirror
+
+A modern, daily-runnable CLI for mirroring bulk biomedical datasets. It tracks every
+file in a SQLite state DB so re-runs do the minimum work: already-verified immutable
+files are skipped with no network request beyond the directory/manifest listing.
+
+## Install
+
+```bash
+pip install -e .
+```
+
+Or use the Makefile:
+
+```bash
+make install
+make dev
+```
+
+## Quick start
+
+```bash
+litsync --data-root /data/literature --email you@institute.org
+```
+
+Common options:
+
+```bash
+litsync --data-root /data/literature --email you@institute.org \
+  --sources pubmed pmc fda clinicaltrials \
+  --fda-endpoints drug/event drug/label
+```
+
+```bash
+--sources pubmed pmc fda clinicaltrials   # which corpora (default: all four)
+--fda-endpoints drug/event drug/label     # default: all openFDA endpoints
+--pmc-groups oa_comm oa_noncomm oa_other
+--pmc-formats xml txt                     # default: xml
+--workers 4                               # concurrent downloads (keep modest; be polite)
+--dry-run                                 # plan only, download nothing
+--reverify                                # re-download local files (integrity audit)
+--prune                                   # delete local files no longer on the server
+--count-articles                          # count articles in already-downloaded files (no network)
+--no-rich                                 # disable Rich progress bars / tables
+```
+
+## On-disk layout
+
+```
+/data/literature/
+  pubmed/baseline/                    pubmed26nXXXX.xml.gz (+ .md5 verified)
+  pubmed/updatefiles/                 daily citation deltas
+  pmc/oa_bulk/<group>/<fmt>/          baseline + dated incremental .tar.gz
+  pmc/oa_file_list.csv                PMCID <-> PMID id map
+  fda/<category>/<endpoint>/          openFDA bulk snapshot zips + extracted JSON
+  clinicaltrials/ctg-public-xml.zip   ClinicalTrials.gov full XML dump
+  clinicaltrials/ctg-public-xml/      extracted study XML files
+  _state/state.sqlite                 file ledger (status, size, mtime, md5, etag, attempts)
+  _state/logs/                        dated run logs
+  _state/litsync.lock                 run lock (prevents overlapping cron runs)
+```
+
+## Cron (daily 02:30)
+
+```cron
+30 2 * * *  /path/to/venv/bin/litsync --data-root /data/literature --email you@institute.org >> /data/literature/_state/cron.log 2>&1
+```
+
+## Extract corpus to sharded JSONL
+
+```bash
+litsync-extract --data-root /data/literature --out /data/corpus \
+  --sources pubmed pmc fda clinicaltrials
+```
+
+Or with Make:
+
+```bash
+make extract DATA_ROOT=/data/literature CORPUS_OUT=/data/corpus
+make extract-test DATA_ROOT=/data/literature
+```
+
+## Integrity model
+
+- **PubMed**: every `.xml.gz` is verified against its NCBI `.md5` sidecar.
+- **PMC**: bulk packages have no md5 sidecar, so they are verified by `Content-Length`
+  and an `ETag` is recorded for change detection.
+- **openFDA / ClinicalTrials.gov**: these sources publish full snapshots. The downloader
+  detects changed snapshots via `ETag` / `Last-Modified` / `Content-Length` and only
+  re-downloads when the snapshot changes. When a snapshot changes it is extracted
+  again next to the zip file.
+- Downloads are atomic (`.part` -> rename) and resumable via HTTP Range.
+- Exit code is non-zero if any file failed, so cron/monitoring can alert.
+
+## Notes on sources
+
+- **openFDA** bulk data is zipped JSON. The manifest is fetched from `https://api.fda.gov/download.json`.
+  Each endpoint partition becomes one downloaded/extracted unit.
+- **ClinicalTrials.gov** bulk data is the full public XML dump from
+  `https://clinicaltrials.gov/api/legacy/public-xml?format=zip`. One XML file per study.
+- Both sources are snapshots, not daily deltas. Daily runs are still cheap because unchanged
+  snapshots are skipped; changed snapshots are replaced in full.
+

litsync-0.0.2/pyproject.toml

@@ -0,0 +1,40 @@
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "litsync"
+version = "0.0.2"
+description = "Incremental mirror for PubMed, PMC, FDA, and ClinicalTrials.gov"
+readme = "README.md"
+requires-python = ">=3.10"
+license = {text = "MIT"}
+authors = [
+    {name = "Rahul Brahma", email = "rahul.brahma@uni-greifswald.de"},
+    {name = "Literature Downloader Contributors"},
+]
+keywords = ["pubmed", "pmc", "fda", "clinicaltrials", "biomedical", "mirror"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Science/Research",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+]
+dependencies = [
+    "requests>=2.31",
+    "rich>=13.0",
+]
+
+[project.scripts]
+litsync = "litsync.cli:main"
+litsync-extract = "litsync.cli:extract_command"
+
+[project.urls]
+Homepage = "https://github.com/takshan/litsync"
+Repository = "https://github.com/takshan/litsync"
+
+[tool.setuptools.packages.find]
+where = ["src"]

litsync-0.0.2/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

litsync-0.0.2/src/litsync/__init__.py

@@ -0,0 +1,3 @@
+"""litsync — incremental mirror for PubMed, PMC, FDA, and ClinicalTrials.gov."""
+
+__version__ = "0.0.2"

litsync-0.0.2/src/litsync/__main__.py

@@ -0,0 +1,3 @@
+from litsync.cli import main
+
+raise SystemExit(main())

litsync-0.0.2/src/litsync/cli.py

@@ -0,0 +1,143 @@
+from __future__ import annotations
+
+import argparse
+import logging
+import os
+import sys
+from pathlib import Path
+from typing import Optional
+
+from rich.logging import RichHandler
+
+from litsync.config import Config
+from litsync.extract import run_extraction
+from litsync.sync import Syncer
+from litsync.ui import RichUI, UI
+from litsync.utils import run_lock
+
+
+LOG = logging.getLogger("litsync")
+
+
+def setup_logging(log_dir: Path, verbose: bool = False) -> None:
+    log_dir.mkdir(parents=True, exist_ok=True)
+    logfile = log_dir / f"litsync_{__import__('datetime').date.today().isoformat()}.log"
+
+    file_handler = logging.FileHandler(logfile)
+    file_handler.setFormatter(logging.Formatter(
+        "%(asctime)s %(levelname)-7s %(message)s"
+    ))
+
+    rich_handler = RichHandler(rich_tracebacks=True, show_path=False)
+    rich_handler.setFormatter(logging.Formatter("%(message)s"))
+
+    logging.basicConfig(
+        level=logging.DEBUG if verbose else logging.INFO,
+        handlers=[rich_handler, file_handler],
+    )
+
+
+def parse_args(argv: Optional[list[str]] = None) -> Config:
+    ap = argparse.ArgumentParser(
+        prog="litsync",
+        description="Incremental mirror for PubMed, PMC, FDA, and ClinicalTrials.gov",
+    )
+    ap.add_argument("--data-root", required=True, type=Path,
+                    help="root directory for the local mirror")
+    ap.add_argument("--email", default=os.environ.get("NCBI_EMAIL", ""),
+                    help="contact email (sent in User-Agent)")
+    ap.add_argument("--sources", nargs="+", default=["pubmed", "pmc"],
+                    choices=["pubmed", "pmc", "fda", "clinicaltrials"])
+    ap.add_argument("--pmc-groups", nargs="+", default=["oa_comm", "oa_noncomm", "oa_other"],
+                    choices=["oa_comm", "oa_noncomm", "oa_other"])
+    ap.add_argument("--pmc-formats", nargs="+", default=["xml"],
+                    choices=["xml", "txt"])
+    ap.add_argument("--fda-endpoints", nargs="+", default=None,
+                    help="openFDA endpoints to mirror, e.g. 'drug/event drug/label'; default: all")
+    ap.add_argument("--workers", type=int, default=4)
+    ap.add_argument("--max-retries", type=int, default=5)
+    ap.add_argument("--timeout", type=int, default=60)
+    ap.add_argument("--dry-run", action="store_true",
+                    help="plan only, download nothing")
+    ap.add_argument("--reverify", action="store_true",
+                    help="re-download already-downloaded files to verify integrity")
+    ap.add_argument("--prune", action="store_true",
+                    help="delete local files no longer present on the server")
+    ap.add_argument("--count-articles", action="store_true",
+                    help="count articles in already-downloaded local files and exit "
+                         "(no network); backfills the per-source article totals")
+    ap.add_argument("--no-rich", action="store_true",
+                    help="disable Rich progress bars and use plain text output")
+    ap.add_argument("--verbose", "-v", action="store_true",
+                    help="enable debug logging")
+    a = ap.parse_args(argv)
+    if not a.email:
+        ap.error("provide --email or set NCBI_EMAIL")
+    return Config(
+        data_root=a.data_root.expanduser().resolve(),
+        email=a.email,
+        sources=tuple(a.sources),
+        pmc_groups=tuple(a.pmc_groups),
+        pmc_formats=tuple(a.pmc_formats),
+        workers=max(1, a.workers),
+        max_retries=a.max_retries,
+        timeout=a.timeout,
+        dry_run=a.dry_run,
+        reverify=a.reverify,
+        prune=a.prune,
+        count_articles=a.count_articles,
+        fda_endpoints=tuple(a.fda_endpoints) if a.fda_endpoints else None,
+    )
+
+
+def sync_command(cfg: Config, ui: UI) -> int:
+    cfg.data_root.mkdir(parents=True, exist_ok=True)
+    setup_logging(cfg.log_dir, verbose=False)
+    LOG.info("litsync starting | root=%s sources=%s dry_run=%s count_articles=%s",
+             cfg.data_root, cfg.sources, cfg.dry_run, cfg.count_articles)
+    with run_lock(cfg.lock_path):
+        syncer = Syncer(cfg, ui)
+        if cfg.count_articles:
+            return syncer.backfill_counts()
+        return syncer.run()
+
+
+def extract_command(argv: Optional[list[str]] = None) -> int:
+    ap = argparse.ArgumentParser(
+        prog="litsync-extract",
+        description="Extract litsync mirror into sharded JSONL",
+    )
+    ap.add_argument("--data-root", type=Path, default=Path("./data/literature"))
+    ap.add_argument("--out", type=Path, default=Path("./data/corpus"))
+    ap.add_argument("--sources", nargs="+", default=["pubmed", "pmc"],
+                    choices=["pubmed", "pmc", "fda", "clinicaltrials"])
+    ap.add_argument("--shard-size-mb", type=int, default=256)
+    ap.add_argument("--limit", type=int, default=None)
+    ap.add_argument("--verbose", "-v", action="store_true")
+    args = ap.parse_args(argv)
+
+    logging.basicConfig(
+        level=logging.DEBUG if args.verbose else logging.INFO,
+        format="%(asctime)s %(levelname)-7s %(message)s",
+    )
+    run_extraction(
+        args.data_root.expanduser().resolve(),
+        args.out.expanduser().resolve(),
+        args.sources,
+        args.shard_size_mb,
+        args.limit,
+    )
+    return 0
+
+
+def main(argv: Optional[list[str]] = None) -> int:
+    # Pre-scan argv for --no-rich so we can choose the UI before argparse runs.
+    raw = argv if argv is not None else sys.argv[1:]
+    use_plain = "--no-rich" in raw
+    cfg = parse_args(argv)
+    ui = UI() if use_plain else RichUI()
+    return sync_command(cfg, ui)
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

litsync-0.0.2/src/litsync/config.py

@@ -0,0 +1,42 @@
+from __future__ import annotations
+
+import dataclasses
+from pathlib import Path
+from typing import Optional
+
+
+DEFAULT_WORKERS = 4
+
+
+@dataclasses.dataclass
+class Config:
+    data_root: Path
+    email: str
+    sources: tuple[str, ...]
+    pmc_groups: tuple[str, ...]
+    pmc_formats: tuple[str, ...]
+    workers: int = DEFAULT_WORKERS
+    max_retries: int = 5
+    backoff_base: float = 2.0
+    timeout: int = 60
+    dry_run: bool = False
+    reverify: bool = False
+    prune: bool = False
+    count_articles: bool = False
+    fda_endpoints: Optional[tuple[str, ...]] = None
+
+    @property
+    def state_dir(self) -> Path:
+        return self.data_root / "_state"
+
+    @property
+    def db_path(self) -> Path:
+        return self.state_dir / "state.sqlite"
+
+    @property
+    def lock_path(self) -> Path:
+        return self.state_dir / "litsync.lock"
+
+    @property
+    def log_dir(self) -> Path:
+        return self.state_dir / "logs"