litsync 0.0.2__py3-none-any.whl

litsync-0.0.2.dist-info/METADATA

@@ -0,0 +1,125 @@
+Metadata-Version: 2.4
+Name: litsync
+Version: 0.0.2
+Summary: Incremental mirror for PubMed, PMC, FDA, and ClinicalTrials.gov
+Author: Literature Downloader Contributors
+Author-email: Rahul Brahma <rahul.brahma@uni-greifswald.de>
+License: MIT
+Project-URL: Homepage, https://github.com/takshan/litsync
+Project-URL: Repository, https://github.com/takshan/litsync
+Keywords: pubmed,pmc,fda,clinicaltrials,biomedical,mirror
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: requests>=2.31
+Requires-Dist: rich>=13.0
+
+# litsync — incremental PubMed + PMC + FDA + ClinicalTrials.gov mirror
+
+A modern, daily-runnable CLI for mirroring bulk biomedical datasets. It tracks every
+file in a SQLite state DB so re-runs do the minimum work: already-verified immutable
+files are skipped with no network request beyond the directory/manifest listing.
+
+## Install
+
+```bash
+pip install -e .
+```
+
+Or use the Makefile:
+
+```bash
+make install
+make dev
+```
+
+## Quick start
+
+```bash
+litsync --data-root /data/literature --email you@institute.org
+```
+
+Common options:
+
+```bash
+litsync --data-root /data/literature --email you@institute.org \
+  --sources pubmed pmc fda clinicaltrials \
+  --fda-endpoints drug/event drug/label
+```
+
+```bash
+--sources pubmed pmc fda clinicaltrials   # which corpora (default: all four)
+--fda-endpoints drug/event drug/label     # default: all openFDA endpoints
+--pmc-groups oa_comm oa_noncomm oa_other
+--pmc-formats xml txt                     # default: xml
+--workers 4                               # concurrent downloads (keep modest; be polite)
+--dry-run                                 # plan only, download nothing
+--reverify                                # re-download local files (integrity audit)
+--prune                                   # delete local files no longer on the server
+--count-articles                          # count articles in already-downloaded files (no network)
+--no-rich                                 # disable Rich progress bars / tables
+```
+
+## On-disk layout
+
+```
+/data/literature/
+  pubmed/baseline/                    pubmed26nXXXX.xml.gz (+ .md5 verified)
+  pubmed/updatefiles/                 daily citation deltas
+  pmc/oa_bulk/<group>/<fmt>/          baseline + dated incremental .tar.gz
+  pmc/oa_file_list.csv                PMCID <-> PMID id map
+  fda/<category>/<endpoint>/          openFDA bulk snapshot zips + extracted JSON
+  clinicaltrials/ctg-public-xml.zip   ClinicalTrials.gov full XML dump
+  clinicaltrials/ctg-public-xml/      extracted study XML files
+  _state/state.sqlite                 file ledger (status, size, mtime, md5, etag, attempts)
+  _state/logs/                        dated run logs
+  _state/litsync.lock                 run lock (prevents overlapping cron runs)
+```
+
+## Cron (daily 02:30)
+
+```cron
+30 2 * * *  /path/to/venv/bin/litsync --data-root /data/literature --email you@institute.org >> /data/literature/_state/cron.log 2>&1
+```
+
+## Extract corpus to sharded JSONL
+
+```bash
+litsync-extract --data-root /data/literature --out /data/corpus \
+  --sources pubmed pmc fda clinicaltrials
+```
+
+Or with Make:
+
+```bash
+make extract DATA_ROOT=/data/literature CORPUS_OUT=/data/corpus
+make extract-test DATA_ROOT=/data/literature
+```
+
+## Integrity model
+
+- **PubMed**: every `.xml.gz` is verified against its NCBI `.md5` sidecar.
+- **PMC**: bulk packages have no md5 sidecar, so they are verified by `Content-Length`
+  and an `ETag` is recorded for change detection.
+- **openFDA / ClinicalTrials.gov**: these sources publish full snapshots. The downloader
+  detects changed snapshots via `ETag` / `Last-Modified` / `Content-Length` and only
+  re-downloads when the snapshot changes. When a snapshot changes it is extracted
+  again next to the zip file.
+- Downloads are atomic (`.part` -> rename) and resumable via HTTP Range.
+- Exit code is non-zero if any file failed, so cron/monitoring can alert.
+
+## Notes on sources
+
+- **openFDA** bulk data is zipped JSON. The manifest is fetched from `https://api.fda.gov/download.json`.
+  Each endpoint partition becomes one downloaded/extracted unit.
+- **ClinicalTrials.gov** bulk data is the full public XML dump from
+  `https://clinicaltrials.gov/api/legacy/public-xml?format=zip`. One XML file per study.
+- Both sources are snapshots, not daily deltas. Daily runs are still cheap because unchanged
+  snapshots are skipped; changed snapshots are replaced in full.
+

litsync-0.0.2.dist-info/RECORD

@@ -0,0 +1,20 @@
+litsync/__init__.py,sha256=6fgq3evCz5TI_zaDMgQlEtNAIIM4LZZI2gQg80nvirk,106
+litsync/__main__.py,sha256=b3d0VEGsuMqtiRbeWxB4gBfpjDz-FGvBB-1JG4BCT0g,55
+litsync/cli.py,sha256=J77IzGia27nJHkqJt32fMQsiEIgx1S0heKQq7oVGHHg,5617
+litsync/config.py,sha256=C0zxfSByjy19pMQHRkLoaEXXIidoCw652EneywQ2qDY,934
+litsync/extract.py,sha256=1Adrno9byh1Nwzbi9GiLEW3eiIC2jz5htQKVEEBNsX4,14034
+litsync/http.py,sha256=ZQtciMiLsm6JORCeXCy4zFW4lT7PytfMD6FPnZIqbFY,4458
+litsync/state.py,sha256=ZbAAyIJ0O3i-AZ75JsdzYs7eZ20fN_wTCNfqUXd82Iw,5536
+litsync/sync.py,sha256=ZWBp6QkyI-z5Yo9kYoWhFkK43A42o2u45EIcALVLZB8,13439
+litsync/ui.py,sha256=Gq9kFgDfbqbvY0sMwK-ailojGzZ3YqIx99VP3yk7kvo,8124
+litsync/utils.py,sha256=c61EOnr2yjZFZQkW56l6mKjPGjILcsQEhf9-hpKzl3o,3788
+litsync/sources/__init__.py,sha256=WGjjAqJTH38kzIKCW_sz8-LjeqN-Rchz_ir0lDDitSQ,312
+litsync/sources/clinicaltrials.py,sha256=4ApojOWxTJqrMmZYrD7xpivLdXD3Ei6yZDmDxTlJgtk,804
+litsync/sources/fda.py,sha256=bVXsnMedMjfpEZBvjnVfWITJZ6-OoSvE1gPsgzG4gAg,1689
+litsync/sources/pmc.py,sha256=SYNrthBskxW-wryVmLAQR7HbBQRhXbsgvLtPN1KIahI,1719
+litsync/sources/pubmed.py,sha256=Dd0up_7G0DReHTSQHa8GD3g7avsdMYPjUcbB13vs2oc,1157
+litsync-0.0.2.dist-info/METADATA,sha256=HwndfiNJ4X2XTn8tCgPJVNbjjUQ3IPkBfUwhLDYZ1g8,4756
+litsync-0.0.2.dist-info/WHEEL,sha256=aeYiig01lYGDzBgS8HxWXOg3uV61G9ijOsup-k9o1sk,91
+litsync-0.0.2.dist-info/entry_points.txt,sha256=ymrMbrSu3BMLjArnxnjrxAs8f3-HTPRsGY6Yzo5JO7E,91
+litsync-0.0.2.dist-info/top_level.txt,sha256=YaYYXYOLc9hpIJY40kKrD22kHKBbxKN8T_XE6oJfyLw,8
+litsync-0.0.2.dist-info/RECORD,,

litsync-0.0.2.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (82.0.1)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

litsync-0.0.2.dist-info/entry_points.txt

@@ -0,0 +1,3 @@
+[console_scripts]
+litsync = litsync.cli:main
+litsync-extract = litsync.cli:extract_command

litsync-0.0.2.dist-info/top_level.txt

@@ -0,0 +1 @@
+litsync