pmcfetch 0.1.0__py3-none-any.whl

pmcfetch-0.1.0.dist-info/METADATA

@@ -0,0 +1,186 @@
+Metadata-Version: 2.4
+Name: pmcfetch
+Version: 0.1.0
+Summary: Targeted PDF retrieval from the PMC Open Access Subset on AWS. Zero dependencies.
+Author: Kavach Shah
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/kms2229/pmcfetch
+Project-URL: Issues, https://github.com/kms2229/pmcfetch/issues
+Keywords: pubmed-central,pmc,open-access,bulk-download,bioinformatics
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Classifier: Topic :: Text Processing :: Indexing
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Dynamic: license-file
+
+# pmcfetch
+
+Targeted PDF retrieval from the **PMC Open Access Subset**, with resume, retries,
+atomic writes, and namespace-aware batch layout. Zero dependencies.
+
+```bash
+pip install pmcfetch
+pmcfetch download --corpus corpus.json --out pdfs/ --workers 16 --rps 0
+```
+
+Requires Python 3.10+. Nothing outside the standard library.
+
+## Why
+
+NLM publishes every PMC Open Access article version in the public bucket
+`s3://pmc-oa-opendata` — anonymous HTTPS, no AWS account, no credentials, no rate
+limit, and the access route PMC sanctions for systematic retrieval.
+
+But `aws s3 sync` is all-or-nothing. There's no `--from-id-list`, so pulling the
+few thousand articles you care about means pulling all ~8 million. And fetching
+them one by one with `aws s3 cp` requires knowing things that aren't obvious:
+
+- **Which version?** Articles live under `PMC10007932.1/`, `.2/`, sometimes `.10/`.
+  You can't know without listing first — and the sort must be numeric, not
+  lexicographic.
+- **Which PDF?** `PMC10007932.1/` also contains `jgo-14-01-302-coif.pdf`,
+  `-dss.pdf`, `-prf.pdf`. Glob `*.pdf` and you get the wrong file.
+- **Not every article has one.** Some licences permit XML and plain-text
+  redistribution but not the PDF.
+
+`pmcfetch` handles all three, then adds the batch machinery you'd otherwise
+rewrite: resume, atomic writes, concurrency, pacing, and structured failure logs.
+
+## Python API
+
+```python
+from pmcfetch import S3OpenAccessClient, load_records, download_many, FailureLog
+
+records = load_records("corpus.json").for_namespace("cardiology")
+
+with S3OpenAccessClient() as client:
+    report = download_many(
+        records, "pdfs/",
+        client=client,
+        workers=16,
+        failure_log=FailureLog("failed.txt"),
+    )
+
+print(report.as_dict())        # {'total': 3139, 'downloaded': 3104, 'skipped': 0, 'failed': 35}
+for r in report.failures:
+    print(r.pmcid, r.reason)
+```
+
+Single article, plus metadata:
+
+```python
+with S3OpenAccessClient() as c:
+    meta = c.metadata("10007932")
+    # {'pmcid': 'PMC10007932', 'doi': '10.21037/jgo-23-76',
+    #  'license_code': 'CC BY-NC-ND', 'is_retracted': False, ...}
+    if not meta["is_retracted"]:
+        pdf = c.fetch_pdf("10007932")   # accepts 10007932 / pmc10007932 / PMC10007932
+```
+
+`license_code` and `is_retracted` come free with every article — filter on licence
+terms and drop retracted papers before they reach your corpus.
+
+## CLI
+
+```bash
+# what's in a corpus file?
+pmcfetch namespaces --corpus corpus.json
+
+# pull one namespace
+pmcfetch download --corpus corpus.json --out pdfs/ --namespace cardiology --workers 16 --rps 0
+
+# dry run, no requests
+pmcfetch download --corpus corpus.json --out pdfs/ --limit 20 --dry-run
+
+# retry failures — the log doubles as an input list
+pmcfetch download --corpus pdfs/failed.txt --out pdfs/ --flat
+```
+
+Exit code is `1` when any article failed, `0` otherwise — so a batch job fails
+loudly instead of silently producing a short corpus. Bad arguments or an
+unreadable corpus file exit `2`, and Ctrl-C exits `130`.
+
+S3 has no rate limit, so `--rps 0` switches pacing off entirely (`--jitter` then
+has no effect) and `--workers` can go wide; throughput is bound by your bandwidth.
+
+## Deployment
+
+- **Azure** — VM or Container Instances with Blob persistence, plus cost notes.
+  See [`docs/azure.md`](https://github.com/kms2229/pmcfetch/blob/main/docs/azure.md).
+- **HPC / SGE** — `--task-id` (defaulting to `$SGE_TASK_ID`) selects a namespace
+  by 1-based index. See
+  [`examples/sge/download_array.qsub`](https://github.com/kms2229/pmcfetch/blob/main/examples/sge/download_array.qsub).
+
+Nothing in the library is platform-specific; those are examples.
+
+## Corpus formats
+
+`load_records()` normalises keys (`10007932`, `pmc10007932`, `PMC10007932`) and
+accepts:
+
+| Input | Shape |
+|---|---|
+| assignment JSON | `{"<id>": {"metadata": {"namespace": "cardiology"}}}` |
+| full-text dump | `{"<id>": "<full text>"}` |
+| extraction output | `{"pmc10007932": {"sections": [...]}}` |
+| plain array | `["PMC1", "2", ...]` |
+| `.txt` | newline-delimited ids, including pmcfetch's own failure logs |
+
+Keys that aren't PMC ids land in `LoadResult.skipped` rather than raising, so one
+bad key can't abort a 20k-article batch.
+
+## Behaviour worth knowing
+
+- **Resume** checks size *and* `%PDF` magic, so a half-written file from a killed
+  job is re-fetched rather than treated as complete. Downloads are held to the
+  same check before they are written, so every file on disk is one the next run
+  skips.
+- **Writes are atomic** — `.part` + `fsync` + rename; a failed write removes its
+  `.part`.
+- **Retries** cover 429/5xx and transport errors with exponential backoff and full
+  jitter, stretched to any `Retry-After` the server sends (no wait exceeds
+  `backoff_cap`, 15 s by default). 404s and non-PDF bodies are not retried.
+- **Rate limiting** is process-wide, so `--workers` and `--rps` are independent.
+- **Ctrl-C stops promptly** — queued articles are cancelled; only those already
+  in flight finish.
+- **Namespaces become one folder name** — path separators are replaced, so a
+  corpus file cannot write outside `--out`.
+- **Articles with no PDF** raise `PDFNotAvailableError`; ones outside the subset
+  raise `ArticleNotInOASubsetError`. Both land in the failure log with a reason.
+- **Any object implementing `fetch_pdf(pmcid) -> bytes`** satisfies the
+  `PdfSource` protocol and works with `download_many`.
+
+## Tests
+
+```bash
+pip install -e '.[dev]'
+python -m pytest tests/ -q
+```
+
+No network required — the S3 layer is exercised against scripted listings and
+responses: multi-version and paginated listings, supplementary-PDF
+disambiguation, 429 storms and `Retry-After`, 404s, truncated resumes, and
+concurrent and interrupted batches.
+
+## Attribution
+
+Data comes from the
+[NIH NLM NCBI PubMed Central Article Datasets](https://registry.opendata.aws/ncbi-pmc/)
+on AWS Open Data. If you redistribute anything built from it, acknowledge NLM as
+the source and don't imply NLM endorsement — see the
+[bucket README](https://pmc-oa-opendata.s3.amazonaws.com/README.txt).

pmcfetch-0.1.0.dist-info/RECORD

@@ -0,0 +1,16 @@
+pmcfetch/__init__.py,sha256=iXn10-NB8MHoi5nXbmNcnLqwV26oHGB23PRt7mFOU5g,2743
+pmcfetch/batch.py,sha256=vM5NqtnPvr3yKxPKG7rRvn9NoLEyiCgqNITDWcIbZxo,8934
+pmcfetch/cli.py,sha256=JxTfix3EFzboZPITdfxwV8OZ7WrIH1FAojlTdq6JPvI,8414
+pmcfetch/errors.py,sha256=dp580Az12sCKHMn91PauBnwLAyHsayP9vMEeUIsobbU,1700
+pmcfetch/ids.py,sha256=Kd5LxgAT1PgEr1-ag2LKeaEWASpahYdoVFfzFKJVIZE,1330
+pmcfetch/manifest.py,sha256=YQ9mCYbj2SxOBQylTWyhHnHwR-rVOweP6BYfUUg2YDg,4856
+pmcfetch/pdf.py,sha256=u6Mx1HBuY63uMBSBq94-dPIuvN6jz9JVG2yT8DMu7Xk,1408
+pmcfetch/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+pmcfetch/ratelimit.py,sha256=MfkMqkcC6NJmvemrdR9Liw6myZ9ed4OKfc8Ym3bkgcM,1751
+pmcfetch/s3source.py,sha256=NJO0VYV2F-ST2hTojAuI8pL6nDA2SxO_LbNwy7h1wAQ,9331
+pmcfetch-0.1.0.dist-info/licenses/LICENSE,sha256=jQlZX-s8ad_E1M6EP_SAWrOMqVVCKIg1SJnove7OBpM,1068
+pmcfetch-0.1.0.dist-info/METADATA,sha256=UNZrQPMbkfV2lQjYAZjN9I8bnFGILl_eAZO3bHejw9o,7342
+pmcfetch-0.1.0.dist-info/WHEEL,sha256=mOdXR9C3wMVov7cSY8PY-XCzo3SmmWCHvGQr5csJoao,91
+pmcfetch-0.1.0.dist-info/entry_points.txt,sha256=OP-uK5C4btBKS-S5OVJ4dB94BPfC8OzeXTIKEeFCt00,47
+pmcfetch-0.1.0.dist-info/top_level.txt,sha256=ISYBRCuQLP7ke6rQsK7Xlt5u79NBlkK30ls1C8DHKyA,9
+pmcfetch-0.1.0.dist-info/RECORD,,

pmcfetch-0.1.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (84.2.1)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

pmcfetch-0.1.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+pmcfetch = pmcfetch.cli:main

pmcfetch-0.1.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Kavach Shah
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pmcfetch-0.1.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+pmcfetch