pmcfetch 0.1.0__tar.gz → 0.2.0__tar.gz

{pmcfetch-0.1.0/src/pmcfetch.egg-info → pmcfetch-0.2.0}/PKG-INFO

@@ -1,11 +1,13 @@
 Metadata-Version: 2.4
 Name: pmcfetch
-Version: 0.1.0
-Summary: Targeted PDF retrieval from the PMC Open Access Subset on AWS. Zero dependencies.
+Version: 0.2.0
+Summary: Search the PMC Open Access Subset by keyword and download article PDFs from AWS. Zero dependencies.
 Author: Kavach Shah
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/kms2229/pmcfetch
+Project-URL: Source, https://github.com/kms2229/pmcfetch
 Project-URL: Issues, https://github.com/kms2229/pmcfetch/issues
+Project-URL: Changelog, https://github.com/kms2229/pmcfetch/blob/main/CHANGELOG.md
 Keywords: pubmed-central,pmc,open-access,bulk-download,bioinformatics
 Classifier: Development Status :: 4 - Beta
 Classifier: Environment :: Console
@@ -31,11 +33,14 @@ Dynamic: license-file
 # pmcfetch
 
 Targeted PDF retrieval from the **PMC Open Access Subset**, with resume, retries,
-atomic writes, and namespace-aware batch layout. Zero dependencies.
+atomic writes, and namespace-aware batch layout. Start from a list of PMC ids,
+or from a keyword search. Zero dependencies.
 
 ```bash
 pip install pmcfetch
-pmcfetch download --corpus corpus.json --out pdfs/ --workers 16 --rps 0
+pmcfetch search "burn scar"                      # how many open-access papers match?
+pmcfetch search "burn scar" --download 50        # fetch the 50 most relevant
+pmcfetch download --corpus corpus.json --out pdfs/ --workers 16
 ```
 
 Requires Python 3.10+. Nothing outside the standard library.
@@ -95,14 +100,39 @@ with S3OpenAccessClient() as c:
 `license_code` and `is_retracted` come free with every article — filter on licence
 terms and drop retracted papers before they reach your corpus.
 
+Keyword search — check the count first, then take as many as you want:
+
+```python
+from pmcfetch import PMCSearch, S3OpenAccessClient, download_many
+
+search = PMCSearch()                    # PMCSearch(api_key=...) for NCBI's higher rate limit
+summary = search.summarize("burn scar AND 2020:2025[pdat]")
+print(summary.count, summary.warnings)  # 12013 ()
+
+records = search.records(summary.query, limit=100, namespace="burns")   # limit=None for all
+with S3OpenAccessClient() as client:
+    report = download_many(records, "pdfs/", client=client, workers=16)
+```
+
 ## CLI
 
 ```bash
+# how many open-access papers match? (in a terminal, it then asks how many to download)
+pmcfetch search "wound healing AND diabetic foot"
+
+# download some or all of them, without the question
+pmcfetch search "burn scar" --download 100 --out pdfs/
+pmcfetch search "burn scar" --download all --namespace burns
+
+# or save every match now and download later, e.g. in an HPC array job
+pmcfetch search "burn scar" --namespace burns --save burns.json
+pmcfetch download --corpus burns.json --out pdfs/
+
 # what's in a corpus file?
 pmcfetch namespaces --corpus corpus.json
 
 # pull one namespace
-pmcfetch download --corpus corpus.json --out pdfs/ --namespace cardiology --workers 16 --rps 0
+pmcfetch download --corpus corpus.json --out pdfs/ --namespace cardiology --workers 16
 
 # dry run, no requests
 pmcfetch download --corpus corpus.json --out pdfs/ --limit 20 --dry-run
@@ -113,10 +143,19 @@ pmcfetch download --corpus pdfs/failed.txt --out pdfs/ --flat
 
 Exit code is `1` when any article failed, `0` otherwise — so a batch job fails
 loudly instead of silently producing a short corpus. Bad arguments or an
-unreadable corpus file exit `2`, and Ctrl-C exits `130`.
+unreadable corpus file exit `2`, and Ctrl-C exits `130`. With `--json`, stdout
+carries only the JSON summary and progress goes to stderr, so it pipes cleanly.
+
+S3 has no rate limit, so downloads are not paced by default and `--workers` can
+go wide; throughput is bound by your bandwidth. `--rps N` caps requests per
+second across all workers if you want to go easier on a shared link.
 
-S3 has no rate limit, so `--rps 0` switches pacing off entirely (`--jitter` then
-has no effect) and `--workers` can go wide; throughput is bound by your bandwidth.
+`search` takes PMC's search syntax: field tags (`smith j[author]`), date ranges
+(`2020:2025[pdat]`), `AND`/`OR`/`NOT`. It prints how NCBI read the query and warns
+about words it dropped. `--sort relevance` (the default) or `--sort newest` picks
+which papers come first when you take fewer than all. `--save` writes the chosen
+ids to a corpus file that `pmcfetch download` reads; without `--download`, it
+saves every match and downloads none.
 
 ## Deployment
 
@@ -164,6 +203,18 @@ bad key can't abort a 20k-article batch.
   raise `ArticleNotInOASubsetError`. Both land in the failure log with a reason.
 - **Any object implementing `fetch_pdf(pmcid) -> bytes`** satisfies the
   `PdfSource` protocol and works with `download_many`.
+- **Search counts only open-access articles** (`"open access"[filter]`), the
+  ones the bucket holds. In a sample of 800 matches, 98% had an article PDF; the
+  rest are licensed for XML and text only, and fail with `PDFNotAvailableError`
+  like any other run.
+- **Search has no 10,000 cap.** NCBI returns at most 9,999 ids per query, so
+  larger result sets are collected by splitting the query into PMC id ranges.
+  NCBI ranks only the first 9,999 by relevance; the rest follow newest first.
+- **Search goes through NCBI E-utilities**, which allow 3 requests per second,
+  or 10 with an NCBI API key from your
+  [NCBI account settings](https://account.ncbi.nlm.nih.gov/settings/)
+  (`--api-key`, or `$NCBI_API_KEY`). Collecting 20,000 ids takes about 15 seconds.
+  NCBI's count can drift by a few between calls.
 
 ## Tests
 
@@ -175,7 +226,8 @@ python -m pytest tests/ -q
 No network required — the S3 layer is exercised against scripted listings and
 responses: multi-version and paginated listings, supplementary-PDF
 disambiguation, 429 storms and `Retry-After`, 404s, truncated resumes, and
-concurrent and interrupted batches.
+concurrent and interrupted batches. Search runs against a scripted ESearch that
+enforces NCBI's per-query cap, so the id-range splitting is tested too.
 
 ## Attribution
 

{pmcfetch-0.1.0 → pmcfetch-0.2.0}/README.md

@@ -1,11 +1,14 @@
 # pmcfetch
 
 Targeted PDF retrieval from the **PMC Open Access Subset**, with resume, retries,
-atomic writes, and namespace-aware batch layout. Zero dependencies.
+atomic writes, and namespace-aware batch layout. Start from a list of PMC ids,
+or from a keyword search. Zero dependencies.
 
 ```bash
 pip install pmcfetch
-pmcfetch download --corpus corpus.json --out pdfs/ --workers 16 --rps 0
+pmcfetch search "burn scar"                      # how many open-access papers match?
+pmcfetch search "burn scar" --download 50        # fetch the 50 most relevant
+pmcfetch download --corpus corpus.json --out pdfs/ --workers 16
 ```
 
 Requires Python 3.10+. Nothing outside the standard library.
@@ -65,14 +68,39 @@ with S3OpenAccessClient() as c:
 `license_code` and `is_retracted` come free with every article — filter on licence
 terms and drop retracted papers before they reach your corpus.
 
+Keyword search — check the count first, then take as many as you want:
+
+```python
+from pmcfetch import PMCSearch, S3OpenAccessClient, download_many
+
+search = PMCSearch()                    # PMCSearch(api_key=...) for NCBI's higher rate limit
+summary = search.summarize("burn scar AND 2020:2025[pdat]")
+print(summary.count, summary.warnings)  # 12013 ()
+
+records = search.records(summary.query, limit=100, namespace="burns")   # limit=None for all
+with S3OpenAccessClient() as client:
+    report = download_many(records, "pdfs/", client=client, workers=16)
+```
+
 ## CLI
 
 ```bash
+# how many open-access papers match? (in a terminal, it then asks how many to download)
+pmcfetch search "wound healing AND diabetic foot"
+
+# download some or all of them, without the question
+pmcfetch search "burn scar" --download 100 --out pdfs/
+pmcfetch search "burn scar" --download all --namespace burns
+
+# or save every match now and download later, e.g. in an HPC array job
+pmcfetch search "burn scar" --namespace burns --save burns.json
+pmcfetch download --corpus burns.json --out pdfs/
+
 # what's in a corpus file?
 pmcfetch namespaces --corpus corpus.json
 
 # pull one namespace
-pmcfetch download --corpus corpus.json --out pdfs/ --namespace cardiology --workers 16 --rps 0
+pmcfetch download --corpus corpus.json --out pdfs/ --namespace cardiology --workers 16
 
 # dry run, no requests
 pmcfetch download --corpus corpus.json --out pdfs/ --limit 20 --dry-run
@@ -83,10 +111,19 @@ pmcfetch download --corpus pdfs/failed.txt --out pdfs/ --flat
 
 Exit code is `1` when any article failed, `0` otherwise — so a batch job fails
 loudly instead of silently producing a short corpus. Bad arguments or an
-unreadable corpus file exit `2`, and Ctrl-C exits `130`.
+unreadable corpus file exit `2`, and Ctrl-C exits `130`. With `--json`, stdout
+carries only the JSON summary and progress goes to stderr, so it pipes cleanly.
+
+S3 has no rate limit, so downloads are not paced by default and `--workers` can
+go wide; throughput is bound by your bandwidth. `--rps N` caps requests per
+second across all workers if you want to go easier on a shared link.
 
-S3 has no rate limit, so `--rps 0` switches pacing off entirely (`--jitter` then
-has no effect) and `--workers` can go wide; throughput is bound by your bandwidth.
+`search` takes PMC's search syntax: field tags (`smith j[author]`), date ranges
+(`2020:2025[pdat]`), `AND`/`OR`/`NOT`. It prints how NCBI read the query and warns
+about words it dropped. `--sort relevance` (the default) or `--sort newest` picks
+which papers come first when you take fewer than all. `--save` writes the chosen
+ids to a corpus file that `pmcfetch download` reads; without `--download`, it
+saves every match and downloads none.
 
 ## Deployment
 
@@ -134,6 +171,18 @@ bad key can't abort a 20k-article batch.
   raise `ArticleNotInOASubsetError`. Both land in the failure log with a reason.
 - **Any object implementing `fetch_pdf(pmcid) -> bytes`** satisfies the
   `PdfSource` protocol and works with `download_many`.
+- **Search counts only open-access articles** (`"open access"[filter]`), the
+  ones the bucket holds. In a sample of 800 matches, 98% had an article PDF; the
+  rest are licensed for XML and text only, and fail with `PDFNotAvailableError`
+  like any other run.
+- **Search has no 10,000 cap.** NCBI returns at most 9,999 ids per query, so
+  larger result sets are collected by splitting the query into PMC id ranges.
+  NCBI ranks only the first 9,999 by relevance; the rest follow newest first.
+- **Search goes through NCBI E-utilities**, which allow 3 requests per second,
+  or 10 with an NCBI API key from your
+  [NCBI account settings](https://account.ncbi.nlm.nih.gov/settings/)
+  (`--api-key`, or `$NCBI_API_KEY`). Collecting 20,000 ids takes about 15 seconds.
+  NCBI's count can drift by a few between calls.
 
 ## Tests
 
@@ -145,7 +194,8 @@ python -m pytest tests/ -q
 No network required — the S3 layer is exercised against scripted listings and
 responses: multi-version and paginated listings, supplementary-PDF
 disambiguation, 429 storms and `Retry-After`, 404s, truncated resumes, and
-concurrent and interrupted batches.
+concurrent and interrupted batches. Search runs against a scripted ESearch that
+enforces NCBI's per-query cap, so the id-range splitting is tested too.
 
 ## Attribution
 

{pmcfetch-0.1.0 → pmcfetch-0.2.0}/pyproject.toml

@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
 [project]
 name = "pmcfetch"
 dynamic = ["version"]
-description = "Targeted PDF retrieval from the PMC Open Access Subset on AWS. Zero dependencies."
+description = "Search the PMC Open Access Subset by keyword and download article PDFs from AWS. Zero dependencies."
 readme = "README.md"
 license = "MIT"
 license-files = ["LICENSE"]
@@ -38,7 +38,9 @@ dev = [
 
 [project.urls]
 Homepage = "https://github.com/kms2229/pmcfetch"
+Source = "https://github.com/kms2229/pmcfetch"
 Issues = "https://github.com/kms2229/pmcfetch/issues"
+Changelog = "https://github.com/kms2229/pmcfetch/blob/main/CHANGELOG.md"
 
 [project.scripts]
 pmcfetch = "pmcfetch.cli:main"

{pmcfetch-0.1.0 → pmcfetch-0.2.0}/src/pmcfetch/__init__.py

@@ -8,7 +8,9 @@ and it is the access route PMC sanctions for systematic retrieval.
 care about. This package fills that gap: give it a list of PMC ids and it
 resolves each article's version, picks the article PDF (not the publisher's
 supplementary PDFs sitting in the same prefix), and writes the results with
-resume, retries, atomic writes, and per-namespace layout.
+resume, retries, atomic writes, and per-namespace layout. Or start from a
+keyword search: :class:`PMCSearch` counts the matching open-access articles
+before anything is downloaded.
 
 Zero third-party dependencies.
 
@@ -22,7 +24,7 @@ Zero third-party dependencies.
 
 from __future__ import annotations
 
-__version__ = "0.1.0"
+__version__ = "0.2.0"
 
 from pmcfetch.batch import (
     BatchReport,
@@ -40,10 +42,11 @@ from pmcfetch.errors import (
     InvalidPMCIDError,
     NotAPDFError,
     PMCFetchError,
+    SearchError,
     TransportError,
 )
 from pmcfetch.ids import normalize_pmcid, try_normalize_pmcid
-from pmcfetch.manifest import CorpusRecord, LoadResult, iter_records, load_records
+from pmcfetch.manifest import CorpusRecord, LoadResult, iter_records, load_records, save_records
 from pmcfetch.pdf import DEFAULT_MIN_PDF_BYTES, DEFAULT_TIMEOUT, PdfSource, is_pdf
 from pmcfetch.ratelimit import MinIntervalLimiter, NullLimiter
 from pmcfetch.s3source import (
@@ -56,9 +59,13 @@ from pmcfetch.s3source import (
     list_article_versions,
     resolve_latest_version,
 )
+from pmcfetch.search import PMCSearch, SearchSummary
 
 __all__ = [
     "__version__",
+    # search
+    "PMCSearch",
+    "SearchSummary",
     # source
     "S3OpenAccessClient",
     "OA_BUCKET",
@@ -77,6 +84,7 @@ __all__ = [
     "write_pdf_atomic",
     # manifest
     "load_records",
+    "save_records",
     "iter_records",
     "CorpusRecord",
     "LoadResult",
@@ -99,4 +107,5 @@ __all__ = [
     "ArticleNotInOASubsetError",
     "InvalidPMCIDError",
     "TransportError",
+    "SearchError",
 ]