pubmed-stream 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- pubmed_stream-0.1.0/PKG-INFO +808 -0
- pubmed_stream-0.1.0/README.md +779 -0
- pubmed_stream-0.1.0/pubmed_stream/__init__.py +25 -0
- pubmed_stream-0.1.0/pubmed_stream/__main__.py +5 -0
- pubmed_stream-0.1.0/pubmed_stream/cli.py +145 -0
- pubmed_stream-0.1.0/pubmed_stream/downloader.py +745 -0
- pubmed_stream-0.1.0/pubmed_stream/py.typed +0 -0
- pubmed_stream-0.1.0/pubmed_stream.egg-info/PKG-INFO +808 -0
- pubmed_stream-0.1.0/pubmed_stream.egg-info/SOURCES.txt +13 -0
- pubmed_stream-0.1.0/pubmed_stream.egg-info/dependency_links.txt +1 -0
- pubmed_stream-0.1.0/pubmed_stream.egg-info/entry_points.txt +2 -0
- pubmed_stream-0.1.0/pubmed_stream.egg-info/requires.txt +5 -0
- pubmed_stream-0.1.0/pubmed_stream.egg-info/top_level.txt +1 -0
- pubmed_stream-0.1.0/pyproject.toml +46 -0
- pubmed_stream-0.1.0/setup.cfg +4 -0
|
@@ -0,0 +1,808 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: pubmed-stream
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Download PubMed Central full-text articles via NCBI E-utilities API. Built for biomedical research.
|
|
5
|
+
Author: Haran Lab, Bucci Lab, and Microbiology & Microbiome Dynamics AI HUB
|
|
6
|
+
Maintainer: University of Massachusetts
|
|
7
|
+
License: MIT
|
|
8
|
+
Project-URL: Repository, https://github.com/melhzy/pubmed-stream
|
|
9
|
+
Project-URL: Homepage, https://github.com/melhzy/pubmed-stream
|
|
10
|
+
Project-URL: Bug Tracker, https://github.com/melhzy/pubmed-stream/issues
|
|
11
|
+
Keywords: pubmed,pmc,ncbi,e-utilities,literature,download,biomedical,research
|
|
12
|
+
Classifier: Development Status :: 4 - Beta
|
|
13
|
+
Classifier: Intended Audience :: Science/Research
|
|
14
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
15
|
+
Classifier: Programming Language :: Python :: 3
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.8
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
21
|
+
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
|
|
22
|
+
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
|
|
23
|
+
Requires-Python: >=3.8
|
|
24
|
+
Description-Content-Type: text/markdown
|
|
25
|
+
Requires-Dist: requests>=2.25
|
|
26
|
+
Provides-Extra: dev
|
|
27
|
+
Requires-Dist: pytest; extra == "dev"
|
|
28
|
+
Requires-Dist: ruff; extra == "dev"
|
|
29
|
+
|
|
30
|
+
# pubmed-stream
|
|
31
|
+
|
|
32
|
+
A Python library for downloading PubMed Central (PMC) full-text articles with support for concurrent downloads, automatic rate limiting, metadata extraction, and multiple output formats.
|
|
33
|
+
|
|
34
|
+
**Built on NCBI E-utilities API** for reliable, standards-compliant access to biomedical literature.
|
|
35
|
+
|
|
36
|
+
## Overview
|
|
37
|
+
|
|
38
|
+
`pubmed-stream` simplifies the process of downloading scientific literature from PubMed Central by leveraging the **NCBI E-utilities API** (ESearch and EFetch):
|
|
39
|
+
- **Searching PMC directly** for guaranteed full-text availability
|
|
40
|
+
- **Extracting structured metadata** from JATS XML (title, authors, abstract, keywords, DOI, etc.)
|
|
41
|
+
- **Supporting multiple formats**: plain text, raw XML, or both in a single JSON file
|
|
42
|
+
- **Handling concurrent downloads** with configurable thread pools
|
|
43
|
+
- **Respecting NCBI rate limits** with automatic detection and retry logic
|
|
44
|
+
- **Providing both CLI and Python API** for flexible integration
|
|
45
|
+
|
|
46
|
+
Built for researchers and data scientists who need efficient, reliable access to biomedical literature.
|
|
47
|
+
|
|
48
|
+
## Quick Start
|
|
49
|
+
|
|
50
|
+
```bash
|
|
51
|
+
# Install
|
|
52
|
+
pip install .
|
|
53
|
+
|
|
54
|
+
# Download articles (CLI)
|
|
55
|
+
pubmed-stream download "gut microbiome" --max-results 50
|
|
56
|
+
|
|
57
|
+
# Python API
|
|
58
|
+
from pubmed_stream import search_and_download
|
|
59
|
+
stats = search_and_download(
|
|
60
|
+
keyword="CRISPR gene editing",
|
|
61
|
+
max_results=100,
|
|
62
|
+
fmt="text"
|
|
63
|
+
)
|
|
64
|
+
print(f"Downloaded {stats.successful} articles in {stats.duration_seconds:.1f}s")
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
## Table of Contents
|
|
68
|
+
|
|
69
|
+
- [Installation](#installation)
|
|
70
|
+
- [How It Works](#how-it-works)
|
|
71
|
+
- [Configuration](#configuration-optional)
|
|
72
|
+
- [CLI Usage](#cli-usage)
|
|
73
|
+
- [Python API](#python-api)
|
|
74
|
+
- [Output Format](#output-format)
|
|
75
|
+
- [Use Cases](#use-cases)
|
|
76
|
+
- [Troubleshooting](#troubleshooting)
|
|
77
|
+
- [Performance Tuning](#performance-tuning)
|
|
78
|
+
- [Testing](#testing)
|
|
79
|
+
- [Contributing](#contributing)
|
|
80
|
+
- [License](#license)
|
|
81
|
+
|
|
82
|
+
## Installation
|
|
83
|
+
|
|
84
|
+
### From Local Clone
|
|
85
|
+
|
|
86
|
+
```bash
|
|
87
|
+
# Clone the repository
|
|
88
|
+
git clone https://github.com/melhzy/pubmed-stream.git
|
|
89
|
+
cd pubmed-stream
|
|
90
|
+
|
|
91
|
+
# Install in standard mode
|
|
92
|
+
pip install .
|
|
93
|
+
|
|
94
|
+
# Or install in development mode (editable install)
|
|
95
|
+
pip install -e ".[dev]"
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
### Requirements
|
|
99
|
+
|
|
100
|
+
- Python 3.8+
|
|
101
|
+
- [requests](https://pypi.org/project/requests/) (only runtime dependency)
|
|
102
|
+
|
|
103
|
+
Development dependencies (optional):
|
|
104
|
+
- pytest (for testing)
|
|
105
|
+
- ruff (for linting)
|
|
106
|
+
|
|
107
|
+
## How It Works
|
|
108
|
+
|
|
109
|
+
### Architecture
|
|
110
|
+
|
|
111
|
+
The library follows a modular design with three main components, all built on **NCBI E-utilities**:
|
|
112
|
+
|
|
113
|
+
1. **Search Module** (`esearch_pmc`)
|
|
114
|
+
- Uses NCBI E-utilities **ESearch** to query PMC database directly
|
|
115
|
+
- Returns PMC IDs for articles with guaranteed full-text availability
|
|
116
|
+
- Handles pagination and result limiting
|
|
117
|
+
|
|
118
|
+
2. **Download Module** (`efetch_pmc`)
|
|
119
|
+
- Uses NCBI E-utilities **EFetch** to retrieve full XML article data from PMC
|
|
120
|
+
- Extracts structured metadata using JATS XML parsing
|
|
121
|
+
- Converts XML to plain text (optional)
|
|
122
|
+
- Saves in requested format(s)
|
|
123
|
+
|
|
124
|
+
3. **Orchestration Layer** (`search_and_download`)
|
|
125
|
+
- Combines search and download operations
|
|
126
|
+
- Manages concurrent downloads with thread pools
|
|
127
|
+
- Handles rate limiting, retries, and error recovery
|
|
128
|
+
- Tracks statistics (success/failure/skipped counts)
|
|
129
|
+
|
|
130
|
+
### Search Strategy
|
|
131
|
+
|
|
132
|
+
**Direct PMC Search**: Unlike some tools that search PubMed and then try to find PMC links, this library searches PMC directly. This ensures:
|
|
133
|
+
- All results have full-text available
|
|
134
|
+
- No broken links or paywalled content
|
|
135
|
+
- Faster, more reliable downloads
|
|
136
|
+
- Better results for generic queries like "microbiome" or "CRISPR"
|
|
137
|
+
|
|
138
|
+
### Data Flow
|
|
139
|
+
|
|
140
|
+
```
|
|
141
|
+
User Query → PMC Search → PMC IDs → Concurrent Downloads → Parse XML → Extract Metadata → Save JSON
|
|
142
|
+
↓ ↓ ↓ ↓ ↓ ↓ ↓
|
|
143
|
+
"microbiome" E-Search [123, 456] Thread Pool JATS Parser {title, authors} PMC*.json
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
### Output Format Options
|
|
147
|
+
|
|
148
|
+
All formats save as `.json` files with different content:
|
|
149
|
+
|
|
150
|
+
**`"text"` (default)**: JSON with plain-text content
|
|
151
|
+
```json
|
|
152
|
+
{
|
|
153
|
+
"pmcid": "PMC12345678",
|
|
154
|
+
"metadata": {"title": "...", "authors": [...]},
|
|
155
|
+
"text": "Introduction\nThis study examines..."
|
|
156
|
+
}
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
**`"xml"`**: JSON with raw JATS XML string
|
|
160
|
+
```json
|
|
161
|
+
{
|
|
162
|
+
"pmcid": "PMC12345678",
|
|
163
|
+
"metadata": {"title": "...", "authors": [...]},
|
|
164
|
+
"xml": "<article>...</article>"
|
|
165
|
+
}
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
**`"both"`**: JSON with both XML and text
|
|
169
|
+
```json
|
|
170
|
+
{
|
|
171
|
+
"pmcid": "PMC12345678",
|
|
172
|
+
"metadata": {"title": "...", "authors": [...]},
|
|
173
|
+
"xml": "<article>...</article>",
|
|
174
|
+
"text": "Introduction\nThis study examines..."
|
|
175
|
+
}
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
**Why unified JSON?**
|
|
179
|
+
- Consistent file handling (always `.json`)
|
|
180
|
+
- Metadata always included
|
|
181
|
+
- Easy to query and filter
|
|
182
|
+
- Smaller storage (text-only is ~50% smaller than XML)
|
|
183
|
+
- Backward compatible with existing pipelines
|
|
184
|
+
|
|
185
|
+
## Configuration (optional)
|
|
186
|
+
|
|
187
|
+
Set these environment variables for best results:
|
|
188
|
+
|
|
189
|
+
```bash
|
|
190
|
+
# PowerShell
|
|
191
|
+
$env:NCBI_API_KEY = "YOUR_KEY" # raises rate limit from 3→10 req/s
|
|
192
|
+
$env:NCBI_EMAIL = "you@example.com"
|
|
193
|
+
|
|
194
|
+
# Bash / Zsh
|
|
195
|
+
export NCBI_API_KEY="YOUR_KEY"
|
|
196
|
+
export NCBI_EMAIL="you@example.com"
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
You can also pass `--api-key` / `--email` on the command line.
|
|
200
|
+
|
|
201
|
+
### Rate Limits
|
|
202
|
+
|
|
203
|
+
NCBI enforces rate limits on API requests:
|
|
204
|
+
|
|
205
|
+
- **Without API key**: 3 requests/second
|
|
206
|
+
- **With API key**: 10 requests/second
|
|
207
|
+
|
|
208
|
+
The library **automatically detects** whether you have an API key and adjusts the rate limit accordingly:
|
|
209
|
+
- With API key: 0.1s between requests (10 req/s)
|
|
210
|
+
- Without API key: 0.34s between requests (3 req/s)
|
|
211
|
+
|
|
212
|
+
You can override this with `--rate-limit` if needed, but the auto-detection works well for most cases.
|
|
213
|
+
|
|
214
|
+
**Recommendation**: Get a free API key to download ~3x faster. Without a key, the library still works but is slower.
|
|
215
|
+
|
|
216
|
+
## CLI usage
|
|
217
|
+
|
|
218
|
+
After installation the `pubmed-stream` command is available:
|
|
219
|
+
|
|
220
|
+
```bash
|
|
221
|
+
# Download up to 50 articles as JSON (default, concurrent)
|
|
222
|
+
pubmed-stream download "frailty cytokines" --max-results 50
|
|
223
|
+
|
|
224
|
+
# Sequential mode for unstable networks
|
|
225
|
+
pubmed-stream download "gut microbiome" --sequential
|
|
226
|
+
|
|
227
|
+
# Custom output directory
|
|
228
|
+
pubmed-stream download "Alzheimer microbiome" -o ./my_papers
|
|
229
|
+
|
|
230
|
+
# Verbose logging
|
|
231
|
+
pubmed-stream download "frailty" -v --max-results 5
|
|
232
|
+
|
|
233
|
+
# Show version
|
|
234
|
+
pubmed-stream --version
|
|
235
|
+
```
|
|
236
|
+
|
|
237
|
+
The same interface is available via `python -m pubmed_stream`:
|
|
238
|
+
|
|
239
|
+
```bash
|
|
240
|
+
python -m pubmed_stream download "frailty cytokines" --max-results 50
|
|
241
|
+
```
|
|
242
|
+
|
|
243
|
+
### CLI options
|
|
244
|
+
|
|
245
|
+
| Flag | Description | Default |
|
|
246
|
+
|------|-------------|---------|
|
|
247
|
+
| `keyword` | PubMed search query (positional) | required |
|
|
248
|
+
| `--max-results` | Maximum articles to download | 100 |
|
|
249
|
+
| `--format` | Output format: `text`, `xml`, `both` (all save as `.json`) | `text` |
|
|
250
|
+
| `--output-dir`, `-o` | Base output directory | `./publications` |
|
|
251
|
+
| `--api-key` | NCBI API key | `NCBI_API_KEY` env var |
|
|
252
|
+
| `--email` | Contact email for NCBI | `NCBI_EMAIL` env var |
|
|
253
|
+
| `--sequential` | Disable concurrent downloads | concurrent |
|
|
254
|
+
| `--workers` | Concurrent thread count | 5 |
|
|
255
|
+
| `--exclude-text` | Omit `text` field in JSON (~30% smaller) | included |
|
|
256
|
+
| `--rate-limit` | Min seconds between requests | auto: 0.1 with key, 0.34 without |
|
|
257
|
+
| `--user-agent` | Custom HTTP `User-Agent` | `pubmed-stream/<ver>` |
|
|
258
|
+
| `--verbose`, `-v` | DEBUG-level logging | INFO |
|
|
259
|
+
|
|
260
|
+
### Exit codes
|
|
261
|
+
|
|
262
|
+
| Code | Meaning |
|
|
263
|
+
|------|---------|
|
|
264
|
+
| 0 | At least one article downloaded (or all already cached) |
|
|
265
|
+
| 1 | No results found for the query |
|
|
266
|
+
| 2 | All download attempts failed |
|
|
267
|
+
|
|
268
|
+
## Python API
|
|
269
|
+
|
|
270
|
+
### Basic Usage
|
|
271
|
+
|
|
272
|
+
```python
|
|
273
|
+
from pubmed_stream import search_and_download
|
|
274
|
+
|
|
275
|
+
# Download articles with default settings
|
|
276
|
+
stats = search_and_download(
|
|
277
|
+
keyword="gut microbiome inflammation",
|
|
278
|
+
max_results=50,
|
|
279
|
+
fmt="text", # or "xml", "both"
|
|
280
|
+
output_dir="publications",
|
|
281
|
+
)
|
|
282
|
+
|
|
283
|
+
print(f"Downloaded: {stats.successful}")
|
|
284
|
+
print(f"Skipped (already exist): {stats.skipped}")
|
|
285
|
+
print(f"Failed: {stats.failed}")
|
|
286
|
+
print(f"Success rate: {stats.success_rate:.1f}%")
|
|
287
|
+
print(f"Duration: {stats.duration_seconds:.1f}s")
|
|
288
|
+
```
|
|
289
|
+
|
|
290
|
+
### Advanced Usage
|
|
291
|
+
|
|
292
|
+
```python
|
|
293
|
+
from pubmed_stream import search_and_download
|
|
294
|
+
from pathlib import Path
|
|
295
|
+
|
|
296
|
+
# Full control over all parameters
|
|
297
|
+
stats = search_and_download(
|
|
298
|
+
keyword="CRISPR gene editing",
|
|
299
|
+
max_results=100,
|
|
300
|
+
fmt="both", # Save both XML and text
|
|
301
|
+
api_key="your_api_key", # Or use NCBI_API_KEY env var
|
|
302
|
+
use_concurrent=True, # Enable parallel downloads
|
|
303
|
+
max_workers=10, # Use 10 threads
|
|
304
|
+
include_text=True, # Include plain-text field
|
|
305
|
+
output_dir=Path("./research/crispr"),
|
|
306
|
+
user_agent="MyResearchBot/1.0",
|
|
307
|
+
email="researcher@university.edu",
|
|
308
|
+
rate_limit=0.15, # Custom rate limit
|
|
309
|
+
)
|
|
310
|
+
|
|
311
|
+
# Access detailed statistics
|
|
312
|
+
print(f"Total found in PMC: {stats.total_found}")
|
|
313
|
+
print(f"Requested: {stats.requested}")
|
|
314
|
+
print(f"Output: {stats.output_dir}")
|
|
315
|
+
```
|
|
316
|
+
|
|
317
|
+
### Low-Level API
|
|
318
|
+
|
|
319
|
+
For more control, use the underlying functions:
|
|
320
|
+
|
|
321
|
+
```python
|
|
322
|
+
from pubmed_stream import esearch_pmc, efetch_pmc, create_session, RateLimiter
|
|
323
|
+
from pathlib import Path
|
|
324
|
+
|
|
325
|
+
# Create session and rate limiter
|
|
326
|
+
session = create_session("MyApp/1.0")
|
|
327
|
+
rate_limiter = RateLimiter(0.1) # 10 req/s
|
|
328
|
+
|
|
329
|
+
# Search for PMC IDs
|
|
330
|
+
pmcids, total_found = esearch_pmc(
|
|
331
|
+
term="microbiome AND aging",
|
|
332
|
+
max_results=20,
|
|
333
|
+
api_key="your_key",
|
|
334
|
+
session=session,
|
|
335
|
+
rate_limiter=rate_limiter
|
|
336
|
+
)
|
|
337
|
+
|
|
338
|
+
print(f"Found {len(pmcids)} PMC IDs (total: {total_found})")
|
|
339
|
+
|
|
340
|
+
# Download individual articles
|
|
341
|
+
for pmcid in pmcids:
|
|
342
|
+
success, status = efetch_pmc(
|
|
343
|
+
pmcid=pmcid,
|
|
344
|
+
out_dir=Path("downloads"),
|
|
345
|
+
fmt="text",
|
|
346
|
+
api_key="your_key",
|
|
347
|
+
session=session,
|
|
348
|
+
rate_limiter=rate_limiter,
|
|
349
|
+
include_text=True
|
|
350
|
+
)
|
|
351
|
+
|
|
352
|
+
if success:
|
|
353
|
+
print(f"✓ Downloaded PMC{pmcid}")
|
|
354
|
+
else:
|
|
355
|
+
print(f"✗ Failed PMC{pmcid}: {status}")
|
|
356
|
+
|
|
357
|
+
session.close()
|
|
358
|
+
```
|
|
359
|
+
|
|
360
|
+
### Metadata Extraction
|
|
361
|
+
|
|
362
|
+
The library automatically extracts metadata from JATS XML:
|
|
363
|
+
|
|
364
|
+
```python
|
|
365
|
+
from pubmed_stream import extract_metadata_from_pmc_xml
|
|
366
|
+
|
|
367
|
+
xml_content = """<article>...</article>""" # Your PMC XML
|
|
368
|
+
|
|
369
|
+
metadata = extract_metadata_from_pmc_xml(xml_content)
|
|
370
|
+
|
|
371
|
+
# Available metadata fields
|
|
372
|
+
print(metadata.get("title")) # Article title
|
|
373
|
+
print(metadata.get("journal")) # Journal name
|
|
374
|
+
print(metadata.get("doi")) # DOI
|
|
375
|
+
print(metadata.get("pmid")) # PubMed ID
|
|
376
|
+
print(metadata.get("pmcid")) # PMC ID
|
|
377
|
+
print(metadata.get("authors")) # List of authors
|
|
378
|
+
print(metadata.get("abstract")) # Abstract text
|
|
379
|
+
print(metadata.get("keywords")) # Keywords list
|
|
380
|
+
print(metadata.get("year")) # Publication year
|
|
381
|
+
print(metadata.get("pub_date")) # Full publication date
|
|
382
|
+
```
|
|
383
|
+
|
|
384
|
+
## Output Format
|
|
385
|
+
|
|
386
|
+
### Directory Structure
|
|
387
|
+
|
|
388
|
+
Articles are organized by search query:
|
|
389
|
+
|
|
390
|
+
```
|
|
391
|
+
publications/
|
|
392
|
+
gut_microbiome_inflammation/
|
|
393
|
+
PMC12345678.json
|
|
394
|
+
PMC12345679.json
|
|
395
|
+
PMC12345680.json
|
|
396
|
+
alzheimer_disease/
|
|
397
|
+
PMC11111111.json
|
|
398
|
+
PMC22222222.json
|
|
399
|
+
```
|
|
400
|
+
|
|
401
|
+
Query strings are slugified (spaces→underscores, special characters removed) for clean directory names.
|
|
402
|
+
|
|
403
|
+
### JSON Schema
|
|
404
|
+
|
|
405
|
+
Each JSON file contains:
|
|
406
|
+
|
|
407
|
+
```json
|
|
408
|
+
{
|
|
409
|
+
"pmcid": "PMC12345678",
|
|
410
|
+
"source": "PMC",
|
|
411
|
+
"download_date": "2026-02-17T12:30:45.123456",
|
|
412
|
+
"metadata": {
|
|
413
|
+
"title": "The Role of Gut Microbiome in Inflammation",
|
|
414
|
+
"journal": "Nature Medicine",
|
|
415
|
+
"journal_title": "Nature Medicine",
|
|
416
|
+
"journal_nlm_ta": "Nat Med",
|
|
417
|
+
"journal_iso_abbrev": "Nat. Med.",
|
|
418
|
+
"doi": "10.1038/s41591-024-12345-6",
|
|
419
|
+
"pmid": "38123456",
|
|
420
|
+
"pmcid": "PMC12345678",
|
|
421
|
+
"year": "2024",
|
|
422
|
+
"month": "11",
|
|
423
|
+
"day": "15",
|
|
424
|
+
"pub_date": {
|
|
425
|
+
"year": "2024",
|
|
426
|
+
"month": "11",
|
|
427
|
+
"day": "15"
|
|
428
|
+
},
|
|
429
|
+
"authors": [
|
|
430
|
+
"Smith, John A.",
|
|
431
|
+
"Johnson, Mary B.",
|
|
432
|
+
"Williams, Robert C."
|
|
433
|
+
],
|
|
434
|
+
"abstract": "Background: The gut microbiome plays a crucial role...",
|
|
435
|
+
"keywords": [
|
|
436
|
+
"microbiome",
|
|
437
|
+
"inflammation",
|
|
438
|
+
"gut health"
|
|
439
|
+
]
|
|
440
|
+
},
|
|
441
|
+
"text": "Introduction\nThe human gut microbiome comprises...",
|
|
442
|
+
"xml": "<article>...</article>"
|
|
443
|
+
}
|
|
444
|
+
```
|
|
445
|
+
|
|
446
|
+
**Field descriptions:**
|
|
447
|
+
- `pmcid`: PubMed Central identifier
|
|
448
|
+
- `source`: Always "PMC"
|
|
449
|
+
- `download_date`: ISO 8601 timestamp
|
|
450
|
+
- `metadata`: Structured article information
|
|
451
|
+
- `text`: Plain-text content (if `fmt="text"` or `fmt="both"`)
|
|
452
|
+
- `xml`: Raw JATS XML string (if `fmt="xml"` or `fmt="both"`)
|
|
453
|
+
|
|
454
|
+
### Format Comparison
|
|
455
|
+
|
|
456
|
+
| Format | File Type | Contains | Use Case | Size |
|
|
457
|
+
|--------|-----------|----------|----------|------|
|
|
458
|
+
| `text` | `.json` | metadata + text | Vector DBs, search, NLP | Smallest (~50% of XML) |
|
|
459
|
+
| `xml` | `.json` | metadata + xml | Section parsing, structure analysis | Medium |
|
|
460
|
+
| `both` | `.json` | metadata + xml + text | Comprehensive analysis | Largest |
|
|
461
|
+
|
|
462
|
+
## Use Cases
|
|
463
|
+
|
|
464
|
+
### 1. Literature Review & Meta-Analysis
|
|
465
|
+
|
|
466
|
+
```python
|
|
467
|
+
# Download recent papers on a specific topic
|
|
468
|
+
stats = search_and_download(
|
|
469
|
+
keyword="COVID-19 vaccine efficacy",
|
|
470
|
+
max_results=200,
|
|
471
|
+
fmt="text",
|
|
472
|
+
output_dir="covid19_review"
|
|
473
|
+
)
|
|
474
|
+
|
|
475
|
+
# Process downloaded articles
|
|
476
|
+
import json
|
|
477
|
+
from pathlib import Path
|
|
478
|
+
|
|
479
|
+
for json_file in Path("covid19_review/covid_19_vaccine_efficacy").glob("*.json"):
|
|
480
|
+
with open(json_file) as f:
|
|
481
|
+
article = json.load(f)
|
|
482
|
+
|
|
483
|
+
print(f"Title: {article['metadata']['title']}")
|
|
484
|
+
print(f"Year: {article['metadata']['year']}")
|
|
485
|
+
print(f"Abstract: {article['metadata']['abstract'][:200]}...")
|
|
486
|
+
```
|
|
487
|
+
|
|
488
|
+
### 2. Vector Database Embedding
|
|
489
|
+
|
|
490
|
+
```python
|
|
491
|
+
# Download for embedding into vector database
|
|
492
|
+
stats = search_and_download(
|
|
493
|
+
keyword="machine learning healthcare",
|
|
494
|
+
max_results=1000,
|
|
495
|
+
fmt="text", # Plain text is ideal for embeddings
|
|
496
|
+
include_text=True
|
|
497
|
+
)
|
|
498
|
+
|
|
499
|
+
# Example: Embed into ChromaDB
|
|
500
|
+
import chromadb
|
|
501
|
+
from pathlib import Path
|
|
502
|
+
import json
|
|
503
|
+
|
|
504
|
+
client = chromadb.Client()
|
|
505
|
+
collection = client.create_collection("medical_papers")
|
|
506
|
+
|
|
507
|
+
for json_file in Path("publications/machine_learning_healthcare").glob("*.json"):
|
|
508
|
+
with open(json_file) as f:
|
|
509
|
+
article = json.load(f)
|
|
510
|
+
|
|
511
|
+
collection.add(
|
|
512
|
+
documents=[article["text"]],
|
|
513
|
+
metadatas=[{
|
|
514
|
+
"title": article["metadata"]["title"],
|
|
515
|
+
"year": article["metadata"]["year"],
|
|
516
|
+
"doi": article["metadata"].get("doi", ""),
|
|
517
|
+
}],
|
|
518
|
+
ids=[article["pmcid"]]
|
|
519
|
+
)
|
|
520
|
+
```
|
|
521
|
+
|
|
522
|
+
### 3. Section-Level Analysis
|
|
523
|
+
|
|
524
|
+
```python
|
|
525
|
+
# Download with XML for detailed structure parsing
|
|
526
|
+
stats = search_and_download(
|
|
527
|
+
keyword="CRISPR therapy clinical trials",
|
|
528
|
+
max_results=50,
|
|
529
|
+
fmt="xml" # Get raw XML for section extraction
|
|
530
|
+
)
|
|
531
|
+
|
|
532
|
+
# Parse sections from XML
|
|
533
|
+
import xml.etree.ElementTree as ET
|
|
534
|
+
import json
|
|
535
|
+
from pathlib import Path
|
|
536
|
+
|
|
537
|
+
for json_file in Path("publications/crispr_therapy_clinical_trials").glob("*.json"):
|
|
538
|
+
with open(json_file) as f:
|
|
539
|
+
article = json.load(f)
|
|
540
|
+
|
|
541
|
+
# Parse XML to extract sections
|
|
542
|
+
root = ET.fromstring(article["xml"])
|
|
543
|
+
|
|
544
|
+
# Find all sections with sec-type attribute
|
|
545
|
+
for sec in root.findall(".//sec[@sec-type]"):
|
|
546
|
+
sec_type = sec.get("sec-type")
|
|
547
|
+
title = sec.find("title")
|
|
548
|
+
|
|
549
|
+
if title is not None:
|
|
550
|
+
print(f"Section: {sec_type} - {title.text}")
|
|
551
|
+
```
|
|
552
|
+
|
|
553
|
+
### 4. Batch Processing with Keywords
|
|
554
|
+
|
|
555
|
+
```python
|
|
556
|
+
# Process multiple queries from CSV
|
|
557
|
+
import pandas as pd
|
|
558
|
+
|
|
559
|
+
keywords_df = pd.read_csv("research_keywords.csv")
|
|
560
|
+
|
|
561
|
+
for _, row in keywords_df.iterrows():
|
|
562
|
+
print(f"Processing: {row['keyword']}")
|
|
563
|
+
|
|
564
|
+
stats = search_and_download(
|
|
565
|
+
keyword=row['keyword'],
|
|
566
|
+
max_results=row['max_articles'],
|
|
567
|
+
fmt="both", # Get both formats for flexibility
|
|
568
|
+
output_dir="batch_download"
|
|
569
|
+
)
|
|
570
|
+
|
|
571
|
+
print(f" Downloaded: {stats.successful}, Success rate: {stats.success_rate:.1f}%")
|
|
572
|
+
```
|
|
573
|
+
|
|
574
|
+
## Troubleshooting
|
|
575
|
+
|
|
576
|
+
### Common Issues
|
|
577
|
+
|
|
578
|
+
**1. "0 downloads" but high PMC result count**
|
|
579
|
+
- **Cause**: Files already downloaded (skipped)
|
|
580
|
+
- **Solution**: Check `stats.skipped` - this is actually success! Delete folder to re-download.
|
|
581
|
+
|
|
582
|
+
**2. Rate limiting (HTTP 429)**
|
|
583
|
+
- **Cause**: Exceeding NCBI rate limits
|
|
584
|
+
- **Solution**:
|
|
585
|
+
- Get a free API key (increases limit from 3→10 req/s)
|
|
586
|
+
- Use `--rate-limit` to slow down: `--rate-limit 0.5`
|
|
587
|
+
- The library automatically retries with backoff
|
|
588
|
+
|
|
589
|
+
**3. "Unavailable" articles**
|
|
590
|
+
- **Cause**: Article not in PMC or embargoed
|
|
591
|
+
- **Solution**: Normal - not all PubMed articles have PMC full-text. The library only returns available articles.
|
|
592
|
+
|
|
593
|
+
**4. Network timeouts**
|
|
594
|
+
- **Cause**: Unstable internet connection
|
|
595
|
+
- **Solution**: Use `--sequential` mode: `pubmed-stream download "query" --sequential`
|
|
596
|
+
|
|
597
|
+
**5. Memory issues with large downloads**
|
|
598
|
+
- **Cause**: Processing too many articles concurrently
|
|
599
|
+
- **Solution**: Reduce workers: `--workers 2` or use `--sequential`
|
|
600
|
+
|
|
601
|
+
### Debugging
|
|
602
|
+
|
|
603
|
+
Enable verbose logging to see detailed information:
|
|
604
|
+
|
|
605
|
+
```bash
|
|
606
|
+
pubmed-stream download "microbiome" -v --max-results 10
|
|
607
|
+
```
|
|
608
|
+
|
|
609
|
+
```python
|
|
610
|
+
import logging
|
|
611
|
+
logging.basicConfig(level=logging.DEBUG)
|
|
612
|
+
|
|
613
|
+
from pubmed_stream import search_and_download
|
|
614
|
+
stats = search_and_download(keyword="test", max_results=5)
|
|
615
|
+
```
|
|
616
|
+
|
|
617
|
+
## Performance Tuning
|
|
618
|
+
|
|
619
|
+
### Concurrent vs Sequential
|
|
620
|
+
|
|
621
|
+
**Concurrent (default)**: Faster for stable networks
|
|
622
|
+
```bash
|
|
623
|
+
pubmed-stream download "query" --workers 10 # 10 parallel downloads
|
|
624
|
+
```
|
|
625
|
+
|
|
626
|
+
**Sequential**: More reliable for unstable networks
|
|
627
|
+
```bash
|
|
628
|
+
pubmed-stream download "query" --sequential
|
|
629
|
+
```
|
|
630
|
+
|
|
631
|
+
### Optimal Settings by Network
|
|
632
|
+
|
|
633
|
+
| Network Quality | Mode | Workers | Rate Limit |
|
|
634
|
+
|----------------|------|---------|------------|
|
|
635
|
+
| Excellent | Concurrent | 10 | 0.1 (with API key) |
|
|
636
|
+
| Good | Concurrent | 5 | 0.1 |
|
|
637
|
+
| Fair | Concurrent | 2-3 | 0.2 |
|
|
638
|
+
| Poor | Sequential | 1 | 0.34 |
|
|
639
|
+
|
|
640
|
+
### Performance Benchmarks
|
|
641
|
+
|
|
642
|
+
Typical download speeds (with API key, 5 workers):
|
|
643
|
+
- **Small articles** (~50KB): 2-5 articles/second
|
|
644
|
+
- **Medium articles** (~200KB): 1-3 articles/second
|
|
645
|
+
- **Large articles** (~1MB): 0.5-1 articles/second
|
|
646
|
+
|
|
647
|
+
Example: 100 medium-sized articles ≈ 45-60 seconds
|
|
648
|
+
|
|
649
|
+
Note: The raw XML is not included in JSON format to save storage. Use `--format xml` if you need the raw XML.
|
|
650
|
+
|
|
651
|
+
## Testing
|
|
652
|
+
|
|
653
|
+
The repository includes a comprehensive Jupyter notebook for testing all features:
|
|
654
|
+
|
|
655
|
+
```bash
|
|
656
|
+
# Install Jupyter
|
|
657
|
+
pip install jupyter
|
|
658
|
+
|
|
659
|
+
# Launch test notebook
|
|
660
|
+
jupyter notebook test_download.ipynb
|
|
661
|
+
```
|
|
662
|
+
|
|
663
|
+
The notebook demonstrates:
|
|
664
|
+
- Basic and advanced download scenarios
|
|
665
|
+
- All format options (text, xml, both)
|
|
666
|
+
- Concurrent vs sequential modes
|
|
667
|
+
- Error handling and edge cases
|
|
668
|
+
- Performance comparisons
|
|
669
|
+
- Low-level API usage
|
|
670
|
+
|
|
671
|
+
### Running Tests
|
|
672
|
+
|
|
673
|
+
```bash
|
|
674
|
+
# Install development dependencies
|
|
675
|
+
pip install -e ".[dev]"
|
|
676
|
+
|
|
677
|
+
# Run tests (if available)
|
|
678
|
+
pytest tests/
|
|
679
|
+
```
|
|
680
|
+
|
|
681
|
+
## Contributing
|
|
682
|
+
|
|
683
|
+
Contributions are welcome! Please:
|
|
684
|
+
|
|
685
|
+
1. Fork the repository
|
|
686
|
+
2. Create a feature branch: `git checkout -b feature-name`
|
|
687
|
+
3. Make your changes with clear commit messages
|
|
688
|
+
4. Add tests if applicable
|
|
689
|
+
5. Ensure code passes linting: `ruff check .`
|
|
690
|
+
6. Submit a pull request
|
|
691
|
+
|
|
692
|
+
### Code Style
|
|
693
|
+
|
|
694
|
+
- Follow PEP 8
|
|
695
|
+
- Use type hints where appropriate
|
|
696
|
+
- Add docstrings for public functions
|
|
697
|
+
- Keep functions focused and modular
|
|
698
|
+
|
|
699
|
+
## Development Roadmap
|
|
700
|
+
|
|
701
|
+
Future enhancements being considered:
|
|
702
|
+
- [ ] Support for PubMed (non-PMC) abstracts
|
|
703
|
+
- [ ] Advanced search query builder
|
|
704
|
+
- [ ] Automatic deduplication
|
|
705
|
+
- [ ] Progress bars for CLI
|
|
706
|
+
- [ ] Export to BibTeX/RIS formats
|
|
707
|
+
- [ ] Integration with citation managers
|
|
708
|
+
- [ ] Parallel processing for metadata extraction
|
|
709
|
+
|
|
710
|
+
## FAQ
|
|
711
|
+
|
|
712
|
+
**Q: Why JSON instead of separate XML/TXT files?**
|
|
713
|
+
A: Unified JSON format provides consistent structure, always includes metadata, and simplifies file operations. You can still get raw XML (as a string field) if needed.
|
|
714
|
+
|
|
715
|
+
**Q: Can I use this for commercial purposes?**
|
|
716
|
+
A: Yes, under the MIT license. However, respect NCBI's usage policies and rate limits.
|
|
717
|
+
|
|
718
|
+
**Q: How do I get an NCBI API key?**
|
|
719
|
+
A: Register at https://www.ncbi.nlm.nih.gov/account/ and generate an API key in your account settings. It's free.
|
|
720
|
+
|
|
721
|
+
**Q: Does this violate NCBI terms of service?**
|
|
722
|
+
A: No - this library is built on official NCBI E-utilities APIs (ESearch and EFetch), respects rate limits, and includes proper user-agent headers in compliance with NCBI usage policies.
|
|
723
|
+
|
|
724
|
+
**Q: Can I resume interrupted downloads?**
|
|
725
|
+
A: Yes - already-downloaded files are automatically skipped (shown in `stats.skipped`).
|
|
726
|
+
|
|
727
|
+
**Q: What's the difference between PMC and PubMed?**
|
|
728
|
+
A: PubMed contains citation abstracts for millions of papers. PMC (PubMed Central) is a subset with full-text articles. This library accesses PMC for complete article text.
|
|
729
|
+
|
|
730
|
+
## Project Structure
|
|
731
|
+
|
|
732
|
+
```
|
|
733
|
+
pubmed-stream/
|
|
734
|
+
├── pubmed_stream/ # Main package
|
|
735
|
+
│ ├── __init__.py # Public API exports
|
|
736
|
+
│ ├── __main__.py # python -m pubmed_stream entry point
|
|
737
|
+
│ ├── cli.py # Command-line interface
|
|
738
|
+
│ ├── downloader.py # Core download logic
|
|
739
|
+
│ └── py.typed # Type hint marker
|
|
740
|
+
├── examples/
|
|
741
|
+
│ ├── keywords/ # Sample keyword CSV files
|
|
742
|
+
│ │ ├── pmc_literature_keywords.csv
|
|
743
|
+
│ │ └── ...
|
|
744
|
+
│ └── utils/ # Helper scripts
|
|
745
|
+
│ ├── manage_text_field.py
|
|
746
|
+
│ └── test_pmc_availability.py
|
|
747
|
+
├── test_download.ipynb # Comprehensive test notebook
|
|
748
|
+
├── pyproject.toml # Project metadata and dependencies
|
|
749
|
+
├── README.md # This file
|
|
750
|
+
└── .gitignore # Git ignore rules
|
|
751
|
+
```
|
|
752
|
+
|
|
753
|
+
## Acknowledgments
|
|
754
|
+
|
|
755
|
+
This library was developed to support biomedical research at the **University of Massachusetts** by:
|
|
756
|
+
- **Haran Lab**
|
|
757
|
+
- **Bucci Lab**
|
|
758
|
+
- **Microbiology & Microbiome Dynamics AI HUB**
|
|
759
|
+
|
|
760
|
+
Special thanks to:
|
|
761
|
+
- **NCBI** for providing the E-utilities API and open access to PubMed Central
|
|
762
|
+
- The open-source community for Python ecosystem tools
|
|
763
|
+
|
|
764
|
+
## Citation
|
|
765
|
+
|
|
766
|
+
If you use this library in your research, please cite:
|
|
767
|
+
|
|
768
|
+
```bibtex
|
|
769
|
+
@software{pubmed_stream_2026,
|
|
770
|
+
title = {pubmed-stream: Python Library for PMC Article Downloads via NCBI E-utilities},
|
|
771
|
+
author = {Haran Lab and Bucci Lab and Microbiology \& Microbiome Dynamics AI HUB},
|
|
772
|
+
organization = {University of Massachusetts},
|
|
773
|
+
year = {2026},
|
|
774
|
+
url = {https://github.com/melhzy/pubmed-stream},
|
|
775
|
+
note = {Built on NCBI E-utilities API}
|
|
776
|
+
}
|
|
777
|
+
```
|
|
778
|
+
|
|
779
|
+
## License
|
|
780
|
+
|
|
781
|
+
MIT License
|
|
782
|
+
|
|
783
|
+
Copyright (c) 2026 University of Massachusetts
|
|
784
|
+
Haran Lab, Bucci Lab, and Microbiology & Microbiome Dynamics AI HUB
|
|
785
|
+
|
|
786
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
787
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
788
|
+
in the Software without restriction, including without limitation the rights
|
|
789
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
790
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
791
|
+
furnished to do so, subject to the following conditions:
|
|
792
|
+
|
|
793
|
+
The above copyright notice and this permission notice shall be included in all
|
|
794
|
+
copies or substantial portions of the Software.
|
|
795
|
+
|
|
796
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
797
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
798
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
799
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
800
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
801
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
802
|
+
SOFTWARE.
|
|
803
|
+
|
|
804
|
+
---
|
|
805
|
+
|
|
806
|
+
**Contact**: For questions or support, please open an issue on GitHub.
|
|
807
|
+
|
|
808
|
+
**Repository**: https://github.com/melhzy/pubmed-stream
|