pubmed-stream 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,808 @@
1
+ Metadata-Version: 2.4
2
+ Name: pubmed-stream
3
+ Version: 0.1.0
4
+ Summary: Download PubMed Central full-text articles via NCBI E-utilities API. Built for biomedical research.
5
+ Author: Haran Lab, Bucci Lab, and Microbiology & Microbiome Dynamics AI HUB
6
+ Maintainer: University of Massachusetts
7
+ License: MIT
8
+ Project-URL: Repository, https://github.com/melhzy/pubmed-stream
9
+ Project-URL: Homepage, https://github.com/melhzy/pubmed-stream
10
+ Project-URL: Bug Tracker, https://github.com/melhzy/pubmed-stream/issues
11
+ Keywords: pubmed,pmc,ncbi,e-utilities,literature,download,biomedical,research
12
+ Classifier: Development Status :: 4 - Beta
13
+ Classifier: Intended Audience :: Science/Research
14
+ Classifier: License :: OSI Approved :: MIT License
15
+ Classifier: Programming Language :: Python :: 3
16
+ Classifier: Programming Language :: Python :: 3.8
17
+ Classifier: Programming Language :: Python :: 3.9
18
+ Classifier: Programming Language :: Python :: 3.10
19
+ Classifier: Programming Language :: Python :: 3.11
20
+ Classifier: Programming Language :: Python :: 3.12
21
+ Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
22
+ Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
23
+ Requires-Python: >=3.8
24
+ Description-Content-Type: text/markdown
25
+ Requires-Dist: requests>=2.25
26
+ Provides-Extra: dev
27
+ Requires-Dist: pytest; extra == "dev"
28
+ Requires-Dist: ruff; extra == "dev"
29
+
30
+ # pubmed-stream
31
+
32
+ A Python library for downloading PubMed Central (PMC) full-text articles with support for concurrent downloads, automatic rate limiting, metadata extraction, and multiple output formats.
33
+
34
+ **Built on NCBI E-utilities API** for reliable, standards-compliant access to biomedical literature.
35
+
36
+ ## Overview
37
+
38
+ `pubmed-stream` simplifies the process of downloading scientific literature from PubMed Central by leveraging the **NCBI E-utilities API** (ESearch and EFetch):
39
+ - **Searching PMC directly** for guaranteed full-text availability
40
+ - **Extracting structured metadata** from JATS XML (title, authors, abstract, keywords, DOI, etc.)
41
+ - **Supporting multiple formats**: plain text, raw XML, or both in a single JSON file
42
+ - **Handling concurrent downloads** with configurable thread pools
43
+ - **Respecting NCBI rate limits** with automatic detection and retry logic
44
+ - **Providing both CLI and Python API** for flexible integration
45
+
46
+ Built for researchers and data scientists who need efficient, reliable access to biomedical literature.
47
+
48
+ ## Quick Start
49
+
50
+ ```bash
51
+ # Install
52
+ pip install .
53
+
54
+ # Download articles (CLI)
55
+ pubmed-stream download "gut microbiome" --max-results 50
56
+
57
+ # Python API
58
+ from pubmed_stream import search_and_download
59
+ stats = search_and_download(
60
+ keyword="CRISPR gene editing",
61
+ max_results=100,
62
+ fmt="text"
63
+ )
64
+ print(f"Downloaded {stats.successful} articles in {stats.duration_seconds:.1f}s")
65
+ ```
66
+
67
+ ## Table of Contents
68
+
69
+ - [Installation](#installation)
70
+ - [How It Works](#how-it-works)
71
+ - [Configuration](#configuration-optional)
72
+ - [CLI Usage](#cli-usage)
73
+ - [Python API](#python-api)
74
+ - [Output Format](#output-format)
75
+ - [Use Cases](#use-cases)
76
+ - [Troubleshooting](#troubleshooting)
77
+ - [Performance Tuning](#performance-tuning)
78
+ - [Testing](#testing)
79
+ - [Contributing](#contributing)
80
+ - [License](#license)
81
+
82
+ ## Installation
83
+
84
+ ### From Local Clone
85
+
86
+ ```bash
87
+ # Clone the repository
88
+ git clone https://github.com/melhzy/pubmed-stream.git
89
+ cd pubmed-stream
90
+
91
+ # Install in standard mode
92
+ pip install .
93
+
94
+ # Or install in development mode (editable install)
95
+ pip install -e ".[dev]"
96
+ ```
97
+
98
+ ### Requirements
99
+
100
+ - Python 3.8+
101
+ - [requests](https://pypi.org/project/requests/) (only runtime dependency)
102
+
103
+ Development dependencies (optional):
104
+ - pytest (for testing)
105
+ - ruff (for linting)
106
+
107
+ ## How It Works
108
+
109
+ ### Architecture
110
+
111
+ The library follows a modular design with three main components, all built on **NCBI E-utilities**:
112
+
113
+ 1. **Search Module** (`esearch_pmc`)
114
+ - Uses NCBI E-utilities **ESearch** to query PMC database directly
115
+ - Returns PMC IDs for articles with guaranteed full-text availability
116
+ - Handles pagination and result limiting
117
+
118
+ 2. **Download Module** (`efetch_pmc`)
119
+ - Uses NCBI E-utilities **EFetch** to retrieve full XML article data from PMC
120
+ - Extracts structured metadata using JATS XML parsing
121
+ - Converts XML to plain text (optional)
122
+ - Saves in requested format(s)
123
+
124
+ 3. **Orchestration Layer** (`search_and_download`)
125
+ - Combines search and download operations
126
+ - Manages concurrent downloads with thread pools
127
+ - Handles rate limiting, retries, and error recovery
128
+ - Tracks statistics (success/failure/skipped counts)
129
+
130
+ ### Search Strategy
131
+
132
+ **Direct PMC Search**: Unlike some tools that search PubMed and then try to find PMC links, this library searches PMC directly. This ensures:
133
+ - All results have full-text available
134
+ - No broken links or paywalled content
135
+ - Faster, more reliable downloads
136
+ - Better results for generic queries like "microbiome" or "CRISPR"
137
+
138
+ ### Data Flow
139
+
140
+ ```
141
+ User Query → PMC Search → PMC IDs → Concurrent Downloads → Parse XML → Extract Metadata → Save JSON
142
+ ↓ ↓ ↓ ↓ ↓ ↓ ↓
143
+ "microbiome" E-Search [123, 456] Thread Pool JATS Parser {title, authors} PMC*.json
144
+ ```
145
+
146
+ ### Output Format Options
147
+
148
+ All formats save as `.json` files with different content:
149
+
150
+ **`"text"` (default)**: JSON with plain-text content
151
+ ```json
152
+ {
153
+ "pmcid": "PMC12345678",
154
+ "metadata": {"title": "...", "authors": [...]},
155
+ "text": "Introduction\nThis study examines..."
156
+ }
157
+ ```
158
+
159
+ **`"xml"`**: JSON with raw JATS XML string
160
+ ```json
161
+ {
162
+ "pmcid": "PMC12345678",
163
+ "metadata": {"title": "...", "authors": [...]},
164
+ "xml": "<article>...</article>"
165
+ }
166
+ ```
167
+
168
+ **`"both"`**: JSON with both XML and text
169
+ ```json
170
+ {
171
+ "pmcid": "PMC12345678",
172
+ "metadata": {"title": "...", "authors": [...]},
173
+ "xml": "<article>...</article>",
174
+ "text": "Introduction\nThis study examines..."
175
+ }
176
+ ```
177
+
178
+ **Why unified JSON?**
179
+ - Consistent file handling (always `.json`)
180
+ - Metadata always included
181
+ - Easy to query and filter
182
+ - Smaller storage (text-only is ~50% smaller than XML)
183
+ - Backward compatible with existing pipelines
184
+
185
+ ## Configuration (optional)
186
+
187
+ Set these environment variables for best results:
188
+
189
+ ```bash
190
+ # PowerShell
191
+ $env:NCBI_API_KEY = "YOUR_KEY" # raises rate limit from 3→10 req/s
192
+ $env:NCBI_EMAIL = "you@example.com"
193
+
194
+ # Bash / Zsh
195
+ export NCBI_API_KEY="YOUR_KEY"
196
+ export NCBI_EMAIL="you@example.com"
197
+ ```
198
+
199
+ You can also pass `--api-key` / `--email` on the command line.
200
+
201
+ ### Rate Limits
202
+
203
+ NCBI enforces rate limits on API requests:
204
+
205
+ - **Without API key**: 3 requests/second
206
+ - **With API key**: 10 requests/second
207
+
208
+ The library **automatically detects** whether you have an API key and adjusts the rate limit accordingly:
209
+ - With API key: 0.1s between requests (10 req/s)
210
+ - Without API key: 0.34s between requests (3 req/s)
211
+
212
+ You can override this with `--rate-limit` if needed, but the auto-detection works well for most cases.
213
+
214
+ **Recommendation**: Get a free API key to download ~3x faster. Without a key, the library still works but is slower.
215
+
216
+ ## CLI usage
217
+
218
+ After installation the `pubmed-stream` command is available:
219
+
220
+ ```bash
221
+ # Download up to 50 articles as JSON (default, concurrent)
222
+ pubmed-stream download "frailty cytokines" --max-results 50
223
+
224
+ # Sequential mode for unstable networks
225
+ pubmed-stream download "gut microbiome" --sequential
226
+
227
+ # Custom output directory
228
+ pubmed-stream download "Alzheimer microbiome" -o ./my_papers
229
+
230
+ # Verbose logging
231
+ pubmed-stream download "frailty" -v --max-results 5
232
+
233
+ # Show version
234
+ pubmed-stream --version
235
+ ```
236
+
237
+ The same interface is available via `python -m pubmed_stream`:
238
+
239
+ ```bash
240
+ python -m pubmed_stream download "frailty cytokines" --max-results 50
241
+ ```
242
+
243
+ ### CLI options
244
+
245
+ | Flag | Description | Default |
246
+ |------|-------------|---------|
247
+ | `keyword` | PubMed search query (positional) | required |
248
+ | `--max-results` | Maximum articles to download | 100 |
249
+ | `--format` | Output format: `text`, `xml`, `both` (all save as `.json`) | `text` |
250
+ | `--output-dir`, `-o` | Base output directory | `./publications` |
251
+ | `--api-key` | NCBI API key | `NCBI_API_KEY` env var |
252
+ | `--email` | Contact email for NCBI | `NCBI_EMAIL` env var |
253
+ | `--sequential` | Disable concurrent downloads | concurrent |
254
+ | `--workers` | Concurrent thread count | 5 |
255
+ | `--exclude-text` | Omit `text` field in JSON (~30% smaller) | included |
256
+ | `--rate-limit` | Min seconds between requests | auto: 0.1 with key, 0.34 without |
257
+ | `--user-agent` | Custom HTTP `User-Agent` | `pubmed-stream/<ver>` |
258
+ | `--verbose`, `-v` | DEBUG-level logging | INFO |
259
+
260
+ ### Exit codes
261
+
262
+ | Code | Meaning |
263
+ |------|---------|
264
+ | 0 | At least one article downloaded (or all already cached) |
265
+ | 1 | No results found for the query |
266
+ | 2 | All download attempts failed |
267
+
268
+ ## Python API
269
+
270
+ ### Basic Usage
271
+
272
+ ```python
273
+ from pubmed_stream import search_and_download
274
+
275
+ # Download articles with default settings
276
+ stats = search_and_download(
277
+ keyword="gut microbiome inflammation",
278
+ max_results=50,
279
+ fmt="text", # or "xml", "both"
280
+ output_dir="publications",
281
+ )
282
+
283
+ print(f"Downloaded: {stats.successful}")
284
+ print(f"Skipped (already exist): {stats.skipped}")
285
+ print(f"Failed: {stats.failed}")
286
+ print(f"Success rate: {stats.success_rate:.1f}%")
287
+ print(f"Duration: {stats.duration_seconds:.1f}s")
288
+ ```
289
+
290
+ ### Advanced Usage
291
+
292
+ ```python
293
+ from pubmed_stream import search_and_download
294
+ from pathlib import Path
295
+
296
+ # Full control over all parameters
297
+ stats = search_and_download(
298
+ keyword="CRISPR gene editing",
299
+ max_results=100,
300
+ fmt="both", # Save both XML and text
301
+ api_key="your_api_key", # Or use NCBI_API_KEY env var
302
+ use_concurrent=True, # Enable parallel downloads
303
+ max_workers=10, # Use 10 threads
304
+ include_text=True, # Include plain-text field
305
+ output_dir=Path("./research/crispr"),
306
+ user_agent="MyResearchBot/1.0",
307
+ email="researcher@university.edu",
308
+ rate_limit=0.15, # Custom rate limit
309
+ )
310
+
311
+ # Access detailed statistics
312
+ print(f"Total found in PMC: {stats.total_found}")
313
+ print(f"Requested: {stats.requested}")
314
+ print(f"Output: {stats.output_dir}")
315
+ ```
316
+
317
+ ### Low-Level API
318
+
319
+ For more control, use the underlying functions:
320
+
321
+ ```python
322
+ from pubmed_stream import esearch_pmc, efetch_pmc, create_session, RateLimiter
323
+ from pathlib import Path
324
+
325
+ # Create session and rate limiter
326
+ session = create_session("MyApp/1.0")
327
+ rate_limiter = RateLimiter(0.1) # 10 req/s
328
+
329
+ # Search for PMC IDs
330
+ pmcids, total_found = esearch_pmc(
331
+ term="microbiome AND aging",
332
+ max_results=20,
333
+ api_key="your_key",
334
+ session=session,
335
+ rate_limiter=rate_limiter
336
+ )
337
+
338
+ print(f"Found {len(pmcids)} PMC IDs (total: {total_found})")
339
+
340
+ # Download individual articles
341
+ for pmcid in pmcids:
342
+ success, status = efetch_pmc(
343
+ pmcid=pmcid,
344
+ out_dir=Path("downloads"),
345
+ fmt="text",
346
+ api_key="your_key",
347
+ session=session,
348
+ rate_limiter=rate_limiter,
349
+ include_text=True
350
+ )
351
+
352
+ if success:
353
+ print(f"✓ Downloaded PMC{pmcid}")
354
+ else:
355
+ print(f"✗ Failed PMC{pmcid}: {status}")
356
+
357
+ session.close()
358
+ ```
359
+
360
+ ### Metadata Extraction
361
+
362
+ The library automatically extracts metadata from JATS XML:
363
+
364
+ ```python
365
+ from pubmed_stream import extract_metadata_from_pmc_xml
366
+
367
+ xml_content = """<article>...</article>""" # Your PMC XML
368
+
369
+ metadata = extract_metadata_from_pmc_xml(xml_content)
370
+
371
+ # Available metadata fields
372
+ print(metadata.get("title")) # Article title
373
+ print(metadata.get("journal")) # Journal name
374
+ print(metadata.get("doi")) # DOI
375
+ print(metadata.get("pmid")) # PubMed ID
376
+ print(metadata.get("pmcid")) # PMC ID
377
+ print(metadata.get("authors")) # List of authors
378
+ print(metadata.get("abstract")) # Abstract text
379
+ print(metadata.get("keywords")) # Keywords list
380
+ print(metadata.get("year")) # Publication year
381
+ print(metadata.get("pub_date")) # Full publication date
382
+ ```
383
+
384
+ ## Output Format
385
+
386
+ ### Directory Structure
387
+
388
+ Articles are organized by search query:
389
+
390
+ ```
391
+ publications/
392
+ gut_microbiome_inflammation/
393
+ PMC12345678.json
394
+ PMC12345679.json
395
+ PMC12345680.json
396
+ alzheimer_disease/
397
+ PMC11111111.json
398
+ PMC22222222.json
399
+ ```
400
+
401
+ Query strings are slugified (spaces→underscores, special characters removed) for clean directory names.
402
+
403
+ ### JSON Schema
404
+
405
+ Each JSON file contains:
406
+
407
+ ```json
408
+ {
409
+ "pmcid": "PMC12345678",
410
+ "source": "PMC",
411
+ "download_date": "2026-02-17T12:30:45.123456",
412
+ "metadata": {
413
+ "title": "The Role of Gut Microbiome in Inflammation",
414
+ "journal": "Nature Medicine",
415
+ "journal_title": "Nature Medicine",
416
+ "journal_nlm_ta": "Nat Med",
417
+ "journal_iso_abbrev": "Nat. Med.",
418
+ "doi": "10.1038/s41591-024-12345-6",
419
+ "pmid": "38123456",
420
+ "pmcid": "PMC12345678",
421
+ "year": "2024",
422
+ "month": "11",
423
+ "day": "15",
424
+ "pub_date": {
425
+ "year": "2024",
426
+ "month": "11",
427
+ "day": "15"
428
+ },
429
+ "authors": [
430
+ "Smith, John A.",
431
+ "Johnson, Mary B.",
432
+ "Williams, Robert C."
433
+ ],
434
+ "abstract": "Background: The gut microbiome plays a crucial role...",
435
+ "keywords": [
436
+ "microbiome",
437
+ "inflammation",
438
+ "gut health"
439
+ ]
440
+ },
441
+ "text": "Introduction\nThe human gut microbiome comprises...",
442
+ "xml": "<article>...</article>"
443
+ }
444
+ ```
445
+
446
+ **Field descriptions:**
447
+ - `pmcid`: PubMed Central identifier
448
+ - `source`: Always "PMC"
449
+ - `download_date`: ISO 8601 timestamp
450
+ - `metadata`: Structured article information
451
+ - `text`: Plain-text content (if `fmt="text"` or `fmt="both"`)
452
+ - `xml`: Raw JATS XML string (if `fmt="xml"` or `fmt="both"`)
453
+
454
+ ### Format Comparison
455
+
456
+ | Format | File Type | Contains | Use Case | Size |
457
+ |--------|-----------|----------|----------|------|
458
+ | `text` | `.json` | metadata + text | Vector DBs, search, NLP | Smallest (~50% of XML) |
459
+ | `xml` | `.json` | metadata + xml | Section parsing, structure analysis | Medium |
460
+ | `both` | `.json` | metadata + xml + text | Comprehensive analysis | Largest |
461
+
462
+ ## Use Cases
463
+
464
+ ### 1. Literature Review & Meta-Analysis
465
+
466
+ ```python
467
+ # Download recent papers on a specific topic
468
+ stats = search_and_download(
469
+ keyword="COVID-19 vaccine efficacy",
470
+ max_results=200,
471
+ fmt="text",
472
+ output_dir="covid19_review"
473
+ )
474
+
475
+ # Process downloaded articles
476
+ import json
477
+ from pathlib import Path
478
+
479
+ for json_file in Path("covid19_review/covid_19_vaccine_efficacy").glob("*.json"):
480
+ with open(json_file) as f:
481
+ article = json.load(f)
482
+
483
+ print(f"Title: {article['metadata']['title']}")
484
+ print(f"Year: {article['metadata']['year']}")
485
+ print(f"Abstract: {article['metadata']['abstract'][:200]}...")
486
+ ```
487
+
488
+ ### 2. Vector Database Embedding
489
+
490
+ ```python
491
+ # Download for embedding into vector database
492
+ stats = search_and_download(
493
+ keyword="machine learning healthcare",
494
+ max_results=1000,
495
+ fmt="text", # Plain text is ideal for embeddings
496
+ include_text=True
497
+ )
498
+
499
+ # Example: Embed into ChromaDB
500
+ import chromadb
501
+ from pathlib import Path
502
+ import json
503
+
504
+ client = chromadb.Client()
505
+ collection = client.create_collection("medical_papers")
506
+
507
+ for json_file in Path("publications/machine_learning_healthcare").glob("*.json"):
508
+ with open(json_file) as f:
509
+ article = json.load(f)
510
+
511
+ collection.add(
512
+ documents=[article["text"]],
513
+ metadatas=[{
514
+ "title": article["metadata"]["title"],
515
+ "year": article["metadata"]["year"],
516
+ "doi": article["metadata"].get("doi", ""),
517
+ }],
518
+ ids=[article["pmcid"]]
519
+ )
520
+ ```
521
+
522
+ ### 3. Section-Level Analysis
523
+
524
+ ```python
525
+ # Download with XML for detailed structure parsing
526
+ stats = search_and_download(
527
+ keyword="CRISPR therapy clinical trials",
528
+ max_results=50,
529
+ fmt="xml" # Get raw XML for section extraction
530
+ )
531
+
532
+ # Parse sections from XML
533
+ import xml.etree.ElementTree as ET
534
+ import json
535
+ from pathlib import Path
536
+
537
+ for json_file in Path("publications/crispr_therapy_clinical_trials").glob("*.json"):
538
+ with open(json_file) as f:
539
+ article = json.load(f)
540
+
541
+ # Parse XML to extract sections
542
+ root = ET.fromstring(article["xml"])
543
+
544
+ # Find all sections with sec-type attribute
545
+ for sec in root.findall(".//sec[@sec-type]"):
546
+ sec_type = sec.get("sec-type")
547
+ title = sec.find("title")
548
+
549
+ if title is not None:
550
+ print(f"Section: {sec_type} - {title.text}")
551
+ ```
552
+
553
+ ### 4. Batch Processing with Keywords
554
+
555
+ ```python
556
+ # Process multiple queries from CSV
557
+ import pandas as pd
558
+
559
+ keywords_df = pd.read_csv("research_keywords.csv")
560
+
561
+ for _, row in keywords_df.iterrows():
562
+ print(f"Processing: {row['keyword']}")
563
+
564
+ stats = search_and_download(
565
+ keyword=row['keyword'],
566
+ max_results=row['max_articles'],
567
+ fmt="both", # Get both formats for flexibility
568
+ output_dir="batch_download"
569
+ )
570
+
571
+ print(f" Downloaded: {stats.successful}, Success rate: {stats.success_rate:.1f}%")
572
+ ```
573
+
574
+ ## Troubleshooting
575
+
576
+ ### Common Issues
577
+
578
+ **1. "0 downloads" but high PMC result count**
579
+ - **Cause**: Files already downloaded (skipped)
580
+ - **Solution**: Check `stats.skipped` - this is actually success! Delete folder to re-download.
581
+
582
+ **2. Rate limiting (HTTP 429)**
583
+ - **Cause**: Exceeding NCBI rate limits
584
+ - **Solution**:
585
+ - Get a free API key (increases limit from 3→10 req/s)
586
+ - Use `--rate-limit` to slow down: `--rate-limit 0.5`
587
+ - The library automatically retries with backoff
588
+
589
+ **3. "Unavailable" articles**
590
+ - **Cause**: Article not in PMC or embargoed
591
+ - **Solution**: Normal - not all PubMed articles have PMC full-text. The library only returns available articles.
592
+
593
+ **4. Network timeouts**
594
+ - **Cause**: Unstable internet connection
595
+ - **Solution**: Use `--sequential` mode: `pubmed-stream download "query" --sequential`
596
+
597
+ **5. Memory issues with large downloads**
598
+ - **Cause**: Processing too many articles concurrently
599
+ - **Solution**: Reduce workers: `--workers 2` or use `--sequential`
600
+
601
+ ### Debugging
602
+
603
+ Enable verbose logging to see detailed information:
604
+
605
+ ```bash
606
+ pubmed-stream download "microbiome" -v --max-results 10
607
+ ```
608
+
609
+ ```python
610
+ import logging
611
+ logging.basicConfig(level=logging.DEBUG)
612
+
613
+ from pubmed_stream import search_and_download
614
+ stats = search_and_download(keyword="test", max_results=5)
615
+ ```
616
+
617
+ ## Performance Tuning
618
+
619
+ ### Concurrent vs Sequential
620
+
621
+ **Concurrent (default)**: Faster for stable networks
622
+ ```bash
623
+ pubmed-stream download "query" --workers 10 # 10 parallel downloads
624
+ ```
625
+
626
+ **Sequential**: More reliable for unstable networks
627
+ ```bash
628
+ pubmed-stream download "query" --sequential
629
+ ```
630
+
631
+ ### Optimal Settings by Network
632
+
633
+ | Network Quality | Mode | Workers | Rate Limit |
634
+ |----------------|------|---------|------------|
635
+ | Excellent | Concurrent | 10 | 0.1 (with API key) |
636
+ | Good | Concurrent | 5 | 0.1 |
637
+ | Fair | Concurrent | 2-3 | 0.2 |
638
+ | Poor | Sequential | 1 | 0.34 |
639
+
640
+ ### Performance Benchmarks
641
+
642
+ Typical download speeds (with API key, 5 workers):
643
+ - **Small articles** (~50KB): 2-5 articles/second
644
+ - **Medium articles** (~200KB): 1-3 articles/second
645
+ - **Large articles** (~1MB): 0.5-1 articles/second
646
+
647
+ Example: 100 medium-sized articles ≈ 45-60 seconds
648
+
649
+ Note: The raw XML is not included in JSON format to save storage. Use `--format xml` if you need the raw XML.
650
+
651
+ ## Testing
652
+
653
+ The repository includes a comprehensive Jupyter notebook for testing all features:
654
+
655
+ ```bash
656
+ # Install Jupyter
657
+ pip install jupyter
658
+
659
+ # Launch test notebook
660
+ jupyter notebook test_download.ipynb
661
+ ```
662
+
663
+ The notebook demonstrates:
664
+ - Basic and advanced download scenarios
665
+ - All format options (text, xml, both)
666
+ - Concurrent vs sequential modes
667
+ - Error handling and edge cases
668
+ - Performance comparisons
669
+ - Low-level API usage
670
+
671
+ ### Running Tests
672
+
673
+ ```bash
674
+ # Install development dependencies
675
+ pip install -e ".[dev]"
676
+
677
+ # Run tests (if available)
678
+ pytest tests/
679
+ ```
680
+
681
+ ## Contributing
682
+
683
+ Contributions are welcome! Please:
684
+
685
+ 1. Fork the repository
686
+ 2. Create a feature branch: `git checkout -b feature-name`
687
+ 3. Make your changes with clear commit messages
688
+ 4. Add tests if applicable
689
+ 5. Ensure code passes linting: `ruff check .`
690
+ 6. Submit a pull request
691
+
692
+ ### Code Style
693
+
694
+ - Follow PEP 8
695
+ - Use type hints where appropriate
696
+ - Add docstrings for public functions
697
+ - Keep functions focused and modular
698
+
699
+ ## Development Roadmap
700
+
701
+ Future enhancements being considered:
702
+ - [ ] Support for PubMed (non-PMC) abstracts
703
+ - [ ] Advanced search query builder
704
+ - [ ] Automatic deduplication
705
+ - [ ] Progress bars for CLI
706
+ - [ ] Export to BibTeX/RIS formats
707
+ - [ ] Integration with citation managers
708
+ - [ ] Parallel processing for metadata extraction
709
+
710
+ ## FAQ
711
+
712
+ **Q: Why JSON instead of separate XML/TXT files?**
713
+ A: Unified JSON format provides consistent structure, always includes metadata, and simplifies file operations. You can still get raw XML (as a string field) if needed.
714
+
715
+ **Q: Can I use this for commercial purposes?**
716
+ A: Yes, under the MIT license. However, respect NCBI's usage policies and rate limits.
717
+
718
+ **Q: How do I get an NCBI API key?**
719
+ A: Register at https://www.ncbi.nlm.nih.gov/account/ and generate an API key in your account settings. It's free.
720
+
721
+ **Q: Does this violate NCBI terms of service?**
722
+ A: No - this library is built on official NCBI E-utilities APIs (ESearch and EFetch), respects rate limits, and includes proper user-agent headers in compliance with NCBI usage policies.
723
+
724
+ **Q: Can I resume interrupted downloads?**
725
+ A: Yes - already-downloaded files are automatically skipped (shown in `stats.skipped`).
726
+
727
+ **Q: What's the difference between PMC and PubMed?**
728
+ A: PubMed contains citation abstracts for millions of papers. PMC (PubMed Central) is a subset with full-text articles. This library accesses PMC for complete article text.
729
+
730
+ ## Project Structure
731
+
732
+ ```
733
+ pubmed-stream/
734
+ ├── pubmed_stream/ # Main package
735
+ │ ├── __init__.py # Public API exports
736
+ │ ├── __main__.py # python -m pubmed_stream entry point
737
+ │ ├── cli.py # Command-line interface
738
+ │ ├── downloader.py # Core download logic
739
+ │ └── py.typed # Type hint marker
740
+ ├── examples/
741
+ │ ├── keywords/ # Sample keyword CSV files
742
+ │ │ ├── pmc_literature_keywords.csv
743
+ │ │ └── ...
744
+ │ └── utils/ # Helper scripts
745
+ │ ├── manage_text_field.py
746
+ │ └── test_pmc_availability.py
747
+ ├── test_download.ipynb # Comprehensive test notebook
748
+ ├── pyproject.toml # Project metadata and dependencies
749
+ ├── README.md # This file
750
+ └── .gitignore # Git ignore rules
751
+ ```
752
+
753
+ ## Acknowledgments
754
+
755
+ This library was developed to support biomedical research at the **University of Massachusetts** by:
756
+ - **Haran Lab**
757
+ - **Bucci Lab**
758
+ - **Microbiology & Microbiome Dynamics AI HUB**
759
+
760
+ Special thanks to:
761
+ - **NCBI** for providing the E-utilities API and open access to PubMed Central
762
+ - The open-source community for Python ecosystem tools
763
+
764
+ ## Citation
765
+
766
+ If you use this library in your research, please cite:
767
+
768
+ ```bibtex
769
+ @software{pubmed_stream_2026,
770
+ title = {pubmed-stream: Python Library for PMC Article Downloads via NCBI E-utilities},
771
+ author = {Haran Lab and Bucci Lab and Microbiology \& Microbiome Dynamics AI HUB},
772
+ organization = {University of Massachusetts},
773
+ year = {2026},
774
+ url = {https://github.com/melhzy/pubmed-stream},
775
+ note = {Built on NCBI E-utilities API}
776
+ }
777
+ ```
778
+
779
+ ## License
780
+
781
+ MIT License
782
+
783
+ Copyright (c) 2026 University of Massachusetts
784
+ Haran Lab, Bucci Lab, and Microbiology & Microbiome Dynamics AI HUB
785
+
786
+ Permission is hereby granted, free of charge, to any person obtaining a copy
787
+ of this software and associated documentation files (the "Software"), to deal
788
+ in the Software without restriction, including without limitation the rights
789
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
790
+ copies of the Software, and to permit persons to whom the Software is
791
+ furnished to do so, subject to the following conditions:
792
+
793
+ The above copyright notice and this permission notice shall be included in all
794
+ copies or substantial portions of the Software.
795
+
796
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
797
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
798
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
799
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
800
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
801
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
802
+ SOFTWARE.
803
+
804
+ ---
805
+
806
+ **Contact**: For questions or support, please open an issue on GitHub.
807
+
808
+ **Repository**: https://github.com/melhzy/pubmed-stream