biolit 0.1.0__tar.gz

biolit-0.1.0/PKG-INFO

@@ -0,0 +1,153 @@
+Metadata-Version: 2.4
+Name: biolit
+Version: 0.1.0
+Summary: LLM-assisted biomedical literature screening and structured extraction for PubMed and GEO.
+Author-email: Rachel Schwartz <raschwaaa@gmail.com>
+License-Expression: MIT
+Project-URL: Repository, https://github.com/rachadele/pubmed-screener
+Project-URL: Homepage, https://github.com/rachadele/pubmed-screener#readme
+Keywords: pubmed,geo,literature-review,llm,bioinformatics,genomics,mcp
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: anthropic
+Requires-Dist: openai
+Requires-Dist: requests
+Requires-Dist: python-dotenv
+Requires-Dist: lxml
+Requires-Dist: pdfminer.six
+
+# biolit
+
+LLM-assisted biomedical literature screening and structured extraction. Accepts PubMed alert emails, plain PMID lists, or GEO accession lists. Supports multiple LLM providers and optional full-text retrieval.
+
+## Setup
+
+**Requirements:** Python 3.8+
+
+Install the package (creates the `biolit` command):
+
+```bash
+pip install -e .
+```
+
+Copy `.env.example` to `.env` and add your API key:
+
+```bash
+cp .env.example .env
+# edit .env and set ANTHROPIC_API_KEY (or OPENAI_API_KEY)
+```
+
+## Usage
+
+The tool accepts several input formats, auto-detected by file extension or content:
+
+| Input | How to pass | Example |
+|---|---|---|
+| PubMed alert email | positional `.eml` file | `alert.eml` |
+| PMID list (file) | positional plain-text file, one PMID per line | `pmids.txt` |
+| GEO accession list (file) | positional plain-text file, one accession per line | `geo_accessions.txt` |
+| PMIDs (inline) | `--pmids` flag, comma-separated | `--pmids 41795042,41792186` |
+| GEO accessions (inline) | `--accessions` flag, comma-separated | `--accessions GSE53987,GSE12345` |
+
+Use `--default` to run with schizophrenia genomics defaults (no prompts):
+
+```bash
+biolit alert.eml --default
+biolit pmids.txt --default
+biolit geo_accessions.txt --default
+biolit --pmids 41795042,41792186 --default
+biolit --accessions GSE53987 --default
+```
+
+Or specify criterion and fields as flags:
+
+```bash
+biolit pmids.txt \
+  --criterion "Is this about treatment-resistant schizophrenia?" \
+  --fields "methodology, sample_size, treatment, outcomes"
+```
+
+Or interactively (prompted if not provided):
+
+```bash
+biolit alert.eml
+```
+
+### GEO accession input
+
+Pass a file of GEO series accessions (GSE, GDS, GSM, or GPL prefixes) to screen GEO records directly. The tool fetches each record's MINiML XML, extracts the summary, overall design, experiment type, and organism, then runs the same LLM screening and extraction pipeline.
+
+```bash
+biolit geo_accessions.txt \
+  --criterion "Does this study perturb a transcription factor?" \
+  --fields "organism, experiment_type, tf_perturbed, perturbation_method, summary"
+```
+
+GEO results include `geo_accession` and `pmids` (linked PubMed IDs) columns in place of `pmid`.
+
+### Full-text retrieval (PubMed inputs only)
+
+Use `--fulltext` to screen and extract from full text instead of just the abstract. The pipeline tries each source in order:
+
+1. PMC JATS XML (open access)
+2. Preprint XML (bioRxiv / medRxiv)
+3. Unpaywall PDF (requires `--unpaywall-email`)
+4. Abstract fallback
+
+```bash
+biolit alert.eml --default --fulltext --unpaywall-email you@example.com
+```
+
+Limit which sections are sent to the LLM:
+
+```bash
+biolit alert.eml --default --fulltext --sections methods,results
+```
+
+### LLM providers
+
+The tool supports Anthropic (default), OpenAI, and local Ollama models:
+
+```bash
+# OpenAI
+biolit pmids.txt --default --provider openai --model gpt-4o
+
+# Ollama (local)
+biolit pmids.txt --default --provider ollama --model llama3
+```
+
+You can also set `LLM_PROVIDER` and `LLM_MODEL` as environment variables.
+
+## Output
+
+Each run creates a timestamped directory (e.g. `run_20260313_142000/`) containing:
+
+- `results.csv` — one row per relevant record
+- `artifacts/<id>/` — per-record folder with the text sent to the LLM, metadata, and any retrieved full-text files
+
+With `--default` on PubMed inputs, the CSV columns are:
+
+| Column | Description |
+|---|---|
+| `title` | Paper title |
+| `url` | PubMed link |
+| `pmid` | PubMed ID |
+| `doi` | DOI |
+| `text_source` | Where the text came from (`abstract`, `pmc_fulltext`, `preprint_fulltext`, `unpaywall_pdf`) |
+| `methodology` | General method (e.g. GWAS, scRNA-seq, proteomics) |
+| `sample_type` | Tissue/sample type and origin |
+| `causal_claims` | Statements about causes of schizophrenia inferred from the data |
+| `genetics_claims` | Claims about specific genes, loci, or pathways |
+| `summary` | 2-3 sentence plain-language summary for triage |
+
+For GEO inputs, `pmid` is replaced by `geo_accession` and `pmids`.
+
+The CSV can be imported directly into Google Sheets (File → Import).
+
+## Known Limitations
+
+- Papers without abstracts or accessible full text are skipped silently.
+- Full-text retrieval (`--fulltext`) applies to PubMed inputs only; GEO records use the record metadata directly.

biolit-0.1.0/README.md

@@ -0,0 +1,132 @@
+# biolit
+
+LLM-assisted biomedical literature screening and structured extraction. Accepts PubMed alert emails, plain PMID lists, or GEO accession lists. Supports multiple LLM providers and optional full-text retrieval.
+
+## Setup
+
+**Requirements:** Python 3.8+
+
+Install the package (creates the `biolit` command):
+
+```bash
+pip install -e .
+```
+
+Copy `.env.example` to `.env` and add your API key:
+
+```bash
+cp .env.example .env
+# edit .env and set ANTHROPIC_API_KEY (or OPENAI_API_KEY)
+```
+
+## Usage
+
+The tool accepts several input formats, auto-detected by file extension or content:
+
+| Input | How to pass | Example |
+|---|---|---|
+| PubMed alert email | positional `.eml` file | `alert.eml` |
+| PMID list (file) | positional plain-text file, one PMID per line | `pmids.txt` |
+| GEO accession list (file) | positional plain-text file, one accession per line | `geo_accessions.txt` |
+| PMIDs (inline) | `--pmids` flag, comma-separated | `--pmids 41795042,41792186` |
+| GEO accessions (inline) | `--accessions` flag, comma-separated | `--accessions GSE53987,GSE12345` |
+
+Use `--default` to run with schizophrenia genomics defaults (no prompts):
+
+```bash
+biolit alert.eml --default
+biolit pmids.txt --default
+biolit geo_accessions.txt --default
+biolit --pmids 41795042,41792186 --default
+biolit --accessions GSE53987 --default
+```
+
+Or specify criterion and fields as flags:
+
+```bash
+biolit pmids.txt \
+  --criterion "Is this about treatment-resistant schizophrenia?" \
+  --fields "methodology, sample_size, treatment, outcomes"
+```
+
+Or interactively (prompted if not provided):
+
+```bash
+biolit alert.eml
+```
+
+### GEO accession input
+
+Pass a file of GEO series accessions (GSE, GDS, GSM, or GPL prefixes) to screen GEO records directly. The tool fetches each record's MINiML XML, extracts the summary, overall design, experiment type, and organism, then runs the same LLM screening and extraction pipeline.
+
+```bash
+biolit geo_accessions.txt \
+  --criterion "Does this study perturb a transcription factor?" \
+  --fields "organism, experiment_type, tf_perturbed, perturbation_method, summary"
+```
+
+GEO results include `geo_accession` and `pmids` (linked PubMed IDs) columns in place of `pmid`.
+
+### Full-text retrieval (PubMed inputs only)
+
+Use `--fulltext` to screen and extract from full text instead of just the abstract. The pipeline tries each source in order:
+
+1. PMC JATS XML (open access)
+2. Preprint XML (bioRxiv / medRxiv)
+3. Unpaywall PDF (requires `--unpaywall-email`)
+4. Abstract fallback
+
+```bash
+biolit alert.eml --default --fulltext --unpaywall-email you@example.com
+```
+
+Limit which sections are sent to the LLM:
+
+```bash
+biolit alert.eml --default --fulltext --sections methods,results
+```
+
+### LLM providers
+
+The tool supports Anthropic (default), OpenAI, and local Ollama models:
+
+```bash
+# OpenAI
+biolit pmids.txt --default --provider openai --model gpt-4o
+
+# Ollama (local)
+biolit pmids.txt --default --provider ollama --model llama3
+```
+
+You can also set `LLM_PROVIDER` and `LLM_MODEL` as environment variables.
+
+## Output
+
+Each run creates a timestamped directory (e.g. `run_20260313_142000/`) containing:
+
+- `results.csv` — one row per relevant record
+- `artifacts/<id>/` — per-record folder with the text sent to the LLM, metadata, and any retrieved full-text files
+
+With `--default` on PubMed inputs, the CSV columns are:
+
+| Column | Description |
+|---|---|
+| `title` | Paper title |
+| `url` | PubMed link |
+| `pmid` | PubMed ID |
+| `doi` | DOI |
+| `text_source` | Where the text came from (`abstract`, `pmc_fulltext`, `preprint_fulltext`, `unpaywall_pdf`) |
+| `methodology` | General method (e.g. GWAS, scRNA-seq, proteomics) |
+| `sample_type` | Tissue/sample type and origin |
+| `causal_claims` | Statements about causes of schizophrenia inferred from the data |
+| `genetics_claims` | Claims about specific genes, loci, or pathways |
+| `summary` | 2-3 sentence plain-language summary for triage |
+
+For GEO inputs, `pmid` is replaced by `geo_accession` and `pmids`.
+
+The CSV can be imported directly into Google Sheets (File → Import).
+
+## Known Limitations
+
+- Papers without abstracts or accessible full text are skipped silently.
+- Full-text retrieval (`--fulltext`) applies to PubMed inputs only; GEO records use the record metadata directly.

biolit-0.1.0/biolit/__init__.py

@@ -0,0 +1,2 @@
+"""PubMed Screener package."""
+

biolit-0.1.0/biolit/cli.py

@@ -0,0 +1,197 @@
+"""CLI entry point for biolit."""
+import argparse
+import os
+import sys
+
+from dotenv import load_dotenv
+
+from biolit.llm import get_llm_client
+from biolit.pipeline import run, run_geo
+from biolit.parsers.utils import DEFAULT_MAX_CHARS
+from biolit.utils import read_eml_body, extract_pmids, read_pmids_file, read_geo_file
+
+load_dotenv()
+
+DEFAULT_CRITERION = (
+    "Is this paper SPECIFICALLY about schizophrenia AND does it use genetics "
+    "or genomics methods (e.g. GWAS, WGS, scRNA-seq, proteomics, gene expression)?"
+)
+DEFAULT_FIELDS = "methodology, sample_type, causal_claims, genetics_claims, summary"
+
+
+def main(argv: list[str] | None = None) -> None:
+    parser = argparse.ArgumentParser(
+        description=(
+            "Screen PubMed alert emails with a configurable criterion and output fields. "
+            "Supports multiple LLM providers and optional full-text fetching."
+        )
+    )
+    parser.add_argument(
+        "input_file", nargs="?", default=None,
+        help="PubMed alert .eml file, plain-text file of PMIDs, or plain-text file of GEO accessions",
+    )
+    parser.add_argument(
+        "--pmids", default=None,
+        help="Comma-separated PMIDs (alternative to input_file)",
+    )
+    parser.add_argument(
+        "--accessions", default=None,
+        help="Comma-separated GEO accessions (alternative to input_file)",
+    )
+
+    # Screening / extraction
+    parser.add_argument("--criterion", default=None,
+                        help="Relevance screening criterion (yes/no question)")
+    parser.add_argument("--fields", default=None,
+                        help="Fields to extract (comma-separated names)")
+    parser.add_argument("--default", action="store_true",
+                        help="Use default schizophrenia genomics criterion and fields")
+    parser.add_argument("--output", default="results.csv",
+                        help="Output CSV path (default: results.csv)")
+
+    # LLM provider
+    parser.add_argument(
+        "--provider", default=os.environ.get("LLM_PROVIDER", "anthropic"),
+        choices=["anthropic", "openai", "ollama"],
+        help="LLM provider (default: anthropic, or LLM_PROVIDER env var)",
+    )
+    parser.add_argument(
+        "--model", default=os.environ.get("LLM_MODEL"),
+        help="Model name for the chosen provider (uses provider default if omitted)",
+    )
+    parser.add_argument(
+        "--openai-base-url", default=None,
+        help="Custom base URL for OpenAI-compatible endpoints (e.g. Azure, local vLLM)",
+    )
+    parser.add_argument(
+        "--ollama-url", default="http://localhost:11434",
+        help="Ollama server URL (default: http://localhost:11434)",
+    )
+
+    # Full-text
+    parser.add_argument(
+        "--fulltext", action="store_true",
+        help="Attempt to fetch full text from PMC, preprint servers, and Unpaywall",
+    )
+    parser.add_argument(
+        "--unpaywall-email", default=os.environ.get("UNPAYWALL_EMAIL"),
+        help="Email for Unpaywall API (required when --fulltext is used; or set UNPAYWALL_EMAIL)",
+    )
+    parser.add_argument(
+        "--sections", default=None,
+        help=(
+            "Comma-separated list of sections to send to the LLM when full text is available "
+            "(e.g. 'methods,results'). Default: all sections."
+        ),
+    )
+    parser.add_argument(
+        "--max-chars", type=int, default=DEFAULT_MAX_CHARS,
+        help=f"Maximum characters of paper text sent to the LLM (default: {DEFAULT_MAX_CHARS})",
+    )
+
+    args = parser.parse_args(argv)
+
+    # Resolve criterion and fields
+    if args.default:
+        criterion = DEFAULT_CRITERION
+        fields = DEFAULT_FIELDS
+    else:
+        criterion = args.criterion
+        if not criterion:
+            criterion = input("Screening criterion (yes/no question about relevance): ").strip()
+        fields = args.fields
+        if not fields:
+            fields = input(
+                "Fields to extract (comma-separated, e.g. methodology, sample_type, summary): "
+            ).strip()
+
+    # Build LLM client
+    try:
+        extra: dict = {}
+        if args.provider == "openai" and args.openai_base_url:
+            extra["base_url"] = args.openai_base_url
+        if args.provider == "ollama":
+            extra["base_url"] = args.ollama_url
+        client = get_llm_client(args.provider, args.model, **extra)
+    except (EnvironmentError, ImportError) as e:
+        print(f"Error: {e}")
+        sys.exit(1)
+
+    print(f"Using LLM: {client}\n")
+
+    # Resolve input — CLI flags take priority over file
+    if not args.input_file and not args.pmids and not args.accessions:
+        print("Error: provide an input_file, --pmids, or --accessions.")
+        sys.exit(1)
+
+    if args.accessions:
+        input_type = "geo"
+        accessions = [a.strip() for a in args.accessions.split(",") if a.strip()]
+        print(f"Using {len(accessions)} GEO accessions from --accessions\n")
+    elif args.pmids:
+        input_type = "pubmed"
+        pmids = [p.strip() for p in args.pmids.split(",") if p.strip()]
+        print(f"Using {len(pmids)} PMIDs from --pmids\n")
+    elif args.input_file.endswith(".eml"):
+        input_type = "pubmed"
+        body = read_eml_body(args.input_file)
+        pmids = extract_pmids(body)
+        print(f"Found {len(pmids)} PMIDs in {args.input_file}\n")
+    else:
+        first_value = _peek_first_value(args.input_file)
+        if first_value and first_value.upper().startswith(("GSE", "GDS", "GSM", "GPL")):
+            input_type = "geo"
+            accessions = read_geo_file(args.input_file)
+            print(f"Read {len(accessions)} GEO accessions from {args.input_file}\n")
+        else:
+            input_type = "pubmed"
+            pmids = read_pmids_file(args.input_file)
+            print(f"Read {len(pmids)} PMIDs from {args.input_file}\n")
+
+    if input_type == "geo":
+        if not accessions:
+            print("No accessions found. Exiting.")
+            sys.exit(1)
+        run_geo(
+            client=client,
+            accessions=accessions,
+            criterion=criterion,
+            fields_description=fields,
+            output_path=args.output,
+            max_chars=args.max_chars,
+        )
+    else:
+        if not pmids:
+            print("No PMIDs found. Exiting.")
+            sys.exit(1)
+        sections_wanted = (
+            [s.strip() for s in args.sections.split(",") if s.strip()]
+            if args.sections
+            else None
+        )
+        run(
+            client=client,
+            pmids=pmids,
+            criterion=criterion,
+            fields_description=fields,
+            output_path=args.output,
+            fulltext=args.fulltext,
+            unpaywall_email=args.unpaywall_email,
+            sections_wanted=sections_wanted,
+            max_chars=args.max_chars,
+        )
+
+
+def _peek_first_value(path: str) -> str | None:
+    """Return the first non-blank, non-comment line of a file."""
+    with open(path, "r", encoding="utf-8") as f:
+        for line in f:
+            line = line.strip()
+            if line and not line.startswith("#"):
+                return line
+    return None
+
+
+if __name__ == "__main__":
+    main()
+

biolit-0.1.0/biolit/fetchers/__init__.py

@@ -0,0 +1,12 @@
+"""Fetcher sub-package."""
+from biolit.fetchers.pubmed import fetch_pubmed_metadata, fetch_pmc_fulltext
+from biolit.fetchers.preprints import fetch_preprint
+from biolit.fetchers.unpaywall import fetch_via_unpaywall
+
+__all__ = [
+    "fetch_pubmed_metadata",
+    "fetch_pmc_fulltext",
+    "fetch_preprint",
+    "fetch_via_unpaywall",
+]
+

biolit-0.1.0/biolit/fetchers/geo.py

@@ -0,0 +1,92 @@
+"""Fetcher for NCBI GEO series records (GSE accessions) via the MINiML XML API."""
+import time
+import xml.etree.ElementTree as ET
+
+import requests
+
+GEO_MINIML_URL = "https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi"
+_RATE_DELAY = 0.4
+
+
+def fetch_geo_record(accession: str) -> dict | None:
+    """Fetch a GEO series record and return a paper-shaped dict.
+
+    Fetches MINiML XML for *accession* (e.g. ``GSE12345``) and extracts:
+    title, summary, overall design, experiment type, organism, and any
+    linked PubMed IDs.
+
+    Returns a dict compatible with the pipeline's ``paper`` format so the
+    same LLM screening and extraction calls work unchanged.
+    """
+    try:
+        resp = requests.get(
+            GEO_MINIML_URL,
+            params={"acc": accession, "targ": "self", "form": "xml", "view": "brief"},
+            timeout=30,
+        )
+        resp.raise_for_status()
+    except Exception as e:
+        raise RuntimeError(f"GEO fetch failed for {accession}: {e}") from e
+
+    time.sleep(_RATE_DELAY)
+    return _parse_miniml(accession, resp.content)
+
+
+def _parse_miniml(accession: str, xml_bytes: bytes) -> dict | None:
+    """Parse a GEO MINiML XML response into a paper-shaped dict."""
+    try:
+        root = ET.fromstring(xml_bytes)
+    except ET.ParseError:
+        return None
+
+    # MINiML uses a namespace; strip it for simpler findtext calls
+    ns = ""
+    if root.tag.startswith("{"):
+        ns = root.tag.split("}")[0] + "}"
+
+    def _text(elem, tag: str, default: str = "") -> str:
+        node = elem.find(f"{ns}{tag}")
+        return (node.text or "").strip() if node is not None else default
+
+    def _all_text(elem, tag: str) -> list[str]:
+        return [(n.text or "").strip() for n in elem.findall(f"{ns}{tag}") if n.text]
+
+    series = root.find(f"{ns}Series")
+    if series is None:
+        return None
+
+    title = _text(series, "Title")
+    summary = _text(series, "Summary")
+    overall_design = _text(series, "Overall-Design")
+    experiment_type = _text(series, "Type")
+    pmids = _all_text(series, "Pubmed-ID")
+
+    # Collect organism from child Sample elements if not on Series directly
+    organisms = list({
+        _text(s, "Organism") or _text(s, "organism")
+        for s in root.findall(f".//{ns}Sample")
+    } - {""})
+
+    # Build an abstract-like blob from the structured fields
+    abstract_parts = []
+    if summary:
+        abstract_parts.append(f"Summary: {summary}")
+    if overall_design:
+        abstract_parts.append(f"Overall design: {overall_design}")
+    if experiment_type:
+        abstract_parts.append(f"Experiment type: {experiment_type}")
+    if organisms:
+        abstract_parts.append(f"Organism(s): {', '.join(organisms)}")
+    abstract = "\n\n".join(abstract_parts)
+
+    return {
+        "pmid": pmids[0] if pmids else None,
+        "accession": accession,
+        "doi": None,
+        "title": title,
+        "abstract": abstract,
+        "mesh_terms": ([experiment_type] if experiment_type else []) + organisms,
+        "url": f"https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc={accession}",
+        "pmids": pmids,
+        "text_source": "geo_record",
+    }

biolit-0.1.0/biolit/fetchers/preprints.py

@@ -0,0 +1,50 @@
+"""Fetch full text from bioRxiv / medRxiv preprint servers."""
+import requests
+
+_API_BASE = "https://api.biorxiv.org/details/{server}/{doi}/na/json"
+_JATS_TEMPLATE = "https://www.biorxiv.org/content/{doi}.source.xml"
+_MEDRXIV_JATS_TEMPLATE = "https://www.medrxiv.org/content/{doi}.source.xml"
+
+
+def _is_preprint_doi(doi: str) -> str | None:
+    """Return 'biorxiv' or 'medrxiv' if the DOI belongs to a preprint server."""
+    if not doi:
+        return None
+    doi_lower = doi.lower()
+    if "10.1101/" in doi_lower:
+        # Both bioRxiv and medRxiv share the 10.1101 prefix; try biorxiv first
+        return "biorxiv"
+    return None
+
+
+def fetch_preprint(doi: str) -> bytes | None:
+    """Try to retrieve JATS XML for a bioRxiv or medRxiv preprint by DOI.
+
+    Returns raw XML bytes, or None if the paper is not a preprint / fetch fails.
+    """
+    if not doi:
+        return None
+    server = _is_preprint_doi(doi)
+    if not server:
+        return None
+
+    # Query the biorxiv API; fall back to medrxiv if no results
+    for srv in ("biorxiv", "medrxiv"):
+        try:
+            api_url = _API_BASE.format(server=srv, doi=doi)
+            resp = requests.get(api_url, timeout=15)
+            resp.raise_for_status()
+            data = resp.json()
+            collection = data.get("collection", [])
+            if not collection:
+                continue
+            # The API confirms this paper exists on this server — grab JATS XML
+            jats_template = _JATS_TEMPLATE if srv == "biorxiv" else _MEDRXIV_JATS_TEMPLATE
+            xml_url = jats_template.format(doi=doi)
+            xml_resp = requests.get(xml_url, timeout=30)
+            xml_resp.raise_for_status()
+            return xml_resp.content
+        except Exception:
+            continue
+    return None
+