@fbraza/pi-cite 0.1.0 → 0.3.0

package/README.md

@@ -1,13 +1,24 @@
 # @fbraza/pi-cite
 
 A standalone [Pi](https://pi.dev) extension providing literature-research tools for
-academic workflows. Registers four tools callable by the agent:
+academic workflows. Registers two tools callable by the agent:
 
-- **`literature_search`** — PubMed-first search with optional Semantic Scholar
-  supplementary metadata.
+- **`literature_search`** — literature workflow search against PubMed using a
+  PubMed-ready query (MeSH `[mh]`, `[tiab]`, `[pt]`, substance `[nm]`, and Boolean
+  logic), with streaming progress and deduplicated results.
 - **`pubmed_search`** — direct PubMed query (MeSH, `[tiab]`, `[pt]`, etc.).
-- **`fetch_fulltext`** — retrieve a paper PDF via PMC → publisher OA → fallback.
-- (`semantic_scholar` helper used internally by the search tools.)
+
+## Bundled skill
+
+Ships with the **`literature`** skill (`skills/literature/`), which turns these
+tools into an end-to-end review workflow: verified-citation search, per-paper
+experiment extraction, and a structured hypothesis synthesis. Its frontmatter
+declares `allowed-tools` covering the extension's tools above, so the skill and
+extension are paired on purpose.
+
+- `references/` — PubMed query syntax, API reference, and common queries.
+- `scripts/` — Python helpers (`extract_experiments.py`, `synthesis.py`,
+  `generate_table.py`, `export_all.py`) invoked by the skill.
 
 ## Install
 
@@ -41,4 +52,3 @@ npm run pack:check  # preview the published tarball contents
 | Variable | Purpose |
 |---|---|
 | `NCBI_API_KEY` / `api_key` env | PubMed rate limit + E-utilities auth |
-| `SEMANTIC_SCHOLAR_API_KEY` | Enables Semantic Scholar supplementary search |

package/package.json

@@ -1,23 +1,26 @@
 {
   "name": "@fbraza/pi-cite",
-  "version": "0.1.0",
-  "description": "Pi extension with PubMed, Semantic Scholar, literature search, and full-text retrieval tools.",
+  "version": "0.3.0",
+  "description": "Pi extension with PubMed and literature search tools.",
   "license": "MIT",
   "type": "module",
   "files": [
     "src",
+    "skills",
     "README.md"
   ],
   "keywords": [
     "pi-package",
     "pi-extension",
     "literature",
-    "pubmed",
-    "semantic-scholar"
+    "pubmed"
   ],
   "pi": {
     "extensions": [
       "./src/index.ts"
+    ],
+    "skills": [
+      "./skills"
     ]
   },
   "scripts": {

package/skills/literature/SKILL.md

@@ -0,0 +1,189 @@
+---
+name: literature
+description: Unified literature search, verification, and synthesis workflow for scientific questions. Use when any biological claim needs a verified citation, when reviewing a gene/pathway/disease/drug/target, when surveying preclinical evidence for a target in a disease, when checking novelty, or when turning a paper set into a structured hypothesis synthesis.
+allowed-tools: Read, Write, WebFetch, WebSearch, literature_search, pubmed_search
+starting-prompt: Conduct a literature review on my research topic with verified citations, structured synthesis, and a per-paper summary table.
+---
+
+# Literature
+
+Unified literature skill replacing the previous split review and preclinical workflows.
+
+## What this skill covers
+
+Use this skill when you need to:
+- verify any biological claim with a real citation
+- review literature on a gene, pathway, disease, drug, or molecular target
+- survey preclinical evidence for a target in a disease context
+- check whether a finding appears novel or already published
+- synthesize a paper set into hypotheses, contradictions, and evidence-weighted conclusions
+
+Do not use this skill for:
+- running computational omics analyses
+- generating figures without a literature component
+- inventing or guessing citations
+
+## Hard rules
+
+- Every citation must be real and verifiable.
+- Never fabricate PMIDs, DOIs, titles, journals, years, or author lists.
+- Distinguish human, animal, and in vitro evidence.
+- Weight evidence quality by study design and replication.
+- Use inline numbered citations like `[1]` or `[1, 2]` in narrative synthesis.
+- Never overwrite outputs from a previous literature search.
+- Never write literature-review outputs directly to generic shared paths under `results/`.
+
+## Standard workflow
+
+### Step 1 — Clarify scope
+
+Always clarify:
+- exact claim, topic, target, or disease
+- desired time range
+- species restrictions
+- study type filters
+- whether the task is **general review** or **preclinical extraction mode**
+
+### Step 2 — Create a dedicated output folder
+
+For every new literature review or literature research task, create a new dedicated folder under `results/literature_review/` before generating files.
+
+Use the path `results/literature_review/<subject_of_study>/`, where `<subject_of_study>` is a short **snake_case title summary of the theme** of the literature search. Derive it from the scope clarified in Step 1: lower case, words separated by single underscores, no spaces, hyphens, or punctuation. For example, a review on **trained immunity in transplantation** becomes:
+
+- `results/literature_review/trained_immunity_in_transplantation/`
+
+Other examples:
+- `results/literature_review/sirna_lung_transplant_new_treatments/`
+- `results/literature_review/multiomics_ml_biomarkers_in_pgd/`
+
+All generated files for that search session must be saved inside this dedicated subject folder, including:
+- `literature_report.md`
+- `paper_summary_table.csv`
+- `search_log.md`
+- any optional analysis/export artifacts such as `analysis_object.pkl`
+
+Never write outputs directly to the parent folder or to the `results/` root, for example:
+- `results/literature_review/literature_report.md`
+- `results/literature_review/paper_summary_table.csv`
+- `results/literature_review/analysis_object.pkl`
+- `results/literature_report.md`
+
+If a folder for a previous search on the same subject already exists, create a new folder with a distinct descriptive `<subject_of_study>` title rather than using versioned filenames.
+
+At the end of the task, clearly report the exact output folder and generated file paths to the user.
+
+### Step 3 — Search
+
+Use the custom literature tool as the primary search path:
+- **Primary:** `literature_search`
+
+When calling `literature_search`:
+- Always construct `pubmed_query` using PubMed-specific syntax from the references below.
+- Use MeSH terms (`[mh]` / `[majr]`), title/abstract terms (`[tiab]`), publication types (`[pt]`), substance names (`[nm]`), date filters, and Boolean logic as appropriate.
+- Do not pass a generic natural-language query as `pubmed_query` when a PubMed/MeSH query can be constructed.
+
+These extension tools are the preferred search path for this skill. Do not fall back to generic `WebFetch` / `WebSearch` first when one of these typed tools fits the task.
+
+Read these references before constructing queries:
+- `references/pubmed_routine.md`
+- `references/pubmed_search_syntax.md`
+- `references/pubmed_common_queries.md`
+
+### Step 4 — Screen and prioritise
+
+- Deduplicate PubMed results.
+- Prioritise by relevance, recency, and study type.
+- Default to deep reading of the top 20 papers unless the user asks otherwise.
+- For preclinical requests, keep studies with experimental target perturbation evidence.
+
+### Step 5 — Synthesis
+
+Always produce:
+1. a narrative synthesis with inline numbered citations
+2. a per-paper structured summary table
+
+When in **preclinical extraction mode**, add:
+- Experiment Type
+- Model System
+- Assay/Endpoint
+- Finding Direction
+
+Use:
+- `scripts/synthesis.py`
+- `scripts/generate_table.py`
+- `scripts/export_all.py`
+
+For preclinical extraction details, read:
+- `references/preclinical-extraction-guide.md`
+- `scripts/extract_experiments.py`
+
+## Evidence quality framework
+
+Rank evidence broadly as:
+- **High:** replicated clinical evidence, meta-analysis, systematic review, strong human studies
+- **Moderate:** strong animal studies, coherent multi-model evidence, robust mechanistic studies
+- **Low/Preliminary:** single-study results, purely computational inference, unreplicated in vitro work
+
+### What to mark as preliminary
+- single-study findings
+- animal-only findings for human claims
+- in vitro findings without in vivo follow-up
+
+### What to refuse without qualification
+- causal claims from correlational studies
+- claims supported only by retracted work
+- claims contradicting the weight of evidence
+
+## Output format
+
+### Narrative section
+
+Use concise prose with inline citations.
+
+### Paper Summary Table
+
+```markdown
+## Paper Summary Table
+
+| # | PMID/DOI | Authors (year) | Key Message | Key Results | Key Methods | Study Type | Evidence Quality |
+|---|---|---|---|---|---|---|---|
+```
+
+### Extra columns for preclinical extraction mode
+
+```markdown
+| Experiment Type | Model System | Assay/Endpoint | Finding Direction |
+```
+
+## Hypothesis synthesis
+
+After reviewing the core paper set, optionally produce:
+- explicit hypotheses stated by authors
+- implicit mechanistic hypotheses inferred from evidence
+- contradiction matrix across papers
+- highest-confidence next-step hypotheses
+
+## Expected files
+
+Typical outputs must be placed in a dedicated subject folder under `./results/literature_review/`, for example `./results/literature_review/<subject_of_study>/`:
+- `literature_report.md`
+- `paper_summary_table.csv`
+- `search_log.md`
+- optional `analysis_object.pkl` or other export artifacts when produced
+
+Do not write these outputs directly to `./results/literature_review/` or to `./results/`, and do not reuse a previous subject folder.
+
+## Companion references
+
+- `references/pubmed_api_reference.md`
+- `references/pubmed_routine.md`
+- `references/pubmed_search_syntax.md`
+- `references/pubmed_common_queries.md`
+- `references/preclinical-extraction-guide.md`
+
+## Companion scripts
+
+- `scripts/extract_experiments.py`
+- `scripts/synthesis.py`
+- `scripts/generate_table.py`
+- `scripts/export_all.py`

package/skills/literature/references/preclinical-extraction-guide.md

@@ -0,0 +1,215 @@
+# Experiment Extraction Guide
+
+**Workflow:** literature
+**Purpose:** Keyword-based extraction approach, dictionaries, and limitations for classifying in vitro and in vivo experiments from paper abstracts.
+
+---
+
+## Overview
+
+The `extract_all_experiments()` function parses paper abstracts to classify each paper into one of three categories:
+- **in_vitro** — only cell-based experiments reported
+- **in_vivo** — only animal model experiments reported
+- **both** — both in vitro and in vivo experiments reported
+- **unclassified** — insufficient information in abstract to classify
+
+Extraction is keyword-based (not AI-based), operating on the abstract text. It is fast and reproducible but has known limitations (see below).
+
+---
+
+## Extraction Dictionaries
+
+### In Vitro Keywords
+
+These terms in the abstract signal cell-based experiments:
+
+```python
+IN_VITRO_KEYWORDS = [
+    # Cell lines
+    "cell line", "cell lines", "cancer cells", "tumor cells",
+    "HeLa", "MCF-7", "A549", "HCT116", "PC-3", "LNCaP", "MDA-MB",
+    "U87", "U251", "Jurkat", "THP-1", "RAW264", "HEK293",
+
+    # Assay types
+    "in vitro", "cell viability", "cell proliferation", "MTT assay",
+    "CCK-8", "colony formation", "clonogenic", "wound healing",
+    "scratch assay", "transwell", "invasion assay", "migration assay",
+    "flow cytometry", "apoptosis", "cell cycle", "western blot",
+    "immunofluorescence", "ELISA", "qPCR", "RNA-seq", "ChIP",
+
+    # Culture conditions
+    "cultured", "culture", "monolayer", "3D culture", "organoid",
+    "spheroid", "primary cells", "iPSC", "stem cells"
+]
+```
+
+### In Vivo Keywords
+
+These terms signal animal model experiments:
+
+```python
+IN_VIVO_KEYWORDS = [
+    # Animal models
+    "in vivo", "mouse model", "murine", "mice", "rat model",
+    "xenograft", "syngeneic", "PDX", "patient-derived xenograft",
+    "transgenic", "knockout mouse", "knock-in", "GEM", "GEMM",
+    "orthotopic", "subcutaneous tumor", "allograft",
+
+    # Animal procedures
+    "tumor volume", "tumor growth", "body weight", "survival",
+    "tumor regression", "metastasis model", "lung metastasis",
+    "liver metastasis", "intracranial", "intraperitoneal injection",
+    "intravenous injection", "oral gavage", "dosing",
+
+    # Animal species
+    "C57BL/6", "BALB/c", "nude mice", "SCID", "NOD-SCID",
+    "NSG", "athymic", "immunodeficient", "Sprague-Dawley",
+    "Wistar rat", "zebrafish"
+]
+```
+
+---
+
+## Extraction Logic
+
+```python
+def classify_paper(abstract: str) -> dict:
+    """
+    Classify a paper abstract into experiment categories.
+    Returns dict with: experiment_type, in_vitro_signals, in_vivo_signals,
+                       cell_lines, animal_models, assays, endpoints, findings
+    """
+    abstract_lower = abstract.lower()
+
+    # Count keyword matches
+    in_vitro_hits = [kw for kw in IN_VITRO_KEYWORDS if kw.lower() in abstract_lower]
+    in_vivo_hits = [kw for kw in IN_VIVO_KEYWORDS if kw.lower() in abstract_lower]
+
+    # Classify
+    has_vitro = len(in_vitro_hits) >= 1
+    has_vivo = len(in_vivo_hits) >= 1
+
+    if has_vitro and has_vivo:
+        experiment_type = "both"
+    elif has_vitro:
+        experiment_type = "in_vitro"
+    elif has_vivo:
+        experiment_type = "in_vivo"
+    else:
+        experiment_type = "unclassified"
+
+    return {
+        "experiment_type": experiment_type,
+        "in_vitro_signals": in_vitro_hits,
+        "in_vivo_signals": in_vivo_hits,
+        "cell_lines": extract_cell_lines(abstract),
+        "animal_models": extract_animal_models(abstract),
+        "assays": extract_assays(abstract),
+        "endpoints": extract_endpoints(abstract),
+        "findings": extract_findings_sentence(abstract)
+    }
+```
+
+---
+
+## Structured Field Extraction
+
+Beyond classification, the script extracts specific structured fields:
+
+### Cell Lines
+Extracted using a curated regex pattern matching common cancer cell line names:
+```python
+CELL_LINE_PATTERN = r'\b(MCF-7|MDA-MB-\d+|A549|HCT116|PC-3|LNCaP|HeLa|U87|U251|Jurkat|THP-1|HEK293[T]?|Caco-2|SW480|HT-29|PANC-1|MiaPaCa|BxPC-3|DU145|22Rv1)\b'
+```
+
+### Animal Models
+```python
+ANIMAL_MODEL_PATTERN = r'\b(xenograft|syngeneic|PDX|patient-derived xenograft|transgenic|knockout|GEMM|orthotopic|allograft|C57BL/6|BALB/c|nude mice|NSG|SCID)\b'
+```
+
+### Assay Types
+```python
+ASSAY_PATTERN = r'\b(MTT|CCK-8|colony formation|clonogenic|wound healing|transwell|flow cytometry|western blot|ELISA|qPCR|RNA-seq|ChIP-seq|ATAC-seq|immunofluorescence|IHC|co-immunoprecipitation)\b'
+```
+
+### Endpoints
+```python
+ENDPOINT_PATTERN = r'\b(tumor volume|tumor growth|survival|body weight|metastasis|apoptosis|cell viability|proliferation|migration|invasion|angiogenesis|tumor regression)\b'
+```
+
+### Key Findings Sentence
+The script extracts the last 1-2 sentences of the abstract as a proxy for the main finding:
+```python
+def extract_findings_sentence(abstract: str) -> str:
+    sentences = abstract.split(". ")
+    return ". ".join(sentences[-2:]).strip()
+```
+
+---
+
+## Output Schema
+
+The `experiment_extraction.csv` file contains one row per paper with these columns:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `pmid` | str | PubMed ID |
+| `doi` | str | DOI |
+| `title` | str | Paper title |
+| `year` | int | Publication year |
+| `experiment_type` | str | in_vitro / in_vivo / both / unclassified |
+| `cell_lines` | str | Comma-separated cell line names found |
+| `animal_models` | str | Comma-separated animal model terms found |
+| `assays` | str | Comma-separated assay types found |
+| `endpoints` | str | Comma-separated endpoints found |
+| `in_vitro_signal_count` | int | Number of in vitro keyword matches |
+| `in_vivo_signal_count` | int | Number of in vivo keyword matches |
+| `key_findings` | str | Last 1-2 sentences of abstract |
+
+---
+
+## Known Limitations
+
+### 1. Abstract-only extraction
+The script only reads abstracts, not full text. Papers that describe experiments only in the methods/results sections (not the abstract) will be misclassified as "unclassified".
+
+**Mitigation:** No full-text enrichment step is currently available; papers whose experiments appear only in the methods/results sections may be misclassified as "unclassified".
+
+### 2. Keyword sensitivity
+- **False positives:** A paper mentioning "mouse model" in the introduction (not as an experiment performed) may be classified as in_vivo.
+- **False negatives:** Novel cell lines or animal models not in the dictionary will be missed.
+
+### 3. No quantitative extraction
+The script does not extract dosing, IC50 values, tumor volumes, or statistical significance — only qualitative classification.
+
+### 4. Language dependency
+Only English-language abstracts are supported. Non-English papers will be classified as "unclassified".
+
+### 5. High "unclassified" rate for some targets
+For niche targets or highly mechanistic papers (e.g., structural biology, computational studies), most papers may be classified as "unclassified". This is expected behavior — not a bug.
+
+---
+
+## Improving Extraction for Your Target
+
+If you find too many "unclassified" papers, you can extend the keyword dictionaries:
+
+```python
+# Add target-specific cell lines
+CUSTOM_CELL_LINES = ["your_cell_line_1", "your_cell_line_2"]
+IN_VITRO_KEYWORDS.extend(CUSTOM_CELL_LINES)
+
+# Add target-specific animal models
+CUSTOM_ANIMAL_MODELS = ["your_model_1", "your_model_2"]
+IN_VIVO_KEYWORDS.extend(CUSTOM_ANIMAL_MODELS)
+```
+
+Edit `scripts/extract_experiments.py` directly and re-run Step 2.
+
+---
+
+## References
+
+- Keyword extraction approach inspired by: Lever J, et al. (2019) *PLOS Biol* — text mining for biomedical literature
+- Cell line nomenclature: ATCC (https://www.atcc.org/)
+- Animal model terminology: NCI Thesaurus (https://ncithesaurus.nci.nih.gov/)