bibliodash 1.0.0__tar.gz

bibliodash-1.0.0/MANIFEST.in

@@ -0,0 +1,9 @@
+include README.md
+include LICENSE
+include pyproject.toml
+
+recursive-include biblioflow *.py *.css *.png *.jpg *.toml *.sql
+recursive-exclude biblioflow/_server/cache *
+recursive-exclude biblioflow/_server/logs *
+recursive-exclude **/__pycache__ *
+recursive-exclude * *.pyc *.pyo

bibliodash-1.0.0/PKG-INFO

@@ -0,0 +1,349 @@
+Metadata-Version: 2.4
+Name: bibliodash
+Version: 1.0.0
+Summary: Local-first bibliometric analysis platform — Research Intelligence, Flowing Insights
+Author-email: Robin Tomar <itsrobintomar@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/imrobintomar/BiblioFlow
+Project-URL: Repository, https://github.com/imrobintomar/BiblioFlow
+Project-URL: Issues, https://github.com/imrobintomar/BiblioFlow/issues
+Keywords: bibliometrics,research,literature review,systematic review,PDF,dash
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: dash>=2.14
+Requires-Dist: dash-bootstrap-components>=1.5
+Requires-Dist: dash-ag-grid>=31
+Requires-Dist: plotly>=5.18
+Requires-Dist: networkx>=3.0
+Requires-Dist: rapidfuzz>=3.0
+Requires-Dist: pydantic>=2.0
+Requires-Dist: requests>=2.28
+Requires-Dist: openpyxl>=3.1
+Requires-Dist: pycountry>=23
+Requires-Dist: geonamescache>=1.6
+Requires-Dist: psutil>=5.9
+Requires-Dist: rich>=13.0
+Requires-Dist: python-dotenv>=1.0
+Provides-Extra: pdf
+Requires-Dist: pymupdf>=1.23; extra == "pdf"
+Requires-Dist: pdfplumber>=0.10; extra == "pdf"
+Requires-Dist: docling>=2.0; extra == "pdf"
+Provides-Extra: semantic
+Requires-Dist: sentence-transformers>=2.6; extra == "semantic"
+Requires-Dist: faiss-cpu>=1.7; extra == "semantic"
+Requires-Dist: scikit-learn>=1.4; extra == "semantic"
+Provides-Extra: ai
+Requires-Dist: ollama>=0.2; extra == "ai"
+Provides-Extra: all
+Requires-Dist: bibliodash[ai,pdf,semantic]; extra == "all"
+
+<div align="center">
+  <img src="src/Public/BiblioFlow_logo.png" alt="BiblioFlow" width="420" />
+
+  <h3>Research Intelligence. Flowing Insights.</h3>
+
+  <p>
+    A local-first platform that extracts structured bibliometric data from research PDFs<br/>
+    and delivers deep scholarly analysis through an interactive, publication-quality dashboard.
+  </p>
+
+  <p>
+    <img src="https://img.shields.io/badge/Python-3.13+-3776AB?style=flat-square&logo=python&logoColor=white" alt="Python 3.13+"/>
+    <img src="https://img.shields.io/badge/Dash-2.x-008DE4?style=flat-square&logo=plotly&logoColor=white" alt="Dash"/>
+    <img src="https://img.shields.io/badge/Plotly-5.x-3F4F75?style=flat-square&logo=plotly&logoColor=white" alt="Plotly"/>
+    <img src="https://img.shields.io/badge/NetworkX-3.x-orange?style=flat-square" alt="NetworkX"/>
+    <img src="https://img.shields.io/badge/SQLite-local--first-003B57?style=flat-square&logo=sqlite&logoColor=white" alt="SQLite"/>
+    <img src="https://img.shields.io/badge/Ollama-local%20AI-black?style=flat-square" alt="Ollama"/>
+    <img src="https://img.shields.io/badge/status-active-brightgreen?style=flat-square" alt="Status"/>
+  </p>
+</div>
+
+---
+
+## What is BiblioFlow?
+
+**BiblioFlow** is a self-hosted bibliometric analysis tool built for researchers. You bring a collection of research PDFs — BiblioFlow does the rest.
+
+It reads every PDF, pulls out the key scholarly metadata (title, DOI, authors, abstract, keywords, funding, citations), automatically enriches each record from open academic databases, and then presents the entire corpus through a rich, interactive analysis dashboard — all running locally on your machine with no data sent to any external service.
+
+**The core workflow is simple:**
+
+```
+Upload PDFs  →  BiblioFlow extracts & enriches metadata  →  Explore bibliometric analysis
+```
+
+Everything — from publication trends to author networks, keyword co-occurrence to AI-generated research summaries — is derived from the data inside your own PDF collection.
+
+---
+
+## Analysis Modules
+
+BiblioFlow provides fourteen dedicated analysis modules across four capability tiers.
+
+### 📊 Overview
+Annual, monthly, quarterly, and decade-level publication production. Growth rates (CAGR), cumulative output, moving averages, publication heatmaps, and forecasting — broken down by document type, country, institution, and journal.
+
+### 📚 Sources
+Identifies the most productive journals and publishers in your corpus. Applies Bradford's Law to reveal the core zone of journals. Shows journal H-index, citation impact, open-access distribution, and publication timelines.
+
+### ✍️ Authors
+Ranks authors by productivity and citation impact. Computes H-index, G-index, M-index, i10-index, citation velocity, and career trajectory for every author. Maps co-author collaboration networks with community detection.
+
+### 📄 Documents
+Lists the most-cited papers in your corpus. Analyses reference lists to find the most frequently co-cited works. Full funding intelligence: funder lollipop chart, category breakdown (Government / Industry / Other), year-on-year funding trends, and multi-funded paper table.
+
+### 🔑 Conceptual Structure
+Identifies the most frequent keywords and concepts. Builds a **Thematic Map** — plotting keyword clusters by Callon centrality versus density — to reveal motor, niche, emerging, and declining themes. Tracks how keyword sets shift over time.
+
+### 🌐 Social Structure
+Maps collaboration at three levels — **Author**, **Institution**, and **Country** — as interactive network graphs with Louvain community detection. Renders a choropleth world map of collaboration arcs. Shows a collaboration timeline across years.
+
+### 🧠 Intellectual Structure
+Uncovers the citation backbone of your field. Identifies the most co-cited reference pairs. Builds a citation historiography showing how foundational works influenced one another over time.
+
+### 🔗 Clustering
+Detects research communities using four network types — co-authorship, keyword co-occurrence, bibliographic coupling, and citation networks — each analysed for degree, betweenness, closeness, eigenvector, and PageRank centrality.
+
+### 🏅 Evidence Quality
+Per-paper **Evidence Score** (0–100) and **Reproducibility Score** (0–100) computed from nine quality signals: trial registration, ethics approval, informed consent, funding declaration, PubMed indexing, open access, figures, tables, and data availability. Signal coverage bar charts, score distribution histograms, and ranked paper tables for both scores.
+
+### 🔍 Duplicate Detection
+Detects duplicate papers by exact DOI match and fuzzy title similarity (≥ 90% token-sort ratio via rapidfuzz). Reports cluster count, duplicate rate, and a full table of suspect pairs grouped by cluster.
+
+### 🗄️ Provenance Explorer
+Field-level data provenance heatmap showing which enrichment source (CrossRef, OpenAlex, PubMed, etc.) filled each metadata field. Average confidence score per source and per field.
+
+### 📅 Research Timeline
+Interactive bubble scatter plot of publications over time — bubble size proportional to citation count, colour-coded by document type. Complementary year × document-type heatmap to spot activity patterns.
+
+### 🔎 Semantic Search
+Meaning-based full-text search over all paper titles and abstracts using `sentence-transformers` (`all-MiniLM-L6-v2`) and a FAISS cosine-similarity index. Finds topically relevant papers even without exact keyword matches. Index built once per project (~15 s) and cached to disk.
+
+**Sub-features:**
+- **Paper Novelty Scores** — novelty = 1 − mean cosine similarity to 5 nearest semantic neighbours. Identifies unique papers and redundant ones.
+- **Paper Similarity** — find papers most similar to any paper in the corpus.
+
+### 🗺️ Research Gap Detection
+Clusters all paper abstracts with KMeans + FAISS embeddings, names each cluster with TF-IDF keywords, and scores it for research gap potential using an age–size formula. Renders an interactive bubble chart (gap level = colour) and a ranked gap card list. Typical first-run ~30 s, subsequent runs instant (cached index).
+
+### 🤖 AI Features *(local — no API key needed)*
+Three AI-powered views, all running via local **Ollama** (`llama3.1:8b`) with zero data leaving your machine:
+
+| View | What it does |
+|---|---|
+| **AI Summary** | Corpus-level synthesis from your 50 most-cited abstracts → structured JSON: overall summary, main themes, emerging trends, research gaps, common methods, future directions, clinical implications. Cached per project. |
+| **Research Chat** | Multi-turn RAG chat — retrieves 8 semantically relevant papers per question, injects them as grounded context into the LLM, returns cited answers with paper title references. Conversation history maintained per session. |
+| **Claim Extraction** | Per-paper structured extraction for your top-50 cited papers: main finding, study design, sample size, key limitation, future work, evidence level. Each result cached individually. |
+
+### 🗂️ Library & Projects
+A full searchable, sortable table of every paper in your corpus with per-record pipeline status and detail view. Supports **multiple independent projects** — keep different research corpora separate and switch between them from the top bar.
+
+---
+
+## How the Data Extraction Works
+
+BiblioFlow reads your PDFs using **Docling** — a layout-aware parser that understands document structure without OCR. From each PDF, it extracts:
+
+- Title, DOI, abstract, authors, publication year
+- Keywords, trial IDs (NCT-pattern), ORCID identifiers
+- Funding acknowledgement text and grant IDs
+- Section structure, table and figure counts
+- Reference list entries
+
+Each extracted record then passes through an **enrichment waterfall** — a chain of open academic APIs where each source fills in only the fields the previous one left empty:
+
+```
+CrossRef  →  OpenAlex  →  Semantic Scholar  →  PubMed  →  Unpaywall  →  ROR
+```
+
+- **CrossRef:** verifies the DOI, fills publisher, document type, journal, page range
+- **OpenAlex:** adds institutions with ROR IDs, country affiliations, full reference list, concepts, open-access status, and grants
+- **Semantic Scholar:** adds abstract backfill, references, and fields of study
+- **PubMed:** adds MeSH terms and grant IDs (when a PMID is resolvable)
+- **Unpaywall:** adds open-access URL and licence information
+- **ROR:** resolves institution names to countries where other sources missed them
+
+Every field fill is recorded with its source, method, and confidence score in a `field_provenance` table — so you always know where each piece of data came from.
+
+---
+
+## Architecture
+
+<div align="center">
+  <img src="src/Public/BiblioFlow_Workflow.png" alt="BiblioFlow Architecture" width="100%" />
+</div>
+
+---
+
+## Installation
+
+BiblioFlow uses **Docling** for PDF extraction, which includes large ML dependencies. A dedicated virtual environment is strongly recommended.
+
+```bash
+# 1. Clone the repository
+git clone https://github.com/imrobintomar/BiblioFlow.git
+cd BiblioFlow
+
+# 2. Create and activate a virtual environment
+python3 -m venv .venv
+source .venv/bin/activate        # Windows: .venv\Scripts\activate
+
+# 3. Install core dependencies
+pip install -r requirements.txt
+
+# 4. (Optional) Semantic search, novelty, gaps, similarity
+pip install sentence-transformers faiss-cpu scikit-learn
+
+# 5. (Optional) AI features — install Ollama then pull a model
+# https://ollama.com
+ollama pull llama3.1:8b
+```
+
+---
+
+## Usage
+
+### Launch the Dashboard
+
+```bash
+cd src
+../.venv/bin/python3 app.py
+```
+
+Open **http://127.0.0.1:8050** — then use **Import** in the sidebar to point BiblioFlow at a folder of PDFs. The pipeline runs as a background job with live progress updates.
+
+### Import from a Scopus CSV Export
+
+If you already have a Scopus export CSV, import it directly without running the PDF pipeline:
+
+```bash
+cd src
+../.venv/bin/python3 import_scopus_csv.py path/to/scopus_export.csv
+```
+
+### Run the Pipeline from the Command Line
+
+```bash
+../.venv/bin/python3 src/pipeline.py --input PDF/ --output out/
+```
+
+| Flag | Effect |
+|---|---|
+| `--no-scopus` | Skip the Scopus enrichment stage |
+| `--force` | Reprocess all PDFs, including unchanged ones |
+
+### Rebuild the Warehouse
+
+The warehouse database (`src/biblioflow.sqlite`) is fully derived from the pipeline output — safe to delete and recreate at any time:
+
+```bash
+cd src
+../.venv/bin/python3 migrate_legacy.py
+```
+
+---
+
+## Tech Stack
+
+| Component | Technology |
+|---|---|
+| **Dashboard** | [Dash](https://dash.plotly.com/) 2.x + [Plotly](https://plotly.com/) 5.x |
+| **PDF extraction** | [Docling](https://github.com/DS4SD/docling) · PyMuPDF · pdfplumber |
+| **Metadata enrichment** | CrossRef · OpenAlex · Semantic Scholar · PubMed · Unpaywall · ROR |
+| **Network analysis** | [NetworkX](https://networkx.org/) 3.x — Louvain · centrality · spring layout |
+| **Semantic search** | sentence-transformers (`all-MiniLM-L6-v2`) + FAISS (`IndexFlatIP`) |
+| **Gap detection** | scikit-learn KMeans + TF-IDF + PCA |
+| **Duplicate detection** | rapidfuzz `token_sort_ratio` |
+| **AI features** | [Ollama](https://ollama.com) local LLM (`llama3.1:8b`) |
+| **Storage** | SQLite — normalised warehouse + checksum cache |
+| **Data tables** | dash-ag-grid · dash-table |
+| **Geocoding** | geonamescache · pycountry |
+| **Language** | Python 3.13+ |
+
+---
+
+## Project Structure
+
+```
+BiblioFlow/
+├── src/
+│   ├── app.py                      # Dashboard entry point
+│   ├── components.py               # Topbar, sidebar, shared panel components
+│   ├── import_scopus_csv.py        # Standalone Scopus CSV importer
+│   ├── pages/                      # One module per analysis section
+│   │   ├── dashboard.py            # KPI cards, trends, donut, gauge
+│   │   ├── overview.py             # Publication production analysis
+│   │   ├── sources.py              # Journals, Bradford's Law, publishers
+│   │   ├── authors_page.py         # Author metrics and co-author networks
+│   │   ├── documents_page.py       # Citations, references, funding intelligence
+│   │   ├── conceptual_structure.py # Keywords and thematic map
+│   │   ├── social_structure.py     # Collaboration networks and world map
+│   │   ├── clustering.py           # Network clustering analysis
+│   │   ├── networks.py             # Intellectual structure / historiograph
+│   │   ├── evidence.py             # Evidence scores, reproducibility, duplicates,
+│   │   │                           #   provenance, novelty
+│   │   ├── timeline.py             # Research timeline bubble + heatmap
+│   │   ├── semantic_search.py      # Semantic search with example chips
+│   │   ├── gaps.py                 # Research gap detection bubble map
+│   │   ├── ai.py                   # AI Summary · Research Chat · Claim Extraction
+│   │   └── library.py              # Full paper corpus table
+│   ├── engine/                     # Analysis engines
+│   │   ├── evidence.py             # Evidence & reproducibility scoring
+│   │   ├── duplicates.py           # Fuzzy + DOI duplicate detection
+│   │   ├── funding.py              # Funder categorisation and trends
+│   │   ├── semantic_search.py      # FAISS index · search · novelty · similarity
+│   │   ├── gap_detection.py        # KMeans gap scoring
+│   │   ├── llm.py                  # Ollama client wrapper
+│   │   ├── ai_summary.py           # Corpus-level AI summary
+│   │   └── claim_extraction.py     # Per-paper claim extraction
+│   ├── repository/                 # Data access layer
+│   ├── services/                   # Import, enrichment, export services
+│   ├── providers/                  # CrossRef, OpenAlex, PubMed, … clients
+│   ├── database/                   # SQLite schema and connection helpers
+│   ├── visualizations/             # Plotly chart builders and theme
+│   │   ├── theme.py                # Dark palette, apply_theme()
+│   │   ├── lollipop.py             # Horizontal lollipop chart
+│   │   ├── donut.py                # Donut / pie chart
+│   │   ├── gauge.py                # Completeness gauge
+│   │   └── worldmap.py             # Choropleth collaboration map
+│   ├── extract/                    # PDF extraction pipeline
+│   ├── cache/                      # FAISS index, AI summary, claim caches
+│   └── assets/                     # CSS and static files
+├── requirements.txt
+└── README.md
+```
+
+---
+
+## Current Limitations
+
+| Area | Status |
+|---|---|
+| Author–institution attribution | Paper-level only — cannot isolate a specific co-author's affiliation |
+| Citation Half-Life | Not computable — APIs return cumulative citation counts, not per-year breakdowns |
+| Author/Journal co-citation | Not implemented — requires resolving individual reference metadata per paper |
+| Bibliographic coupling | Accuracy bounded by DOI resolution in reference lists |
+| Entity deduplication | Institution and country name variants across providers are not yet merged |
+| Evidence signals (PDF) | Trial IDs, ethics statements, figure counts require PDF extraction to be populated; Scopus CSV imports will show 0 for these fields |
+| AI chat streaming | Responses appear after full generation — streaming output not yet implemented |
+| Report export (PDF/Word/PPT) | Planned — marked "Soon" in the UI |
+
+---
+
+## Author
+
+**Robin Tomar** — [itsrobintomar@gmail.com](mailto:itsrobintomar@gmail.com)  
+**Dr Prabudh Goel's Lab, AIIMS New Delhi**
+
+---
+
+<div align="center">
+  <sub>BiblioFlow © 2026 · Built for researchers, by researchers · Made with ♥ in India</sub>
+</div>

bibliodash-1.0.0/README.md

@@ -0,0 +1,303 @@
+<div align="center">
+  <img src="src/Public/BiblioFlow_logo.png" alt="BiblioFlow" width="420" />
+
+  <h3>Research Intelligence. Flowing Insights.</h3>
+
+  <p>
+    A local-first platform that extracts structured bibliometric data from research PDFs<br/>
+    and delivers deep scholarly analysis through an interactive, publication-quality dashboard.
+  </p>
+
+  <p>
+    <img src="https://img.shields.io/badge/Python-3.13+-3776AB?style=flat-square&logo=python&logoColor=white" alt="Python 3.13+"/>
+    <img src="https://img.shields.io/badge/Dash-2.x-008DE4?style=flat-square&logo=plotly&logoColor=white" alt="Dash"/>
+    <img src="https://img.shields.io/badge/Plotly-5.x-3F4F75?style=flat-square&logo=plotly&logoColor=white" alt="Plotly"/>
+    <img src="https://img.shields.io/badge/NetworkX-3.x-orange?style=flat-square" alt="NetworkX"/>
+    <img src="https://img.shields.io/badge/SQLite-local--first-003B57?style=flat-square&logo=sqlite&logoColor=white" alt="SQLite"/>
+    <img src="https://img.shields.io/badge/Ollama-local%20AI-black?style=flat-square" alt="Ollama"/>
+    <img src="https://img.shields.io/badge/status-active-brightgreen?style=flat-square" alt="Status"/>
+  </p>
+</div>
+
+---
+
+## What is BiblioFlow?
+
+**BiblioFlow** is a self-hosted bibliometric analysis tool built for researchers. You bring a collection of research PDFs — BiblioFlow does the rest.
+
+It reads every PDF, pulls out the key scholarly metadata (title, DOI, authors, abstract, keywords, funding, citations), automatically enriches each record from open academic databases, and then presents the entire corpus through a rich, interactive analysis dashboard — all running locally on your machine with no data sent to any external service.
+
+**The core workflow is simple:**
+
+```
+Upload PDFs  →  BiblioFlow extracts & enriches metadata  →  Explore bibliometric analysis
+```
+
+Everything — from publication trends to author networks, keyword co-occurrence to AI-generated research summaries — is derived from the data inside your own PDF collection.
+
+---
+
+## Analysis Modules
+
+BiblioFlow provides fourteen dedicated analysis modules across four capability tiers.
+
+### 📊 Overview
+Annual, monthly, quarterly, and decade-level publication production. Growth rates (CAGR), cumulative output, moving averages, publication heatmaps, and forecasting — broken down by document type, country, institution, and journal.
+
+### 📚 Sources
+Identifies the most productive journals and publishers in your corpus. Applies Bradford's Law to reveal the core zone of journals. Shows journal H-index, citation impact, open-access distribution, and publication timelines.
+
+### ✍️ Authors
+Ranks authors by productivity and citation impact. Computes H-index, G-index, M-index, i10-index, citation velocity, and career trajectory for every author. Maps co-author collaboration networks with community detection.
+
+### 📄 Documents
+Lists the most-cited papers in your corpus. Analyses reference lists to find the most frequently co-cited works. Full funding intelligence: funder lollipop chart, category breakdown (Government / Industry / Other), year-on-year funding trends, and multi-funded paper table.
+
+### 🔑 Conceptual Structure
+Identifies the most frequent keywords and concepts. Builds a **Thematic Map** — plotting keyword clusters by Callon centrality versus density — to reveal motor, niche, emerging, and declining themes. Tracks how keyword sets shift over time.
+
+### 🌐 Social Structure
+Maps collaboration at three levels — **Author**, **Institution**, and **Country** — as interactive network graphs with Louvain community detection. Renders a choropleth world map of collaboration arcs. Shows a collaboration timeline across years.
+
+### 🧠 Intellectual Structure
+Uncovers the citation backbone of your field. Identifies the most co-cited reference pairs. Builds a citation historiography showing how foundational works influenced one another over time.
+
+### 🔗 Clustering
+Detects research communities using four network types — co-authorship, keyword co-occurrence, bibliographic coupling, and citation networks — each analysed for degree, betweenness, closeness, eigenvector, and PageRank centrality.
+
+### 🏅 Evidence Quality
+Per-paper **Evidence Score** (0–100) and **Reproducibility Score** (0–100) computed from nine quality signals: trial registration, ethics approval, informed consent, funding declaration, PubMed indexing, open access, figures, tables, and data availability. Signal coverage bar charts, score distribution histograms, and ranked paper tables for both scores.
+
+### 🔍 Duplicate Detection
+Detects duplicate papers by exact DOI match and fuzzy title similarity (≥ 90% token-sort ratio via rapidfuzz). Reports cluster count, duplicate rate, and a full table of suspect pairs grouped by cluster.
+
+### 🗄️ Provenance Explorer
+Field-level data provenance heatmap showing which enrichment source (CrossRef, OpenAlex, PubMed, etc.) filled each metadata field. Average confidence score per source and per field.
+
+### 📅 Research Timeline
+Interactive bubble scatter plot of publications over time — bubble size proportional to citation count, colour-coded by document type. Complementary year × document-type heatmap to spot activity patterns.
+
+### 🔎 Semantic Search
+Meaning-based full-text search over all paper titles and abstracts using `sentence-transformers` (`all-MiniLM-L6-v2`) and a FAISS cosine-similarity index. Finds topically relevant papers even without exact keyword matches. Index built once per project (~15 s) and cached to disk.
+
+**Sub-features:**
+- **Paper Novelty Scores** — novelty = 1 − mean cosine similarity to 5 nearest semantic neighbours. Identifies unique papers and redundant ones.
+- **Paper Similarity** — find papers most similar to any paper in the corpus.
+
+### 🗺️ Research Gap Detection
+Clusters all paper abstracts with KMeans + FAISS embeddings, names each cluster with TF-IDF keywords, and scores it for research gap potential using an age–size formula. Renders an interactive bubble chart (gap level = colour) and a ranked gap card list. Typical first-run ~30 s, subsequent runs instant (cached index).
+
+### 🤖 AI Features *(local — no API key needed)*
+Three AI-powered views, all running via local **Ollama** (`llama3.1:8b`) with zero data leaving your machine:
+
+| View | What it does |
+|---|---|
+| **AI Summary** | Corpus-level synthesis from your 50 most-cited abstracts → structured JSON: overall summary, main themes, emerging trends, research gaps, common methods, future directions, clinical implications. Cached per project. |
+| **Research Chat** | Multi-turn RAG chat — retrieves 8 semantically relevant papers per question, injects them as grounded context into the LLM, returns cited answers with paper title references. Conversation history maintained per session. |
+| **Claim Extraction** | Per-paper structured extraction for your top-50 cited papers: main finding, study design, sample size, key limitation, future work, evidence level. Each result cached individually. |
+
+### 🗂️ Library & Projects
+A full searchable, sortable table of every paper in your corpus with per-record pipeline status and detail view. Supports **multiple independent projects** — keep different research corpora separate and switch between them from the top bar.
+
+---
+
+## How the Data Extraction Works
+
+BiblioFlow reads your PDFs using **Docling** — a layout-aware parser that understands document structure without OCR. From each PDF, it extracts:
+
+- Title, DOI, abstract, authors, publication year
+- Keywords, trial IDs (NCT-pattern), ORCID identifiers
+- Funding acknowledgement text and grant IDs
+- Section structure, table and figure counts
+- Reference list entries
+
+Each extracted record then passes through an **enrichment waterfall** — a chain of open academic APIs where each source fills in only the fields the previous one left empty:
+
+```
+CrossRef  →  OpenAlex  →  Semantic Scholar  →  PubMed  →  Unpaywall  →  ROR
+```
+
+- **CrossRef:** verifies the DOI, fills publisher, document type, journal, page range
+- **OpenAlex:** adds institutions with ROR IDs, country affiliations, full reference list, concepts, open-access status, and grants
+- **Semantic Scholar:** adds abstract backfill, references, and fields of study
+- **PubMed:** adds MeSH terms and grant IDs (when a PMID is resolvable)
+- **Unpaywall:** adds open-access URL and licence information
+- **ROR:** resolves institution names to countries where other sources missed them
+
+Every field fill is recorded with its source, method, and confidence score in a `field_provenance` table — so you always know where each piece of data came from.
+
+---
+
+## Architecture
+
+<div align="center">
+  <img src="src/Public/BiblioFlow_Workflow.png" alt="BiblioFlow Architecture" width="100%" />
+</div>
+
+---
+
+## Installation
+
+BiblioFlow uses **Docling** for PDF extraction, which includes large ML dependencies. A dedicated virtual environment is strongly recommended.
+
+```bash
+# 1. Clone the repository
+git clone https://github.com/imrobintomar/BiblioFlow.git
+cd BiblioFlow
+
+# 2. Create and activate a virtual environment
+python3 -m venv .venv
+source .venv/bin/activate        # Windows: .venv\Scripts\activate
+
+# 3. Install core dependencies
+pip install -r requirements.txt
+
+# 4. (Optional) Semantic search, novelty, gaps, similarity
+pip install sentence-transformers faiss-cpu scikit-learn
+
+# 5. (Optional) AI features — install Ollama then pull a model
+# https://ollama.com
+ollama pull llama3.1:8b
+```
+
+---
+
+## Usage
+
+### Launch the Dashboard
+
+```bash
+cd src
+../.venv/bin/python3 app.py
+```
+
+Open **http://127.0.0.1:8050** — then use **Import** in the sidebar to point BiblioFlow at a folder of PDFs. The pipeline runs as a background job with live progress updates.
+
+### Import from a Scopus CSV Export
+
+If you already have a Scopus export CSV, import it directly without running the PDF pipeline:
+
+```bash
+cd src
+../.venv/bin/python3 import_scopus_csv.py path/to/scopus_export.csv
+```
+
+### Run the Pipeline from the Command Line
+
+```bash
+../.venv/bin/python3 src/pipeline.py --input PDF/ --output out/
+```
+
+| Flag | Effect |
+|---|---|
+| `--no-scopus` | Skip the Scopus enrichment stage |
+| `--force` | Reprocess all PDFs, including unchanged ones |
+
+### Rebuild the Warehouse
+
+The warehouse database (`src/biblioflow.sqlite`) is fully derived from the pipeline output — safe to delete and recreate at any time:
+
+```bash
+cd src
+../.venv/bin/python3 migrate_legacy.py
+```
+
+---
+
+## Tech Stack
+
+| Component | Technology |
+|---|---|
+| **Dashboard** | [Dash](https://dash.plotly.com/) 2.x + [Plotly](https://plotly.com/) 5.x |
+| **PDF extraction** | [Docling](https://github.com/DS4SD/docling) · PyMuPDF · pdfplumber |
+| **Metadata enrichment** | CrossRef · OpenAlex · Semantic Scholar · PubMed · Unpaywall · ROR |
+| **Network analysis** | [NetworkX](https://networkx.org/) 3.x — Louvain · centrality · spring layout |
+| **Semantic search** | sentence-transformers (`all-MiniLM-L6-v2`) + FAISS (`IndexFlatIP`) |
+| **Gap detection** | scikit-learn KMeans + TF-IDF + PCA |
+| **Duplicate detection** | rapidfuzz `token_sort_ratio` |
+| **AI features** | [Ollama](https://ollama.com) local LLM (`llama3.1:8b`) |
+| **Storage** | SQLite — normalised warehouse + checksum cache |
+| **Data tables** | dash-ag-grid · dash-table |
+| **Geocoding** | geonamescache · pycountry |
+| **Language** | Python 3.13+ |
+
+---
+
+## Project Structure
+
+```
+BiblioFlow/
+├── src/
+│   ├── app.py                      # Dashboard entry point
+│   ├── components.py               # Topbar, sidebar, shared panel components
+│   ├── import_scopus_csv.py        # Standalone Scopus CSV importer
+│   ├── pages/                      # One module per analysis section
+│   │   ├── dashboard.py            # KPI cards, trends, donut, gauge
+│   │   ├── overview.py             # Publication production analysis
+│   │   ├── sources.py              # Journals, Bradford's Law, publishers
+│   │   ├── authors_page.py         # Author metrics and co-author networks
+│   │   ├── documents_page.py       # Citations, references, funding intelligence
+│   │   ├── conceptual_structure.py # Keywords and thematic map
+│   │   ├── social_structure.py     # Collaboration networks and world map
+│   │   ├── clustering.py           # Network clustering analysis
+│   │   ├── networks.py             # Intellectual structure / historiograph
+│   │   ├── evidence.py             # Evidence scores, reproducibility, duplicates,
+│   │   │                           #   provenance, novelty
+│   │   ├── timeline.py             # Research timeline bubble + heatmap
+│   │   ├── semantic_search.py      # Semantic search with example chips
+│   │   ├── gaps.py                 # Research gap detection bubble map
+│   │   ├── ai.py                   # AI Summary · Research Chat · Claim Extraction
+│   │   └── library.py              # Full paper corpus table
+│   ├── engine/                     # Analysis engines
+│   │   ├── evidence.py             # Evidence & reproducibility scoring
+│   │   ├── duplicates.py           # Fuzzy + DOI duplicate detection
+│   │   ├── funding.py              # Funder categorisation and trends
+│   │   ├── semantic_search.py      # FAISS index · search · novelty · similarity
+│   │   ├── gap_detection.py        # KMeans gap scoring
+│   │   ├── llm.py                  # Ollama client wrapper
+│   │   ├── ai_summary.py           # Corpus-level AI summary
+│   │   └── claim_extraction.py     # Per-paper claim extraction
+│   ├── repository/                 # Data access layer
+│   ├── services/                   # Import, enrichment, export services
+│   ├── providers/                  # CrossRef, OpenAlex, PubMed, … clients
+│   ├── database/                   # SQLite schema and connection helpers
+│   ├── visualizations/             # Plotly chart builders and theme
+│   │   ├── theme.py                # Dark palette, apply_theme()
+│   │   ├── lollipop.py             # Horizontal lollipop chart
+│   │   ├── donut.py                # Donut / pie chart
+│   │   ├── gauge.py                # Completeness gauge
+│   │   └── worldmap.py             # Choropleth collaboration map
+│   ├── extract/                    # PDF extraction pipeline
+│   ├── cache/                      # FAISS index, AI summary, claim caches
+│   └── assets/                     # CSS and static files
+├── requirements.txt
+└── README.md
+```
+
+---
+
+## Current Limitations
+
+| Area | Status |
+|---|---|
+| Author–institution attribution | Paper-level only — cannot isolate a specific co-author's affiliation |
+| Citation Half-Life | Not computable — APIs return cumulative citation counts, not per-year breakdowns |
+| Author/Journal co-citation | Not implemented — requires resolving individual reference metadata per paper |
+| Bibliographic coupling | Accuracy bounded by DOI resolution in reference lists |
+| Entity deduplication | Institution and country name variants across providers are not yet merged |
+| Evidence signals (PDF) | Trial IDs, ethics statements, figure counts require PDF extraction to be populated; Scopus CSV imports will show 0 for these fields |
+| AI chat streaming | Responses appear after full generation — streaming output not yet implemented |
+| Report export (PDF/Word/PPT) | Planned — marked "Soon" in the UI |
+
+---
+
+## Author
+
+**Robin Tomar** — [itsrobintomar@gmail.com](mailto:itsrobintomar@gmail.com)  
+**Dr Prabudh Goel's Lab, AIIMS New Delhi**
+
+---
+
+<div align="center">
+  <sub>BiblioFlow © 2026 · Built for researchers, by researchers · Made with ♥ in India</sub>
+</div>