paper-librarian 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- paper_librarian-0.1.0/.gitignore +8 -0
- paper_librarian-0.1.0/PKG-INFO +400 -0
- paper_librarian-0.1.0/README.md +382 -0
- paper_librarian-0.1.0/demo/app.py +247 -0
- paper_librarian-0.1.0/demo/onnx_embed.py +62 -0
- paper_librarian-0.1.0/demo/remote_mcp.py +31 -0
- paper_librarian-0.1.0/demo/requirements.txt +13 -0
- paper_librarian-0.1.0/demo/seed.py +30 -0
- paper_librarian-0.1.0/demo/sqlite_shim.py +10 -0
- paper_librarian-0.1.0/demo/static/index.html +268 -0
- paper_librarian-0.1.0/docs/screenshots/01-add-paper.png +0 -0
- paper_librarian-0.1.0/docs/screenshots/02-calling-tool.png +0 -0
- paper_librarian-0.1.0/docs/screenshots/03-second-paper-linked.png +0 -0
- paper_librarian-0.1.0/docs/screenshots/04-search-with-citations.png +0 -0
- paper_librarian-0.1.0/docs/screenshots/05-add-note.png +0 -0
- paper_librarian-0.1.0/pyproject.toml +35 -0
- paper_librarian-0.1.0/src/librarian/__init__.py +3 -0
- paper_librarian-0.1.0/src/librarian/citations.py +161 -0
- paper_librarian-0.1.0/src/librarian/embed.py +53 -0
- paper_librarian-0.1.0/src/librarian/ingest.py +457 -0
- paper_librarian-0.1.0/src/librarian/search.py +198 -0
- paper_librarian-0.1.0/src/librarian/server.py +333 -0
- paper_librarian-0.1.0/src/librarian/store.py +325 -0
- paper_librarian-0.1.0/tests/__init__.py +0 -0
- paper_librarian-0.1.0/tests/fixtures/.gitkeep +0 -0
- paper_librarian-0.1.0/tests/test_citations.py +167 -0
- paper_librarian-0.1.0/tests/test_ingest.py +259 -0
- paper_librarian-0.1.0/tests/test_search.py +109 -0
- paper_librarian-0.1.0/tests/test_server.py +153 -0
- paper_librarian-0.1.0/tests/test_smoke.py +24 -0
- paper_librarian-0.1.0/tests/test_store.py +100 -0
|
@@ -0,0 +1,400 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: paper-librarian
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: MCP server for a local, searchable library of research papers
|
|
5
|
+
Project-URL: Homepage, https://github.com/NeelGhoshal/MCP/tree/main/paper-librarian
|
|
6
|
+
Project-URL: Repository, https://github.com/NeelGhoshal/MCP
|
|
7
|
+
Keywords: arxiv,claude,mcp,papers,research,semantic-search
|
|
8
|
+
Requires-Python: >=3.11
|
|
9
|
+
Requires-Dist: arxiv>=2.1
|
|
10
|
+
Requires-Dist: mcp<3,>=2.3
|
|
11
|
+
Requires-Dist: pymupdf>=1.24
|
|
12
|
+
Requires-Dist: requests>=2.31
|
|
13
|
+
Requires-Dist: sentence-transformers>=3.0
|
|
14
|
+
Requires-Dist: sqlite-vec>=0.1.6
|
|
15
|
+
Provides-Extra: dev
|
|
16
|
+
Requires-Dist: pytest>=8; extra == 'dev'
|
|
17
|
+
Description-Content-Type: text/markdown
|
|
18
|
+
|
|
19
|
+
# Paper Librarian
|
|
20
|
+
|
|
21
|
+
<!-- mcp-name: io.github.NeelGhoshal/paper-librarian -->
|
|
22
|
+
|
|
23
|
+
An MCP server that gives Claude a local, searchable library of research papers (arXiv + PDFs).
|
|
24
|
+
Built on the **MCP Python SDK 2.x** (`MCPServer`, the 2.x successor to `FastMCP`).
|
|
25
|
+
|
|
26
|
+
Add papers by arXiv id or PDF path, then ask questions about them. Answers cite the paper, section
|
|
27
|
+
and page, and references between papers in the library are linked in both directions.
|
|
28
|
+
|
|
29
|
+
**Quick install** (needs [uv](https://docs.astral.sh/uv/)):
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
claude mcp add paper-librarian --scope user -- uvx paper-librarian
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
or in Claude Desktop's `claude_desktop_config.json`:
|
|
36
|
+
|
|
37
|
+
```json
|
|
38
|
+
{ "mcpServers": { "paper-librarian": { "command": "uvx", "args": ["paper-librarian"] } } }
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
---
|
|
42
|
+
|
|
43
|
+
## Demo
|
|
44
|
+
|
|
45
|
+
A session in Claude Code with the server connected.
|
|
46
|
+
|
|
47
|
+
**1. Add a paper by arXiv id.** `add_paper` downloads *Attention Is All You Need*, splits it into
|
|
48
|
+
44 searchable chunks and parses its 40 references. None are linked yet because the library was empty.
|
|
49
|
+
|
|
50
|
+

|
|
51
|
+
|
|
52
|
+
**2. Claude calls the MCP tool.** Asked to add BERT (`1810.04805`), Claude Code routes the request to
|
|
53
|
+
the `paper-librarian` server.
|
|
54
|
+
|
|
55
|
+

|
|
56
|
+
|
|
57
|
+
**3. Add a second paper; citations link automatically.** BERT is added with 71 chunks and 56 references.
|
|
58
|
+
One of them matches the Transformer paper already in the library, so the two papers are now linked.
|
|
59
|
+
|
|
60
|
+

|
|
61
|
+
|
|
62
|
+
**4. Ask a question; answers cite section and page.** Hybrid search finds the relevant passages in both
|
|
63
|
+
papers, and Claude cites each claim, e.g. *(Attention Is All You Need, Positional Encoding, p.6)*.
|
|
64
|
+
|
|
65
|
+

|
|
66
|
+
|
|
67
|
+
**5. Save a note to a paper.** `add_note` attaches a summary to the paper; it is returned whenever the
|
|
68
|
+
paper is looked up with `get_paper`.
|
|
69
|
+
|
|
70
|
+

|
|
71
|
+
|
|
72
|
+
---
|
|
73
|
+
|
|
74
|
+
## Problem statement
|
|
75
|
+
|
|
76
|
+
Researchers and students keep dozens of papers as PDFs, but an AI assistant can't use them well:
|
|
77
|
+
|
|
78
|
+
- **No memory of your library.** Each chat starts from nothing; pasting whole PDFs runs into context
|
|
79
|
+
limits and has to be repeated every time.
|
|
80
|
+
- **Answers from memory, not sources.** Without access to the actual text, a model summarizes papers
|
|
81
|
+
from training data, which may be outdated, vague or wrong, and can't point to where a claim comes from.
|
|
82
|
+
- **PDFs are hard to read programmatically.** Two-column layouts, section headings, hyphenation,
|
|
83
|
+
ligatures and reference lists in several citation styles all break naive text extraction.
|
|
84
|
+
- **Papers are connected, but the connections are invisible.** Which of my papers cite each other?
|
|
85
|
+
That information is buried in reference lists.
|
|
86
|
+
|
|
87
|
+
**Paper Librarian** solves this with an MCP server that runs locally and:
|
|
88
|
+
|
|
89
|
+
1. ingests papers from arXiv or local PDFs into a structured library (sections, pages, references);
|
|
90
|
+
2. indexes them for **hybrid search** (meaning-based + keyword), returning passages with
|
|
91
|
+
section and page so every answer can be cited;
|
|
92
|
+
3. parses reference lists and **links citations** between papers in the library;
|
|
93
|
+
4. exposes all of this to Claude (or any MCP client) as tools, a resource and a prompt.
|
|
94
|
+
|
|
95
|
+
Everything — database, PDFs, embedding model — stays on your machine.
|
|
96
|
+
|
|
97
|
+
---
|
|
98
|
+
|
|
99
|
+
## Features
|
|
100
|
+
|
|
101
|
+
- **Add papers** by arXiv id (`1706.03762`, `arXiv:1706.03762v5`), arXiv URL, or local PDF path, with
|
|
102
|
+
duplicate detection by arXiv id and file hash.
|
|
103
|
+
- **Layout-aware PDF extraction**: two-column reading order, section detection using font weight and
|
|
104
|
+
size, page tracking, ligature and hyphenation repair.
|
|
105
|
+
- **Hybrid search**: vector similarity (sqlite-vec) + BM25 keyword search (SQLite FTS5), merged with
|
|
106
|
+
reciprocal rank fusion; filters by year and paper.
|
|
107
|
+
- **Citation graph**: reference lists split in `[n]`, `n.` and author-year styles; matched to library
|
|
108
|
+
papers by arXiv id, exact title or fuzzy title; linked forwards and backwards as papers are added.
|
|
109
|
+
- **Section reader**: fetch the full text of any section ("Method", "Positional Encoding", …).
|
|
110
|
+
- **Comparison helper**: gathers the most relevant passages per paper per aspect for comparison tables.
|
|
111
|
+
- **Notes** attached to papers.
|
|
112
|
+
- **Local and private**: one SQLite file plus a PDF folder; no external services except arXiv
|
|
113
|
+
downloads and the one-time model download.
|
|
114
|
+
|
|
115
|
+
---
|
|
116
|
+
|
|
117
|
+
## Tech stack
|
|
118
|
+
|
|
119
|
+
| Layer | Technology | Why |
|
|
120
|
+
|---|---|---|
|
|
121
|
+
| Language | Python 3.11+ (developed on 3.13) | Ecosystem for PDFs, ML and MCP |
|
|
122
|
+
| MCP server | [MCP Python SDK](https://github.com/modelcontextprotocol/python-sdk) **2.3** (`MCPServer`) | Tools, resources, prompts, progress reporting, stdio transport |
|
|
123
|
+
| PDF parsing | [PyMuPDF](https://pymupdf.readthedocs.io/) 2.0 | Fast text extraction with font, size and position data per span |
|
|
124
|
+
| arXiv | [`arxiv`](https://pypi.org/project/arxiv/) 4.0 + `requests` | Metadata lookup (title, authors, abstract) and PDF download |
|
|
125
|
+
| Embeddings | [sentence-transformers](https://www.sbert.net/) 6.2, model `all-MiniLM-L6-v2` (384-dim) | Small (~90 MB), CPU-friendly, good quality for retrieval |
|
|
126
|
+
| Vector index | [sqlite-vec](https://github.com/asg017/sqlite-vec) 0.1.9 (`vec0` table) | KNN search inside the same SQLite file |
|
|
127
|
+
| Keyword index | SQLite FTS5 (BM25 ranking) | Built into Python's `sqlite3`, no extra service |
|
|
128
|
+
| Storage | SQLite | Single file, transactional, zero setup |
|
|
129
|
+
| Testing | pytest 9, anyio, MCP in-process `Client` | Unit tests plus end-to-end tests through a real MCP client |
|
|
130
|
+
| Build | hatchling (`pyproject.toml`) | Standard packaging; `paper-librarian` console script |
|
|
131
|
+
|
|
132
|
+
---
|
|
133
|
+
|
|
134
|
+
## Architecture
|
|
135
|
+
|
|
136
|
+
```mermaid
|
|
137
|
+
flowchart LR
|
|
138
|
+
Claude["Claude / MCP client"] <-- "stdio (JSON-RPC)" --> Server["server.py<br/>MCPServer: 8 tools,<br/>1 resource, 1 prompt"]
|
|
139
|
+
|
|
140
|
+
Server --> Ingest["ingest.py<br/>fetch · extract · sections · chunks"]
|
|
141
|
+
Server --> Search["search.py<br/>vector + BM25 + RRF"]
|
|
142
|
+
Server --> Store
|
|
143
|
+
|
|
144
|
+
Ingest --> ArXiv[("arXiv API")]
|
|
145
|
+
Ingest --> Embed["embed.py<br/>MiniLM embeddings"]
|
|
146
|
+
Ingest --> Cit["citations.py<br/>split · match · link"]
|
|
147
|
+
Search --> Embed
|
|
148
|
+
|
|
149
|
+
Ingest --> Store["store.py<br/>SQLite + FTS5 + sqlite-vec"]
|
|
150
|
+
Cit --> Store
|
|
151
|
+
Search --> Store
|
|
152
|
+
Store --> DB[("~/.librarian/library.db")]
|
|
153
|
+
Ingest --> PDFs[("~/.librarian/pdfs/")]
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
### How a paper is added (`add_paper`)
|
|
157
|
+
|
|
158
|
+
```mermaid
|
|
159
|
+
flowchart TD
|
|
160
|
+
A["source: arXiv id / URL / PDF path"] --> B{"arXiv id?"}
|
|
161
|
+
B -- yes --> C["duplicate check by arXiv id"] --> D["fetch metadata + download PDF"]
|
|
162
|
+
B -- no --> E["duplicate check by sha256"]
|
|
163
|
+
D --> F["extract_pdf: column-aware page text"]
|
|
164
|
+
E --> F
|
|
165
|
+
F --> G["split_sections: headings → segments (section, page)"]
|
|
166
|
+
G --> H["chunk_segments: 180-word windows, 30-word overlap"]
|
|
167
|
+
H --> I["embed chunks (MiniLM, normalized)"]
|
|
168
|
+
G --> J["References section → split_references"]
|
|
169
|
+
I --> K["one transaction: paper + chunks + vectors + FTS + citations + backlinks"]
|
|
170
|
+
J --> K
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
### How a question is answered (`search`)
|
|
174
|
+
|
|
175
|
+
```mermaid
|
|
176
|
+
flowchart LR
|
|
177
|
+
Q["query"] --> V["embed → KNN on chunk_vectors"]
|
|
178
|
+
Q --> K["sanitize → FTS5 MATCH, BM25"]
|
|
179
|
+
V --> R["reciprocal rank fusion<br/>score = Σ 1/(60 + rank)"]
|
|
180
|
+
K --> R
|
|
181
|
+
R --> F["filters: year, paper ids<br/>cap: N hits per paper"]
|
|
182
|
+
F --> S["hits: title · section · page · snippet · matched_by"]
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
---
|
|
186
|
+
|
|
187
|
+
## Project structure
|
|
188
|
+
|
|
189
|
+
```
|
|
190
|
+
paper-librarian/
|
|
191
|
+
├── pyproject.toml # dependencies, console script, pytest config
|
|
192
|
+
├── README.md
|
|
193
|
+
├── docs/screenshots/ # demo screenshots used in this README
|
|
194
|
+
├── src/librarian/
|
|
195
|
+
│ ├── __init__.py # package version
|
|
196
|
+
│ ├── server.py # MCP server: tools, resource, prompt, lifespan (333 lines)
|
|
197
|
+
│ ├── ingest.py # arXiv fetch, PDF extraction, sections, chunking (457 lines)
|
|
198
|
+
│ ├── search.py # hybrid search, RRF, snippets (198 lines)
|
|
199
|
+
│ ├── citations.py # reference splitting, matching, linking (161 lines)
|
|
200
|
+
│ ├── store.py # SQLite schema and data access (325 lines)
|
|
201
|
+
│ └── embed.py # embedding model wrapper ( 53 lines)
|
|
202
|
+
└── tests/ # 90 tests
|
|
203
|
+
├── test_server.py # end-to-end through an in-process MCP client (9)
|
|
204
|
+
├── test_ingest.py # arXiv ids, headings, sections, chunking, ingestion (39)
|
|
205
|
+
├── test_citations.py # splitting styles, matching, linking (20)
|
|
206
|
+
├── test_search.py # keyword / semantic / hybrid, filters, snippets (10)
|
|
207
|
+
├── test_store.py # schema, constraints, cascades (9)
|
|
208
|
+
└── test_smoke.py # dependencies load (3)
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
---
|
|
212
|
+
|
|
213
|
+
## Code details
|
|
214
|
+
|
|
215
|
+
### `store.py` — data model and storage
|
|
216
|
+
|
|
217
|
+
All data lives in one SQLite file. The schema:
|
|
218
|
+
|
|
219
|
+
| Table | Columns | Notes |
|
|
220
|
+
|---|---|---|
|
|
221
|
+
| `papers` | `id`, `title`, `authors`, `year`, `abstract`, `source`, `arxiv_id`, `sha256`, `path`, `added_at` | `id` is `arxiv:<id>` or `pdf:<sha256[:16]>`; `arxiv_id` and `sha256` are `UNIQUE` (duplicate protection); `source` is `CHECK`ed to `arxiv`/`pdf` |
|
|
222
|
+
| `chunks` | `id`, `paper_id`, `seq`, `section`, `page`, `text` | Searchable passages; `UNIQUE(paper_id, seq)` |
|
|
223
|
+
| `chunks_fts` | FTS5 over `text`, `section` | External-content index kept in sync by `AFTER INSERT/DELETE` triggers |
|
|
224
|
+
| `chunk_vectors` | `vec0(chunk_id, embedding float[384])` | sqlite-vec KNN index; same id as `chunks.id` so results can be fused |
|
|
225
|
+
| `citations` | `id`, `paper_id`, `seq`, `raw_reference`, `matched_paper_id` | `matched_paper_id` → `papers.id`, `ON DELETE SET NULL` |
|
|
226
|
+
| `notes` | `id`, `paper_id`, `text`, `created_at` | |
|
|
227
|
+
| `meta` | `key`, `value` | Stores `embedding_dim`; reopening with a different model raises instead of corrupting the index |
|
|
228
|
+
|
|
229
|
+
Deleting a paper cascades to its chunks, FTS entries, vectors (via trigger), citations and notes.
|
|
230
|
+
|
|
231
|
+
`Store` provides typed access (`Paper`, `Chunk`, `Citation`, `Note` dataclasses):
|
|
232
|
+
`add_paper`, `get_paper`, `find_duplicate`, `list_papers`, `delete_paper`, `add_chunks` (with vectors),
|
|
233
|
+
`get_chunks(section=…)`, `add_citations`, `get_citations`, `get_cited_by`, `unmatched_citations`,
|
|
234
|
+
`set_citation_match`, `delete_citations`, `add_note`, `get_notes`.
|
|
235
|
+
|
|
236
|
+
`store.transaction(commit=True)` wraps writes; passing `commit=False` to individual methods lets a
|
|
237
|
+
caller group several writes into one atomic transaction (used by ingestion).
|
|
238
|
+
|
|
239
|
+
The connection is opened with `check_same_thread=False` because MCP tools run in worker threads; the
|
|
240
|
+
server serializes access with a lock.
|
|
241
|
+
|
|
242
|
+
### `ingest.py` — getting papers in
|
|
243
|
+
|
|
244
|
+
| Function | What it does |
|
|
245
|
+
|---|---|
|
|
246
|
+
| `parse_arxiv_id(source)` | Recognizes new-style (`2310.06825v2`), old-style (`hep-th/9901001`), `arXiv:` prefixes and `abs/`/`pdf/` URLs; returns the version-less id |
|
|
247
|
+
| `fetch_arxiv(id, pdf_dir)` | Metadata via the `arxiv` API, PDF via `requests` (arxiv 4.x dropped `download_pdf`); validates the `%PDF` header |
|
|
248
|
+
| `extract_pdf(path)` | Title (PDF metadata, else largest horizontal text on page 1), authors, year, section segments, raw References text |
|
|
249
|
+
| `page_text(page)` | **Column-aware reading order**: blocks are classified left / right / full-width around the page centre; within bands separated by full-width blocks, the left column is read before the right. Marks indented lines (for reference splitting) and emphasized lines (bold or ≥1.1× body font size, for heading detection). Merges spans on the same baseline and applies NFKC normalization (`fi` → `fi`) |
|
|
250
|
+
| `heading_name(line, appendix)` | Decides whether a line is a section heading. Known names (Introduction, Methods, …) map to canonical names; other numbered headings (`3 Model Architecture`) must be emphasized. Rejects years and decimals as section numbers, lines with column gaps, and sentence fragments. Letter numbering (`A`, `B.2`) only counts after References/Appendix |
|
|
251
|
+
| `split_sections(pages)` | Walks lines, producing `Segment(section, page, text, raw)`; a new segment starts on each page so page numbers stay accurate |
|
|
252
|
+
| `chunk_segments(...)` | 180-word windows with 30-word overlap, never crossing a section boundary. 180 words fits MiniLM's 256-token input limit, so nothing is silently truncated |
|
|
253
|
+
| `ingest(source, store, embedder, progress)` | The whole pipeline (diagram above); writes paper, chunks, vectors and citations in **one transaction**, so a failure never leaves half a paper |
|
|
254
|
+
| `rebuild_citations(store)` | Re-parses every stored PDF's references, e.g. after improving the parser |
|
|
255
|
+
|
|
256
|
+
### `embed.py` — embeddings
|
|
257
|
+
|
|
258
|
+
`Embedder` is a `Protocol` (`dim`, `embed(texts)`), so tests use small fake embedders.
|
|
259
|
+
`SentenceTransformerEmbedder` loads the model lazily (thread-safe), normalizes vectors, and offers
|
|
260
|
+
`warm_up()` to load in the background at server start so the first search doesn't wait.
|
|
261
|
+
|
|
262
|
+
### `search.py` — hybrid retrieval
|
|
263
|
+
|
|
264
|
+
| Function | What it does |
|
|
265
|
+
|---|---|
|
|
266
|
+
| `fts_query(text)` | Turns free text into a safe FTS5 query (each word quoted, OR-ed), so input like `"x"? AND NEAR*` can't cause syntax errors |
|
|
267
|
+
| `keyword_search(...)` | FTS5 `MATCH` ranked by BM25, joined to papers for filtering |
|
|
268
|
+
| `vector_search(...)` | sqlite-vec KNN; with filters it over-fetches (10×, min 200) and filters after, since `vec0` can't filter on paper columns during the scan |
|
|
269
|
+
| `rrf(rankings)` | Reciprocal rank fusion, `score = Σ 1/(60 + rank)`; records which methods found each chunk |
|
|
270
|
+
| `search(...)` | Runs one or both retrievers, fuses, applies a per-paper cap, and builds `SearchHit`s |
|
|
271
|
+
| `make_snippet(text, terms)` | Picks the window containing the most distinct meaningful query words (short words like "of" ignored) |
|
|
272
|
+
|
|
273
|
+
On a 4-paper, 10-query check (Transformer, BERT, ResNet, word2vec), hybrid search ranked the
|
|
274
|
+
correct paper first 10/10 times.
|
|
275
|
+
|
|
276
|
+
### `citations.py` — the citation graph
|
|
277
|
+
|
|
278
|
+
| Function / class | What it does |
|
|
279
|
+
|---|---|
|
|
280
|
+
| `split_references(raw)` | Tries `[n]` markers (≥2), then sequential `n.` markers (≥3, mostly counting up), then **hanging indents** for author-year styles (first line at the column edge, continuation lines indented). Rejoins words hyphenated across lines |
|
|
281
|
+
| `reference_arxiv_id(ref)` | Finds `arXiv:1607.06450`, `abs/1409.0473`, arxiv.org URLs |
|
|
282
|
+
| `Matcher` | Matches a reference to a library paper: **arXiv id** → **exact title** (compared as lowercase alphanumerics, so `Pre-\ntraining` = `Pretraining`) → **fuzzy title** (≥92% of title characters found in order, after a cheap word-overlap pre-filter). Titles under 12 characters ("BERT") never match on title alone; the longest matching title wins |
|
|
283
|
+
| `link_paper(store, paper, refs)` | Stores the paper's references matched against the library, **and** re-checks older papers' unmatched references against the new paper (backlinks). Self-citations are excluded. Re-running replaces the paper's previous citations |
|
|
284
|
+
|
|
285
|
+
### `server.py` — the MCP interface
|
|
286
|
+
|
|
287
|
+
`create_server(db_path=None, embedder=None)` builds an `MCPServer` whose **lifespan** opens the
|
|
288
|
+
`Store`, creates the embedder (warming it up), and yields an `AppState(store, embedder, lock)`.
|
|
289
|
+
Tools get it from `ctx.request_context.lifespan_context`. The factory lets tests inject a temporary
|
|
290
|
+
database and a fake embedder; `main()` runs the default server over stdio and quiets
|
|
291
|
+
model-hub HTTP logging (`httpx`, `httpx2`, `huggingface_hub`).
|
|
292
|
+
|
|
293
|
+
**Tools**
|
|
294
|
+
|
|
295
|
+
| Tool | Arguments | Returns | Annotations |
|
|
296
|
+
|---|---|---|---|
|
|
297
|
+
| `add_paper` | `source` | paper summary, `already_in_library`, chunk and reference counts; reports progress | open-world, idempotent |
|
|
298
|
+
| `search` | `query`, `limit=8` (max 30), `mode` (`hybrid`/`semantic`/`keyword`), `year_from`, `year_to`, `paper_ids` | `results[]`: paper id, title, year, section, page, snippet, matched_by | read-only |
|
|
299
|
+
| `get_paper` | `paper_id`, `section?` | metadata, abstract, sections (with start page), notes — or one section's full text (chunk overlaps removed, partial names accepted) | read-only |
|
|
300
|
+
| `get_citations` | `paper_id` | `references[]` with `in_library` link, `cited_by[]` | read-only |
|
|
301
|
+
| `compare_papers` | `paper_ids` (2–8), `aspects` (1–8), `passages_per_aspect=2` | passages per paper per aspect | read-only |
|
|
302
|
+
| `add_note` | `paper_id`, `text` | note id, timestamp | |
|
|
303
|
+
| `list_papers` | — | count, paper summaries | read-only |
|
|
304
|
+
| `remove_paper` | `paper_id` | removed id and title (PDF file kept) | destructive |
|
|
305
|
+
|
|
306
|
+
**Resource:** `library://papers` — the library as a Markdown list.
|
|
307
|
+
**Prompt:** `literature_review(topic)` — guides Claude through search → compare → citations → a cited review.
|
|
308
|
+
|
|
309
|
+
Paper ids are accepted in full (`arxiv:1706.03762`) or bare (`1706.03762`). Expected failures (unknown
|
|
310
|
+
paper, missing file, download error, unknown section) are raised as `ToolError`, so Claude gets an
|
|
311
|
+
actionable message such as *"No paper '…' in the library. Use list_papers to see paper ids."*
|
|
312
|
+
|
|
313
|
+
---
|
|
314
|
+
|
|
315
|
+
## Setup
|
|
316
|
+
|
|
317
|
+
```bash
|
|
318
|
+
cd paper-librarian
|
|
319
|
+
python -m venv .venv
|
|
320
|
+
.venv\Scripts\activate # Windows (source .venv/bin/activate on macOS/Linux)
|
|
321
|
+
pip install -e ".[dev]"
|
|
322
|
+
pytest -q # 90 tests
|
|
323
|
+
```
|
|
324
|
+
|
|
325
|
+
The first run downloads the `all-MiniLM-L6-v2` embedding model (~90 MB).
|
|
326
|
+
|
|
327
|
+
## Connect to Claude
|
|
328
|
+
|
|
329
|
+
**Claude Code**
|
|
330
|
+
|
|
331
|
+
```bash
|
|
332
|
+
claude mcp add paper-librarian --scope user -- <path-to-repo>/.venv/Scripts/python.exe -m librarian.server
|
|
333
|
+
claude mcp get paper-librarian # should show: Status: ✔ Connected
|
|
334
|
+
```
|
|
335
|
+
|
|
336
|
+
Start a new Claude Code session afterwards; servers are loaded when a session starts.
|
|
337
|
+
Remove with `claude mcp remove paper-librarian -s user`.
|
|
338
|
+
|
|
339
|
+
**Claude Desktop** — add to `claude_desktop_config.json` (Settings → Developer → Edit Config), then restart:
|
|
340
|
+
|
|
341
|
+
```json
|
|
342
|
+
{
|
|
343
|
+
"mcpServers": {
|
|
344
|
+
"paper-librarian": {
|
|
345
|
+
"command": "C:/path/to/paper-librarian/.venv/Scripts/python.exe",
|
|
346
|
+
"args": ["-m", "librarian.server"]
|
|
347
|
+
}
|
|
348
|
+
}
|
|
349
|
+
}
|
|
350
|
+
```
|
|
351
|
+
|
|
352
|
+
## Try it
|
|
353
|
+
|
|
354
|
+
- "Add arXiv 1706.03762 and 1810.04805 to my paper library"
|
|
355
|
+
- "What do my papers say about positional encoding? Cite sections."
|
|
356
|
+
- "Compare BERT and the Transformer on pre-training corpus, model size and results"
|
|
357
|
+
- "Which papers in my library cite Attention Is All You Need?"
|
|
358
|
+
- Use the `literature_review` prompt with a topic
|
|
359
|
+
|
|
360
|
+
## Configuration
|
|
361
|
+
|
|
362
|
+
| Variable | Default | Meaning |
|
|
363
|
+
|---|---|---|
|
|
364
|
+
| `LIBRARIAN_HOME` | `~/.librarian` | Library folder (`library.db`, downloaded `pdfs/`) |
|
|
365
|
+
| `LIBRARIAN_EMBED_MODEL` | `sentence-transformers/all-MiniLM-L6-v2` | Embedding model (must produce 384-dim vectors) |
|
|
366
|
+
| `HF_HUB_OFFLINE` | unset | Set to `1` once the model is downloaded to skip Hugging Face checks at startup |
|
|
367
|
+
|
|
368
|
+
## Testing and debugging
|
|
369
|
+
|
|
370
|
+
```bash
|
|
371
|
+
pytest -q # unit + end-to-end MCP tests
|
|
372
|
+
npx @modelcontextprotocol/inspector .venv/Scripts/python.exe -m librarian.server
|
|
373
|
+
```
|
|
374
|
+
|
|
375
|
+
`tests/test_server.py` connects the SDK's `Client` to the server in-process and exercises every
|
|
376
|
+
tool, the resource and the prompt, including progress notifications and error cases.
|
|
377
|
+
|
|
378
|
+
---
|
|
379
|
+
|
|
380
|
+
## Known limitations
|
|
381
|
+
|
|
382
|
+
- **Scanned PDFs** (no text layer) are rejected; there is no OCR.
|
|
383
|
+
- **Unusual layouts**: three-column pages, or reference lists that fit none of the three styles,
|
|
384
|
+
may extract or split imperfectly.
|
|
385
|
+
- **Subsections replace parent names**: text under 3.2.1 is labelled `Scaled Dot-Product Attention`,
|
|
386
|
+
not `Model Architecture`.
|
|
387
|
+
- **Filtered semantic search** over-fetches and filters afterwards; with many thousands of papers, a
|
|
388
|
+
very narrow filter could miss results.
|
|
389
|
+
- **Ingestion holds the database lock**, so other tool calls wait while a paper is being added
|
|
390
|
+
(typically 5–30 s).
|
|
391
|
+
- **Hyphen rejoining** also merges genuine compounds broken at a line end (`sequence-aligned` →
|
|
392
|
+
`sequencealigned`).
|
|
393
|
+
- References to papers **outside** the library are stored as text only (no Semantic Scholar lookup yet).
|
|
394
|
+
|
|
395
|
+
## Roadmap
|
|
396
|
+
|
|
397
|
+
- Semantic Scholar lookup for references not in the library
|
|
398
|
+
- OCR fallback for scanned PDFs
|
|
399
|
+
- Parent/child section hierarchy
|
|
400
|
+
- Packaging for `uvx paper-librarian`
|