citewell 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,26 @@
1
+ # Python
2
+ __pycache__/
3
+ *.py[cod]
4
+ *.egg-info/
5
+ build/
6
+ dist/
7
+ .venv/
8
+ .mypy_cache/
9
+ .pytest_cache/
10
+ .ruff_cache/
11
+ .coverage
12
+ .coverage.*
13
+ coverage.xml
14
+ htmlcov/
15
+
16
+ # Docs
17
+ site/
18
+
19
+ # Local
20
+ .env
21
+ data/*
22
+ eval-report*.json
23
+ .DS_Store
24
+ .idea/
25
+ .vscode/
26
+ !data/.gitkeep
@@ -0,0 +1,32 @@
1
+ # Changelog
2
+
3
+ ## 0.1.0 (2026-11-03)
4
+
5
+
6
+ ### Features
7
+
8
+ * drop near-duplicate passages, time searches without model loading, preload models ([cdd5de0](https://github.com/omerozkilic/citewell/commit/cdd5de04b46ccac19114ee3b5ed41d3cd62d256f))
9
+ * **eval:** Turkish legislation eval set with article-based page resolution ([8276f23](https://github.com/omerozkilic/citewell/commit/8276f230fa50bb2c6d92a65583b8a0e0f9f97897))
10
+ * hybrid search with RRF fusion and optional cross-encoder rerank ([4b3829c](https://github.com/omerozkilic/citewell/commit/4b3829ce9a27f32e0c03df3a04b6470371f94598))
11
+ * MCP server, REST API and CLI ([1f97659](https://github.com/omerozkilic/citewell/commit/1f9765999d64aec9adbdc5ae0247c4e3fb1d148e))
12
+ * page-aware ingest pipeline with Turkish-aware text handling ([8a7e0ef](https://github.com/omerozkilic/citewell/commit/8a7e0ef0b689c0ed319199f80ee2acaf2476289b))
13
+ * retrieval eval harness and Turkish sample corpus ([6a2e67f](https://github.com/omerozkilic/citewell/commit/6a2e67fa52c930b83ff3937172b885bcf4801327))
14
+
15
+
16
+ ### Bug Fixes
17
+
18
+ * fail fast with an actionable message when PostgreSQL is unreachable ([a3d1892](https://github.com/omerozkilic/citewell/commit/a3d1892daba8df70ef335a889d9001452aca1894))
19
+ * load cached models without network calls and quiet model-library logs ([29973cb](https://github.com/omerozkilic/citewell/commit/29973cb98065de8cbffbef610a5fef7352661453))
20
+ * stay offline once models are cached and flag PDF pages without text ([478efc6](https://github.com/omerozkilic/citewell/commit/478efc683056a0f8757ef3fc7c436ad1aad66e29))
21
+
22
+
23
+ ### Documentation
24
+
25
+ * publish first bge-m3 results on the Turkish Labour Law eval ([967246f](https://github.com/omerozkilic/citewell/commit/967246f9e47a9257998701066a5b065ab97eac53))
26
+ * README, MkDocs site, ADRs and community files ([9db71d1](https://github.com/omerozkilic/citewell/commit/9db71d1b295b5f023e4c4aa85a2ecd8fcc8e34ab))
27
+
28
+ ## Changelog
29
+
30
+ All notable changes to this project are documented here. This file is maintained by
31
+ [release-please](https://github.com/googleapis/release-please) from
32
+ [Conventional Commits](https://www.conventionalcommits.org/).
citewell-0.1.0/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Ömer Özkılıç
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,306 @@
1
+ Metadata-Version: 2.5
2
+ Name: citewell
3
+ Version: 0.1.0
4
+ Summary: Citation-grounded document memory for AI assistants. Offline, MCP-native, Turkish-ready.
5
+ Project-URL: Homepage, https://github.com/omerozkilic/citewell
6
+ Project-URL: Documentation, https://omerozkilic.github.io/citewell
7
+ Project-URL: Repository, https://github.com/omerozkilic/citewell
8
+ Project-URL: Issues, https://github.com/omerozkilic/citewell/issues
9
+ Project-URL: Changelog, https://github.com/omerozkilic/citewell/blob/main/CHANGELOG.md
10
+ Author: Ömer Özkılıç
11
+ License-Expression: MIT
12
+ License-File: LICENSE
13
+ Keywords: citations,claude,mcp,model-context-protocol,pgvector,rag,retrieval,turkish
14
+ Classifier: Development Status :: 3 - Alpha
15
+ Classifier: Intended Audience :: Developers
16
+ Classifier: Natural Language :: English
17
+ Classifier: Natural Language :: Turkish
18
+ Classifier: Operating System :: OS Independent
19
+ Classifier: Programming Language :: Python :: 3
20
+ Classifier: Programming Language :: Python :: 3.11
21
+ Classifier: Programming Language :: Python :: 3.12
22
+ Classifier: Programming Language :: Python :: 3.13
23
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
24
+ Classifier: Topic :: Text Processing :: Indexing
25
+ Classifier: Typing :: Typed
26
+ Requires-Python: >=3.11
27
+ Requires-Dist: fastapi>=0.115
28
+ Requires-Dist: httpx>=0.27
29
+ Requires-Dist: mcp<3,>=2.3
30
+ Requires-Dist: numpy>=1.26
31
+ Requires-Dist: pgvector>=0.3
32
+ Requires-Dist: psycopg-pool>=3.2
33
+ Requires-Dist: psycopg[binary]>=3.2
34
+ Requires-Dist: pydantic-settings>=2.4
35
+ Requires-Dist: pydantic>=2.7
36
+ Requires-Dist: pypdf>=5.0
37
+ Requires-Dist: python-multipart>=0.0.9
38
+ Requires-Dist: rich>=13.7
39
+ Requires-Dist: typer>=0.12
40
+ Requires-Dist: uvicorn[standard]>=0.30
41
+ Provides-Extra: docx
42
+ Requires-Dist: python-docx>=1.1; extra == 'docx'
43
+ Provides-Extra: models
44
+ Requires-Dist: sentence-transformers>=3.0; extra == 'models'
45
+ Requires-Dist: torch>=2.3; extra == 'models'
46
+ Description-Content-Type: text/markdown
47
+
48
+ <div align="center">
49
+
50
+ # Citewell
51
+
52
+ **Citation-grounded document memory for AI assistants.**<br>
53
+ Offline · MCP-native · Turkish-ready
54
+
55
+ [![CI](https://github.com/omerozkilic/citewell/actions/workflows/ci.yml/badge.svg)](https://github.com/omerozkilic/citewell/actions/workflows/ci.yml)
56
+ [![CodeQL](https://github.com/omerozkilic/citewell/actions/workflows/codeql.yml/badge.svg)](https://github.com/omerozkilic/citewell/actions/workflows/codeql.yml)
57
+ [![codecov](https://codecov.io/gh/omerozkilic/citewell/branch/main/graph/badge.svg)](https://codecov.io/gh/omerozkilic/citewell)
58
+ [![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13-blue)](pyproject.toml)
59
+ [![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
60
+ [![Docs](https://img.shields.io/badge/docs-mkdocs-blueviolet)](https://omerozkilic.github.io/citewell)
61
+
62
+ [Türkçe README](README.tr.md) · [Documentation](https://omerozkilic.github.io/citewell) · [Architecture decisions](docs/adr/index.md)
63
+
64
+ </div>
65
+
66
+ ---
67
+
68
+ Citewell is a **headless RAG server** that gives Claude Desktop, Claude Code, Cursor and any other
69
+ [MCP](https://modelcontextprotocol.io) client access to *your* documents — and every passage it
70
+ returns comes with **file, page, chunk id and a verbatim quote**, so answers can be checked.
71
+
72
+ It has **no LLM of its own and needs no API key**. It retrieves; your assistant writes the answer.
73
+
74
+ ```text
75
+ You ▸ Ankara'da otelde bir gece için en fazla ne kadar harcayabilirim?
76
+
77
+ Claude ▸ (calls citewell.search_documents)
78
+ Ankara, İstanbul ve İzmir'de gecelik konaklama üst sınırı 3.500 TL'dir [1].
79
+
80
+ Kaynaklar
81
+ [1] calisan-el-kitabi.pdf, s. 4 — "İş seyahatlerinde konaklama için gecelik üst sınır
82
+ İstanbul, Ankara ve İzmir'de 3.500 TL, diğer şehirlerde 2.500 TL'dir."
83
+ ```
84
+
85
+ ## Why Citewell?
86
+
87
+ - **Cited, verifiable passages** — page-aware chunking that never crosses page boundaries, verbatim
88
+ quotes, stable chunk ids ([ADR-0006](docs/adr/0006-page-aware-chunking.md)).
89
+ - **MCP-native** — stdio for desktop clients, Streamable HTTP at `/mcp`, structured tool output,
90
+ resources and a citation prompt. Read-only by design ([ADR-0007](docs/adr/0007-read-only-mcp.md)).
91
+ - **Hybrid search** — pgvector (HNSW) + PostgreSQL full-text search fused with Reciprocal Rank Fusion,
92
+ optional `bge-reranker-v2-m3` cross-encoder ([ADR-0004](docs/adr/0004-hybrid-search-rrf.md)).
93
+ - **Turkish first** — Turkish stemming (`tsvector` `turkish` config), correct `I/ı` – `İ/i` casing,
94
+ cp1254 decoding, multilingual `bge-m3` embeddings, a Turkish eval set.
95
+ - **Offline & private** — local models; after a one-time download nothing leaves your machine.
96
+ - **Measured** — built-in eval: Recall@k, MRR, hit rate, page hit rate, p50/p95 latency.
97
+ - **One command** — `docker compose up`. PostgreSQL is the only dependency.
98
+
99
+ ## Quickstart (Docker)
100
+
101
+ ```bash
102
+ git clone https://github.com/omerozkilic/citewell && cd citewell
103
+ docker compose up -d # PostgreSQL + Citewell on localhost:8000
104
+
105
+ cp -r ~/Documents/handbooks/* data/ # PDF, Markdown, TXT, HTML, DOCX
106
+ docker compose run --rm citewell models download # once (~2–3 GB), needs internet
107
+ docker compose run --rm citewell ingest /data
108
+ docker compose run --rm citewell search "yıllık izin kaç gün?"
109
+ ```
110
+
111
+ - REST API & OpenAPI docs: <http://localhost:8000/docs>
112
+ - MCP endpoint (Streamable HTTP): <http://localhost:8000/mcp>
113
+
114
+ No documents at hand? Use the bundled sample corpus (a fictional Turkish company handbook):
115
+
116
+ ```bash
117
+ docker compose run --rm -v ./examples:/examples:ro citewell ingest /examples/corpus
118
+ docker compose run --rm citewell search "Ankara'da otelde bir gece için en fazla ne kadar harcayabilirim?"
119
+ ```
120
+
121
+ > A database is tied to the embedding model it was initialised with (`citewell doctor` shows it).
122
+ > To switch models, use a fresh database volume and re-ingest.
123
+
124
+ ## Quickstart (local, uv)
125
+
126
+ ```bash
127
+ uv sync --extra models
128
+ docker compose up -d db
129
+ uv run citewell init
130
+ uv run citewell ingest ./my-docs --collection handbook
131
+ uv run citewell search "parola en az kaç karakter olmalı?" -c handbook
132
+ ```
133
+
134
+ ## Connect your assistant
135
+
136
+ <details open>
137
+ <summary><b>Claude Code</b></summary>
138
+
139
+ ```bash
140
+ # HTTP (server running via docker compose or `citewell serve`)
141
+ claude mcp add --transport http citewell http://localhost:8000/mcp
142
+
143
+ # or stdio, from a local checkout
144
+ claude mcp add citewell -- uv --directory /path/to/citewell run citewell mcp
145
+ ```
146
+ </details>
147
+
148
+ <details>
149
+ <summary><b>Claude Desktop</b> — <code>claude_desktop_config.json</code></summary>
150
+
151
+ ```json
152
+ {
153
+ "mcpServers": {
154
+ "citewell": {
155
+ "command": "docker",
156
+ "args": ["exec", "-i", "citewell-citewell-1", "citewell", "mcp"]
157
+ }
158
+ }
159
+ }
160
+ ```
161
+
162
+ Local checkout instead of Docker:
163
+
164
+ ```json
165
+ {
166
+ "mcpServers": {
167
+ "citewell": {
168
+ "command": "uv",
169
+ "args": ["--directory", "/path/to/citewell", "run", "citewell", "mcp"],
170
+ "env": { "CITEWELL_DATABASE_URL": "postgresql://citewell:citewell@localhost:5432/citewell" }
171
+ }
172
+ }
173
+ }
174
+ ```
175
+ </details>
176
+
177
+ <details>
178
+ <summary><b>Cursor</b> — <code>.cursor/mcp.json</code></summary>
179
+
180
+ ```json
181
+ { "mcpServers": { "citewell": { "url": "http://localhost:8000/mcp" } } }
182
+ ```
183
+ </details>
184
+
185
+ ### MCP surface
186
+
187
+ | Kind | Name | Purpose |
188
+ |---|---|---|
189
+ | tool | `search_documents(query, top_k, collection, mode, rerank, source_prefix)` | Cited passages (`ref`, `citation`, `source`, `page`, `chunk_id`, `quote`, `text`, `score`) |
190
+ | tool | `get_chunk(chunk_id, context)` | A chunk plus neighbouring chunks, to read around a quote |
191
+ | tool | `list_collections()` / `list_documents(collection)` | What is indexed |
192
+ | resource | `citewell://collections`, `citewell://chunks/{id}` | Same data as resources |
193
+ | prompt | `answer_with_citations(question, collection)` | "Answer only from passages, cite with [n]" |
194
+
195
+ ## CLI
196
+
197
+ ```text
198
+ citewell init create extension + schema (idempotent)
199
+ citewell ingest PATH... [-c COL] [-l turkish|english|simple] [--force]
200
+ citewell search "query" [-k 5] [-m hybrid|vector|keyword] [-c COL|*] [--rerank] [--json]
201
+ citewell serve [--init] [--host] [--port] REST API + MCP at /mcp
202
+ citewell mcp [--transport stdio|http] MCP server only
203
+ citewell eval DATASET.jsonl [-k 5] [-r vector,keyword,hybrid,hybrid+rerank] [-o report.json]
204
+ citewell collections list|create|delete · documents list|delete · models download · doctor
205
+ ```
206
+
207
+ ## REST API
208
+
209
+ ```bash
210
+ curl -s localhost:8000/v1/search -H 'content-type: application/json' \
211
+ -d '{"query": "uzaktan çalışma kaç gün?", "top_k": 3, "mode": "hybrid"}'
212
+ curl -s -F file=@politika.pdf -F collection=handbook localhost:8000/v1/documents
213
+ ```
214
+
215
+ `GET /health` · `POST|GET /v1/search` · `GET /v1/chunks/{id}?context=1` · `GET|POST|DELETE /v1/collections` ·
216
+ `GET|POST|DELETE /v1/documents` — full schema at `/docs`. Set `CITEWELL_API_TOKEN` to require a bearer token.
217
+
218
+ ## How it works
219
+
220
+ ```mermaid
221
+ flowchart LR
222
+ subgraph Ingest
223
+ F[PDF / MD / TXT / HTML / DOCX] --> L[Loaders<br/>pages & sections]
224
+ L --> C[Page-aware chunker<br/>verbatim slices]
225
+ C --> E[bge-m3 embeddings<br/>+ title/section header]
226
+ end
227
+ E --> PG[(PostgreSQL<br/>pgvector HNSW<br/>tsvector GIN)]
228
+ subgraph Query
229
+ Q[query] --> V[vector top-40]
230
+ Q --> K[keyword top-40<br/>turkish stemmer]
231
+ V --> R[RRF k=60]
232
+ K --> R
233
+ R --> X[optional rerank<br/>bge-reranker-v2-m3]
234
+ X --> O[passages + citations<br/>+ best quote]
235
+ end
236
+ PG --- V
237
+ PG --- K
238
+ O --> M[MCP · REST · CLI]
239
+ ```
240
+
241
+ ## Evaluation
242
+
243
+ ```bash
244
+ citewell eval examples/eval/tr-sample.jsonl -k 5 -r vector,keyword,hybrid,hybrid+rerank --markdown
245
+ ```
246
+
247
+ Datasets are JSON Lines: `{"id", "query", "relevant": [{"source", "page"?}], "collection"?}`.
248
+ Metrics: **Recall@k**, **MRR@k**, **Hit@k**, **Page hit@k** (exact page), **p50/p95 latency**.
249
+ CI runs the sample set end-to-end on every push (with the lexical test backends) and fails below a
250
+ recall floor; see [docs/evaluation.md](docs/evaluation.md).
251
+
252
+ ### First results: Turkish Labour Law (İş Kanunu, 4857)
253
+
254
+ 20 natural-language questions from [`examples/eval/mevzuat-questions.jsonl`](examples/eval/mevzuat-questions.jsonl)
255
+ against the official PDF from mevzuat.gov.tr (121 articles). A hit counts only if it lands on a page
256
+ of the right article. `BAAI/bge-m3` + `bge-reranker-v2-m3`, default chunking, Apple Silicon (MPS),
257
+ models already loaded:
258
+
259
+ | Run | Recall@5 | MRR@5 | Hit@5 | Page hit@5 | p50 ms | p95 ms |
260
+ |---|---:|---:|---:|---:|---:|---:|
261
+ | vector | 1.000 | 0.885 | 1.000 | 1.000 | 45 | 95 |
262
+ | keyword | 0.850 | 0.614 | 0.850 | 0.850 | 8 | 11 |
263
+ | **hybrid** (default) | **1.000** | **0.893** | **1.000** | **1.000** | **48** | **54** |
264
+ | hybrid + rerank | 1.000 | 0.950 | 1.000 | 1.000 | 3259 | 3771 |
265
+
266
+ - Every question finds the right article **and page** in the top 5 with hybrid search; the right
267
+ passage is usually ranked first or second (MRR 0.89).
268
+ - Dense retrieval does the heavy lifting on paraphrased questions; keyword search alone misses 15 %.
269
+ - The cross-encoder lifts MRR to 0.95 but costs ~3 s per query on a laptop, so it is opt-in
270
+ (`--rerank` / `rerank: true`).
271
+
272
+ > Small and single-domain (20 questions, one law, written by the maintainers) — a first data point,
273
+ > not a leaderboard. Reproduce it with [docs/evaluation.md](docs/evaluation.md#turkish-legislation-set);
274
+ > larger independent Turkish eval sets are very welcome.
275
+
276
+ ## Configuration
277
+
278
+ All settings are environment variables prefixed with `CITEWELL_` (or a `.env` file) — see
279
+ [`.env.example`](.env.example) and [docs/configuration.md](docs/configuration.md).
280
+
281
+ | Variable | Default |
282
+ |---|---|
283
+ | `CITEWELL_DATABASE_URL` | `postgresql://citewell:citewell@localhost:5432/citewell` |
284
+ | `CITEWELL_EMBEDDING_MODEL` / `_DIM` | `BAAI/bge-m3` / `1024` |
285
+ | `CITEWELL_RERANKER_MODEL` | `BAAI/bge-reranker-v2-m3` |
286
+ | `CITEWELL_DEFAULT_MODE` / `_TOP_K` | `hybrid` / `5` |
287
+ | `CITEWELL_CHUNK_SIZE` / `_OVERLAP` | `1000` / `150` characters |
288
+ | `CITEWELL_API_TOKEN` | unset (no auth; bind to localhost) |
289
+
290
+ ## Roadmap
291
+
292
+ - [x] **v0.1** — ingest (PDF/MD/TXT/HTML/DOCX), vector search, cited passages, CLI, REST, MCP, Docker
293
+ - [x] **v0.2** — hybrid search (pgvector + tsvector, RRF), cross-encoder rerank, Turkish eval harness
294
+ - [ ] **v0.3** — MCP: completions, progress notifications, resource subscriptions; per-collection
295
+ access, collection-scoped tokens *(collections, resources and prompts already ship)*
296
+ - [ ] Later — OpenAI-compatible endpoint (Open WebUI), OCR for scanned PDFs, sparse/BM25 retriever,
297
+ larger public Turkish benchmark, SQLite single-file mode
298
+
299
+ ## Contributing
300
+
301
+ See [CONTRIBUTING.md](CONTRIBUTING.md). Security reports: [SECURITY.md](SECURITY.md).
302
+ Architecture decisions live in [docs/adr](docs/adr/index.md).
303
+
304
+ ## License
305
+
306
+ [MIT](LICENSE) © Ömer Özkılıç
@@ -0,0 +1,259 @@
1
+ <div align="center">
2
+
3
+ # Citewell
4
+
5
+ **Citation-grounded document memory for AI assistants.**<br>
6
+ Offline · MCP-native · Turkish-ready
7
+
8
+ [![CI](https://github.com/omerozkilic/citewell/actions/workflows/ci.yml/badge.svg)](https://github.com/omerozkilic/citewell/actions/workflows/ci.yml)
9
+ [![CodeQL](https://github.com/omerozkilic/citewell/actions/workflows/codeql.yml/badge.svg)](https://github.com/omerozkilic/citewell/actions/workflows/codeql.yml)
10
+ [![codecov](https://codecov.io/gh/omerozkilic/citewell/branch/main/graph/badge.svg)](https://codecov.io/gh/omerozkilic/citewell)
11
+ [![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13-blue)](pyproject.toml)
12
+ [![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
13
+ [![Docs](https://img.shields.io/badge/docs-mkdocs-blueviolet)](https://omerozkilic.github.io/citewell)
14
+
15
+ [Türkçe README](README.tr.md) · [Documentation](https://omerozkilic.github.io/citewell) · [Architecture decisions](docs/adr/index.md)
16
+
17
+ </div>
18
+
19
+ ---
20
+
21
+ Citewell is a **headless RAG server** that gives Claude Desktop, Claude Code, Cursor and any other
22
+ [MCP](https://modelcontextprotocol.io) client access to *your* documents — and every passage it
23
+ returns comes with **file, page, chunk id and a verbatim quote**, so answers can be checked.
24
+
25
+ It has **no LLM of its own and needs no API key**. It retrieves; your assistant writes the answer.
26
+
27
+ ```text
28
+ You ▸ Ankara'da otelde bir gece için en fazla ne kadar harcayabilirim?
29
+
30
+ Claude ▸ (calls citewell.search_documents)
31
+ Ankara, İstanbul ve İzmir'de gecelik konaklama üst sınırı 3.500 TL'dir [1].
32
+
33
+ Kaynaklar
34
+ [1] calisan-el-kitabi.pdf, s. 4 — "İş seyahatlerinde konaklama için gecelik üst sınır
35
+ İstanbul, Ankara ve İzmir'de 3.500 TL, diğer şehirlerde 2.500 TL'dir."
36
+ ```
37
+
38
+ ## Why Citewell?
39
+
40
+ - **Cited, verifiable passages** — page-aware chunking that never crosses page boundaries, verbatim
41
+ quotes, stable chunk ids ([ADR-0006](docs/adr/0006-page-aware-chunking.md)).
42
+ - **MCP-native** — stdio for desktop clients, Streamable HTTP at `/mcp`, structured tool output,
43
+ resources and a citation prompt. Read-only by design ([ADR-0007](docs/adr/0007-read-only-mcp.md)).
44
+ - **Hybrid search** — pgvector (HNSW) + PostgreSQL full-text search fused with Reciprocal Rank Fusion,
45
+ optional `bge-reranker-v2-m3` cross-encoder ([ADR-0004](docs/adr/0004-hybrid-search-rrf.md)).
46
+ - **Turkish first** — Turkish stemming (`tsvector` `turkish` config), correct `I/ı` – `İ/i` casing,
47
+ cp1254 decoding, multilingual `bge-m3` embeddings, a Turkish eval set.
48
+ - **Offline & private** — local models; after a one-time download nothing leaves your machine.
49
+ - **Measured** — built-in eval: Recall@k, MRR, hit rate, page hit rate, p50/p95 latency.
50
+ - **One command** — `docker compose up`. PostgreSQL is the only dependency.
51
+
52
+ ## Quickstart (Docker)
53
+
54
+ ```bash
55
+ git clone https://github.com/omerozkilic/citewell && cd citewell
56
+ docker compose up -d # PostgreSQL + Citewell on localhost:8000
57
+
58
+ cp -r ~/Documents/handbooks/* data/ # PDF, Markdown, TXT, HTML, DOCX
59
+ docker compose run --rm citewell models download # once (~2–3 GB), needs internet
60
+ docker compose run --rm citewell ingest /data
61
+ docker compose run --rm citewell search "yıllık izin kaç gün?"
62
+ ```
63
+
64
+ - REST API & OpenAPI docs: <http://localhost:8000/docs>
65
+ - MCP endpoint (Streamable HTTP): <http://localhost:8000/mcp>
66
+
67
+ No documents at hand? Use the bundled sample corpus (a fictional Turkish company handbook):
68
+
69
+ ```bash
70
+ docker compose run --rm -v ./examples:/examples:ro citewell ingest /examples/corpus
71
+ docker compose run --rm citewell search "Ankara'da otelde bir gece için en fazla ne kadar harcayabilirim?"
72
+ ```
73
+
74
+ > A database is tied to the embedding model it was initialised with (`citewell doctor` shows it).
75
+ > To switch models, use a fresh database volume and re-ingest.
76
+
77
+ ## Quickstart (local, uv)
78
+
79
+ ```bash
80
+ uv sync --extra models
81
+ docker compose up -d db
82
+ uv run citewell init
83
+ uv run citewell ingest ./my-docs --collection handbook
84
+ uv run citewell search "parola en az kaç karakter olmalı?" -c handbook
85
+ ```
86
+
87
+ ## Connect your assistant
88
+
89
+ <details open>
90
+ <summary><b>Claude Code</b></summary>
91
+
92
+ ```bash
93
+ # HTTP (server running via docker compose or `citewell serve`)
94
+ claude mcp add --transport http citewell http://localhost:8000/mcp
95
+
96
+ # or stdio, from a local checkout
97
+ claude mcp add citewell -- uv --directory /path/to/citewell run citewell mcp
98
+ ```
99
+ </details>
100
+
101
+ <details>
102
+ <summary><b>Claude Desktop</b> — <code>claude_desktop_config.json</code></summary>
103
+
104
+ ```json
105
+ {
106
+ "mcpServers": {
107
+ "citewell": {
108
+ "command": "docker",
109
+ "args": ["exec", "-i", "citewell-citewell-1", "citewell", "mcp"]
110
+ }
111
+ }
112
+ }
113
+ ```
114
+
115
+ Local checkout instead of Docker:
116
+
117
+ ```json
118
+ {
119
+ "mcpServers": {
120
+ "citewell": {
121
+ "command": "uv",
122
+ "args": ["--directory", "/path/to/citewell", "run", "citewell", "mcp"],
123
+ "env": { "CITEWELL_DATABASE_URL": "postgresql://citewell:citewell@localhost:5432/citewell" }
124
+ }
125
+ }
126
+ }
127
+ ```
128
+ </details>
129
+
130
+ <details>
131
+ <summary><b>Cursor</b> — <code>.cursor/mcp.json</code></summary>
132
+
133
+ ```json
134
+ { "mcpServers": { "citewell": { "url": "http://localhost:8000/mcp" } } }
135
+ ```
136
+ </details>
137
+
138
+ ### MCP surface
139
+
140
+ | Kind | Name | Purpose |
141
+ |---|---|---|
142
+ | tool | `search_documents(query, top_k, collection, mode, rerank, source_prefix)` | Cited passages (`ref`, `citation`, `source`, `page`, `chunk_id`, `quote`, `text`, `score`) |
143
+ | tool | `get_chunk(chunk_id, context)` | A chunk plus neighbouring chunks, to read around a quote |
144
+ | tool | `list_collections()` / `list_documents(collection)` | What is indexed |
145
+ | resource | `citewell://collections`, `citewell://chunks/{id}` | Same data as resources |
146
+ | prompt | `answer_with_citations(question, collection)` | "Answer only from passages, cite with [n]" |
147
+
148
+ ## CLI
149
+
150
+ ```text
151
+ citewell init create extension + schema (idempotent)
152
+ citewell ingest PATH... [-c COL] [-l turkish|english|simple] [--force]
153
+ citewell search "query" [-k 5] [-m hybrid|vector|keyword] [-c COL|*] [--rerank] [--json]
154
+ citewell serve [--init] [--host] [--port] REST API + MCP at /mcp
155
+ citewell mcp [--transport stdio|http] MCP server only
156
+ citewell eval DATASET.jsonl [-k 5] [-r vector,keyword,hybrid,hybrid+rerank] [-o report.json]
157
+ citewell collections list|create|delete · documents list|delete · models download · doctor
158
+ ```
159
+
160
+ ## REST API
161
+
162
+ ```bash
163
+ curl -s localhost:8000/v1/search -H 'content-type: application/json' \
164
+ -d '{"query": "uzaktan çalışma kaç gün?", "top_k": 3, "mode": "hybrid"}'
165
+ curl -s -F file=@politika.pdf -F collection=handbook localhost:8000/v1/documents
166
+ ```
167
+
168
+ `GET /health` · `POST|GET /v1/search` · `GET /v1/chunks/{id}?context=1` · `GET|POST|DELETE /v1/collections` ·
169
+ `GET|POST|DELETE /v1/documents` — full schema at `/docs`. Set `CITEWELL_API_TOKEN` to require a bearer token.
170
+
171
+ ## How it works
172
+
173
+ ```mermaid
174
+ flowchart LR
175
+ subgraph Ingest
176
+ F[PDF / MD / TXT / HTML / DOCX] --> L[Loaders<br/>pages & sections]
177
+ L --> C[Page-aware chunker<br/>verbatim slices]
178
+ C --> E[bge-m3 embeddings<br/>+ title/section header]
179
+ end
180
+ E --> PG[(PostgreSQL<br/>pgvector HNSW<br/>tsvector GIN)]
181
+ subgraph Query
182
+ Q[query] --> V[vector top-40]
183
+ Q --> K[keyword top-40<br/>turkish stemmer]
184
+ V --> R[RRF k=60]
185
+ K --> R
186
+ R --> X[optional rerank<br/>bge-reranker-v2-m3]
187
+ X --> O[passages + citations<br/>+ best quote]
188
+ end
189
+ PG --- V
190
+ PG --- K
191
+ O --> M[MCP · REST · CLI]
192
+ ```
193
+
194
+ ## Evaluation
195
+
196
+ ```bash
197
+ citewell eval examples/eval/tr-sample.jsonl -k 5 -r vector,keyword,hybrid,hybrid+rerank --markdown
198
+ ```
199
+
200
+ Datasets are JSON Lines: `{"id", "query", "relevant": [{"source", "page"?}], "collection"?}`.
201
+ Metrics: **Recall@k**, **MRR@k**, **Hit@k**, **Page hit@k** (exact page), **p50/p95 latency**.
202
+ CI runs the sample set end-to-end on every push (with the lexical test backends) and fails below a
203
+ recall floor; see [docs/evaluation.md](docs/evaluation.md).
204
+
205
+ ### First results: Turkish Labour Law (İş Kanunu, 4857)
206
+
207
+ 20 natural-language questions from [`examples/eval/mevzuat-questions.jsonl`](examples/eval/mevzuat-questions.jsonl)
208
+ against the official PDF from mevzuat.gov.tr (121 articles). A hit counts only if it lands on a page
209
+ of the right article. `BAAI/bge-m3` + `bge-reranker-v2-m3`, default chunking, Apple Silicon (MPS),
210
+ models already loaded:
211
+
212
+ | Run | Recall@5 | MRR@5 | Hit@5 | Page hit@5 | p50 ms | p95 ms |
213
+ |---|---:|---:|---:|---:|---:|---:|
214
+ | vector | 1.000 | 0.885 | 1.000 | 1.000 | 45 | 95 |
215
+ | keyword | 0.850 | 0.614 | 0.850 | 0.850 | 8 | 11 |
216
+ | **hybrid** (default) | **1.000** | **0.893** | **1.000** | **1.000** | **48** | **54** |
217
+ | hybrid + rerank | 1.000 | 0.950 | 1.000 | 1.000 | 3259 | 3771 |
218
+
219
+ - Every question finds the right article **and page** in the top 5 with hybrid search; the right
220
+ passage is usually ranked first or second (MRR 0.89).
221
+ - Dense retrieval does the heavy lifting on paraphrased questions; keyword search alone misses 15 %.
222
+ - The cross-encoder lifts MRR to 0.95 but costs ~3 s per query on a laptop, so it is opt-in
223
+ (`--rerank` / `rerank: true`).
224
+
225
+ > Small and single-domain (20 questions, one law, written by the maintainers) — a first data point,
226
+ > not a leaderboard. Reproduce it with [docs/evaluation.md](docs/evaluation.md#turkish-legislation-set);
227
+ > larger independent Turkish eval sets are very welcome.
228
+
229
+ ## Configuration
230
+
231
+ All settings are environment variables prefixed with `CITEWELL_` (or a `.env` file) — see
232
+ [`.env.example`](.env.example) and [docs/configuration.md](docs/configuration.md).
233
+
234
+ | Variable | Default |
235
+ |---|---|
236
+ | `CITEWELL_DATABASE_URL` | `postgresql://citewell:citewell@localhost:5432/citewell` |
237
+ | `CITEWELL_EMBEDDING_MODEL` / `_DIM` | `BAAI/bge-m3` / `1024` |
238
+ | `CITEWELL_RERANKER_MODEL` | `BAAI/bge-reranker-v2-m3` |
239
+ | `CITEWELL_DEFAULT_MODE` / `_TOP_K` | `hybrid` / `5` |
240
+ | `CITEWELL_CHUNK_SIZE` / `_OVERLAP` | `1000` / `150` characters |
241
+ | `CITEWELL_API_TOKEN` | unset (no auth; bind to localhost) |
242
+
243
+ ## Roadmap
244
+
245
+ - [x] **v0.1** — ingest (PDF/MD/TXT/HTML/DOCX), vector search, cited passages, CLI, REST, MCP, Docker
246
+ - [x] **v0.2** — hybrid search (pgvector + tsvector, RRF), cross-encoder rerank, Turkish eval harness
247
+ - [ ] **v0.3** — MCP: completions, progress notifications, resource subscriptions; per-collection
248
+ access, collection-scoped tokens *(collections, resources and prompts already ship)*
249
+ - [ ] Later — OpenAI-compatible endpoint (Open WebUI), OCR for scanned PDFs, sparse/BM25 retriever,
250
+ larger public Turkish benchmark, SQLite single-file mode
251
+
252
+ ## Contributing
253
+
254
+ See [CONTRIBUTING.md](CONTRIBUTING.md). Security reports: [SECURITY.md](SECURITY.md).
255
+ Architecture decisions live in [docs/adr](docs/adr/index.md).
256
+
257
+ ## License
258
+
259
+ [MIT](LICENSE) © Ömer Özkılıç