mkd-retriever 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- mkd_retriever-0.1.0/CHANGELOG.md +34 -0
- mkd_retriever-0.1.0/LICENSE +21 -0
- mkd_retriever-0.1.0/MANIFEST.in +4 -0
- mkd_retriever-0.1.0/PKG-INFO +173 -0
- mkd_retriever-0.1.0/README.md +129 -0
- mkd_retriever-0.1.0/pyproject.toml +116 -0
- mkd_retriever-0.1.0/setup.cfg +4 -0
- mkd_retriever-0.1.0/src/mkd_retriever.egg-info/PKG-INFO +173 -0
- mkd_retriever-0.1.0/src/mkd_retriever.egg-info/SOURCES.txt +34 -0
- mkd_retriever-0.1.0/src/mkd_retriever.egg-info/dependency_links.txt +1 -0
- mkd_retriever-0.1.0/src/mkd_retriever.egg-info/requires.txt +29 -0
- mkd_retriever-0.1.0/src/mkd_retriever.egg-info/top_level.txt +1 -0
- mkd_retriever-0.1.0/src/retrieval/__init__.py +34 -0
- mkd_retriever-0.1.0/src/retrieval/core/__init__.py +51 -0
- mkd_retriever-0.1.0/src/retrieval/core/interfaces.py +112 -0
- mkd_retriever-0.1.0/src/retrieval/core/schemas.py +257 -0
- mkd_retriever-0.1.0/src/retrieval/eval/__init__.py +43 -0
- mkd_retriever-0.1.0/src/retrieval/eval/golden.py +97 -0
- mkd_retriever-0.1.0/src/retrieval/eval/metrics.py +178 -0
- mkd_retriever-0.1.0/src/retrieval/eval/runner.py +99 -0
- mkd_retriever-0.1.0/src/retrieval/modules/__init__.py +0 -0
- mkd_retriever-0.1.0/src/retrieval/modules/agentic/__init__.py +0 -0
- mkd_retriever-0.1.0/src/retrieval/modules/db_query/__init__.py +58 -0
- mkd_retriever-0.1.0/src/retrieval/modules/db_query/executor.py +91 -0
- mkd_retriever-0.1.0/src/retrieval/modules/db_query/factory.py +84 -0
- mkd_retriever-0.1.0/src/retrieval/modules/db_query/generator.py +91 -0
- mkd_retriever-0.1.0/src/retrieval/modules/db_query/retriever.py +99 -0
- mkd_retriever-0.1.0/src/retrieval/modules/intent_classification/__init__.py +40 -0
- mkd_retriever-0.1.0/src/retrieval/modules/intent_classification/classifier.py +73 -0
- mkd_retriever-0.1.0/src/retrieval/modules/intent_classification/factory.py +74 -0
- mkd_retriever-0.1.0/src/retrieval/modules/intent_classification/parsing.py +79 -0
- mkd_retriever-0.1.0/src/retrieval/modules/intent_classification/prompt.py +89 -0
- mkd_retriever-0.1.0/src/retrieval/modules/query_rewriting/__init__.py +0 -0
- mkd_retriever-0.1.0/src/retrieval/modules/raptor/__init__.py +0 -0
- mkd_retriever-0.1.0/src/retrieval/modules/top_n/__init__.py +0 -0
- mkd_retriever-0.1.0/src/retrieval/py.typed +0 -0
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project are documented here. The format loosely
|
|
4
|
+
follows [Keep a Changelog](https://keepachangelog.com/); versions follow
|
|
5
|
+
[SemVer](https://semver.org/).
|
|
6
|
+
|
|
7
|
+
## [0.1.0] — 2026-07-22
|
|
8
|
+
|
|
9
|
+
First public release.
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
- **`retrieval.core`** — the small, stable shared surface every module depends
|
|
13
|
+
on: `RetrievalResult`, the async `Retriever` protocol (+ `run_sync`),
|
|
14
|
+
`IntentLabel` (9 classes) / `IntentResult`, and the ingestion-contract
|
|
15
|
+
pydantic schemas (`ChunkFile`/`ChunkNode`, `ProductFile`/`ProductNode`,
|
|
16
|
+
`DocumentFile`/`DocumentNode`) with file loaders.
|
|
17
|
+
- **`retrieval.eval`** — golden-set format, hand-written metrics
|
|
18
|
+
(accuracy / macro-micro F1 / confusion, recall@k / MRR) and an async runner.
|
|
19
|
+
- **`intent_classification`** module (extra `intent_classification`) — an
|
|
20
|
+
LLM-prompt classifier over any `langchain-core` `BaseChatModel`, returning a
|
|
21
|
+
single `IntentResult`. Provider-agnostic; a factory wires a real
|
|
22
|
+
OpenAI-compatible endpoint. Defensive label parsing, graceful
|
|
23
|
+
logprobs→confidence.
|
|
24
|
+
- **`db_query`** module (extra `db_query`) — retrieval by generating SQL
|
|
25
|
+
(via the `text2sql-engine` package) and running it against a ready-made
|
|
26
|
+
database. Two injected seams: `SqlGenerator` (NL→SQL) and `SqlExecutor`
|
|
27
|
+
(`SQLiteExecutor`, read-only). Rows map to `RetrievalResult` with a constant
|
|
28
|
+
score of `1.0`.
|
|
29
|
+
|
|
30
|
+
### Notes
|
|
31
|
+
- Every module depends only on `retrieval.core`, never on another module.
|
|
32
|
+
- Model inference is expected behind a remote OpenAI-compatible endpoint; the
|
|
33
|
+
library only sends requests and never runs local inference. The default test
|
|
34
|
+
run is fully mocked and never touches a network endpoint.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Mehmet Kaan Durupunar
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,173 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: mkd-retriever
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Modular, project-agnostic RAG retrieval layer: independent retrieval strategies behind a small shared core.
|
|
5
|
+
Author: Mehmet Kaan Durupunar
|
|
6
|
+
License: MIT
|
|
7
|
+
Project-URL: Homepage, https://pypi.org/project/mkd-retriever/
|
|
8
|
+
Keywords: rag,retrieval,information-retrieval,intent-classification,text-to-sql,llm,nlp,modular
|
|
9
|
+
Classifier: Development Status :: 4 - Beta
|
|
10
|
+
Classifier: Intended Audience :: Developers
|
|
11
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
12
|
+
Classifier: Operating System :: OS Independent
|
|
13
|
+
Classifier: Programming Language :: Python :: 3
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
16
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
17
|
+
Classifier: Topic :: Software Development :: Libraries
|
|
18
|
+
Classifier: Topic :: Text Processing :: Linguistic
|
|
19
|
+
Classifier: Typing :: Typed
|
|
20
|
+
Requires-Python: >=3.11
|
|
21
|
+
Description-Content-Type: text/markdown
|
|
22
|
+
License-File: LICENSE
|
|
23
|
+
Requires-Dist: pydantic>=2
|
|
24
|
+
Requires-Dist: python-dotenv>=1
|
|
25
|
+
Requires-Dist: langchain-core>=0.3
|
|
26
|
+
Provides-Extra: top-n
|
|
27
|
+
Provides-Extra: query-rewriting
|
|
28
|
+
Provides-Extra: intent-classification
|
|
29
|
+
Requires-Dist: langchain-openai>=0.2; extra == "intent-classification"
|
|
30
|
+
Provides-Extra: raptor
|
|
31
|
+
Provides-Extra: agentic
|
|
32
|
+
Provides-Extra: db-query
|
|
33
|
+
Requires-Dist: text2sql-engine[openai]>=0.2; extra == "db-query"
|
|
34
|
+
Provides-Extra: eval
|
|
35
|
+
Provides-Extra: all
|
|
36
|
+
Requires-Dist: langchain-openai>=0.2; extra == "all"
|
|
37
|
+
Requires-Dist: text2sql-engine[openai]>=0.2; extra == "all"
|
|
38
|
+
Provides-Extra: dev
|
|
39
|
+
Requires-Dist: pytest>=8; extra == "dev"
|
|
40
|
+
Requires-Dist: ruff>=0.6; extra == "dev"
|
|
41
|
+
Requires-Dist: build>=1; extra == "dev"
|
|
42
|
+
Requires-Dist: twine>=5; extra == "dev"
|
|
43
|
+
Dynamic: license-file
|
|
44
|
+
|
|
45
|
+
# mkd-retriever
|
|
46
|
+
|
|
47
|
+
A **modular, project-agnostic RAG retrieval layer**. Several retrieval
|
|
48
|
+
strategies — intent classification, text-to-SQL database query, and more to
|
|
49
|
+
come (top-N vector search, query rewriting, RAPTOR, agentic flows) — live side
|
|
50
|
+
by side in one package **without depending on one another**.
|
|
51
|
+
|
|
52
|
+
The PyPI name is `mkd-retriever`; the import name is `retrieval`
|
|
53
|
+
(`import retrieval`).
|
|
54
|
+
|
|
55
|
+
```bash
|
|
56
|
+
pip install mkd-retriever # core only
|
|
57
|
+
pip install "mkd-retriever[intent_classification]"
|
|
58
|
+
pip install "mkd-retriever[db_query]"
|
|
59
|
+
pip install "mkd-retriever[all]" # everything implemented so far
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
## Why it is built this way
|
|
63
|
+
|
|
64
|
+
Building retrieval methods on top of one another is a Jenga tower: changing or
|
|
65
|
+
removing one piece topples the rest. Instead, every method is an **independent
|
|
66
|
+
module**. The hard rules (enforced, see [`ARCHITECTURE.md`](./ARCHITECTURE.md)):
|
|
67
|
+
|
|
68
|
+
1. **A module never imports another module.** Cross-module communication is the
|
|
69
|
+
consuming project's job, not this library's.
|
|
70
|
+
2. **Everything shared lives in `retrieval.core`, and core stays small.** Only
|
|
71
|
+
minimal interfaces and common data schemas — never a module's implementation
|
|
72
|
+
detail. The dependency arrow is always *module → core*.
|
|
73
|
+
3. **Dependencies are isolated per module.** Each module has its own
|
|
74
|
+
optional-dependency group; a library one module needs never leaks into
|
|
75
|
+
another module's environment.
|
|
76
|
+
4. **Each module is understandable and testable on its own.**
|
|
77
|
+
5. **Intent classification only returns a label — it does not route.** Mapping a
|
|
78
|
+
label to a retrieval method is the consuming project's decision.
|
|
79
|
+
|
|
80
|
+
## What's inside
|
|
81
|
+
|
|
82
|
+
```
|
|
83
|
+
retrieval/
|
|
84
|
+
core/ shared interfaces + ingestion-contract schemas (small, stable)
|
|
85
|
+
eval/ golden-set format + metrics + async runner (cross-cutting)
|
|
86
|
+
modules/
|
|
87
|
+
intent_classification/ query -> intent label (usable)
|
|
88
|
+
db_query/ query -> SQL -> rows (usable)
|
|
89
|
+
top_n/ vector search (planned)
|
|
90
|
+
query_rewriting/ rewriting / expansion (planned)
|
|
91
|
+
raptor/ hierarchical retrieval (planned)
|
|
92
|
+
agentic/ multi-step flows (planned)
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
### `retrieval.core`
|
|
96
|
+
|
|
97
|
+
The whole surface every module shares:
|
|
98
|
+
|
|
99
|
+
- `RetrievalResult` — a single scored item, returned identically by every
|
|
100
|
+
backend (`id`, `score`, `text`, `metadata`).
|
|
101
|
+
- `Retriever` — the one async method a backend implements:
|
|
102
|
+
`async def retrieve(query, k) -> list[RetrievalResult]`. `run_sync` is a thin
|
|
103
|
+
convenience for synchronous callers.
|
|
104
|
+
- `IntentLabel` (9 classes) / `IntentResult`.
|
|
105
|
+
- Ingestion-contract schemas — pydantic models for the JSON an upstream
|
|
106
|
+
ingestion/parsing project emits (chunks, product graph, document metadata).
|
|
107
|
+
The contract is defined by *shape*, not by importing the producer, so any
|
|
108
|
+
project emitting these shapes can feed this layer. Models set `extra="ignore"`
|
|
109
|
+
to stay resilient to upstream format drift.
|
|
110
|
+
|
|
111
|
+
### `intent_classification`
|
|
112
|
+
|
|
113
|
+
A prompt-based classifier over any `langchain-core` `BaseChatModel`. It returns
|
|
114
|
+
one `IntentResult` and nothing more — the label → strategy mapping is the
|
|
115
|
+
consuming project's job.
|
|
116
|
+
|
|
117
|
+
```python
|
|
118
|
+
from retrieval.modules.intent_classification import (
|
|
119
|
+
LLMIntentClassifier, build_default_chat_model,
|
|
120
|
+
)
|
|
121
|
+
|
|
122
|
+
clf = LLMIntentClassifier(build_default_chat_model())
|
|
123
|
+
result = await clf.classify("de1000 fiyati ne kadar") # -> IntentResult
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
In tests, inject a fake `BaseChatModel` instead of the factory — no network,
|
|
127
|
+
no model download.
|
|
128
|
+
|
|
129
|
+
### `db_query`
|
|
130
|
+
|
|
131
|
+
Retrieval by generating SQL and running it. A natural-language query is turned
|
|
132
|
+
into a SQL string by the [`text2sql-engine`](https://pypi.org/project/text2sql-engine/)
|
|
133
|
+
package (a 2-stage LLM pipeline), that SQL runs against a **ready-made**
|
|
134
|
+
database, and the rows come back as `RetrievalResult` items.
|
|
135
|
+
|
|
136
|
+
```python
|
|
137
|
+
from retrieval.modules.db_query import build_default_retriever
|
|
138
|
+
|
|
139
|
+
retriever = build_default_retriever(db_path="specs.db")
|
|
140
|
+
results = await retriever.retrieve("operating temperature of DE9001", k=5)
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
Two injected seams keep it testable and dependency-isolated:
|
|
144
|
+
|
|
145
|
+
- `SqlGenerator` (NL → SQL) — the production impl wraps a `text2sql` engine
|
|
146
|
+
hitting a remote endpoint; unit tests inject a fake returning canned SQL.
|
|
147
|
+
- `SqlExecutor` (SQL → rows) — the shipped `SQLiteExecutor` uses only stdlib
|
|
148
|
+
`sqlite3` and opens the database **read-only** by default, so a stray write in
|
|
149
|
+
generated SQL fails at the engine level.
|
|
150
|
+
|
|
151
|
+
The score is a constant `1.0` (a SQL result is an exact match set, not a
|
|
152
|
+
similarity ranking; ordering is the query's own `ORDER BY`). The full row is in
|
|
153
|
+
`RetrievalResult.metadata`. This module connects to an existing database — it
|
|
154
|
+
does not build one.
|
|
155
|
+
|
|
156
|
+
## Inference & the GPU rule
|
|
157
|
+
|
|
158
|
+
Model inference (LLM/embedding) is expected to run on a **separate machine
|
|
159
|
+
behind an OpenAI-compatible endpoint**. This library only sends requests to that
|
|
160
|
+
endpoint; it never downloads models or runs local inference. The default test
|
|
161
|
+
run (`pytest`) is fully mocked and never reaches a network endpoint — tests that
|
|
162
|
+
hit a live endpoint are marked `integration` and skipped by default.
|
|
163
|
+
|
|
164
|
+
## Development
|
|
165
|
+
|
|
166
|
+
```bash
|
|
167
|
+
pip install -e ".[all,dev]"
|
|
168
|
+
pytest
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
## License
|
|
172
|
+
|
|
173
|
+
MIT. See [LICENSE](LICENSE).
|
|
@@ -0,0 +1,129 @@
|
|
|
1
|
+
# mkd-retriever
|
|
2
|
+
|
|
3
|
+
A **modular, project-agnostic RAG retrieval layer**. Several retrieval
|
|
4
|
+
strategies — intent classification, text-to-SQL database query, and more to
|
|
5
|
+
come (top-N vector search, query rewriting, RAPTOR, agentic flows) — live side
|
|
6
|
+
by side in one package **without depending on one another**.
|
|
7
|
+
|
|
8
|
+
The PyPI name is `mkd-retriever`; the import name is `retrieval`
|
|
9
|
+
(`import retrieval`).
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
pip install mkd-retriever # core only
|
|
13
|
+
pip install "mkd-retriever[intent_classification]"
|
|
14
|
+
pip install "mkd-retriever[db_query]"
|
|
15
|
+
pip install "mkd-retriever[all]" # everything implemented so far
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
## Why it is built this way
|
|
19
|
+
|
|
20
|
+
Building retrieval methods on top of one another is a Jenga tower: changing or
|
|
21
|
+
removing one piece topples the rest. Instead, every method is an **independent
|
|
22
|
+
module**. The hard rules (enforced, see [`ARCHITECTURE.md`](./ARCHITECTURE.md)):
|
|
23
|
+
|
|
24
|
+
1. **A module never imports another module.** Cross-module communication is the
|
|
25
|
+
consuming project's job, not this library's.
|
|
26
|
+
2. **Everything shared lives in `retrieval.core`, and core stays small.** Only
|
|
27
|
+
minimal interfaces and common data schemas — never a module's implementation
|
|
28
|
+
detail. The dependency arrow is always *module → core*.
|
|
29
|
+
3. **Dependencies are isolated per module.** Each module has its own
|
|
30
|
+
optional-dependency group; a library one module needs never leaks into
|
|
31
|
+
another module's environment.
|
|
32
|
+
4. **Each module is understandable and testable on its own.**
|
|
33
|
+
5. **Intent classification only returns a label — it does not route.** Mapping a
|
|
34
|
+
label to a retrieval method is the consuming project's decision.
|
|
35
|
+
|
|
36
|
+
## What's inside
|
|
37
|
+
|
|
38
|
+
```
|
|
39
|
+
retrieval/
|
|
40
|
+
core/ shared interfaces + ingestion-contract schemas (small, stable)
|
|
41
|
+
eval/ golden-set format + metrics + async runner (cross-cutting)
|
|
42
|
+
modules/
|
|
43
|
+
intent_classification/ query -> intent label (usable)
|
|
44
|
+
db_query/ query -> SQL -> rows (usable)
|
|
45
|
+
top_n/ vector search (planned)
|
|
46
|
+
query_rewriting/ rewriting / expansion (planned)
|
|
47
|
+
raptor/ hierarchical retrieval (planned)
|
|
48
|
+
agentic/ multi-step flows (planned)
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
### `retrieval.core`
|
|
52
|
+
|
|
53
|
+
The whole surface every module shares:
|
|
54
|
+
|
|
55
|
+
- `RetrievalResult` — a single scored item, returned identically by every
|
|
56
|
+
backend (`id`, `score`, `text`, `metadata`).
|
|
57
|
+
- `Retriever` — the one async method a backend implements:
|
|
58
|
+
`async def retrieve(query, k) -> list[RetrievalResult]`. `run_sync` is a thin
|
|
59
|
+
convenience for synchronous callers.
|
|
60
|
+
- `IntentLabel` (9 classes) / `IntentResult`.
|
|
61
|
+
- Ingestion-contract schemas — pydantic models for the JSON an upstream
|
|
62
|
+
ingestion/parsing project emits (chunks, product graph, document metadata).
|
|
63
|
+
The contract is defined by *shape*, not by importing the producer, so any
|
|
64
|
+
project emitting these shapes can feed this layer. Models set `extra="ignore"`
|
|
65
|
+
to stay resilient to upstream format drift.
|
|
66
|
+
|
|
67
|
+
### `intent_classification`
|
|
68
|
+
|
|
69
|
+
A prompt-based classifier over any `langchain-core` `BaseChatModel`. It returns
|
|
70
|
+
one `IntentResult` and nothing more — the label → strategy mapping is the
|
|
71
|
+
consuming project's job.
|
|
72
|
+
|
|
73
|
+
```python
|
|
74
|
+
from retrieval.modules.intent_classification import (
|
|
75
|
+
LLMIntentClassifier, build_default_chat_model,
|
|
76
|
+
)
|
|
77
|
+
|
|
78
|
+
clf = LLMIntentClassifier(build_default_chat_model())
|
|
79
|
+
result = await clf.classify("de1000 fiyati ne kadar") # -> IntentResult
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
In tests, inject a fake `BaseChatModel` instead of the factory — no network,
|
|
83
|
+
no model download.
|
|
84
|
+
|
|
85
|
+
### `db_query`
|
|
86
|
+
|
|
87
|
+
Retrieval by generating SQL and running it. A natural-language query is turned
|
|
88
|
+
into a SQL string by the [`text2sql-engine`](https://pypi.org/project/text2sql-engine/)
|
|
89
|
+
package (a 2-stage LLM pipeline), that SQL runs against a **ready-made**
|
|
90
|
+
database, and the rows come back as `RetrievalResult` items.
|
|
91
|
+
|
|
92
|
+
```python
|
|
93
|
+
from retrieval.modules.db_query import build_default_retriever
|
|
94
|
+
|
|
95
|
+
retriever = build_default_retriever(db_path="specs.db")
|
|
96
|
+
results = await retriever.retrieve("operating temperature of DE9001", k=5)
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
Two injected seams keep it testable and dependency-isolated:
|
|
100
|
+
|
|
101
|
+
- `SqlGenerator` (NL → SQL) — the production impl wraps a `text2sql` engine
|
|
102
|
+
hitting a remote endpoint; unit tests inject a fake returning canned SQL.
|
|
103
|
+
- `SqlExecutor` (SQL → rows) — the shipped `SQLiteExecutor` uses only stdlib
|
|
104
|
+
`sqlite3` and opens the database **read-only** by default, so a stray write in
|
|
105
|
+
generated SQL fails at the engine level.
|
|
106
|
+
|
|
107
|
+
The score is a constant `1.0` (a SQL result is an exact match set, not a
|
|
108
|
+
similarity ranking; ordering is the query's own `ORDER BY`). The full row is in
|
|
109
|
+
`RetrievalResult.metadata`. This module connects to an existing database — it
|
|
110
|
+
does not build one.
|
|
111
|
+
|
|
112
|
+
## Inference & the GPU rule
|
|
113
|
+
|
|
114
|
+
Model inference (LLM/embedding) is expected to run on a **separate machine
|
|
115
|
+
behind an OpenAI-compatible endpoint**. This library only sends requests to that
|
|
116
|
+
endpoint; it never downloads models or runs local inference. The default test
|
|
117
|
+
run (`pytest`) is fully mocked and never reaches a network endpoint — tests that
|
|
118
|
+
hit a live endpoint are marked `integration` and skipped by default.
|
|
119
|
+
|
|
120
|
+
## Development
|
|
121
|
+
|
|
122
|
+
```bash
|
|
123
|
+
pip install -e ".[all,dev]"
|
|
124
|
+
pytest
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
## License
|
|
128
|
+
|
|
129
|
+
MIT. See [LICENSE](LICENSE).
|
|
@@ -0,0 +1,116 @@
|
|
|
1
|
+
[build-system]
|
|
2
|
+
requires = ["setuptools>=68", "wheel"]
|
|
3
|
+
build-backend = "setuptools.build_meta"
|
|
4
|
+
|
|
5
|
+
[project]
|
|
6
|
+
# Distribution name on PyPI. The IMPORT name stays `retrieval`:
|
|
7
|
+
# pip install mkd-retriever -> import retrieval
|
|
8
|
+
name = "mkd-retriever"
|
|
9
|
+
version = "0.1.0"
|
|
10
|
+
description = "Modular, project-agnostic RAG retrieval layer: independent retrieval strategies behind a small shared core."
|
|
11
|
+
readme = "README.md"
|
|
12
|
+
requires-python = ">=3.11"
|
|
13
|
+
license = { text = "MIT" }
|
|
14
|
+
authors = [{ name = "Mehmet Kaan Durupunar" }]
|
|
15
|
+
keywords = [
|
|
16
|
+
"rag",
|
|
17
|
+
"retrieval",
|
|
18
|
+
"information-retrieval",
|
|
19
|
+
"intent-classification",
|
|
20
|
+
"text-to-sql",
|
|
21
|
+
"llm",
|
|
22
|
+
"nlp",
|
|
23
|
+
"modular",
|
|
24
|
+
]
|
|
25
|
+
classifiers = [
|
|
26
|
+
"Development Status :: 4 - Beta",
|
|
27
|
+
"Intended Audience :: Developers",
|
|
28
|
+
"License :: OSI Approved :: MIT License",
|
|
29
|
+
"Operating System :: OS Independent",
|
|
30
|
+
"Programming Language :: Python :: 3",
|
|
31
|
+
"Programming Language :: Python :: 3.11",
|
|
32
|
+
"Programming Language :: Python :: 3.12",
|
|
33
|
+
"Topic :: Scientific/Engineering :: Artificial Intelligence",
|
|
34
|
+
"Topic :: Software Development :: Libraries",
|
|
35
|
+
"Topic :: Text Processing :: Linguistic",
|
|
36
|
+
"Typing :: Typed",
|
|
37
|
+
]
|
|
38
|
+
|
|
39
|
+
dependencies = [
|
|
40
|
+
# Kept minimal on purpose: only what core/ (shared interfaces + schemas)
|
|
41
|
+
# actually needs. Anything a specific retrieval method needs goes into
|
|
42
|
+
# that method's own extra below, never here.
|
|
43
|
+
"pydantic>=2",
|
|
44
|
+
"python-dotenv>=1",
|
|
45
|
+
# Thin abstraction layer only (Embeddings/VectorStore/chat interfaces).
|
|
46
|
+
# Heavy integrations (langchain-openai, langgraph, store clients, ...)
|
|
47
|
+
# belong in module extras below — never here. See ARCHITECTURE.md #8.
|
|
48
|
+
"langchain-core>=0.3",
|
|
49
|
+
]
|
|
50
|
+
|
|
51
|
+
# Each retrieval method is an installable extra. A consumer (or a dev working
|
|
52
|
+
# on one method) installs only what that method needs:
|
|
53
|
+
# pip install "mkd-retriever[intent_classification]"
|
|
54
|
+
# pip install "mkd-retriever[db_query]"
|
|
55
|
+
# No extra should ever depend on another extra — that dependency belongs in
|
|
56
|
+
# `dependencies` above (shared/core) instead, or the boundary is wrong.
|
|
57
|
+
[project.optional-dependencies]
|
|
58
|
+
top_n = []
|
|
59
|
+
query_rewriting = []
|
|
60
|
+
intent_classification = [
|
|
61
|
+
# v1 backend calls a remote OpenAI-compatible endpoint (e.g. Ollama /v1)
|
|
62
|
+
# via LangChain's ChatOpenAI. langchain-core is in base deps; only the
|
|
63
|
+
# heavy integration lives here (ARCHITECTURE.md #8).
|
|
64
|
+
"langchain-openai>=0.2",
|
|
65
|
+
]
|
|
66
|
+
raptor = []
|
|
67
|
+
agentic = []
|
|
68
|
+
db_query = [
|
|
69
|
+
# SQL is generated by the text2sql-engine package (import name: text2sql):
|
|
70
|
+
# a 2-stage LLM pipeline that produces a SQL string and never touches a DB.
|
|
71
|
+
# We run that SQL locally with the stdlib sqlite3 (no dep). The [openai]
|
|
72
|
+
# extra pulls the OpenAI SDK to talk to a remote OpenAI-compatible endpoint
|
|
73
|
+
# (e.g. Ollama /v1) — the only heavy piece, isolated here (ARCHITECTURE.md #3).
|
|
74
|
+
"text2sql-engine[openai]>=0.2",
|
|
75
|
+
]
|
|
76
|
+
eval = []
|
|
77
|
+
|
|
78
|
+
# Everything the implemented modules need, in one shot.
|
|
79
|
+
all = [
|
|
80
|
+
"langchain-openai>=0.2",
|
|
81
|
+
"text2sql-engine[openai]>=0.2",
|
|
82
|
+
]
|
|
83
|
+
|
|
84
|
+
dev = [
|
|
85
|
+
"pytest>=8",
|
|
86
|
+
"ruff>=0.6",
|
|
87
|
+
"build>=1",
|
|
88
|
+
"twine>=5",
|
|
89
|
+
]
|
|
90
|
+
|
|
91
|
+
[project.urls]
|
|
92
|
+
Homepage = "https://pypi.org/project/mkd-retriever/"
|
|
93
|
+
|
|
94
|
+
[tool.setuptools.packages.find]
|
|
95
|
+
where = ["src"]
|
|
96
|
+
|
|
97
|
+
[tool.setuptools.package-data]
|
|
98
|
+
# Ship the PEP 561 marker so downstream type-checkers see this as a typed package.
|
|
99
|
+
"retrieval" = ["py.typed"]
|
|
100
|
+
|
|
101
|
+
[tool.setuptools]
|
|
102
|
+
license-files = ["LICENSE"]
|
|
103
|
+
|
|
104
|
+
[tool.pytest.ini_options]
|
|
105
|
+
testpaths = ["tests"]
|
|
106
|
+
# GPU/inference rule (CLAUDE.md): the default run never touches a real endpoint.
|
|
107
|
+
# Integration tests that hit a live inference endpoint are marked `integration`
|
|
108
|
+
# and skipped by default — run them only on the GPU machine with `-m integration`.
|
|
109
|
+
addopts = "-m 'not integration'"
|
|
110
|
+
markers = [
|
|
111
|
+
"integration: hits a real inference endpoint; run only on the GPU machine",
|
|
112
|
+
]
|
|
113
|
+
|
|
114
|
+
[tool.ruff]
|
|
115
|
+
line-length = 100
|
|
116
|
+
src = ["src"]
|
|
@@ -0,0 +1,173 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: mkd-retriever
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Modular, project-agnostic RAG retrieval layer: independent retrieval strategies behind a small shared core.
|
|
5
|
+
Author: Mehmet Kaan Durupunar
|
|
6
|
+
License: MIT
|
|
7
|
+
Project-URL: Homepage, https://pypi.org/project/mkd-retriever/
|
|
8
|
+
Keywords: rag,retrieval,information-retrieval,intent-classification,text-to-sql,llm,nlp,modular
|
|
9
|
+
Classifier: Development Status :: 4 - Beta
|
|
10
|
+
Classifier: Intended Audience :: Developers
|
|
11
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
12
|
+
Classifier: Operating System :: OS Independent
|
|
13
|
+
Classifier: Programming Language :: Python :: 3
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
16
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
17
|
+
Classifier: Topic :: Software Development :: Libraries
|
|
18
|
+
Classifier: Topic :: Text Processing :: Linguistic
|
|
19
|
+
Classifier: Typing :: Typed
|
|
20
|
+
Requires-Python: >=3.11
|
|
21
|
+
Description-Content-Type: text/markdown
|
|
22
|
+
License-File: LICENSE
|
|
23
|
+
Requires-Dist: pydantic>=2
|
|
24
|
+
Requires-Dist: python-dotenv>=1
|
|
25
|
+
Requires-Dist: langchain-core>=0.3
|
|
26
|
+
Provides-Extra: top-n
|
|
27
|
+
Provides-Extra: query-rewriting
|
|
28
|
+
Provides-Extra: intent-classification
|
|
29
|
+
Requires-Dist: langchain-openai>=0.2; extra == "intent-classification"
|
|
30
|
+
Provides-Extra: raptor
|
|
31
|
+
Provides-Extra: agentic
|
|
32
|
+
Provides-Extra: db-query
|
|
33
|
+
Requires-Dist: text2sql-engine[openai]>=0.2; extra == "db-query"
|
|
34
|
+
Provides-Extra: eval
|
|
35
|
+
Provides-Extra: all
|
|
36
|
+
Requires-Dist: langchain-openai>=0.2; extra == "all"
|
|
37
|
+
Requires-Dist: text2sql-engine[openai]>=0.2; extra == "all"
|
|
38
|
+
Provides-Extra: dev
|
|
39
|
+
Requires-Dist: pytest>=8; extra == "dev"
|
|
40
|
+
Requires-Dist: ruff>=0.6; extra == "dev"
|
|
41
|
+
Requires-Dist: build>=1; extra == "dev"
|
|
42
|
+
Requires-Dist: twine>=5; extra == "dev"
|
|
43
|
+
Dynamic: license-file
|
|
44
|
+
|
|
45
|
+
# mkd-retriever
|
|
46
|
+
|
|
47
|
+
A **modular, project-agnostic RAG retrieval layer**. Several retrieval
|
|
48
|
+
strategies — intent classification, text-to-SQL database query, and more to
|
|
49
|
+
come (top-N vector search, query rewriting, RAPTOR, agentic flows) — live side
|
|
50
|
+
by side in one package **without depending on one another**.
|
|
51
|
+
|
|
52
|
+
The PyPI name is `mkd-retriever`; the import name is `retrieval`
|
|
53
|
+
(`import retrieval`).
|
|
54
|
+
|
|
55
|
+
```bash
|
|
56
|
+
pip install mkd-retriever # core only
|
|
57
|
+
pip install "mkd-retriever[intent_classification]"
|
|
58
|
+
pip install "mkd-retriever[db_query]"
|
|
59
|
+
pip install "mkd-retriever[all]" # everything implemented so far
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
## Why it is built this way
|
|
63
|
+
|
|
64
|
+
Building retrieval methods on top of one another is a Jenga tower: changing or
|
|
65
|
+
removing one piece topples the rest. Instead, every method is an **independent
|
|
66
|
+
module**. The hard rules (enforced, see [`ARCHITECTURE.md`](./ARCHITECTURE.md)):
|
|
67
|
+
|
|
68
|
+
1. **A module never imports another module.** Cross-module communication is the
|
|
69
|
+
consuming project's job, not this library's.
|
|
70
|
+
2. **Everything shared lives in `retrieval.core`, and core stays small.** Only
|
|
71
|
+
minimal interfaces and common data schemas — never a module's implementation
|
|
72
|
+
detail. The dependency arrow is always *module → core*.
|
|
73
|
+
3. **Dependencies are isolated per module.** Each module has its own
|
|
74
|
+
optional-dependency group; a library one module needs never leaks into
|
|
75
|
+
another module's environment.
|
|
76
|
+
4. **Each module is understandable and testable on its own.**
|
|
77
|
+
5. **Intent classification only returns a label — it does not route.** Mapping a
|
|
78
|
+
label to a retrieval method is the consuming project's decision.
|
|
79
|
+
|
|
80
|
+
## What's inside
|
|
81
|
+
|
|
82
|
+
```
|
|
83
|
+
retrieval/
|
|
84
|
+
core/ shared interfaces + ingestion-contract schemas (small, stable)
|
|
85
|
+
eval/ golden-set format + metrics + async runner (cross-cutting)
|
|
86
|
+
modules/
|
|
87
|
+
intent_classification/ query -> intent label (usable)
|
|
88
|
+
db_query/ query -> SQL -> rows (usable)
|
|
89
|
+
top_n/ vector search (planned)
|
|
90
|
+
query_rewriting/ rewriting / expansion (planned)
|
|
91
|
+
raptor/ hierarchical retrieval (planned)
|
|
92
|
+
agentic/ multi-step flows (planned)
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
### `retrieval.core`
|
|
96
|
+
|
|
97
|
+
The whole surface every module shares:
|
|
98
|
+
|
|
99
|
+
- `RetrievalResult` — a single scored item, returned identically by every
|
|
100
|
+
backend (`id`, `score`, `text`, `metadata`).
|
|
101
|
+
- `Retriever` — the one async method a backend implements:
|
|
102
|
+
`async def retrieve(query, k) -> list[RetrievalResult]`. `run_sync` is a thin
|
|
103
|
+
convenience for synchronous callers.
|
|
104
|
+
- `IntentLabel` (9 classes) / `IntentResult`.
|
|
105
|
+
- Ingestion-contract schemas — pydantic models for the JSON an upstream
|
|
106
|
+
ingestion/parsing project emits (chunks, product graph, document metadata).
|
|
107
|
+
The contract is defined by *shape*, not by importing the producer, so any
|
|
108
|
+
project emitting these shapes can feed this layer. Models set `extra="ignore"`
|
|
109
|
+
to stay resilient to upstream format drift.
|
|
110
|
+
|
|
111
|
+
### `intent_classification`
|
|
112
|
+
|
|
113
|
+
A prompt-based classifier over any `langchain-core` `BaseChatModel`. It returns
|
|
114
|
+
one `IntentResult` and nothing more — the label → strategy mapping is the
|
|
115
|
+
consuming project's job.
|
|
116
|
+
|
|
117
|
+
```python
|
|
118
|
+
from retrieval.modules.intent_classification import (
|
|
119
|
+
LLMIntentClassifier, build_default_chat_model,
|
|
120
|
+
)
|
|
121
|
+
|
|
122
|
+
clf = LLMIntentClassifier(build_default_chat_model())
|
|
123
|
+
result = await clf.classify("de1000 fiyati ne kadar") # -> IntentResult
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
In tests, inject a fake `BaseChatModel` instead of the factory — no network,
|
|
127
|
+
no model download.
|
|
128
|
+
|
|
129
|
+
### `db_query`
|
|
130
|
+
|
|
131
|
+
Retrieval by generating SQL and running it. A natural-language query is turned
|
|
132
|
+
into a SQL string by the [`text2sql-engine`](https://pypi.org/project/text2sql-engine/)
|
|
133
|
+
package (a 2-stage LLM pipeline), that SQL runs against a **ready-made**
|
|
134
|
+
database, and the rows come back as `RetrievalResult` items.
|
|
135
|
+
|
|
136
|
+
```python
|
|
137
|
+
from retrieval.modules.db_query import build_default_retriever
|
|
138
|
+
|
|
139
|
+
retriever = build_default_retriever(db_path="specs.db")
|
|
140
|
+
results = await retriever.retrieve("operating temperature of DE9001", k=5)
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
Two injected seams keep it testable and dependency-isolated:
|
|
144
|
+
|
|
145
|
+
- `SqlGenerator` (NL → SQL) — the production impl wraps a `text2sql` engine
|
|
146
|
+
hitting a remote endpoint; unit tests inject a fake returning canned SQL.
|
|
147
|
+
- `SqlExecutor` (SQL → rows) — the shipped `SQLiteExecutor` uses only stdlib
|
|
148
|
+
`sqlite3` and opens the database **read-only** by default, so a stray write in
|
|
149
|
+
generated SQL fails at the engine level.
|
|
150
|
+
|
|
151
|
+
The score is a constant `1.0` (a SQL result is an exact match set, not a
|
|
152
|
+
similarity ranking; ordering is the query's own `ORDER BY`). The full row is in
|
|
153
|
+
`RetrievalResult.metadata`. This module connects to an existing database — it
|
|
154
|
+
does not build one.
|
|
155
|
+
|
|
156
|
+
## Inference & the GPU rule
|
|
157
|
+
|
|
158
|
+
Model inference (LLM/embedding) is expected to run on a **separate machine
|
|
159
|
+
behind an OpenAI-compatible endpoint**. This library only sends requests to that
|
|
160
|
+
endpoint; it never downloads models or runs local inference. The default test
|
|
161
|
+
run (`pytest`) is fully mocked and never reaches a network endpoint — tests that
|
|
162
|
+
hit a live endpoint are marked `integration` and skipped by default.
|
|
163
|
+
|
|
164
|
+
## Development
|
|
165
|
+
|
|
166
|
+
```bash
|
|
167
|
+
pip install -e ".[all,dev]"
|
|
168
|
+
pytest
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
## License
|
|
172
|
+
|
|
173
|
+
MIT. See [LICENSE](LICENSE).
|