mkd-retriever 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (36) hide show
  1. mkd_retriever-0.1.0/CHANGELOG.md +34 -0
  2. mkd_retriever-0.1.0/LICENSE +21 -0
  3. mkd_retriever-0.1.0/MANIFEST.in +4 -0
  4. mkd_retriever-0.1.0/PKG-INFO +173 -0
  5. mkd_retriever-0.1.0/README.md +129 -0
  6. mkd_retriever-0.1.0/pyproject.toml +116 -0
  7. mkd_retriever-0.1.0/setup.cfg +4 -0
  8. mkd_retriever-0.1.0/src/mkd_retriever.egg-info/PKG-INFO +173 -0
  9. mkd_retriever-0.1.0/src/mkd_retriever.egg-info/SOURCES.txt +34 -0
  10. mkd_retriever-0.1.0/src/mkd_retriever.egg-info/dependency_links.txt +1 -0
  11. mkd_retriever-0.1.0/src/mkd_retriever.egg-info/requires.txt +29 -0
  12. mkd_retriever-0.1.0/src/mkd_retriever.egg-info/top_level.txt +1 -0
  13. mkd_retriever-0.1.0/src/retrieval/__init__.py +34 -0
  14. mkd_retriever-0.1.0/src/retrieval/core/__init__.py +51 -0
  15. mkd_retriever-0.1.0/src/retrieval/core/interfaces.py +112 -0
  16. mkd_retriever-0.1.0/src/retrieval/core/schemas.py +257 -0
  17. mkd_retriever-0.1.0/src/retrieval/eval/__init__.py +43 -0
  18. mkd_retriever-0.1.0/src/retrieval/eval/golden.py +97 -0
  19. mkd_retriever-0.1.0/src/retrieval/eval/metrics.py +178 -0
  20. mkd_retriever-0.1.0/src/retrieval/eval/runner.py +99 -0
  21. mkd_retriever-0.1.0/src/retrieval/modules/__init__.py +0 -0
  22. mkd_retriever-0.1.0/src/retrieval/modules/agentic/__init__.py +0 -0
  23. mkd_retriever-0.1.0/src/retrieval/modules/db_query/__init__.py +58 -0
  24. mkd_retriever-0.1.0/src/retrieval/modules/db_query/executor.py +91 -0
  25. mkd_retriever-0.1.0/src/retrieval/modules/db_query/factory.py +84 -0
  26. mkd_retriever-0.1.0/src/retrieval/modules/db_query/generator.py +91 -0
  27. mkd_retriever-0.1.0/src/retrieval/modules/db_query/retriever.py +99 -0
  28. mkd_retriever-0.1.0/src/retrieval/modules/intent_classification/__init__.py +40 -0
  29. mkd_retriever-0.1.0/src/retrieval/modules/intent_classification/classifier.py +73 -0
  30. mkd_retriever-0.1.0/src/retrieval/modules/intent_classification/factory.py +74 -0
  31. mkd_retriever-0.1.0/src/retrieval/modules/intent_classification/parsing.py +79 -0
  32. mkd_retriever-0.1.0/src/retrieval/modules/intent_classification/prompt.py +89 -0
  33. mkd_retriever-0.1.0/src/retrieval/modules/query_rewriting/__init__.py +0 -0
  34. mkd_retriever-0.1.0/src/retrieval/modules/raptor/__init__.py +0 -0
  35. mkd_retriever-0.1.0/src/retrieval/modules/top_n/__init__.py +0 -0
  36. mkd_retriever-0.1.0/src/retrieval/py.typed +0 -0
@@ -0,0 +1,34 @@
1
+ # Changelog
2
+
3
+ All notable changes to this project are documented here. The format loosely
4
+ follows [Keep a Changelog](https://keepachangelog.com/); versions follow
5
+ [SemVer](https://semver.org/).
6
+
7
+ ## [0.1.0] — 2026-07-22
8
+
9
+ First public release.
10
+
11
+ ### Added
12
+ - **`retrieval.core`** — the small, stable shared surface every module depends
13
+ on: `RetrievalResult`, the async `Retriever` protocol (+ `run_sync`),
14
+ `IntentLabel` (9 classes) / `IntentResult`, and the ingestion-contract
15
+ pydantic schemas (`ChunkFile`/`ChunkNode`, `ProductFile`/`ProductNode`,
16
+ `DocumentFile`/`DocumentNode`) with file loaders.
17
+ - **`retrieval.eval`** — golden-set format, hand-written metrics
18
+ (accuracy / macro-micro F1 / confusion, recall@k / MRR) and an async runner.
19
+ - **`intent_classification`** module (extra `intent_classification`) — an
20
+ LLM-prompt classifier over any `langchain-core` `BaseChatModel`, returning a
21
+ single `IntentResult`. Provider-agnostic; a factory wires a real
22
+ OpenAI-compatible endpoint. Defensive label parsing, graceful
23
+ logprobs→confidence.
24
+ - **`db_query`** module (extra `db_query`) — retrieval by generating SQL
25
+ (via the `text2sql-engine` package) and running it against a ready-made
26
+ database. Two injected seams: `SqlGenerator` (NL→SQL) and `SqlExecutor`
27
+ (`SQLiteExecutor`, read-only). Rows map to `RetrievalResult` with a constant
28
+ score of `1.0`.
29
+
30
+ ### Notes
31
+ - Every module depends only on `retrieval.core`, never on another module.
32
+ - Model inference is expected behind a remote OpenAI-compatible endpoint; the
33
+ library only sends requests and never runs local inference. The default test
34
+ run is fully mocked and never touches a network endpoint.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Mehmet Kaan Durupunar
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,4 @@
1
+ include LICENSE
2
+ include README.md
3
+ include CHANGELOG.md
4
+ recursive-include src/retrieval py.typed
@@ -0,0 +1,173 @@
1
+ Metadata-Version: 2.4
2
+ Name: mkd-retriever
3
+ Version: 0.1.0
4
+ Summary: Modular, project-agnostic RAG retrieval layer: independent retrieval strategies behind a small shared core.
5
+ Author: Mehmet Kaan Durupunar
6
+ License: MIT
7
+ Project-URL: Homepage, https://pypi.org/project/mkd-retriever/
8
+ Keywords: rag,retrieval,information-retrieval,intent-classification,text-to-sql,llm,nlp,modular
9
+ Classifier: Development Status :: 4 - Beta
10
+ Classifier: Intended Audience :: Developers
11
+ Classifier: License :: OSI Approved :: MIT License
12
+ Classifier: Operating System :: OS Independent
13
+ Classifier: Programming Language :: Python :: 3
14
+ Classifier: Programming Language :: Python :: 3.11
15
+ Classifier: Programming Language :: Python :: 3.12
16
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
17
+ Classifier: Topic :: Software Development :: Libraries
18
+ Classifier: Topic :: Text Processing :: Linguistic
19
+ Classifier: Typing :: Typed
20
+ Requires-Python: >=3.11
21
+ Description-Content-Type: text/markdown
22
+ License-File: LICENSE
23
+ Requires-Dist: pydantic>=2
24
+ Requires-Dist: python-dotenv>=1
25
+ Requires-Dist: langchain-core>=0.3
26
+ Provides-Extra: top-n
27
+ Provides-Extra: query-rewriting
28
+ Provides-Extra: intent-classification
29
+ Requires-Dist: langchain-openai>=0.2; extra == "intent-classification"
30
+ Provides-Extra: raptor
31
+ Provides-Extra: agentic
32
+ Provides-Extra: db-query
33
+ Requires-Dist: text2sql-engine[openai]>=0.2; extra == "db-query"
34
+ Provides-Extra: eval
35
+ Provides-Extra: all
36
+ Requires-Dist: langchain-openai>=0.2; extra == "all"
37
+ Requires-Dist: text2sql-engine[openai]>=0.2; extra == "all"
38
+ Provides-Extra: dev
39
+ Requires-Dist: pytest>=8; extra == "dev"
40
+ Requires-Dist: ruff>=0.6; extra == "dev"
41
+ Requires-Dist: build>=1; extra == "dev"
42
+ Requires-Dist: twine>=5; extra == "dev"
43
+ Dynamic: license-file
44
+
45
+ # mkd-retriever
46
+
47
+ A **modular, project-agnostic RAG retrieval layer**. Several retrieval
48
+ strategies — intent classification, text-to-SQL database query, and more to
49
+ come (top-N vector search, query rewriting, RAPTOR, agentic flows) — live side
50
+ by side in one package **without depending on one another**.
51
+
52
+ The PyPI name is `mkd-retriever`; the import name is `retrieval`
53
+ (`import retrieval`).
54
+
55
+ ```bash
56
+ pip install mkd-retriever # core only
57
+ pip install "mkd-retriever[intent_classification]"
58
+ pip install "mkd-retriever[db_query]"
59
+ pip install "mkd-retriever[all]" # everything implemented so far
60
+ ```
61
+
62
+ ## Why it is built this way
63
+
64
+ Building retrieval methods on top of one another is a Jenga tower: changing or
65
+ removing one piece topples the rest. Instead, every method is an **independent
66
+ module**. The hard rules (enforced, see [`ARCHITECTURE.md`](./ARCHITECTURE.md)):
67
+
68
+ 1. **A module never imports another module.** Cross-module communication is the
69
+ consuming project's job, not this library's.
70
+ 2. **Everything shared lives in `retrieval.core`, and core stays small.** Only
71
+ minimal interfaces and common data schemas — never a module's implementation
72
+ detail. The dependency arrow is always *module → core*.
73
+ 3. **Dependencies are isolated per module.** Each module has its own
74
+ optional-dependency group; a library one module needs never leaks into
75
+ another module's environment.
76
+ 4. **Each module is understandable and testable on its own.**
77
+ 5. **Intent classification only returns a label — it does not route.** Mapping a
78
+ label to a retrieval method is the consuming project's decision.
79
+
80
+ ## What's inside
81
+
82
+ ```
83
+ retrieval/
84
+ core/ shared interfaces + ingestion-contract schemas (small, stable)
85
+ eval/ golden-set format + metrics + async runner (cross-cutting)
86
+ modules/
87
+ intent_classification/ query -> intent label (usable)
88
+ db_query/ query -> SQL -> rows (usable)
89
+ top_n/ vector search (planned)
90
+ query_rewriting/ rewriting / expansion (planned)
91
+ raptor/ hierarchical retrieval (planned)
92
+ agentic/ multi-step flows (planned)
93
+ ```
94
+
95
+ ### `retrieval.core`
96
+
97
+ The whole surface every module shares:
98
+
99
+ - `RetrievalResult` — a single scored item, returned identically by every
100
+ backend (`id`, `score`, `text`, `metadata`).
101
+ - `Retriever` — the one async method a backend implements:
102
+ `async def retrieve(query, k) -> list[RetrievalResult]`. `run_sync` is a thin
103
+ convenience for synchronous callers.
104
+ - `IntentLabel` (9 classes) / `IntentResult`.
105
+ - Ingestion-contract schemas — pydantic models for the JSON an upstream
106
+ ingestion/parsing project emits (chunks, product graph, document metadata).
107
+ The contract is defined by *shape*, not by importing the producer, so any
108
+ project emitting these shapes can feed this layer. Models set `extra="ignore"`
109
+ to stay resilient to upstream format drift.
110
+
111
+ ### `intent_classification`
112
+
113
+ A prompt-based classifier over any `langchain-core` `BaseChatModel`. It returns
114
+ one `IntentResult` and nothing more — the label → strategy mapping is the
115
+ consuming project's job.
116
+
117
+ ```python
118
+ from retrieval.modules.intent_classification import (
119
+ LLMIntentClassifier, build_default_chat_model,
120
+ )
121
+
122
+ clf = LLMIntentClassifier(build_default_chat_model())
123
+ result = await clf.classify("de1000 fiyati ne kadar") # -> IntentResult
124
+ ```
125
+
126
+ In tests, inject a fake `BaseChatModel` instead of the factory — no network,
127
+ no model download.
128
+
129
+ ### `db_query`
130
+
131
+ Retrieval by generating SQL and running it. A natural-language query is turned
132
+ into a SQL string by the [`text2sql-engine`](https://pypi.org/project/text2sql-engine/)
133
+ package (a 2-stage LLM pipeline), that SQL runs against a **ready-made**
134
+ database, and the rows come back as `RetrievalResult` items.
135
+
136
+ ```python
137
+ from retrieval.modules.db_query import build_default_retriever
138
+
139
+ retriever = build_default_retriever(db_path="specs.db")
140
+ results = await retriever.retrieve("operating temperature of DE9001", k=5)
141
+ ```
142
+
143
+ Two injected seams keep it testable and dependency-isolated:
144
+
145
+ - `SqlGenerator` (NL → SQL) — the production impl wraps a `text2sql` engine
146
+ hitting a remote endpoint; unit tests inject a fake returning canned SQL.
147
+ - `SqlExecutor` (SQL → rows) — the shipped `SQLiteExecutor` uses only stdlib
148
+ `sqlite3` and opens the database **read-only** by default, so a stray write in
149
+ generated SQL fails at the engine level.
150
+
151
+ The score is a constant `1.0` (a SQL result is an exact match set, not a
152
+ similarity ranking; ordering is the query's own `ORDER BY`). The full row is in
153
+ `RetrievalResult.metadata`. This module connects to an existing database — it
154
+ does not build one.
155
+
156
+ ## Inference & the GPU rule
157
+
158
+ Model inference (LLM/embedding) is expected to run on a **separate machine
159
+ behind an OpenAI-compatible endpoint**. This library only sends requests to that
160
+ endpoint; it never downloads models or runs local inference. The default test
161
+ run (`pytest`) is fully mocked and never reaches a network endpoint — tests that
162
+ hit a live endpoint are marked `integration` and skipped by default.
163
+
164
+ ## Development
165
+
166
+ ```bash
167
+ pip install -e ".[all,dev]"
168
+ pytest
169
+ ```
170
+
171
+ ## License
172
+
173
+ MIT. See [LICENSE](LICENSE).
@@ -0,0 +1,129 @@
1
+ # mkd-retriever
2
+
3
+ A **modular, project-agnostic RAG retrieval layer**. Several retrieval
4
+ strategies — intent classification, text-to-SQL database query, and more to
5
+ come (top-N vector search, query rewriting, RAPTOR, agentic flows) — live side
6
+ by side in one package **without depending on one another**.
7
+
8
+ The PyPI name is `mkd-retriever`; the import name is `retrieval`
9
+ (`import retrieval`).
10
+
11
+ ```bash
12
+ pip install mkd-retriever # core only
13
+ pip install "mkd-retriever[intent_classification]"
14
+ pip install "mkd-retriever[db_query]"
15
+ pip install "mkd-retriever[all]" # everything implemented so far
16
+ ```
17
+
18
+ ## Why it is built this way
19
+
20
+ Building retrieval methods on top of one another is a Jenga tower: changing or
21
+ removing one piece topples the rest. Instead, every method is an **independent
22
+ module**. The hard rules (enforced, see [`ARCHITECTURE.md`](./ARCHITECTURE.md)):
23
+
24
+ 1. **A module never imports another module.** Cross-module communication is the
25
+ consuming project's job, not this library's.
26
+ 2. **Everything shared lives in `retrieval.core`, and core stays small.** Only
27
+ minimal interfaces and common data schemas — never a module's implementation
28
+ detail. The dependency arrow is always *module → core*.
29
+ 3. **Dependencies are isolated per module.** Each module has its own
30
+ optional-dependency group; a library one module needs never leaks into
31
+ another module's environment.
32
+ 4. **Each module is understandable and testable on its own.**
33
+ 5. **Intent classification only returns a label — it does not route.** Mapping a
34
+ label to a retrieval method is the consuming project's decision.
35
+
36
+ ## What's inside
37
+
38
+ ```
39
+ retrieval/
40
+ core/ shared interfaces + ingestion-contract schemas (small, stable)
41
+ eval/ golden-set format + metrics + async runner (cross-cutting)
42
+ modules/
43
+ intent_classification/ query -> intent label (usable)
44
+ db_query/ query -> SQL -> rows (usable)
45
+ top_n/ vector search (planned)
46
+ query_rewriting/ rewriting / expansion (planned)
47
+ raptor/ hierarchical retrieval (planned)
48
+ agentic/ multi-step flows (planned)
49
+ ```
50
+
51
+ ### `retrieval.core`
52
+
53
+ The whole surface every module shares:
54
+
55
+ - `RetrievalResult` — a single scored item, returned identically by every
56
+ backend (`id`, `score`, `text`, `metadata`).
57
+ - `Retriever` — the one async method a backend implements:
58
+ `async def retrieve(query, k) -> list[RetrievalResult]`. `run_sync` is a thin
59
+ convenience for synchronous callers.
60
+ - `IntentLabel` (9 classes) / `IntentResult`.
61
+ - Ingestion-contract schemas — pydantic models for the JSON an upstream
62
+ ingestion/parsing project emits (chunks, product graph, document metadata).
63
+ The contract is defined by *shape*, not by importing the producer, so any
64
+ project emitting these shapes can feed this layer. Models set `extra="ignore"`
65
+ to stay resilient to upstream format drift.
66
+
67
+ ### `intent_classification`
68
+
69
+ A prompt-based classifier over any `langchain-core` `BaseChatModel`. It returns
70
+ one `IntentResult` and nothing more — the label → strategy mapping is the
71
+ consuming project's job.
72
+
73
+ ```python
74
+ from retrieval.modules.intent_classification import (
75
+ LLMIntentClassifier, build_default_chat_model,
76
+ )
77
+
78
+ clf = LLMIntentClassifier(build_default_chat_model())
79
+ result = await clf.classify("de1000 fiyati ne kadar") # -> IntentResult
80
+ ```
81
+
82
+ In tests, inject a fake `BaseChatModel` instead of the factory — no network,
83
+ no model download.
84
+
85
+ ### `db_query`
86
+
87
+ Retrieval by generating SQL and running it. A natural-language query is turned
88
+ into a SQL string by the [`text2sql-engine`](https://pypi.org/project/text2sql-engine/)
89
+ package (a 2-stage LLM pipeline), that SQL runs against a **ready-made**
90
+ database, and the rows come back as `RetrievalResult` items.
91
+
92
+ ```python
93
+ from retrieval.modules.db_query import build_default_retriever
94
+
95
+ retriever = build_default_retriever(db_path="specs.db")
96
+ results = await retriever.retrieve("operating temperature of DE9001", k=5)
97
+ ```
98
+
99
+ Two injected seams keep it testable and dependency-isolated:
100
+
101
+ - `SqlGenerator` (NL → SQL) — the production impl wraps a `text2sql` engine
102
+ hitting a remote endpoint; unit tests inject a fake returning canned SQL.
103
+ - `SqlExecutor` (SQL → rows) — the shipped `SQLiteExecutor` uses only stdlib
104
+ `sqlite3` and opens the database **read-only** by default, so a stray write in
105
+ generated SQL fails at the engine level.
106
+
107
+ The score is a constant `1.0` (a SQL result is an exact match set, not a
108
+ similarity ranking; ordering is the query's own `ORDER BY`). The full row is in
109
+ `RetrievalResult.metadata`. This module connects to an existing database — it
110
+ does not build one.
111
+
112
+ ## Inference & the GPU rule
113
+
114
+ Model inference (LLM/embedding) is expected to run on a **separate machine
115
+ behind an OpenAI-compatible endpoint**. This library only sends requests to that
116
+ endpoint; it never downloads models or runs local inference. The default test
117
+ run (`pytest`) is fully mocked and never reaches a network endpoint — tests that
118
+ hit a live endpoint are marked `integration` and skipped by default.
119
+
120
+ ## Development
121
+
122
+ ```bash
123
+ pip install -e ".[all,dev]"
124
+ pytest
125
+ ```
126
+
127
+ ## License
128
+
129
+ MIT. See [LICENSE](LICENSE).
@@ -0,0 +1,116 @@
1
+ [build-system]
2
+ requires = ["setuptools>=68", "wheel"]
3
+ build-backend = "setuptools.build_meta"
4
+
5
+ [project]
6
+ # Distribution name on PyPI. The IMPORT name stays `retrieval`:
7
+ # pip install mkd-retriever -> import retrieval
8
+ name = "mkd-retriever"
9
+ version = "0.1.0"
10
+ description = "Modular, project-agnostic RAG retrieval layer: independent retrieval strategies behind a small shared core."
11
+ readme = "README.md"
12
+ requires-python = ">=3.11"
13
+ license = { text = "MIT" }
14
+ authors = [{ name = "Mehmet Kaan Durupunar" }]
15
+ keywords = [
16
+ "rag",
17
+ "retrieval",
18
+ "information-retrieval",
19
+ "intent-classification",
20
+ "text-to-sql",
21
+ "llm",
22
+ "nlp",
23
+ "modular",
24
+ ]
25
+ classifiers = [
26
+ "Development Status :: 4 - Beta",
27
+ "Intended Audience :: Developers",
28
+ "License :: OSI Approved :: MIT License",
29
+ "Operating System :: OS Independent",
30
+ "Programming Language :: Python :: 3",
31
+ "Programming Language :: Python :: 3.11",
32
+ "Programming Language :: Python :: 3.12",
33
+ "Topic :: Scientific/Engineering :: Artificial Intelligence",
34
+ "Topic :: Software Development :: Libraries",
35
+ "Topic :: Text Processing :: Linguistic",
36
+ "Typing :: Typed",
37
+ ]
38
+
39
+ dependencies = [
40
+ # Kept minimal on purpose: only what core/ (shared interfaces + schemas)
41
+ # actually needs. Anything a specific retrieval method needs goes into
42
+ # that method's own extra below, never here.
43
+ "pydantic>=2",
44
+ "python-dotenv>=1",
45
+ # Thin abstraction layer only (Embeddings/VectorStore/chat interfaces).
46
+ # Heavy integrations (langchain-openai, langgraph, store clients, ...)
47
+ # belong in module extras below — never here. See ARCHITECTURE.md #8.
48
+ "langchain-core>=0.3",
49
+ ]
50
+
51
+ # Each retrieval method is an installable extra. A consumer (or a dev working
52
+ # on one method) installs only what that method needs:
53
+ # pip install "mkd-retriever[intent_classification]"
54
+ # pip install "mkd-retriever[db_query]"
55
+ # No extra should ever depend on another extra — that dependency belongs in
56
+ # `dependencies` above (shared/core) instead, or the boundary is wrong.
57
+ [project.optional-dependencies]
58
+ top_n = []
59
+ query_rewriting = []
60
+ intent_classification = [
61
+ # v1 backend calls a remote OpenAI-compatible endpoint (e.g. Ollama /v1)
62
+ # via LangChain's ChatOpenAI. langchain-core is in base deps; only the
63
+ # heavy integration lives here (ARCHITECTURE.md #8).
64
+ "langchain-openai>=0.2",
65
+ ]
66
+ raptor = []
67
+ agentic = []
68
+ db_query = [
69
+ # SQL is generated by the text2sql-engine package (import name: text2sql):
70
+ # a 2-stage LLM pipeline that produces a SQL string and never touches a DB.
71
+ # We run that SQL locally with the stdlib sqlite3 (no dep). The [openai]
72
+ # extra pulls the OpenAI SDK to talk to a remote OpenAI-compatible endpoint
73
+ # (e.g. Ollama /v1) — the only heavy piece, isolated here (ARCHITECTURE.md #3).
74
+ "text2sql-engine[openai]>=0.2",
75
+ ]
76
+ eval = []
77
+
78
+ # Everything the implemented modules need, in one shot.
79
+ all = [
80
+ "langchain-openai>=0.2",
81
+ "text2sql-engine[openai]>=0.2",
82
+ ]
83
+
84
+ dev = [
85
+ "pytest>=8",
86
+ "ruff>=0.6",
87
+ "build>=1",
88
+ "twine>=5",
89
+ ]
90
+
91
+ [project.urls]
92
+ Homepage = "https://pypi.org/project/mkd-retriever/"
93
+
94
+ [tool.setuptools.packages.find]
95
+ where = ["src"]
96
+
97
+ [tool.setuptools.package-data]
98
+ # Ship the PEP 561 marker so downstream type-checkers see this as a typed package.
99
+ "retrieval" = ["py.typed"]
100
+
101
+ [tool.setuptools]
102
+ license-files = ["LICENSE"]
103
+
104
+ [tool.pytest.ini_options]
105
+ testpaths = ["tests"]
106
+ # GPU/inference rule (CLAUDE.md): the default run never touches a real endpoint.
107
+ # Integration tests that hit a live inference endpoint are marked `integration`
108
+ # and skipped by default — run them only on the GPU machine with `-m integration`.
109
+ addopts = "-m 'not integration'"
110
+ markers = [
111
+ "integration: hits a real inference endpoint; run only on the GPU machine",
112
+ ]
113
+
114
+ [tool.ruff]
115
+ line-length = 100
116
+ src = ["src"]
@@ -0,0 +1,4 @@
1
+ [egg_info]
2
+ tag_build =
3
+ tag_date = 0
4
+
@@ -0,0 +1,173 @@
1
+ Metadata-Version: 2.4
2
+ Name: mkd-retriever
3
+ Version: 0.1.0
4
+ Summary: Modular, project-agnostic RAG retrieval layer: independent retrieval strategies behind a small shared core.
5
+ Author: Mehmet Kaan Durupunar
6
+ License: MIT
7
+ Project-URL: Homepage, https://pypi.org/project/mkd-retriever/
8
+ Keywords: rag,retrieval,information-retrieval,intent-classification,text-to-sql,llm,nlp,modular
9
+ Classifier: Development Status :: 4 - Beta
10
+ Classifier: Intended Audience :: Developers
11
+ Classifier: License :: OSI Approved :: MIT License
12
+ Classifier: Operating System :: OS Independent
13
+ Classifier: Programming Language :: Python :: 3
14
+ Classifier: Programming Language :: Python :: 3.11
15
+ Classifier: Programming Language :: Python :: 3.12
16
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
17
+ Classifier: Topic :: Software Development :: Libraries
18
+ Classifier: Topic :: Text Processing :: Linguistic
19
+ Classifier: Typing :: Typed
20
+ Requires-Python: >=3.11
21
+ Description-Content-Type: text/markdown
22
+ License-File: LICENSE
23
+ Requires-Dist: pydantic>=2
24
+ Requires-Dist: python-dotenv>=1
25
+ Requires-Dist: langchain-core>=0.3
26
+ Provides-Extra: top-n
27
+ Provides-Extra: query-rewriting
28
+ Provides-Extra: intent-classification
29
+ Requires-Dist: langchain-openai>=0.2; extra == "intent-classification"
30
+ Provides-Extra: raptor
31
+ Provides-Extra: agentic
32
+ Provides-Extra: db-query
33
+ Requires-Dist: text2sql-engine[openai]>=0.2; extra == "db-query"
34
+ Provides-Extra: eval
35
+ Provides-Extra: all
36
+ Requires-Dist: langchain-openai>=0.2; extra == "all"
37
+ Requires-Dist: text2sql-engine[openai]>=0.2; extra == "all"
38
+ Provides-Extra: dev
39
+ Requires-Dist: pytest>=8; extra == "dev"
40
+ Requires-Dist: ruff>=0.6; extra == "dev"
41
+ Requires-Dist: build>=1; extra == "dev"
42
+ Requires-Dist: twine>=5; extra == "dev"
43
+ Dynamic: license-file
44
+
45
+ # mkd-retriever
46
+
47
+ A **modular, project-agnostic RAG retrieval layer**. Several retrieval
48
+ strategies — intent classification, text-to-SQL database query, and more to
49
+ come (top-N vector search, query rewriting, RAPTOR, agentic flows) — live side
50
+ by side in one package **without depending on one another**.
51
+
52
+ The PyPI name is `mkd-retriever`; the import name is `retrieval`
53
+ (`import retrieval`).
54
+
55
+ ```bash
56
+ pip install mkd-retriever # core only
57
+ pip install "mkd-retriever[intent_classification]"
58
+ pip install "mkd-retriever[db_query]"
59
+ pip install "mkd-retriever[all]" # everything implemented so far
60
+ ```
61
+
62
+ ## Why it is built this way
63
+
64
+ Building retrieval methods on top of one another is a Jenga tower: changing or
65
+ removing one piece topples the rest. Instead, every method is an **independent
66
+ module**. The hard rules (enforced, see [`ARCHITECTURE.md`](./ARCHITECTURE.md)):
67
+
68
+ 1. **A module never imports another module.** Cross-module communication is the
69
+ consuming project's job, not this library's.
70
+ 2. **Everything shared lives in `retrieval.core`, and core stays small.** Only
71
+ minimal interfaces and common data schemas — never a module's implementation
72
+ detail. The dependency arrow is always *module → core*.
73
+ 3. **Dependencies are isolated per module.** Each module has its own
74
+ optional-dependency group; a library one module needs never leaks into
75
+ another module's environment.
76
+ 4. **Each module is understandable and testable on its own.**
77
+ 5. **Intent classification only returns a label — it does not route.** Mapping a
78
+ label to a retrieval method is the consuming project's decision.
79
+
80
+ ## What's inside
81
+
82
+ ```
83
+ retrieval/
84
+ core/ shared interfaces + ingestion-contract schemas (small, stable)
85
+ eval/ golden-set format + metrics + async runner (cross-cutting)
86
+ modules/
87
+ intent_classification/ query -> intent label (usable)
88
+ db_query/ query -> SQL -> rows (usable)
89
+ top_n/ vector search (planned)
90
+ query_rewriting/ rewriting / expansion (planned)
91
+ raptor/ hierarchical retrieval (planned)
92
+ agentic/ multi-step flows (planned)
93
+ ```
94
+
95
+ ### `retrieval.core`
96
+
97
+ The whole surface every module shares:
98
+
99
+ - `RetrievalResult` — a single scored item, returned identically by every
100
+ backend (`id`, `score`, `text`, `metadata`).
101
+ - `Retriever` — the one async method a backend implements:
102
+ `async def retrieve(query, k) -> list[RetrievalResult]`. `run_sync` is a thin
103
+ convenience for synchronous callers.
104
+ - `IntentLabel` (9 classes) / `IntentResult`.
105
+ - Ingestion-contract schemas — pydantic models for the JSON an upstream
106
+ ingestion/parsing project emits (chunks, product graph, document metadata).
107
+ The contract is defined by *shape*, not by importing the producer, so any
108
+ project emitting these shapes can feed this layer. Models set `extra="ignore"`
109
+ to stay resilient to upstream format drift.
110
+
111
+ ### `intent_classification`
112
+
113
+ A prompt-based classifier over any `langchain-core` `BaseChatModel`. It returns
114
+ one `IntentResult` and nothing more — the label → strategy mapping is the
115
+ consuming project's job.
116
+
117
+ ```python
118
+ from retrieval.modules.intent_classification import (
119
+ LLMIntentClassifier, build_default_chat_model,
120
+ )
121
+
122
+ clf = LLMIntentClassifier(build_default_chat_model())
123
+ result = await clf.classify("de1000 fiyati ne kadar") # -> IntentResult
124
+ ```
125
+
126
+ In tests, inject a fake `BaseChatModel` instead of the factory — no network,
127
+ no model download.
128
+
129
+ ### `db_query`
130
+
131
+ Retrieval by generating SQL and running it. A natural-language query is turned
132
+ into a SQL string by the [`text2sql-engine`](https://pypi.org/project/text2sql-engine/)
133
+ package (a 2-stage LLM pipeline), that SQL runs against a **ready-made**
134
+ database, and the rows come back as `RetrievalResult` items.
135
+
136
+ ```python
137
+ from retrieval.modules.db_query import build_default_retriever
138
+
139
+ retriever = build_default_retriever(db_path="specs.db")
140
+ results = await retriever.retrieve("operating temperature of DE9001", k=5)
141
+ ```
142
+
143
+ Two injected seams keep it testable and dependency-isolated:
144
+
145
+ - `SqlGenerator` (NL → SQL) — the production impl wraps a `text2sql` engine
146
+ hitting a remote endpoint; unit tests inject a fake returning canned SQL.
147
+ - `SqlExecutor` (SQL → rows) — the shipped `SQLiteExecutor` uses only stdlib
148
+ `sqlite3` and opens the database **read-only** by default, so a stray write in
149
+ generated SQL fails at the engine level.
150
+
151
+ The score is a constant `1.0` (a SQL result is an exact match set, not a
152
+ similarity ranking; ordering is the query's own `ORDER BY`). The full row is in
153
+ `RetrievalResult.metadata`. This module connects to an existing database — it
154
+ does not build one.
155
+
156
+ ## Inference & the GPU rule
157
+
158
+ Model inference (LLM/embedding) is expected to run on a **separate machine
159
+ behind an OpenAI-compatible endpoint**. This library only sends requests to that
160
+ endpoint; it never downloads models or runs local inference. The default test
161
+ run (`pytest`) is fully mocked and never reaches a network endpoint — tests that
162
+ hit a live endpoint are marked `integration` and skipped by default.
163
+
164
+ ## Development
165
+
166
+ ```bash
167
+ pip install -e ".[all,dev]"
168
+ pytest
169
+ ```
170
+
171
+ ## License
172
+
173
+ MIT. See [LICENSE](LICENSE).