misayafff-modular-rag 0.1.0__tar.gz

misayafff_modular_rag-0.1.0/PKG-INFO

@@ -0,0 +1,83 @@
+Metadata-Version: 2.4
+Name: misayafff-modular-rag
+Version: 0.1.0
+Summary: A modular RAG retrieval class based on LangChain and Chroma
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: chromadb
+Requires-Dist: langchain-chroma
+Requires-Dist: langchain-community
+Requires-Dist: langchain-core
+Requires-Dist: langchain-openai
+Requires-Dist: langchain-classic
+Requires-Dist: jieba
+Requires-Dist: rank-bm25
+Requires-Dist: python-dotenv
+Requires-Dist: dashscope
+Provides-Extra: publish
+Requires-Dist: build; extra == "publish"
+Requires-Dist: twine; extra == "publish"
+
+# langchain-modular-rag
+
+提供一个 `ModularRAG` 类，用于连接已有的 Chroma 知识库并执行在线检索。
+
+Python 3.11 及以上。
+
+## 安装
+
+```bash
+pip install langchain-modular-rag
+```
+
+## 使用
+
+```python
+from langchain_modular_rag import ModularRAG
+
+rag = ModularRAG({
+    "persist_directory": "./chroma_db",
+    "collection_name": "study",
+    "embedding_model": "text-embedding-v4",
+    "top_k": 5,
+    "recall_top_k": 20,
+    "pre_retrieval": {
+        "enabled": False,
+        "rewrite_query": False,
+        "hyde": False,
+        "model": "qwen-plus",
+    },
+    "hybrid_retrieval": {
+        "enabled": True,
+        "weights": [0.5, 0.5],
+    },
+    "post_retrieval": {
+        "enabled": True,
+        "deduplicate": True,
+        "rerank": False,
+        "rerank_model": "qwen3-rerank",
+        "similarity_filter": False,
+        "similarity_threshold": 0.4,
+        "compress_context": False,
+        "llm_model": "qwen-plus",
+        "reorder": False,
+    },
+})
+
+results = rag.search("公司打卡时间？")
+for document in results:
+    print(document.metadata)
+    print(document.page_content)
+```
+
+## 环境变量
+
+需要配置通义 API Key：
+
+```bash
+export DASHSCOPE_API_KEY="你的 API Key"
+或
+setx DASHSCOPE_API_KEY="你的 API Key"
+```
+
+`persist_directory` 指向已经完成入库的 Chroma 数据库。这个包当前只提供在线检索，不包含你的个人知识库数据，也不会自动创建或导入知识库。

misayafff_modular_rag-0.1.0/README.md

@@ -0,0 +1,63 @@
+# langchain-modular-rag
+
+提供一个 `ModularRAG` 类，用于连接已有的 Chroma 知识库并执行在线检索。
+
+Python 3.11 及以上。
+
+## 安装
+
+```bash
+pip install langchain-modular-rag
+```
+
+## 使用
+
+```python
+from langchain_modular_rag import ModularRAG
+
+rag = ModularRAG({
+    "persist_directory": "./chroma_db",
+    "collection_name": "study",
+    "embedding_model": "text-embedding-v4",
+    "top_k": 5,
+    "recall_top_k": 20,
+    "pre_retrieval": {
+        "enabled": False,
+        "rewrite_query": False,
+        "hyde": False,
+        "model": "qwen-plus",
+    },
+    "hybrid_retrieval": {
+        "enabled": True,
+        "weights": [0.5, 0.5],
+    },
+    "post_retrieval": {
+        "enabled": True,
+        "deduplicate": True,
+        "rerank": False,
+        "rerank_model": "qwen3-rerank",
+        "similarity_filter": False,
+        "similarity_threshold": 0.4,
+        "compress_context": False,
+        "llm_model": "qwen-plus",
+        "reorder": False,
+    },
+})
+
+results = rag.search("公司打卡时间？")
+for document in results:
+    print(document.metadata)
+    print(document.page_content)
+```
+
+## 环境变量
+
+需要配置通义 API Key：
+
+```bash
+export DASHSCOPE_API_KEY="你的 API Key"
+或
+setx DASHSCOPE_API_KEY="你的 API Key"
+```
+
+`persist_directory` 指向已经完成入库的 Chroma 数据库。这个包当前只提供在线检索，不包含你的个人知识库数据，也不会自动创建或导入知识库。

misayafff_modular_rag-0.1.0/misayafff_modular_rag.egg-info/PKG-INFO

@@ -0,0 +1,83 @@
+Metadata-Version: 2.4
+Name: misayafff-modular-rag
+Version: 0.1.0
+Summary: A modular RAG retrieval class based on LangChain and Chroma
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: chromadb
+Requires-Dist: langchain-chroma
+Requires-Dist: langchain-community
+Requires-Dist: langchain-core
+Requires-Dist: langchain-openai
+Requires-Dist: langchain-classic
+Requires-Dist: jieba
+Requires-Dist: rank-bm25
+Requires-Dist: python-dotenv
+Requires-Dist: dashscope
+Provides-Extra: publish
+Requires-Dist: build; extra == "publish"
+Requires-Dist: twine; extra == "publish"
+
+# langchain-modular-rag
+
+提供一个 `ModularRAG` 类，用于连接已有的 Chroma 知识库并执行在线检索。
+
+Python 3.11 及以上。
+
+## 安装
+
+```bash
+pip install langchain-modular-rag
+```
+
+## 使用
+
+```python
+from langchain_modular_rag import ModularRAG
+
+rag = ModularRAG({
+    "persist_directory": "./chroma_db",
+    "collection_name": "study",
+    "embedding_model": "text-embedding-v4",
+    "top_k": 5,
+    "recall_top_k": 20,
+    "pre_retrieval": {
+        "enabled": False,
+        "rewrite_query": False,
+        "hyde": False,
+        "model": "qwen-plus",
+    },
+    "hybrid_retrieval": {
+        "enabled": True,
+        "weights": [0.5, 0.5],
+    },
+    "post_retrieval": {
+        "enabled": True,
+        "deduplicate": True,
+        "rerank": False,
+        "rerank_model": "qwen3-rerank",
+        "similarity_filter": False,
+        "similarity_threshold": 0.4,
+        "compress_context": False,
+        "llm_model": "qwen-plus",
+        "reorder": False,
+    },
+})
+
+results = rag.search("公司打卡时间？")
+for document in results:
+    print(document.metadata)
+    print(document.page_content)
+```
+
+## 环境变量
+
+需要配置通义 API Key：
+
+```bash
+export DASHSCOPE_API_KEY="你的 API Key"
+或
+setx DASHSCOPE_API_KEY="你的 API Key"
+```
+
+`persist_directory` 指向已经完成入库的 Chroma 数据库。这个包当前只提供在线检索，不包含你的个人知识库数据，也不会自动创建或导入知识库。

misayafff_modular_rag-0.1.0/misayafff_modular_rag.egg-info/SOURCES.txt

@@ -0,0 +1,8 @@
+README.md
+misayafff_modular_rag.py
+pyproject.toml
+misayafff_modular_rag.egg-info/PKG-INFO
+misayafff_modular_rag.egg-info/SOURCES.txt
+misayafff_modular_rag.egg-info/dependency_links.txt
+misayafff_modular_rag.egg-info/requires.txt
+misayafff_modular_rag.egg-info/top_level.txt

misayafff_modular_rag-0.1.0/misayafff_modular_rag.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

misayafff_modular_rag-0.1.0/misayafff_modular_rag.egg-info/requires.txt

@@ -0,0 +1,14 @@
+chromadb
+langchain-chroma
+langchain-community
+langchain-core
+langchain-openai
+langchain-classic
+jieba
+rank-bm25
+python-dotenv
+dashscope
+
+[publish]
+build
+twine

misayafff_modular_rag-0.1.0/misayafff_modular_rag.egg-info/top_level.txt

@@ -0,0 +1 @@
+misayafff_modular_rag

misayafff_modular_rag-0.1.0/misayafff_modular_rag.py

@@ -0,0 +1,299 @@
+class ModularRAG:
+    def __init__(self, config: dict):
+        import os
+        from pathlib import Path
+        from dotenv import load_dotenv
+        from langchain_chroma import Chroma
+        from langchain_community.embeddings import DashScopeEmbeddings
+
+        load_dotenv()
+        self.config = config
+        directory = config.get("persist_directory", "./chroma_db")
+        if not (Path(directory).expanduser() / "chroma.sqlite3").is_file():
+            raise FileNotFoundError(f"知识库不存在：{directory}")
+        self.embeddings = DashScopeEmbeddings(
+            model=config.get("embedding_model", "text-embedding-v4"),
+            dashscope_api_key=os.getenv("DASHSCOPE_API_KEY"),
+        )
+        self.db = Chroma(
+            collection_name=config.get("collection_name", "study"),
+            persist_directory=str(Path(directory).expanduser()),
+            embedding_function=self.embeddings,
+            create_collection_if_not_exists=False,
+        )
+
+    def search(self, question: str, top_k=None, filters=None):
+        from langchain_core.documents import Document
+
+        if not question.strip():
+            raise ValueError("问题不能为空")
+        limit = self.config.get("top_k", 5) if top_k is None else top_k
+        if not isinstance(limit, int) or limit <= 0:
+            raise ValueError("top_k 必须是正整数")
+        count = max(self.config.get("recall_top_k", 20), limit)
+        pre = self.config.get("pre_retrieval", {})
+        hybrid = self.config.get("hybrid_retrieval", {})
+        post = self.config.get("post_retrieval", {})
+
+        query = question
+        if pre.get("enabled", False):
+            query = self.normalize_query(query)
+            if pre.get("rewrite_query", False):
+                query = self.rewrite_query(query, model=pre.get("model", "qwen-plus"))
+        keyword_query = query
+        dense_query = query
+        if pre.get("enabled", False) and pre.get("hyde", False):
+            dense_query = self.generate_hyde(query, model=pre.get("model", "qwen-plus"))
+
+        conditions = []
+        for condition in (self.config.get("filters"), filters):
+            if condition:
+                conditions.extend({key: value} for key, value in condition.items())
+        where = None
+        if len(conditions) == 1:
+            where = conditions[0]
+        elif len(conditions) > 1:
+            where = {"$and": conditions}
+
+        if hybrid.get("enabled", False):
+            import jieba
+            from langchain_community.retrievers import BM25Retriever
+            from langchain_core.runnables import RunnableLambda
+
+            stored = self.db.get(where=where, include=["documents", "metadatas"])
+            corpus = []
+            for index, text in enumerate(stored["documents"] or []):
+                corpus.append(Document(
+                    id=stored["ids"][index],
+                    page_content=text,
+                    metadata=stored["metadatas"][index] or {},
+                ))
+            if not corpus:
+                return []
+            vector_retriever = self.db.as_retriever(
+                search_type="similarity",
+                search_kwargs={"k": count, "filter": where},
+            )
+            if dense_query != keyword_query:
+                vector_retriever = (
+                    RunnableLambda(lambda incoming_query: dense_query) | vector_retriever
+                )
+            bm25_retriever = BM25Retriever.from_documents(
+                corpus,
+                preprocess_func=lambda text: list(jieba.cut(text)),
+                k=count,
+            )
+            documents = self.rrf_fusion(
+                keyword_query, vector_retriever, bm25_retriever,
+                weights=hybrid.get("weights", [0.5, 0.5]),
+            )
+        else:
+            documents = self.vector_search(dense_query, self.db, count, where)
+
+        if post.get("enabled", False):
+            if post.get("deduplicate", True):
+                documents = self.deduplicate(documents)
+            if post.get("rerank", False):
+                documents = self.rerank_documents(
+                    question, documents, top_k=len(documents),
+                    model=post.get("rerank_model", "qwen3-rerank"),
+                )
+            if post.get("similarity_filter", False):
+                documents = self.similarity_filter(
+                    question, documents, self.embeddings,
+                    threshold=post.get("similarity_threshold", 0.4),
+                )
+
+        documents = documents[:limit]
+        for rank, document in enumerate(documents, start=1):
+            document.metadata["retrieval_rank"] = rank
+
+        if post.get("enabled", False):
+            if post.get("compress_context", False):
+                documents = self.compress_context(
+                    question, documents, model=post.get("llm_model", "qwen-plus")
+                )
+            if post.get("reorder", False):
+                documents = self.reorder_documents(documents)
+        for position, document in enumerate(documents, start=1):
+            document.metadata["context_position"] = position
+        return documents
+        
+    @staticmethod
+    # ==================== 1. 归一化查询词：移除所有空格并首尾去空格 ====================
+    def normalize_query(question: str):
+        import re
+        return re.sub(r"\s+", " ", question).strip() # 移除所有空格并首尾去空格
+
+    @staticmethod
+    # ==================== 2. 改写查询词：根据模型提示改写为适合知识库检索的独立查询 ====================
+    def rewrite_query(question: str, model="qwen-plus"):
+        import os
+        from dotenv import load_dotenv
+        from langchain_openai import ChatOpenAI
+
+        load_dotenv()
+        llm = ChatOpenAI(
+            model=model,
+            api_key=os.getenv("DASHSCOPE_API_KEY"),
+            base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
+            temperature=0,
+        )
+        response = llm.invoke([
+            ("system", "把问题改写为适合知识库检索的独立查询。保留原意，不编造缺失上下文，只输出查询。"),
+            ("human", question),
+        ])
+        return response.content.strip()
+
+    @staticmethod
+    # ==================== 3. 生成假设性答案：根据模型提示生成约100字的假设性答案 ====================
+    def generate_hyde(question: str, model="qwen-plus"):
+        import os
+        from dotenv import load_dotenv
+        from langchain_openai import ChatOpenAI
+
+        load_dotenv()
+        llm = ChatOpenAI(
+            model=model,
+            api_key=os.getenv("DASHSCOPE_API_KEY"),
+            base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
+            temperature=0,
+        )
+        response = llm.invoke([
+            ("system", "为问题生成约100字的假设性答案，用于语义检索。只输出假设答案，不将其视为事实证据。"),
+            ("human", question),
+        ])
+        return response.content.strip()
+
+    @staticmethod
+    # ==================== 4. 稠密：向量检索召回文档 ====================
+    def vector_search(question: str, db, top_k=5, filters=None):
+        retriever = db.as_retriever(
+            search_type="similarity",
+            search_kwargs={"k": top_k, "filter": filters},
+        )
+        documents = retriever.invoke(question)
+        for document in documents:
+            document.metadata["chunk_id"] = document.id
+        return documents
+
+    @staticmethod
+    # ==================== 5. 稀疏：BM25检索召回文档 ====================
+    def bm25_search(question: str, documents, top_k=5):
+        import jieba
+        from langchain_community.retrievers import BM25Retriever
+
+        if not documents:
+            return []
+        bm25_retriever = BM25Retriever.from_documents(
+            documents,
+            preprocess_func=lambda text: list(jieba.cut(text)),
+            k=top_k,
+        )
+        return bm25_retriever.invoke(question)
+
+    @staticmethod
+    # ==================== 6. RRF融合召回文档 ====================
+    def rrf_fusion(question: str, vector_retriever, bm25_retriever, weights=None, rrf_k=60):
+        from langchain_classic.retrievers import EnsembleRetriever
+
+        retriever = EnsembleRetriever(
+            retrievers=[vector_retriever, bm25_retriever],
+            weights=weights or [0.5, 0.5],
+            c=rrf_k,
+        )
+        return retriever.invoke(question)
+
+    @staticmethod
+    # ==================== 7. 去重：根据文档内容去重 ====================
+    def deduplicate(documents):
+        seen = set()
+        results = []
+        for document in documents:
+            if document.page_content not in seen:
+                seen.add(document.page_content)
+                results.append(document)
+        return results
+
+    from langchain_core.documents import Document
+
+    @staticmethod
+    # ==================== 8. 重排：根据模型提示重排召回文档 ====================
+    def rerank_documents(question: str, documents, top_k=5, model="qwen3-rerank"):
+        import os
+        from dotenv import load_dotenv
+        from langchain_community.document_compressors import DashScopeRerank
+
+        if not documents:
+            return []
+        load_dotenv()
+        reranker = DashScopeRerank(
+            model=model,
+            dashscope_api_key=os.getenv("DASHSCOPE_API_KEY"),
+            top_n=top_k,
+        )
+        return list(reranker.compress_documents(documents, question))
+
+    @staticmethod
+    # ==================== 9. 筛选：根据文档相似度筛选出相关文档 ====================
+    def similarity_filter(question: str, documents, embeddings, threshold=0.4):
+        import math
+
+        if not documents:
+            return []
+        query_vector = embeddings.embed_query(question)
+        vectors = embeddings.embed_documents([
+            document.page_content for document in documents
+        ])
+        query_norm = math.sqrt(sum(value * value for value in query_vector))
+
+        results = []
+        for document, vector in zip(documents, vectors):
+            vector_norm = math.sqrt(sum(value * value for value in vector))
+            if query_norm == 0 or vector_norm == 0:
+                continue
+            similarity = sum(
+                query_value * value for query_value, value in zip(query_vector, vector)
+            ) / (query_norm * vector_norm)
+            if similarity >= threshold:
+                result = document.model_copy(deep=True)
+                result.metadata["cosine_similarity"] = similarity
+                results.append(result)
+        return results
+
+    @staticmethod
+    # ==================== 10. 压缩：根据模型提示压缩文档上下文 ====================
+    def compress_context(question: str, documents, model="qwen-plus"):
+        import os
+        from dotenv import load_dotenv
+        from langchain_openai import ChatOpenAI
+        from langchain_classic.retrievers.document_compressors import LLMChainExtractor
+
+        if not documents:
+            return []
+        load_dotenv()
+        llm = ChatOpenAI(
+            model=model,
+            api_key=os.getenv("DASHSCOPE_API_KEY"),
+            base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
+            temperature=0,
+        )
+        compressor = LLMChainExtractor.from_llm(llm)
+        results = []
+        for document in documents:
+            is_table = (
+                document.metadata.get("content_type") == "table"
+                or "<table" in document.page_content.lower()
+            )
+            if is_table:
+                results.append(document)
+            else:
+                results.extend(compressor.compress_documents([document], question))
+        return results
+
+    @staticmethod
+    # ==================== 11. 压缩：根据模型提示压缩文档上下文 ====================
+    def reorder_documents(documents):
+        from langchain_community.document_transformers import LongContextReorder
+
+        return list(LongContextReorder().transform_documents(documents))

misayafff_modular_rag-0.1.0/pyproject.toml

@@ -0,0 +1,28 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "misayafff-modular-rag"
+version = "0.1.0"
+description = "A modular RAG retrieval class based on LangChain and Chroma"
+readme = "README.md"
+requires-python = ">=3.11"
+dependencies = [
+    "chromadb",
+    "langchain-chroma",
+    "langchain-community",
+    "langchain-core",
+    "langchain-openai",
+    "langchain-classic",
+    "jieba",
+    "rank-bm25",
+    "python-dotenv",
+    "dashscope",
+]
+
+[project.optional-dependencies]
+publish = ["build", "twine"]
+
+[tool.setuptools]
+py-modules = ["misayafff_modular_rag"]

misayafff_modular_rag-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+