flashrag-dev 0.3.0.dev20250924__tar.gz → 0.3.0.dev20251106__tar.gz

{flashrag_dev-0.3.0.dev20250924 → flashrag_dev-0.3.0.dev20251106}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
 Name: flashrag_dev
-Version: 0.3.0.dev20250924
+Version: 0.3.0.dev20251106
 Summary: A library for efficient Retrieval-Augmented Generation research
 Home-page: https://github.com/RUC-NLPIR/FlashRAG
 Author: Jiajie Jin, Yutao Zhu, Chenghao Zhang, Xinyu Yang, Zhicheng Dou
@@ -8,12 +8,103 @@ Author-email: jinjiajie@ruc.edu.cn
 License: MIT License
 Requires-Python: >=3.9
 Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: datasets
+Requires-Dist: base58
+Requires-Dist: nltk
+Requires-Dist: numpy
+Requires-Dist: langid
+Requires-Dist: openai
+Requires-Dist: peft
+Requires-Dist: PyYAML
+Requires-Dist: rank_bm25
+Requires-Dist: rouge
+Requires-Dist: spacy
+Requires-Dist: tiktoken
+Requires-Dist: torch
+Requires-Dist: tqdm
+Requires-Dist: transformers>=4.40.0
+Requires-Dist: bm25s[core]==0.2.1
+Requires-Dist: fschat
+Requires-Dist: streamlit
+Requires-Dist: chonkie<1.1.0,>=1.0.2
+Requires-Dist: gradio>=5.0.0
+Requires-Dist: rouge-chinese
+Requires-Dist: jieba
 Provides-Extra: core
+Requires-Dist: datasets; extra == "core"
+Requires-Dist: base58; extra == "core"
+Requires-Dist: nltk; extra == "core"
+Requires-Dist: numpy; extra == "core"
+Requires-Dist: langid; extra == "core"
+Requires-Dist: openai; extra == "core"
+Requires-Dist: peft; extra == "core"
+Requires-Dist: PyYAML; extra == "core"
+Requires-Dist: rank_bm25; extra == "core"
+Requires-Dist: rouge; extra == "core"
+Requires-Dist: spacy; extra == "core"
+Requires-Dist: tiktoken; extra == "core"
+Requires-Dist: torch; extra == "core"
+Requires-Dist: tqdm; extra == "core"
+Requires-Dist: transformers>=4.40.0; extra == "core"
+Requires-Dist: bm25s[core]==0.2.1; extra == "core"
+Requires-Dist: fschat; extra == "core"
+Requires-Dist: streamlit; extra == "core"
+Requires-Dist: chonkie<1.1.0,>=1.0.2; extra == "core"
+Requires-Dist: gradio>=5.0.0; extra == "core"
+Requires-Dist: rouge-chinese; extra == "core"
+Requires-Dist: jieba; extra == "core"
 Provides-Extra: retriever
+Requires-Dist: pyserini; extra == "retriever"
+Requires-Dist: sentence-transformers>=3.0.1; extra == "retriever"
 Provides-Extra: generator
+Requires-Dist: vllm; extra == "generator"
 Provides-Extra: multimodal
+Requires-Dist: timm; extra == "multimodal"
+Requires-Dist: torchvision; extra == "multimodal"
+Requires-Dist: pillow; extra == "multimodal"
+Requires-Dist: qwen_vl_utils; extra == "multimodal"
 Provides-Extra: full
-License-File: LICENSE
+Requires-Dist: datasets; extra == "full"
+Requires-Dist: base58; extra == "full"
+Requires-Dist: nltk; extra == "full"
+Requires-Dist: numpy; extra == "full"
+Requires-Dist: langid; extra == "full"
+Requires-Dist: openai; extra == "full"
+Requires-Dist: peft; extra == "full"
+Requires-Dist: PyYAML; extra == "full"
+Requires-Dist: rank_bm25; extra == "full"
+Requires-Dist: rouge; extra == "full"
+Requires-Dist: spacy; extra == "full"
+Requires-Dist: tiktoken; extra == "full"
+Requires-Dist: torch; extra == "full"
+Requires-Dist: tqdm; extra == "full"
+Requires-Dist: transformers>=4.40.0; extra == "full"
+Requires-Dist: bm25s[core]==0.2.1; extra == "full"
+Requires-Dist: fschat; extra == "full"
+Requires-Dist: streamlit; extra == "full"
+Requires-Dist: chonkie<1.1.0,>=1.0.2; extra == "full"
+Requires-Dist: gradio>=5.0.0; extra == "full"
+Requires-Dist: rouge-chinese; extra == "full"
+Requires-Dist: jieba; extra == "full"
+Requires-Dist: pyserini; extra == "full"
+Requires-Dist: sentence-transformers>=3.0.1; extra == "full"
+Requires-Dist: vllm; extra == "full"
+Requires-Dist: timm; extra == "full"
+Requires-Dist: torchvision; extra == "full"
+Requires-Dist: pillow; extra == "full"
+Requires-Dist: qwen_vl_utils; extra == "full"
+Dynamic: author
+Dynamic: author-email
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license
+Dynamic: license-file
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
 
 # <div align="center">⚡FlashRAG: A Python Toolkit for Efficient RAG Research<div>
 \[ English | [中文](README_zh.md) \]
@@ -100,6 +191,8 @@ FlashRAG is still under development and there are many issues and room for impro
 - [ ] Add support for api-based retriever (vllm server)
 
 ## :page_with_curl: Changelog
+[25/11/06] 🎯 NEW Retriever! We have integrated a Web Search Engine-based Retriever, which seamlessly integrates with existing methods and can be enabled quickly with just a Serper API key! This enhancement significantly expands retrieval coverage and real-time capability, supporting dynamic information access and external knowledge augmentation. Experience a more flexible and powerful retrieval workflow now! 
+
 [25/08/06] 🎯 **NEW!** We have added support for **Reasoning Pipeline**, which is a new paradigm that combines reasoning ability and retrieval, representing work that includes [R1-Searcher](https://github.com/SsmallSong/R1-Searcher), [Search-R1](https://github.com/PeterGriffinJin/Search-R1),.... We evaluate the performance of the pipeline on various RAG benchmarks, it can achieve F1 scores close to 60 on multi hop inference datasets such as HotpotQA. See it in [**result table**](#robot-supporting-methods).
 
 [25/03/21] 🚀 **Major Update!** We have expanded our toolkit to support **23 state-of-the-art RAG algorithms**, including **7 reasoning-based methods** that significantly improve performance on complex reasoning tasks. This represents a major milestone in our toolkit's evolution!
@@ -767,6 +860,7 @@ Please kindly cite our paper if helps your research:
 }
 ```
 
+
 ## Star History
 
 [![Star History Chart](https://api.star-history.com/svg?repos=RUC-NLPIR/FlashRAG&type=Date)](https://star-history.com/#RUC-NLPIR/FlashRAG&Date)

{flashrag_dev-0.3.0.dev20250924 → flashrag_dev-0.3.0.dev20251106}/README.md

@@ -83,6 +83,8 @@ FlashRAG is still under development and there are many issues and room for impro
 - [ ] Add support for api-based retriever (vllm server)
 
 ## :page_with_curl: Changelog
+[25/11/06] 🎯 NEW Retriever! We have integrated a Web Search Engine-based Retriever, which seamlessly integrates with existing methods and can be enabled quickly with just a Serper API key! This enhancement significantly expands retrieval coverage and real-time capability, supporting dynamic information access and external knowledge augmentation. Experience a more flexible and powerful retrieval workflow now! 
+
 [25/08/06] 🎯 **NEW!** We have added support for **Reasoning Pipeline**, which is a new paradigm that combines reasoning ability and retrieval, representing work that includes [R1-Searcher](https://github.com/SsmallSong/R1-Searcher), [Search-R1](https://github.com/PeterGriffinJin/Search-R1),.... We evaluate the performance of the pipeline on various RAG benchmarks, it can achieve F1 scores close to 60 on multi hop inference datasets such as HotpotQA. See it in [**result table**](#robot-supporting-methods).
 
 [25/03/21] 🚀 **Major Update!** We have expanded our toolkit to support **23 state-of-the-art RAG algorithms**, including **7 reasoning-based methods** that significantly improve performance on complex reasoning tasks. This represents a major milestone in our toolkit's evolution!
@@ -750,6 +752,7 @@ Please kindly cite our paper if helps your research:
 }
 ```
 
+
 ## Star History
 
 [![Star History Chart](https://api.star-history.com/svg?repos=RUC-NLPIR/FlashRAG&type=Date)](https://star-history.com/#RUC-NLPIR/FlashRAG&Date)

{flashrag_dev-0.3.0.dev20250924 → flashrag_dev-0.3.0.dev20251106}/flashrag/retriever/retriever.py

@@ -1,6 +1,7 @@
 import json
 import os
 import time
+import requests
 
 os.environ["TOKENIZERS_PARALLELISM"] = "false"
 import warnings
@@ -1141,3 +1142,149 @@ class SparseRetriever(BaseTextRetriever):
             num_threads=int(self.cores)
         )
         return search_results
+
+class SerperRetriever(BaseRetriever):
+    """Retriever based on Google Serper API for web search."""
+
+    def __init__(self, config):
+        super().__init__(config)
+        
+        # Serper API specific configuration
+        self.api_key = config["serper_api_key"]
+        if not self.api_key:
+            raise ValueError("serper_api_key is required in config")
+        
+        self.api_url = "https://google.serper.dev/search"
+        self.search_type = config["serper_search_type"] if config["serper_search_type"] else "search"  # search, news, images, etc.
+        self.location = config["serper_location"] if config["serper_location"] else None  # e.g., "United States"
+        self.gl = config["serper_gl"] if config["serper_gl"] else None  # Country code, e.g., "us"
+        self.hl = config["serper_hl"] if config["serper_hl"] else "en"  # Language, e.g., "en"
+        
+    def _search(self, query: str, num: int) -> List[Dict[str, str]]:
+        """
+        Retrieve top-k relevant documents using Google Serper API.
+        
+        Args:
+            query: Search query string
+            num: Number of results to return
+            return_score: Whether to return relevance scores
+            
+        Returns:
+            List of dictionaries containing search results with keys:
+                - contents: The snippet/description
+                - title: Page title
+                - text: Full text (same as contents for web search)
+                - url: Page URL
+                - score: Relevance score (if return_score=True)
+        """
+        headers = {
+            'X-API-KEY': self.api_key,
+            'Content-Type': 'application/json'
+        }
+        
+        payload = {
+            'q': query,
+            'num': num,
+            'hl': self.hl
+        }
+        
+        if self.location:
+            payload['location'] = self.location
+        if self.gl:
+            payload['gl'] = self.gl
+        
+        try:
+            response = requests.post(
+                self.api_url,
+                headers=headers,
+                json=payload,
+                timeout=30
+            )
+            response.raise_for_status()
+            data = response.json()
+            
+            results = []
+            
+            # Parse organic results
+            organic_results = data.get('organic', [])
+            for idx, item in enumerate(organic_results[:num]):
+                result = {
+                    'title': item.get('title', ''),
+                    'text': item.get('snippet', ''),
+                    'url': item.get('link', ''),
+                }
+                
+                results.append(result)
+            
+            return results
+            
+        except requests.exceptions.RequestException as e:
+            print(f"Error calling Serper API: {e}")
+            return []
+        except Exception as e:
+            print(f"Unexpected error in _search: {e}")
+            return []
+    
+    def search(self, query: str, num: int = None) -> List[Dict[str, str]]:
+        """
+        Single search wrapper for SerperRetriever.
+        """
+        if num is None:
+            num = self.topk
+        return self._search(query, num)
+    
+    def _batch_search(self, query_list: List[str], num: int) -> List[List[Dict[str, str]]]:
+        """
+        Batch search for multiple queries.
+        
+        Args:
+            query_list: List of query strings
+            num: Number of results per query
+            return_score: Whether to return relevance scores
+            
+        Returns:
+            List of result lists, one for each query
+        """
+        results = []
+        if num is None:
+            num = self.topk
+        
+        for query in query_list:
+            result = self._search(query, num)
+            results.append(result)
+            # Add a small delay to avoid rate limiting
+            time.sleep(0.1)
+        
+        return results
+
+    def batch_search(self, query_list: List[str], num: int = None):
+        return self._batch_search(query_list, num)
+
+def main():
+# Example configuration
+    config = {
+        # Base retriever config
+        "retrieval_method": "serper",
+        "retrieval_topk": 10,
+        "index_path": None,  # Not used for Serper
+        "corpus_path": None,  # Not used for Serper
+        "save_dir": "./output",
+        
+        # Serper specific config
+        "serper_api_key": "your-api-key",
+        "serper_search_type": "search",
+        "serper_location": "United States",
+        "serper_gl": "us",
+        "serper_hl": "en"
+    }
+    from flashrag.config import Config
+    config = Config("basic_config.yaml",config)
+    retriever = SerperRetriever(config)
+
+    # Batch search
+    queries = ["Python programming", "Machine learning"]
+    batch_results = retriever.batch_search(queries)
+    print(batch_results)
+
+if __name__ == "__main__":
+    main()

flashrag_dev-0.3.0.dev20251106/flashrag/version.py

@@ -0,0 +1 @@
+__version__ = "0.3.0dev20251106"

{flashrag_dev-0.3.0.dev20250924 → flashrag_dev-0.3.0.dev20251106}/flashrag_dev.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.1
-Name: flashrag-dev
-Version: 0.3.0.dev20250924
+Metadata-Version: 2.4
+Name: flashrag_dev
+Version: 0.3.0.dev20251106
 Summary: A library for efficient Retrieval-Augmented Generation research
 Home-page: https://github.com/RUC-NLPIR/FlashRAG
 Author: Jiajie Jin, Yutao Zhu, Chenghao Zhang, Xinyu Yang, Zhicheng Dou
@@ -8,12 +8,103 @@ Author-email: jinjiajie@ruc.edu.cn
 License: MIT License
 Requires-Python: >=3.9
 Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: datasets
+Requires-Dist: base58
+Requires-Dist: nltk
+Requires-Dist: numpy
+Requires-Dist: langid
+Requires-Dist: openai
+Requires-Dist: peft
+Requires-Dist: PyYAML
+Requires-Dist: rank_bm25
+Requires-Dist: rouge
+Requires-Dist: spacy
+Requires-Dist: tiktoken
+Requires-Dist: torch
+Requires-Dist: tqdm
+Requires-Dist: transformers>=4.40.0
+Requires-Dist: bm25s[core]==0.2.1
+Requires-Dist: fschat
+Requires-Dist: streamlit
+Requires-Dist: chonkie<1.1.0,>=1.0.2
+Requires-Dist: gradio>=5.0.0
+Requires-Dist: rouge-chinese
+Requires-Dist: jieba
 Provides-Extra: core
+Requires-Dist: datasets; extra == "core"
+Requires-Dist: base58; extra == "core"
+Requires-Dist: nltk; extra == "core"
+Requires-Dist: numpy; extra == "core"
+Requires-Dist: langid; extra == "core"
+Requires-Dist: openai; extra == "core"
+Requires-Dist: peft; extra == "core"
+Requires-Dist: PyYAML; extra == "core"
+Requires-Dist: rank_bm25; extra == "core"
+Requires-Dist: rouge; extra == "core"
+Requires-Dist: spacy; extra == "core"
+Requires-Dist: tiktoken; extra == "core"
+Requires-Dist: torch; extra == "core"
+Requires-Dist: tqdm; extra == "core"
+Requires-Dist: transformers>=4.40.0; extra == "core"
+Requires-Dist: bm25s[core]==0.2.1; extra == "core"
+Requires-Dist: fschat; extra == "core"
+Requires-Dist: streamlit; extra == "core"
+Requires-Dist: chonkie<1.1.0,>=1.0.2; extra == "core"
+Requires-Dist: gradio>=5.0.0; extra == "core"
+Requires-Dist: rouge-chinese; extra == "core"
+Requires-Dist: jieba; extra == "core"
 Provides-Extra: retriever
+Requires-Dist: pyserini; extra == "retriever"
+Requires-Dist: sentence-transformers>=3.0.1; extra == "retriever"
 Provides-Extra: generator
+Requires-Dist: vllm; extra == "generator"
 Provides-Extra: multimodal
+Requires-Dist: timm; extra == "multimodal"
+Requires-Dist: torchvision; extra == "multimodal"
+Requires-Dist: pillow; extra == "multimodal"
+Requires-Dist: qwen_vl_utils; extra == "multimodal"
 Provides-Extra: full
-License-File: LICENSE
+Requires-Dist: datasets; extra == "full"
+Requires-Dist: base58; extra == "full"
+Requires-Dist: nltk; extra == "full"
+Requires-Dist: numpy; extra == "full"
+Requires-Dist: langid; extra == "full"
+Requires-Dist: openai; extra == "full"
+Requires-Dist: peft; extra == "full"
+Requires-Dist: PyYAML; extra == "full"
+Requires-Dist: rank_bm25; extra == "full"
+Requires-Dist: rouge; extra == "full"
+Requires-Dist: spacy; extra == "full"
+Requires-Dist: tiktoken; extra == "full"
+Requires-Dist: torch; extra == "full"
+Requires-Dist: tqdm; extra == "full"
+Requires-Dist: transformers>=4.40.0; extra == "full"
+Requires-Dist: bm25s[core]==0.2.1; extra == "full"
+Requires-Dist: fschat; extra == "full"
+Requires-Dist: streamlit; extra == "full"
+Requires-Dist: chonkie<1.1.0,>=1.0.2; extra == "full"
+Requires-Dist: gradio>=5.0.0; extra == "full"
+Requires-Dist: rouge-chinese; extra == "full"
+Requires-Dist: jieba; extra == "full"
+Requires-Dist: pyserini; extra == "full"
+Requires-Dist: sentence-transformers>=3.0.1; extra == "full"
+Requires-Dist: vllm; extra == "full"
+Requires-Dist: timm; extra == "full"
+Requires-Dist: torchvision; extra == "full"
+Requires-Dist: pillow; extra == "full"
+Requires-Dist: qwen_vl_utils; extra == "full"
+Dynamic: author
+Dynamic: author-email
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license
+Dynamic: license-file
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
 
 # <div align="center">⚡FlashRAG: A Python Toolkit for Efficient RAG Research<div>
 \[ English | [中文](README_zh.md) \]
@@ -100,6 +191,8 @@ FlashRAG is still under development and there are many issues and room for impro
 - [ ] Add support for api-based retriever (vllm server)
 
 ## :page_with_curl: Changelog
+[25/11/06] 🎯 NEW Retriever! We have integrated a Web Search Engine-based Retriever, which seamlessly integrates with existing methods and can be enabled quickly with just a Serper API key! This enhancement significantly expands retrieval coverage and real-time capability, supporting dynamic information access and external knowledge augmentation. Experience a more flexible and powerful retrieval workflow now! 
+
 [25/08/06] 🎯 **NEW!** We have added support for **Reasoning Pipeline**, which is a new paradigm that combines reasoning ability and retrieval, representing work that includes [R1-Searcher](https://github.com/SsmallSong/R1-Searcher), [Search-R1](https://github.com/PeterGriffinJin/Search-R1),.... We evaluate the performance of the pipeline on various RAG benchmarks, it can achieve F1 scores close to 60 on multi hop inference datasets such as HotpotQA. See it in [**result table**](#robot-supporting-methods).
 
 [25/03/21] 🚀 **Major Update!** We have expanded our toolkit to support **23 state-of-the-art RAG algorithms**, including **7 reasoning-based methods** that significantly improve performance on complex reasoning tasks. This represents a major milestone in our toolkit's evolution!
@@ -767,6 +860,7 @@ Please kindly cite our paper if helps your research:
 }
 ```
 
+
 ## Star History
 
 [![Star History Chart](https://api.star-history.com/svg?repos=RUC-NLPIR/FlashRAG&type=Date)](https://star-history.com/#RUC-NLPIR/FlashRAG&Date)

flashrag_dev-0.3.0.dev20250924/flashrag/version.py

@@ -1 +0,0 @@
-__version__ = "0.3.0dev20250924"