flashrag-dev 0.1.3.dev20241102__tar.gz → 0.1.3.dev20241104__tar.gz

{flashrag-dev-0.1.3.dev20241102/flashrag_dev.egg-info → flashrag-dev-0.1.3.dev20241104}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: flashrag-dev
-Version: 0.1.3.dev20241102
+Version: 0.1.3.dev20241104
 Summary: A library for efficient Retrieval-Augmented Generation research
 Home-page: https://github.com/RUC-NLPIR/FlashRAG
 Author: Jiajie Jin, Yutao Zhu, Chenghao Zhang, Xinyu Yang, Zhicheng Dou
@@ -20,6 +20,7 @@ License-File: LICENSE
 <div align="center">
 <a href="https://arxiv.org/abs/2405.13576" target="_blank"><img src=https://img.shields.io/badge/arXiv-b5212f.svg?logo=arxiv></a>
 <a href="https://huggingface.co/datasets/RUC-NLPIR/FlashRAG_datasets/" target="_blank"><img src=https://img.shields.io/badge/%F0%9F%A4%97%20HuggingFace%20Datasets-27b3b4.svg></a>
+<a href="https://www.modelscope.cn/datasets/hhjinjiajie/FlashRAG_Dataset" target="_blank"><img src=https://custom-icon-badges.demolab.com/badge/ModelScope%20Datasets-624aff?style=flat&logo=modelscope&logoColor=white></a>
 <a href="https://github.com/RUC-NLPIR/FlashRAG/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/badge/LICENSE-MIT-green"></a>
 <a><img alt="Static Badge" src="https://img.shields.io/badge/made_with-Python-blue"></a>
 </div>
@@ -112,7 +113,7 @@ FlashRAG is still under development and there are many issues and room for impro
 
 To get started with FlashRAG, you can simply install it with pip:
 ```base
-pip install flashrag-dev[core]
+pip install flashrag-dev --pre
 ```
 
 Or you can clone it from Github and install (requires Python 3.9+): 
@@ -120,7 +121,7 @@ Or you can clone it from Github and install (requires Python 3.9+):
 ```bash
 git clone https://github.com/RUC-NLPIR/FlashRAG.git
 cd FlashRAG
-pip install -e .[core] 
+pip install -e .
 ```
 
 If you want to use vllm, sentence-transformers or pyserini, you can install the optional dependencies:
@@ -493,7 +494,7 @@ Below is the list of datasets along with the corresponding sample sizes:
 | QA                        | WebQuestions    | Google Freebase  | 3,778     | /       | 2,032  |
 | QA                        | AmbigQA         | wiki             | 10,036    | 2,002   | /      |
 | QA                        | SIQA            | -                | 33,410    | 1,954   | /      |
-| QA                        | CommenseQA      | -                | 9,741     | 1,221   | /      |
+| QA                        | CommonSenseQA      | -                | 9,741     | 1,221   | /      |
 | QA                        | BoolQ           | wiki             | 9,427     | 3,270   | /      |
 | QA                        | PIQA            | -                | 16,113    | 1,838   | /      |
 | QA                        | Fermi           | wiki             | 8,000     | 1,000   | 1,000  |

{flashrag-dev-0.1.3.dev20241102 → flashrag-dev-0.1.3.dev20241104}/README.md

@@ -4,6 +4,7 @@
 <div align="center">
 <a href="https://arxiv.org/abs/2405.13576" target="_blank"><img src=https://img.shields.io/badge/arXiv-b5212f.svg?logo=arxiv></a>
 <a href="https://huggingface.co/datasets/RUC-NLPIR/FlashRAG_datasets/" target="_blank"><img src=https://img.shields.io/badge/%F0%9F%A4%97%20HuggingFace%20Datasets-27b3b4.svg></a>
+<a href="https://www.modelscope.cn/datasets/hhjinjiajie/FlashRAG_Dataset" target="_blank"><img src=https://custom-icon-badges.demolab.com/badge/ModelScope%20Datasets-624aff?style=flat&logo=modelscope&logoColor=white></a>
 <a href="https://github.com/RUC-NLPIR/FlashRAG/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/badge/LICENSE-MIT-green"></a>
 <a><img alt="Static Badge" src="https://img.shields.io/badge/made_with-Python-blue"></a>
 </div>
@@ -96,7 +97,7 @@ FlashRAG is still under development and there are many issues and room for impro
 
 To get started with FlashRAG, you can simply install it with pip:
 ```base
-pip install flashrag-dev[core]
+pip install flashrag-dev --pre
 ```
 
 Or you can clone it from Github and install (requires Python 3.9+): 
@@ -104,7 +105,7 @@ Or you can clone it from Github and install (requires Python 3.9+):
 ```bash
 git clone https://github.com/RUC-NLPIR/FlashRAG.git
 cd FlashRAG
-pip install -e .[core] 
+pip install -e .
 ```
 
 If you want to use vllm, sentence-transformers or pyserini, you can install the optional dependencies:
@@ -477,7 +478,7 @@ Below is the list of datasets along with the corresponding sample sizes:
 | QA                        | WebQuestions    | Google Freebase  | 3,778     | /       | 2,032  |
 | QA                        | AmbigQA         | wiki             | 10,036    | 2,002   | /      |
 | QA                        | SIQA            | -                | 33,410    | 1,954   | /      |
-| QA                        | CommenseQA      | -                | 9,741     | 1,221   | /      |
+| QA                        | CommonSenseQA      | -                | 9,741     | 1,221   | /      |
 | QA                        | BoolQ           | wiki             | 9,427     | 3,270   | /      |
 | QA                        | PIQA            | -                | 16,113    | 1,838   | /      |
 | QA                        | Fermi           | wiki             | 8,000     | 1,000   | 1,000  |

{flashrag-dev-0.1.3.dev20241102 → flashrag-dev-0.1.3.dev20241104}/flashrag/generator/openai_generator.py

@@ -27,8 +27,12 @@ class OpenaiGenerator:
             self.client = AsyncAzureOpenAI(**self.openai_setting)
         else:
             self.client = AsyncOpenAI(**self.openai_setting)
-
-        self.tokenizer = tiktoken.encoding_for_model(self.model_name)
+        try:
+            self.tokenizer = tiktoken.encoding_for_model(self.model_name)
+        except Exception as e:
+            print("Error: ", e)
+            warnings.warn("This model is not supported by tiktoken. Use gpt-3.5-turbo instead.")
+            self.tokenizer = tiktoken.encoding_for_model('gpt-3.5-turbo')
 
     async def get_response(self, input: List, **params):
         response = await self.client.chat.completions.create(model=self.model_name, messages=input, **params)
@@ -46,7 +50,7 @@ class OpenaiGenerator:
 
     def generate(self, input_list: List[List], batch_size=None, return_scores=False, **params) -> List[str]:
         # deal with single input
-        if len(input_list) == 1:
+        if len(input_list) == 1 and isinstance(input_list[0], dict):
             input_list = [input_list]
         if batch_size is None:
             batch_size = self.batch_size

{flashrag-dev-0.1.3.dev20241102 → flashrag-dev-0.1.3.dev20241104}/flashrag/prompt/base_prompt.py

@@ -1,5 +1,6 @@
 from transformers import AutoTokenizer, AutoConfig
 import tiktoken
+import warnings
 
 class PromptTemplate:
     placeholders = ["reference", "question"]
@@ -26,7 +27,12 @@ class PromptTemplate:
         else:
             self.is_chat = True
             self.enable_chat = True
-            self.tokenizer = tiktoken.encoding_for_model("gpt-4o")
+            try:
+                self.tokenizer = tiktoken.encoding_for_model(config['generator_model'])
+            except Exception as e:
+                print("Error: ", e)
+                warnings.warn("This model is not supported by tiktoken. Use gpt-3.5-turbo instead.")
+                self.tokenizer = tiktoken.encoding_for_model('gpt-3.5-turbo')
 
         if len(system_prompt) == 0 and len(user_prompt) == 0:
             system_prompt = self.base_system_prompt
@@ -91,9 +97,6 @@ class PromptTemplate:
                 return self.truncate_prompt(messages)
             if self.is_chat and self.enable_chat:
                 if self.is_openai:
-                    for item in input:
-                        if item["role"] == "system":
-                            item["role"] = "assistant"
                     self.truncate_prompt(messages)
                 else:
                     prompt = self.tokenizer.apply_chat_template(
@@ -124,11 +127,7 @@ class PromptTemplate:
                 input.append({"role": "system", "content": system_prompt})
             if user_prompt != "":
                 input.append({"role": "user", "content": user_prompt})
-            if self.is_openai:
-                for item in input:
-                    if item["role"] == "system":
-                        item["role"] = "assistant"
-            else:
+            if not self.is_openai:
                 input = self.tokenizer.apply_chat_template(input, tokenize=False, add_generation_prompt=True)
         else:
             input = "\n\n".join([prompt for prompt in [system_prompt, user_prompt] if prompt != ""])

{flashrag-dev-0.1.3.dev20241102 → flashrag-dev-0.1.3.dev20241104}/flashrag/retriever/encoder.py

@@ -37,7 +37,7 @@ class Encoder:
             query_emb = pooling(
                 output.pooler_output, output.last_hidden_state, inputs["attention_mask"], self.pooling_method
             )
-
+        query_emb = torch.nn.functional.normalize(query_emb, dim=-1)
         query_emb = query_emb.detach().cpu().numpy()
         query_emb = query_emb.astype(np.float32, order="C")
         return query_emb

flashrag-dev-0.1.3.dev20241104/flashrag/version.py

@@ -0,0 +1 @@
+__version__ = "0.1.3dev20241104"

{flashrag-dev-0.1.3.dev20241102 → flashrag-dev-0.1.3.dev20241104/flashrag_dev.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: flashrag-dev
-Version: 0.1.3.dev20241102
+Version: 0.1.3.dev20241104
 Summary: A library for efficient Retrieval-Augmented Generation research
 Home-page: https://github.com/RUC-NLPIR/FlashRAG
 Author: Jiajie Jin, Yutao Zhu, Chenghao Zhang, Xinyu Yang, Zhicheng Dou
@@ -20,6 +20,7 @@ License-File: LICENSE
 <div align="center">
 <a href="https://arxiv.org/abs/2405.13576" target="_blank"><img src=https://img.shields.io/badge/arXiv-b5212f.svg?logo=arxiv></a>
 <a href="https://huggingface.co/datasets/RUC-NLPIR/FlashRAG_datasets/" target="_blank"><img src=https://img.shields.io/badge/%F0%9F%A4%97%20HuggingFace%20Datasets-27b3b4.svg></a>
+<a href="https://www.modelscope.cn/datasets/hhjinjiajie/FlashRAG_Dataset" target="_blank"><img src=https://custom-icon-badges.demolab.com/badge/ModelScope%20Datasets-624aff?style=flat&logo=modelscope&logoColor=white></a>
 <a href="https://github.com/RUC-NLPIR/FlashRAG/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/badge/LICENSE-MIT-green"></a>
 <a><img alt="Static Badge" src="https://img.shields.io/badge/made_with-Python-blue"></a>
 </div>
@@ -112,7 +113,7 @@ FlashRAG is still under development and there are many issues and room for impro
 
 To get started with FlashRAG, you can simply install it with pip:
 ```base
-pip install flashrag-dev[core]
+pip install flashrag-dev --pre
 ```
 
 Or you can clone it from Github and install (requires Python 3.9+): 
@@ -120,7 +121,7 @@ Or you can clone it from Github and install (requires Python 3.9+):
 ```bash
 git clone https://github.com/RUC-NLPIR/FlashRAG.git
 cd FlashRAG
-pip install -e .[core] 
+pip install -e .
 ```
 
 If you want to use vllm, sentence-transformers or pyserini, you can install the optional dependencies:
@@ -493,7 +494,7 @@ Below is the list of datasets along with the corresponding sample sizes:
 | QA                        | WebQuestions    | Google Freebase  | 3,778     | /       | 2,032  |
 | QA                        | AmbigQA         | wiki             | 10,036    | 2,002   | /      |
 | QA                        | SIQA            | -                | 33,410    | 1,954   | /      |
-| QA                        | CommenseQA      | -                | 9,741     | 1,221   | /      |
+| QA                        | CommonSenseQA      | -                | 9,741     | 1,221   | /      |
 | QA                        | BoolQ           | wiki             | 9,427     | 3,270   | /      |
 | QA                        | PIQA            | -                | 16,113    | 1,838   | /      |
 | QA                        | Fermi           | wiki             | 8,000     | 1,000   | 1,000  |

flashrag-dev-0.1.3.dev20241102/flashrag/version.py

@@ -1 +0,0 @@
-__version__ = "0.1.3dev20241102"