flashrag-dev 0.1.4.dev20250112__tar.gz → 0.1.4.dev20250113__tar.gz

{flashrag_dev-0.1.4.dev20250112 → flashrag_dev-0.1.4.dev20250113}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: flashrag_dev
-Version: 0.1.4.dev20250112
+Version: 0.1.4.dev20250113
 Summary: A library for efficient Retrieval-Augmented Generation research
 Home-page: https://github.com/RUC-NLPIR/FlashRAG
 Author: Jiajie Jin, Yutao Zhu, Chenghao Zhang, Xinyu Yang, Zhicheng Dou
@@ -32,7 +32,7 @@ License-File: LICENSE
 <a href="#sparkles-features">Features</a> |
 <a href="#rocket-quick-start">Quick-Start</a> |
 <a href="#gear-components"> Components</a> |
-<a href="#computer-flashrag-ui"> FlashRAG-UI</a> |
+<a href="#art-flashrag-ui"> FlashRAG-UI</a> |
 <a href="#robot-supporting-methods"> Supporting Methods</a> |
 <a href="#notebook-supporting-datasets--document-corpus"> Supporting Datasets</a> |
 <a href="#raised_hands-additional-faqs"> FAQs</a>
@@ -47,68 +47,14 @@ FlashRAG is a Python toolkit for the reproduction and development of Retrieval A
 <img src="asset/framework.jpg">
 </p>
 
-With FlashRAG and provided resources, you can effortlessly reproduce existing SOTA works in the RAG domain or implement your custom RAG processes and components.
+With FlashRAG and provided resources, you can effortlessly reproduce existing SOTA works in the RAG domain or implement your custom RAG processes and components. Besides, we provide an easy-to-use UI:
+
+https://github.com/user-attachments/assets/8ca00873-5df2-48a7-b853-89e7b18bc6e9
 
 <p>
 <a href="https://trendshift.io/repositories/10454" target="_blank"><img src="https://trendshift.io/api/badge/repositories/10454" alt="RUC-NLPIR%2FFlashRAG | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
 </p>
 
-<h2 align="center">✨ Introduce FlashRAG-UI ✨</h2>
-<p>With <strong>FlashRAG-UI</strong>, you can easily and quickly configure and experience the supported RAG methods through our meticulously designed visual interface, and evaluate these methods on benchmarks, making complex research work more efficient!</p>
-
-
-https://github.com/user-attachments/assets/8ca00873-5df2-48a7-b853-89e7b18bc6e9
-
-
-<h3>🌟 Features</h3>
-<ul>
-  <li>🚀 <strong>One-Click Configuration Loading</strong>
-    <ul>
-      <li>You can load parameters and configuration files for various RAG methods through simple clicks, selections, and inputs.</li>
-      <li>🖼️ Supports preview interface for intuitive parameter settings.</li>
-      <li>💾 Provides save functionality to easily store configurations for future use.</li>
-    </ul>
-  </li>
-  <li>🔍 <strong>Quick Method Experience</strong>
-    <ul>
-      <li>⚡ Quickly load corpora and index files to explore the characteristics and application scenarios of various RAG methods.</li>
-      <li>🔄 Supports loading and switching different components and hyperparameters, seamlessly connecting different RAG Pipelines to quickly experience their performance and differences!</li>
-    </ul>
-  </li>
-  <li>📊 <strong>Efficient Benchmark Reproduction</strong>
-    <ul>
-      <li>🎯 Easily reproduce the built-in baseline methods and carefully collected benchmarks on FlashRAG-UI.</li>
-      <li>🛠️ Use cutting-edge research tools directly without complex settings, providing a smooth experience for your research work!</li>
-    </ul>
-  </li>
-</ul>
-
-<details>
-<summary>Show more</summary>
-<table align="center">
-  <tr>
-    <td align="center">
-      <img src="./asset/demo_en1.jpg" alt="Image 1" width="505"/>
-    </td>
-    <td align="center">
-      <img src="./asset/demo_en2.jpg" alt="Image 2" width="505"/>
-    </td>
-  </tr>
-  <tr>
-    <td align="center">
-      <img src="./asset/demo_en4.png" alt="Image 3" width="500"/>
-    </td>
-    <td align="center">
-      <img src="./asset/demo_en3.jpg" alt="Image 4" width="500"/>
-    </td>
-  </tr>
-</table>
-</details>
-
-<p>To experience FlashRAG-UI, please go to <a href="#computer-flashrag-ui">FlashRAG-UI</a>.</p>
-<center><h3>🎨 Experience our meticulously designed FlashRAG-UI—both user-friendly and visually appealing</h3></center>
-
-
 ## :link: Navigation
 - [Features](#sparkles-features)
 - [Roadmap](#mag_right-roadmap)
@@ -116,7 +62,7 @@ https://github.com/user-attachments/assets/8ca00873-5df2-48a7-b853-89e7b18bc6e9
 - [Installation](#wrench-installation)
 - [Quick Start](#rocket-quick-start)
 - [Components](#gear-components)
-- [FlashRAG-UI](#computer-flashrag-ui)
+- [FlashRAG-UI](#art-flashrag-ui)
 - [Supporting Methods](#robot-supporting-methods)
 - [Supporting Datasets & Document Corpus](#notebook-supporting-datasets--document-corpus)
 - [Additional FAQs](#raised_hands-additional-faqs)
@@ -554,9 +500,44 @@ In each category, we have implemented corresponding common pipelines. Some pipel
     </tbody>
 </table>
 
-## :computer: FlashRAG-UI
-To use FlashRAG-UI, run the following code:
+## :art: FlashRAG-UI
+<p>With <strong>FlashRAG-UI</strong>, you can easily and quickly configure and experience the supported RAG methods through our meticulously designed visual interface, and evaluate these methods on benchmarks, making complex research work more efficient!</p>
+
+### :star2: Features
+- **One-Click Configuration Loading**
+  - You can load parameters and configuration files for various RAG methods through simple clicks, selections, and inputs.</li>
+  - Supports preview interface for intuitive parameter settings.</li>
+  - Provides save functionality to easily store configurations for future use.</li>
+- **Quick Method Experience**
+  - Quickly load corpora and index files to explore the characteristics and application scenarios of various RAG methods.</li>
+  - Supports loading and switching different components and hyperparameters, seamlessly connecting different RAG Pipelines to quickly experience their performance and differences!</li>
+- **Efficient Benchmark Reproduction**
+  - Easily reproduce the built-in baseline methods and carefully collected benchmarks on FlashRAG-UI.</li>
+  - Use cutting-edge research tools directly without complex settings, providing a smooth experience for your research work!</li>
+  
+<details>
+<summary>Show more</summary>
+<table align="center">
+  <tr>
+    <td align="center">
+      <img src="./asset/demo_en1.jpg" alt="Image 1" width="505"/>
+    </td>
+    <td align="center">
+      <img src="./asset/demo_en2.jpg" alt="Image 2" width="505"/>
+    </td>
+  </tr>
+  <tr>
+    <td align="center">
+      <img src="./asset/demo_en4.png" alt="Image 3" width="500"/>
+    </td>
+    <td align="center">
+      <img src="./asset/demo_en3.jpg" alt="Image 4" width="500"/>
+    </td>
+  </tr>
+</table>
+</details>
 
+#### Experience our meticulously designed FlashRAG-UI—both user-friendly and visually appealing:
 ```bash
 cd webui
 python interface.py

{flashrag_dev-0.1.4.dev20250112 → flashrag_dev-0.1.4.dev20250113}/README.md

@@ -15,7 +15,7 @@
 <a href="#sparkles-features">Features</a> |
 <a href="#rocket-quick-start">Quick-Start</a> |
 <a href="#gear-components"> Components</a> |
-<a href="#computer-flashrag-ui"> FlashRAG-UI</a> |
+<a href="#art-flashrag-ui"> FlashRAG-UI</a> |
 <a href="#robot-supporting-methods"> Supporting Methods</a> |
 <a href="#notebook-supporting-datasets--document-corpus"> Supporting Datasets</a> |
 <a href="#raised_hands-additional-faqs"> FAQs</a>
@@ -30,68 +30,14 @@ FlashRAG is a Python toolkit for the reproduction and development of Retrieval A
 <img src="asset/framework.jpg">
 </p>
 
-With FlashRAG and provided resources, you can effortlessly reproduce existing SOTA works in the RAG domain or implement your custom RAG processes and components.
+With FlashRAG and provided resources, you can effortlessly reproduce existing SOTA works in the RAG domain or implement your custom RAG processes and components. Besides, we provide an easy-to-use UI:
+
+https://github.com/user-attachments/assets/8ca00873-5df2-48a7-b853-89e7b18bc6e9
 
 <p>
 <a href="https://trendshift.io/repositories/10454" target="_blank"><img src="https://trendshift.io/api/badge/repositories/10454" alt="RUC-NLPIR%2FFlashRAG | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
 </p>
 
-<h2 align="center">✨ Introduce FlashRAG-UI ✨</h2>
-<p>With <strong>FlashRAG-UI</strong>, you can easily and quickly configure and experience the supported RAG methods through our meticulously designed visual interface, and evaluate these methods on benchmarks, making complex research work more efficient!</p>
-
-
-https://github.com/user-attachments/assets/8ca00873-5df2-48a7-b853-89e7b18bc6e9
-
-
-<h3>🌟 Features</h3>
-<ul>
-  <li>🚀 <strong>One-Click Configuration Loading</strong>
-    <ul>
-      <li>You can load parameters and configuration files for various RAG methods through simple clicks, selections, and inputs.</li>
-      <li>🖼️ Supports preview interface for intuitive parameter settings.</li>
-      <li>💾 Provides save functionality to easily store configurations for future use.</li>
-    </ul>
-  </li>
-  <li>🔍 <strong>Quick Method Experience</strong>
-    <ul>
-      <li>⚡ Quickly load corpora and index files to explore the characteristics and application scenarios of various RAG methods.</li>
-      <li>🔄 Supports loading and switching different components and hyperparameters, seamlessly connecting different RAG Pipelines to quickly experience their performance and differences!</li>
-    </ul>
-  </li>
-  <li>📊 <strong>Efficient Benchmark Reproduction</strong>
-    <ul>
-      <li>🎯 Easily reproduce the built-in baseline methods and carefully collected benchmarks on FlashRAG-UI.</li>
-      <li>🛠️ Use cutting-edge research tools directly without complex settings, providing a smooth experience for your research work!</li>
-    </ul>
-  </li>
-</ul>
-
-<details>
-<summary>Show more</summary>
-<table align="center">
-  <tr>
-    <td align="center">
-      <img src="./asset/demo_en1.jpg" alt="Image 1" width="505"/>
-    </td>
-    <td align="center">
-      <img src="./asset/demo_en2.jpg" alt="Image 2" width="505"/>
-    </td>
-  </tr>
-  <tr>
-    <td align="center">
-      <img src="./asset/demo_en4.png" alt="Image 3" width="500"/>
-    </td>
-    <td align="center">
-      <img src="./asset/demo_en3.jpg" alt="Image 4" width="500"/>
-    </td>
-  </tr>
-</table>
-</details>
-
-<p>To experience FlashRAG-UI, please go to <a href="#computer-flashrag-ui">FlashRAG-UI</a>.</p>
-<center><h3>🎨 Experience our meticulously designed FlashRAG-UI—both user-friendly and visually appealing</h3></center>
-
-
 ## :link: Navigation
 - [Features](#sparkles-features)
 - [Roadmap](#mag_right-roadmap)
@@ -99,7 +45,7 @@ https://github.com/user-attachments/assets/8ca00873-5df2-48a7-b853-89e7b18bc6e9
 - [Installation](#wrench-installation)
 - [Quick Start](#rocket-quick-start)
 - [Components](#gear-components)
-- [FlashRAG-UI](#computer-flashrag-ui)
+- [FlashRAG-UI](#art-flashrag-ui)
 - [Supporting Methods](#robot-supporting-methods)
 - [Supporting Datasets & Document Corpus](#notebook-supporting-datasets--document-corpus)
 - [Additional FAQs](#raised_hands-additional-faqs)
@@ -537,9 +483,44 @@ In each category, we have implemented corresponding common pipelines. Some pipel
     </tbody>
 </table>
 
-## :computer: FlashRAG-UI
-To use FlashRAG-UI, run the following code:
+## :art: FlashRAG-UI
+<p>With <strong>FlashRAG-UI</strong>, you can easily and quickly configure and experience the supported RAG methods through our meticulously designed visual interface, and evaluate these methods on benchmarks, making complex research work more efficient!</p>
+
+### :star2: Features
+- **One-Click Configuration Loading**
+  - You can load parameters and configuration files for various RAG methods through simple clicks, selections, and inputs.</li>
+  - Supports preview interface for intuitive parameter settings.</li>
+  - Provides save functionality to easily store configurations for future use.</li>
+- **Quick Method Experience**
+  - Quickly load corpora and index files to explore the characteristics and application scenarios of various RAG methods.</li>
+  - Supports loading and switching different components and hyperparameters, seamlessly connecting different RAG Pipelines to quickly experience their performance and differences!</li>
+- **Efficient Benchmark Reproduction**
+  - Easily reproduce the built-in baseline methods and carefully collected benchmarks on FlashRAG-UI.</li>
+  - Use cutting-edge research tools directly without complex settings, providing a smooth experience for your research work!</li>
+  
+<details>
+<summary>Show more</summary>
+<table align="center">
+  <tr>
+    <td align="center">
+      <img src="./asset/demo_en1.jpg" alt="Image 1" width="505"/>
+    </td>
+    <td align="center">
+      <img src="./asset/demo_en2.jpg" alt="Image 2" width="505"/>
+    </td>
+  </tr>
+  <tr>
+    <td align="center">
+      <img src="./asset/demo_en4.png" alt="Image 3" width="500"/>
+    </td>
+    <td align="center">
+      <img src="./asset/demo_en3.jpg" alt="Image 4" width="500"/>
+    </td>
+  </tr>
+</table>
+</details>
 
+#### Experience our meticulously designed FlashRAG-UI—both user-friendly and visually appealing:
 ```bash
 cd webui
 python interface.py

{flashrag_dev-0.1.4.dev20250112 → flashrag_dev-0.1.4.dev20250113}/flashrag/refiner/refiner.py

@@ -73,7 +73,6 @@ class LLMLinguaRefiner(BaseRefiner):
         output = []
         for item in tqdm(dataset, desc="Refining process: "):
             question = item.question
-            retrieval_result = item.retrieval_result
             # TODO: suit more cases
             if self.input_prompt_flag:
                 input_prompt = item.prompt
@@ -88,6 +87,7 @@ class LLMLinguaRefiner(BaseRefiner):
                     **self.compress_config,
                 )
             else:
+                retrieval_result = item.retrieval_result
                 docs = self.format_reference(retrieval_result).split("\n")
                 docs = [i for i in docs if i != ""]
                 item_output = self.refiner.compress_prompt(

{flashrag_dev-0.1.4.dev20250112 → flashrag_dev-0.1.4.dev20250113}/flashrag/retriever/utils.py

@@ -5,6 +5,7 @@ from typing import Dict, Any, Union, List, Dict
 import numpy as np
 import datasets
 import re
+import langid
 from transformers import AutoTokenizer, AutoModel, AutoConfig
 
 def convert_numpy(obj: Union[Dict, list, np.ndarray, np.generic]) -> Any:
@@ -27,7 +28,14 @@ def judge_zh(input_str: str):
     assert isinstance(input_str, str), input_str
     if len(input_str) == 0:
         return False
-    return bool(re.search(r'[\u4e00-\u9fff]', input_str))
+    detect_result = langid.classify(input_str)
+    if detect_result[0] == 'zh':
+        return True
+    else:
+        return False
+    #return bool(re.search(r'[\u4e00-\u9fff]', input_str))
+
+
 def convert_numpy(obj: Union[Dict, list, np.ndarray, np.generic]) -> Any:
     """Recursively convert numpy objects in nested dictionaries or lists to native Python types."""
     if isinstance(obj, dict):

flashrag_dev-0.1.4.dev20250113/flashrag/version.py

@@ -0,0 +1 @@
+__version__ = "0.1.4dev20250113"

{flashrag_dev-0.1.4.dev20250112 → flashrag_dev-0.1.4.dev20250113}/flashrag_dev.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: flashrag-dev
-Version: 0.1.4.dev20250112
+Version: 0.1.4.dev20250113
 Summary: A library for efficient Retrieval-Augmented Generation research
 Home-page: https://github.com/RUC-NLPIR/FlashRAG
 Author: Jiajie Jin, Yutao Zhu, Chenghao Zhang, Xinyu Yang, Zhicheng Dou
@@ -32,7 +32,7 @@ License-File: LICENSE
 <a href="#sparkles-features">Features</a> |
 <a href="#rocket-quick-start">Quick-Start</a> |
 <a href="#gear-components"> Components</a> |
-<a href="#computer-flashrag-ui"> FlashRAG-UI</a> |
+<a href="#art-flashrag-ui"> FlashRAG-UI</a> |
 <a href="#robot-supporting-methods"> Supporting Methods</a> |
 <a href="#notebook-supporting-datasets--document-corpus"> Supporting Datasets</a> |
 <a href="#raised_hands-additional-faqs"> FAQs</a>
@@ -47,68 +47,14 @@ FlashRAG is a Python toolkit for the reproduction and development of Retrieval A
 <img src="asset/framework.jpg">
 </p>
 
-With FlashRAG and provided resources, you can effortlessly reproduce existing SOTA works in the RAG domain or implement your custom RAG processes and components.
+With FlashRAG and provided resources, you can effortlessly reproduce existing SOTA works in the RAG domain or implement your custom RAG processes and components. Besides, we provide an easy-to-use UI:
+
+https://github.com/user-attachments/assets/8ca00873-5df2-48a7-b853-89e7b18bc6e9
 
 <p>
 <a href="https://trendshift.io/repositories/10454" target="_blank"><img src="https://trendshift.io/api/badge/repositories/10454" alt="RUC-NLPIR%2FFlashRAG | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
 </p>
 
-<h2 align="center">✨ Introduce FlashRAG-UI ✨</h2>
-<p>With <strong>FlashRAG-UI</strong>, you can easily and quickly configure and experience the supported RAG methods through our meticulously designed visual interface, and evaluate these methods on benchmarks, making complex research work more efficient!</p>
-
-
-https://github.com/user-attachments/assets/8ca00873-5df2-48a7-b853-89e7b18bc6e9
-
-
-<h3>🌟 Features</h3>
-<ul>
-  <li>🚀 <strong>One-Click Configuration Loading</strong>
-    <ul>
-      <li>You can load parameters and configuration files for various RAG methods through simple clicks, selections, and inputs.</li>
-      <li>🖼️ Supports preview interface for intuitive parameter settings.</li>
-      <li>💾 Provides save functionality to easily store configurations for future use.</li>
-    </ul>
-  </li>
-  <li>🔍 <strong>Quick Method Experience</strong>
-    <ul>
-      <li>⚡ Quickly load corpora and index files to explore the characteristics and application scenarios of various RAG methods.</li>
-      <li>🔄 Supports loading and switching different components and hyperparameters, seamlessly connecting different RAG Pipelines to quickly experience their performance and differences!</li>
-    </ul>
-  </li>
-  <li>📊 <strong>Efficient Benchmark Reproduction</strong>
-    <ul>
-      <li>🎯 Easily reproduce the built-in baseline methods and carefully collected benchmarks on FlashRAG-UI.</li>
-      <li>🛠️ Use cutting-edge research tools directly without complex settings, providing a smooth experience for your research work!</li>
-    </ul>
-  </li>
-</ul>
-
-<details>
-<summary>Show more</summary>
-<table align="center">
-  <tr>
-    <td align="center">
-      <img src="./asset/demo_en1.jpg" alt="Image 1" width="505"/>
-    </td>
-    <td align="center">
-      <img src="./asset/demo_en2.jpg" alt="Image 2" width="505"/>
-    </td>
-  </tr>
-  <tr>
-    <td align="center">
-      <img src="./asset/demo_en4.png" alt="Image 3" width="500"/>
-    </td>
-    <td align="center">
-      <img src="./asset/demo_en3.jpg" alt="Image 4" width="500"/>
-    </td>
-  </tr>
-</table>
-</details>
-
-<p>To experience FlashRAG-UI, please go to <a href="#computer-flashrag-ui">FlashRAG-UI</a>.</p>
-<center><h3>🎨 Experience our meticulously designed FlashRAG-UI—both user-friendly and visually appealing</h3></center>
-
-
 ## :link: Navigation
 - [Features](#sparkles-features)
 - [Roadmap](#mag_right-roadmap)
@@ -116,7 +62,7 @@ https://github.com/user-attachments/assets/8ca00873-5df2-48a7-b853-89e7b18bc6e9
 - [Installation](#wrench-installation)
 - [Quick Start](#rocket-quick-start)
 - [Components](#gear-components)
-- [FlashRAG-UI](#computer-flashrag-ui)
+- [FlashRAG-UI](#art-flashrag-ui)
 - [Supporting Methods](#robot-supporting-methods)
 - [Supporting Datasets & Document Corpus](#notebook-supporting-datasets--document-corpus)
 - [Additional FAQs](#raised_hands-additional-faqs)
@@ -554,9 +500,44 @@ In each category, we have implemented corresponding common pipelines. Some pipel
     </tbody>
 </table>
 
-## :computer: FlashRAG-UI
-To use FlashRAG-UI, run the following code:
+## :art: FlashRAG-UI
+<p>With <strong>FlashRAG-UI</strong>, you can easily and quickly configure and experience the supported RAG methods through our meticulously designed visual interface, and evaluate these methods on benchmarks, making complex research work more efficient!</p>
+
+### :star2: Features
+- **One-Click Configuration Loading**
+  - You can load parameters and configuration files for various RAG methods through simple clicks, selections, and inputs.</li>
+  - Supports preview interface for intuitive parameter settings.</li>
+  - Provides save functionality to easily store configurations for future use.</li>
+- **Quick Method Experience**
+  - Quickly load corpora and index files to explore the characteristics and application scenarios of various RAG methods.</li>
+  - Supports loading and switching different components and hyperparameters, seamlessly connecting different RAG Pipelines to quickly experience their performance and differences!</li>
+- **Efficient Benchmark Reproduction**
+  - Easily reproduce the built-in baseline methods and carefully collected benchmarks on FlashRAG-UI.</li>
+  - Use cutting-edge research tools directly without complex settings, providing a smooth experience for your research work!</li>
+  
+<details>
+<summary>Show more</summary>
+<table align="center">
+  <tr>
+    <td align="center">
+      <img src="./asset/demo_en1.jpg" alt="Image 1" width="505"/>
+    </td>
+    <td align="center">
+      <img src="./asset/demo_en2.jpg" alt="Image 2" width="505"/>
+    </td>
+  </tr>
+  <tr>
+    <td align="center">
+      <img src="./asset/demo_en4.png" alt="Image 3" width="500"/>
+    </td>
+    <td align="center">
+      <img src="./asset/demo_en3.jpg" alt="Image 4" width="500"/>
+    </td>
+  </tr>
+</table>
+</details>
 
+#### Experience our meticulously designed FlashRAG-UI—both user-friendly and visually appealing:
 ```bash
 cd webui
 python interface.py

{flashrag_dev-0.1.4.dev20250112 → flashrag_dev-0.1.4.dev20250113}/flashrag_dev.egg-info/requires.txt

@@ -2,6 +2,7 @@ datasets
 base58
 nltk
 numpy
+langid
 openai
 peft
 PyYAML
@@ -23,6 +24,7 @@ datasets
 base58
 nltk
 numpy
+langid
 openai
 peft
 PyYAML
@@ -44,6 +46,7 @@ datasets
 base58
 nltk
 numpy
+langid
 openai
 peft
 PyYAML

flashrag_dev-0.1.4.dev20250112/flashrag/version.py

@@ -1 +0,0 @@
-__version__ = "0.1.4dev20250112"