flashrag-dev 0.1.4.dev20250310__tar.gz → 0.1.4.dev20250321__tar.gz

{flashrag_dev-0.1.4.dev20250310 → flashrag_dev-0.1.4.dev20250321}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: flashrag_dev
-Version: 0.1.4.dev20250310
+Version: 0.1.4.dev20250321
 Summary: A library for efficient Retrieval-Augmented Generation research
 Home-page: https://github.com/RUC-NLPIR/FlashRAG
 Author: Jiajie Jin, Yutao Zhu, Chenghao Zhang, Xinyu Yang, Zhicheng Dou
@@ -41,7 +41,7 @@ License-File: LICENSE
 </h4>
 
 
-FlashRAG is a Python toolkit for the reproduction and development of Retrieval Augmented Generation (RAG) research. Our toolkit includes 36 pre-processed benchmark RAG datasets and 16 state-of-the-art RAG algorithms.
+FlashRAG is a Python toolkit for the reproduction and development of Retrieval Augmented Generation (RAG) research. Our toolkit includes 36 pre-processed benchmark RAG datasets and 17 state-of-the-art RAG algorithms.
 
 <p align="center">
 <img src="asset/framework.jpg">
@@ -75,7 +75,7 @@ https://github.com/user-attachments/assets/8ca00873-5df2-48a7-b853-89e7b18bc6e9
 
 - **Comprehensive Benchmark Datasets**: A collection of 36 pre-processed RAG benchmark datasets to test and validate RAG models' performances.
 
-- **Pre-implemented Advanced RAG Algorithms**: Features 16 advancing RAG algorithms with reported results, based on our framework. Easily reproducing results under different settings.
+- **Pre-implemented Advanced RAG Algorithms**: Features 17 advancing RAG algorithms with reported results, based on our framework. Easily reproducing results under different settings.
 
 - **Efficient Preprocessing Stage**: Simplifies the RAG workflow preparation by providing various scripts like corpus processing for retrieval, retrieval index building, and pre-retrieval of documents.
 
@@ -96,17 +96,19 @@ FlashRAG is still under development and there are many issues and room for impro
 - [ ] Enhance code adaptability and readability
 
 ## :page_with_curl: Changelog
+[25/03/21] We have added support for **Reasoning Pipeline**, which is a new paradigm that combines reasoning ability and retrieval, representing work that includes [Search-o1](https://github.com/sunnynexus/Search-o1), [R1-Searcher](https://github.com/SsmallSong/R1-Searcher), [ReSearch](https://github.com/Agent-RL/ReSearch). We evaluate the performance of the pipeline on various RAG benchmarks, it can achieve F1 scores close to 60 on multi hop inference datasets such as Hotpotqe. See it in [**result table**](#robot-supporting-methods).
+
 [25/02/24] 🔥🔥🔥 We have added support for **multimodal RAG**, including [**MLLMs like Llava, Qwen, InternVL**](https://ruc-nlpir.github.io/FlashRAG/#/zh-cn/component/generator?id=%e5%a4%9a%e6%a8%a1%e6%80%81%e7%94%9f%e6%88%90%e5%99%a8), and various [**multimodal retrievers with Clip architecture**](https://ruc-nlpir.github.io/FlashRAG/#/zh-cn/component/retriever?id=%e5%a4%9a%e6%a8%a1%e6%80%81%e6%a3%80%e7%b4%a2%e5%99%a8). More information can be found in our new version of arxiv article and our documentation. Try it!
 
 [25/01/21] Our technical paper [FlashRAG: A Python Toolkit for Efficient RAG Research](https://arxiv.org/abs/2405.13576) is honored to have been accepted to the Resource Track of the 2025 **ACM Web Conference (WWW 2025)**. Please Check it out!
 
 [25/01/12] Introduce <strong>FlashRAG-UI</strong>, an easy to use interface. You can easily and quickly configure and experience the supported RAG methods and evaluate them on the benchmarks.
 
-[25/01/11] We have added support for a new method [<u>RQRAG</u>](https://arxiv.org/abs/2404.00610) method, see it in [**reproduce_experiment**](docs/reproduce_experiment.md).
+[25/01/11] We have added support for a new method [<u>RQRAG</u>](https://arxiv.org/abs/2404.00610) method, see it in [**reproduce_experiment**](docs/original_docs/reproduce_experiment.md).
 
-[25/01/07] We have currently support the aggregation of multiple retrievers, see it in [**multi retriever usage**](https://github.com/RUC-NLPIR/FlashRAG/blob/main/docs/multi_retriever_usage.md).
+[25/01/07] We have currently support the aggregation of multiple retrievers, see it in [**multi retriever usage**](https://github.com/RUC-NLPIR/FlashRAG/blob/main/docs/original_docs/multi_retriever_usage.md).
 
-[25/01/07] We have integrated a very flexible and lightweight corpus chunking library [**Chunkie**](https://github.com/chonkie-ai/chonkie?tab=readme-ov-file#usage), which supports various custom chunking methods (tokens, sentences, semantic, etc.). Use it in [<u>chunking doc corpus</u>](docs/chunk-doc-corpus.md).
+[25/01/07] We have integrated a very flexible and lightweight corpus chunking library [**Chunkie**](https://github.com/chonkie-ai/chonkie?tab=readme-ov-file#usage), which supports various custom chunking methods (tokens, sentences, semantic, etc.). Use it in [<u>chunking doc corpus</u>](docs/original_docs/chunk-doc-corpus.md).
 
 [24/10/21] We have released a version based on the Paddle framework that supports Chinese hardware platforms. Please refer to [FlashRAG Paddle](https://github.com/RUC-NLPIR/FlashRAG-Paddle) for details.
 
@@ -125,7 +127,7 @@ FlashRAG is still under development and there are many issues and room for impro
 
 [24/07/17] Due to some unknown issues with HuggingFace, our original dataset link has been invalid. We have updated it. Please check the [new link](https://huggingface.co/datasets/RUC-NLPIR/FlashRAG_datasets/) if you encounter any problems.
 
-[24/07/06] We add support for a new method: [<u>Trace</u>](https://arxiv.org/abs/2406.11460), which refine text by constructing a knowledge graph. See it [<u>results</u>](#robot-supporting-methods) and [<u>details</u>](./docs/baseline_details.md).
+[24/07/06] We add support for a new method: [<u>Trace</u>](https://arxiv.org/abs/2406.11460), which refine text by constructing a knowledge graph. See it [<u>results</u>](#robot-supporting-methods) and [<u>details</u>](./docs/original_docs/baseline_details.md).
 
 [24/06/19] We add support for a new method: [<u>IRCoT</u>](https://arxiv.org/abs/2212.10509), and update the [<u>result table</u>](#robot-supporting-methods).
 
@@ -133,9 +135,9 @@ FlashRAG is still under development and there are many issues and room for impro
 
 [24/06/11] We have integrated `sentence transformers` in the retriever module. Now it's easier to use the retriever without setting pooling methods.
 
-[24/06/05] We have provided detailed document for reproducing existing methods (see [how to reproduce](./docs/reproduce_experiment.md), [baseline details](./docs/baseline_details.md)), and [<u>configurations settings</u>](./docs/configuration.md).
+[24/06/05] We have provided detailed document for reproducing existing methods (see [how to reproduce](./docs/original_docs/reproduce_experiment.md), [baseline details](./docs/original_docs/baseline_details.md)), and [<u>configurations settings</u>](./docs/original_docs/configuration.md).
 
-[24/06/02] We have provided an introduction of FlashRAG for beginners, see [<u>an introduction to flashrag</u>](./docs/introduction_for_beginners_en.md) ([<u>中文版</u>](./docs/introduction_for_beginners_zh.md) [<u>한국어</u>](./docs/introduction_for_beginners_kr.md)).
+[24/06/02] We have provided an introduction of FlashRAG for beginners, see [<u>an introduction to flashrag</u>](./docs/original_docs/introduction_for_beginners_en.md) ([<u>中文版</u>](./docs/original_docs/introduction_for_beginners_zh.md) [<u>한국어</u>](./docs/original_docs/introduction_for_beginners_kr.md)).
 
 [24/05/31] We supported Openai-series models as generator.
 
@@ -200,7 +202,7 @@ To build an index, you first need to save your corpus as a `jsonl` file with eac
 {"id": "1", "contents": "..."}
 ```
 
-If you want to use Wikipedia as your corpus, you can refer to our documentation [Processing Wikipedia](./docs/process-wiki.md) to convert it into an indexable format.
+If you want to use Wikipedia as your corpus, you can refer to our documentation [Processing Wikipedia](./docs/original_docs/process-wiki.md) to convert it into an indexable format.
 
 ### Index Construction
 
@@ -289,7 +291,7 @@ my_config = Config(
     config_dict = config_dict
 ```
 
-We provide comprehensive guidance on how to set configurations, you can see our [<u>configuration guidance</u>](./docs/configuration.md).
+We provide comprehensive guidance on how to set configurations, you can see our [<u>configuration guidance</u>](./docs/original_docs/configuration.md).
 You can also refer to the [<u>basic yaml file</u>](./flashrag/config/basic_config.yaml) we provide to set your own parameters.
 
 Next, load the corresponding dataset and initialize the pipeline. The components in the pipeline will be automatically loaded.
@@ -360,11 +362,11 @@ class ToyPipeline(BasicPipeline):
     return dataset
 ```
 
-Please first understand the input and output forms of the components you need to use from our [<u>documentation</u>](./docs/basic_usage.md).
+Please first understand the input and output forms of the components you need to use from our [<u>documentation</u>](./docs/original_docs/basic_usage.md).
 
 ### Just use components
 
-If you already have your own code and only want to use our components to embed the original code, you can refer to the [<u>basic introduction of the components</u>](./docs/basic_usage.md) to obtain the input and output formats of each component.
+If you already have your own code and only want to use our components to embed the original code, you can refer to the [<u>basic introduction of the components</u>](./docs/original_docs/basic_usage.md) to obtain the input and output formats of each component.
 
 ## :gear: Components
 
@@ -482,7 +484,7 @@ In each category, we have implemented corresponding common pipelines. Some pipel
           <td>Ranking and merging generated results based on each document</td>
         </tr>
         <tr>
-            <td rowspan="5">Loop</td>
+            <td rowspan="6">Loop</td>
             <td>Iterative Pipeline</td>
             <td>Alternating retrieval and generation</td>
         </tr>
@@ -502,6 +504,10 @@ In each category, we have implemented corresponding common pipelines. Some pipel
             <td>IRCoT Pipeline</td>
             <td>Integrate retrieval process with CoT</td>
         </tr>
+        <tr>
+            <td>Reasoning Pipeline</td>
+            <td>Reasoning with retrieval</td>
+        </tr>
     </tbody>
 </table>
 
@@ -554,11 +560,11 @@ We have implemented 15 works with a consistent setting of:
 
 - **Generator:** LLAMA3-8B-instruct with input length of 2048
 - **Retriever:** e5-base-v2 as embedding model, retrieve 5 docs per query
-- **Prompt:** A consistent default prompt, template can be found in the [<u>method details</u>](./docs/baseline_details.md).
+- **Prompt:** A consistent default prompt, template can be found in the [<u>method details</u>](./docs/original_docs/baseline_details.md).
 
 For open-source methods, we implemented their processes using our framework. For methods where the author did not provide source code, we will try our best to follow the methods in the original paper for implementation.
 
-For necessary settings and hyperparameters specific to some methods, we have documented them in the **specific settings** column. For more details, please consult our [<u>reproduce guidance</u>](./docs/reproduce_experiment.md) and [<u>method details</u>](./docs/baseline_details.md).
+For necessary settings and hyperparameters specific to some methods, we have documented them in the **specific settings** column. For more details, please consult our [<u>reproduce guidance</u>](./docs/original_docs/reproduce_experiment.md) and [<u>method details</u>](./docs/original_docs/baseline_details.md).
 
 It’s important to note that, to ensure consistency, we have utilized a uniform setting. However, this setting may differ from the original setting of the method, leading to variations in results compared to the original outcomes.
 
@@ -582,6 +588,8 @@ It’s important to note that, to ensure consistency, we have utilized a uniform
 | [Iter-Retgen](https://arxiv.org/abs/2305.15294), [ITRG](https://arxiv.org/abs/2310.05149) | Loop        | 36.8    | 60.1          | 38.3          | 21.6       | 37.9       | 18.2      |                                                 |
 | [IRCoT](https://aclanthology.org/2023.acl-long.557.pdf)                                   | Loop        | 33.3    | 56.9          | 41.5          | 32.4       | 45.6       | 20.7      |                                                 |
 | [RQRAG](https://arxiv.org/abs/2404.00610)                                   | Loop        | 32.6    | 52.5          | 33.5          | 35.8       | 46.4       | 26.2      |  Use trained rqrag-llama2-7B                                               |
+| [R1-Searcher](https://arxiv.org/pdf/2503.05592) | Reasoning | 37.3 | 58.7 | 59.5 | 55.5 | 43.9 | 27.5 | Using trained Qwen2.5-7B model |
+ 
 
 
 ## :notebook: Supporting Datasets & Document Corpus
@@ -653,7 +661,7 @@ Our toolkit supports jsonl format for retrieval document collections, with the f
 
 The `contents` key is essential for building the index. For documents that include both text and title, we recommend setting the value of `contents` to `{title}\n{text}`. The corpus file can also contain other keys to record additional characteristics of the documents.
 
-In the academic research, Wikipedia and MS MARCO are the most commonly used retrieval document collections. For Wikipedia, we provide a [<u>comprehensive script</u>](./docs/process-wiki.md) to process any Wikipedia dump into a clean corpus. Additionally, various processed versions of the Wikipedia corpus are available in many works, and we have listed some reference links.
+In the academic research, Wikipedia and MS MARCO are the most commonly used retrieval document collections. For Wikipedia, we provide a [<u>comprehensive script</u>](./docs/original_docs/process-wiki.md) to process any Wikipedia dump into a clean corpus. Additionally, various processed versions of the Wikipedia corpus are available in many works, and we have listed some reference links.
 
 For MS MARCO, it is already processed upon release and can be directly downloaded from its [<u>hosting link</u>](https://huggingface.co/datasets/Tevatron/msmarco-passage-corpus) on Hugging Face.
 
@@ -671,10 +679,10 @@ The index was created using the e5-base-v2 retriever on our uploaded wiki18_100w
 
 ## :raised_hands: Additional FAQs
 
-- [How should I set different experimental parameters?](./docs/configuration.md)
-- [How to build my own corpus, such as a specific segmented Wikipedia?](./docs/process-wiki.md)
-- [How to index my own corpus?](./docs/building-index.md)
-- [How to reproduce supporting methods?](./docs/reproduce_experiment.md)
+- [How should I set different experimental parameters?](./docs/original_docs/configuration.md)
+- [How to build my own corpus, such as a specific segmented Wikipedia?](./docs/original_docs/process-wiki.md)
+- [How to index my own corpus?](./docs/original_docs/building-index.md)
+- [How to reproduce supporting methods?](./docs/original_docs/reproduce_experiment.md)
 
 ## :bookmark: License
 

{flashrag_dev-0.1.4.dev20250310 → flashrag_dev-0.1.4.dev20250321}/README.md

@@ -24,7 +24,7 @@
 </h4>
 
 
-FlashRAG is a Python toolkit for the reproduction and development of Retrieval Augmented Generation (RAG) research. Our toolkit includes 36 pre-processed benchmark RAG datasets and 16 state-of-the-art RAG algorithms.
+FlashRAG is a Python toolkit for the reproduction and development of Retrieval Augmented Generation (RAG) research. Our toolkit includes 36 pre-processed benchmark RAG datasets and 17 state-of-the-art RAG algorithms.
 
 <p align="center">
 <img src="asset/framework.jpg">
@@ -58,7 +58,7 @@ https://github.com/user-attachments/assets/8ca00873-5df2-48a7-b853-89e7b18bc6e9
 
 - **Comprehensive Benchmark Datasets**: A collection of 36 pre-processed RAG benchmark datasets to test and validate RAG models' performances.
 
-- **Pre-implemented Advanced RAG Algorithms**: Features 16 advancing RAG algorithms with reported results, based on our framework. Easily reproducing results under different settings.
+- **Pre-implemented Advanced RAG Algorithms**: Features 17 advancing RAG algorithms with reported results, based on our framework. Easily reproducing results under different settings.
 
 - **Efficient Preprocessing Stage**: Simplifies the RAG workflow preparation by providing various scripts like corpus processing for retrieval, retrieval index building, and pre-retrieval of documents.
 
@@ -79,17 +79,19 @@ FlashRAG is still under development and there are many issues and room for impro
 - [ ] Enhance code adaptability and readability
 
 ## :page_with_curl: Changelog
+[25/03/21] We have added support for **Reasoning Pipeline**, which is a new paradigm that combines reasoning ability and retrieval, representing work that includes [Search-o1](https://github.com/sunnynexus/Search-o1), [R1-Searcher](https://github.com/SsmallSong/R1-Searcher), [ReSearch](https://github.com/Agent-RL/ReSearch). We evaluate the performance of the pipeline on various RAG benchmarks, it can achieve F1 scores close to 60 on multi hop inference datasets such as Hotpotqe. See it in [**result table**](#robot-supporting-methods).
+
 [25/02/24] 🔥🔥🔥 We have added support for **multimodal RAG**, including [**MLLMs like Llava, Qwen, InternVL**](https://ruc-nlpir.github.io/FlashRAG/#/zh-cn/component/generator?id=%e5%a4%9a%e6%a8%a1%e6%80%81%e7%94%9f%e6%88%90%e5%99%a8), and various [**multimodal retrievers with Clip architecture**](https://ruc-nlpir.github.io/FlashRAG/#/zh-cn/component/retriever?id=%e5%a4%9a%e6%a8%a1%e6%80%81%e6%a3%80%e7%b4%a2%e5%99%a8). More information can be found in our new version of arxiv article and our documentation. Try it!
 
 [25/01/21] Our technical paper [FlashRAG: A Python Toolkit for Efficient RAG Research](https://arxiv.org/abs/2405.13576) is honored to have been accepted to the Resource Track of the 2025 **ACM Web Conference (WWW 2025)**. Please Check it out!
 
 [25/01/12] Introduce <strong>FlashRAG-UI</strong>, an easy to use interface. You can easily and quickly configure and experience the supported RAG methods and evaluate them on the benchmarks.
 
-[25/01/11] We have added support for a new method [<u>RQRAG</u>](https://arxiv.org/abs/2404.00610) method, see it in [**reproduce_experiment**](docs/reproduce_experiment.md).
+[25/01/11] We have added support for a new method [<u>RQRAG</u>](https://arxiv.org/abs/2404.00610) method, see it in [**reproduce_experiment**](docs/original_docs/reproduce_experiment.md).
 
-[25/01/07] We have currently support the aggregation of multiple retrievers, see it in [**multi retriever usage**](https://github.com/RUC-NLPIR/FlashRAG/blob/main/docs/multi_retriever_usage.md).
+[25/01/07] We have currently support the aggregation of multiple retrievers, see it in [**multi retriever usage**](https://github.com/RUC-NLPIR/FlashRAG/blob/main/docs/original_docs/multi_retriever_usage.md).
 
-[25/01/07] We have integrated a very flexible and lightweight corpus chunking library [**Chunkie**](https://github.com/chonkie-ai/chonkie?tab=readme-ov-file#usage), which supports various custom chunking methods (tokens, sentences, semantic, etc.). Use it in [<u>chunking doc corpus</u>](docs/chunk-doc-corpus.md).
+[25/01/07] We have integrated a very flexible and lightweight corpus chunking library [**Chunkie**](https://github.com/chonkie-ai/chonkie?tab=readme-ov-file#usage), which supports various custom chunking methods (tokens, sentences, semantic, etc.). Use it in [<u>chunking doc corpus</u>](docs/original_docs/chunk-doc-corpus.md).
 
 [24/10/21] We have released a version based on the Paddle framework that supports Chinese hardware platforms. Please refer to [FlashRAG Paddle](https://github.com/RUC-NLPIR/FlashRAG-Paddle) for details.
 
@@ -108,7 +110,7 @@ FlashRAG is still under development and there are many issues and room for impro
 
 [24/07/17] Due to some unknown issues with HuggingFace, our original dataset link has been invalid. We have updated it. Please check the [new link](https://huggingface.co/datasets/RUC-NLPIR/FlashRAG_datasets/) if you encounter any problems.
 
-[24/07/06] We add support for a new method: [<u>Trace</u>](https://arxiv.org/abs/2406.11460), which refine text by constructing a knowledge graph. See it [<u>results</u>](#robot-supporting-methods) and [<u>details</u>](./docs/baseline_details.md).
+[24/07/06] We add support for a new method: [<u>Trace</u>](https://arxiv.org/abs/2406.11460), which refine text by constructing a knowledge graph. See it [<u>results</u>](#robot-supporting-methods) and [<u>details</u>](./docs/original_docs/baseline_details.md).
 
 [24/06/19] We add support for a new method: [<u>IRCoT</u>](https://arxiv.org/abs/2212.10509), and update the [<u>result table</u>](#robot-supporting-methods).
 
@@ -116,9 +118,9 @@ FlashRAG is still under development and there are many issues and room for impro
 
 [24/06/11] We have integrated `sentence transformers` in the retriever module. Now it's easier to use the retriever without setting pooling methods.
 
-[24/06/05] We have provided detailed document for reproducing existing methods (see [how to reproduce](./docs/reproduce_experiment.md), [baseline details](./docs/baseline_details.md)), and [<u>configurations settings</u>](./docs/configuration.md).
+[24/06/05] We have provided detailed document for reproducing existing methods (see [how to reproduce](./docs/original_docs/reproduce_experiment.md), [baseline details](./docs/original_docs/baseline_details.md)), and [<u>configurations settings</u>](./docs/original_docs/configuration.md).
 
-[24/06/02] We have provided an introduction of FlashRAG for beginners, see [<u>an introduction to flashrag</u>](./docs/introduction_for_beginners_en.md) ([<u>中文版</u>](./docs/introduction_for_beginners_zh.md) [<u>한국어</u>](./docs/introduction_for_beginners_kr.md)).
+[24/06/02] We have provided an introduction of FlashRAG for beginners, see [<u>an introduction to flashrag</u>](./docs/original_docs/introduction_for_beginners_en.md) ([<u>中文版</u>](./docs/original_docs/introduction_for_beginners_zh.md) [<u>한국어</u>](./docs/original_docs/introduction_for_beginners_kr.md)).
 
 [24/05/31] We supported Openai-series models as generator.
 
@@ -183,7 +185,7 @@ To build an index, you first need to save your corpus as a `jsonl` file with eac
 {"id": "1", "contents": "..."}
 ```
 
-If you want to use Wikipedia as your corpus, you can refer to our documentation [Processing Wikipedia](./docs/process-wiki.md) to convert it into an indexable format.
+If you want to use Wikipedia as your corpus, you can refer to our documentation [Processing Wikipedia](./docs/original_docs/process-wiki.md) to convert it into an indexable format.
 
 ### Index Construction
 
@@ -272,7 +274,7 @@ my_config = Config(
     config_dict = config_dict
 ```
 
-We provide comprehensive guidance on how to set configurations, you can see our [<u>configuration guidance</u>](./docs/configuration.md).
+We provide comprehensive guidance on how to set configurations, you can see our [<u>configuration guidance</u>](./docs/original_docs/configuration.md).
 You can also refer to the [<u>basic yaml file</u>](./flashrag/config/basic_config.yaml) we provide to set your own parameters.
 
 Next, load the corresponding dataset and initialize the pipeline. The components in the pipeline will be automatically loaded.
@@ -343,11 +345,11 @@ class ToyPipeline(BasicPipeline):
     return dataset
 ```
 
-Please first understand the input and output forms of the components you need to use from our [<u>documentation</u>](./docs/basic_usage.md).
+Please first understand the input and output forms of the components you need to use from our [<u>documentation</u>](./docs/original_docs/basic_usage.md).
 
 ### Just use components
 
-If you already have your own code and only want to use our components to embed the original code, you can refer to the [<u>basic introduction of the components</u>](./docs/basic_usage.md) to obtain the input and output formats of each component.
+If you already have your own code and only want to use our components to embed the original code, you can refer to the [<u>basic introduction of the components</u>](./docs/original_docs/basic_usage.md) to obtain the input and output formats of each component.
 
 ## :gear: Components
 
@@ -465,7 +467,7 @@ In each category, we have implemented corresponding common pipelines. Some pipel
           <td>Ranking and merging generated results based on each document</td>
         </tr>
         <tr>
-            <td rowspan="5">Loop</td>
+            <td rowspan="6">Loop</td>
             <td>Iterative Pipeline</td>
             <td>Alternating retrieval and generation</td>
         </tr>
@@ -485,6 +487,10 @@ In each category, we have implemented corresponding common pipelines. Some pipel
             <td>IRCoT Pipeline</td>
             <td>Integrate retrieval process with CoT</td>
         </tr>
+        <tr>
+            <td>Reasoning Pipeline</td>
+            <td>Reasoning with retrieval</td>
+        </tr>
     </tbody>
 </table>
 
@@ -537,11 +543,11 @@ We have implemented 15 works with a consistent setting of:
 
 - **Generator:** LLAMA3-8B-instruct with input length of 2048
 - **Retriever:** e5-base-v2 as embedding model, retrieve 5 docs per query
-- **Prompt:** A consistent default prompt, template can be found in the [<u>method details</u>](./docs/baseline_details.md).
+- **Prompt:** A consistent default prompt, template can be found in the [<u>method details</u>](./docs/original_docs/baseline_details.md).
 
 For open-source methods, we implemented their processes using our framework. For methods where the author did not provide source code, we will try our best to follow the methods in the original paper for implementation.
 
-For necessary settings and hyperparameters specific to some methods, we have documented them in the **specific settings** column. For more details, please consult our [<u>reproduce guidance</u>](./docs/reproduce_experiment.md) and [<u>method details</u>](./docs/baseline_details.md).
+For necessary settings and hyperparameters specific to some methods, we have documented them in the **specific settings** column. For more details, please consult our [<u>reproduce guidance</u>](./docs/original_docs/reproduce_experiment.md) and [<u>method details</u>](./docs/original_docs/baseline_details.md).
 
 It’s important to note that, to ensure consistency, we have utilized a uniform setting. However, this setting may differ from the original setting of the method, leading to variations in results compared to the original outcomes.
 
@@ -565,6 +571,8 @@ It’s important to note that, to ensure consistency, we have utilized a uniform
 | [Iter-Retgen](https://arxiv.org/abs/2305.15294), [ITRG](https://arxiv.org/abs/2310.05149) | Loop        | 36.8    | 60.1          | 38.3          | 21.6       | 37.9       | 18.2      |                                                 |
 | [IRCoT](https://aclanthology.org/2023.acl-long.557.pdf)                                   | Loop        | 33.3    | 56.9          | 41.5          | 32.4       | 45.6       | 20.7      |                                                 |
 | [RQRAG](https://arxiv.org/abs/2404.00610)                                   | Loop        | 32.6    | 52.5          | 33.5          | 35.8       | 46.4       | 26.2      |  Use trained rqrag-llama2-7B                                               |
+| [R1-Searcher](https://arxiv.org/pdf/2503.05592) | Reasoning | 37.3 | 58.7 | 59.5 | 55.5 | 43.9 | 27.5 | Using trained Qwen2.5-7B model |
+ 
 
 
 ## :notebook: Supporting Datasets & Document Corpus
@@ -636,7 +644,7 @@ Our toolkit supports jsonl format for retrieval document collections, with the f
 
 The `contents` key is essential for building the index. For documents that include both text and title, we recommend setting the value of `contents` to `{title}\n{text}`. The corpus file can also contain other keys to record additional characteristics of the documents.
 
-In the academic research, Wikipedia and MS MARCO are the most commonly used retrieval document collections. For Wikipedia, we provide a [<u>comprehensive script</u>](./docs/process-wiki.md) to process any Wikipedia dump into a clean corpus. Additionally, various processed versions of the Wikipedia corpus are available in many works, and we have listed some reference links.
+In the academic research, Wikipedia and MS MARCO are the most commonly used retrieval document collections. For Wikipedia, we provide a [<u>comprehensive script</u>](./docs/original_docs/process-wiki.md) to process any Wikipedia dump into a clean corpus. Additionally, various processed versions of the Wikipedia corpus are available in many works, and we have listed some reference links.
 
 For MS MARCO, it is already processed upon release and can be directly downloaded from its [<u>hosting link</u>](https://huggingface.co/datasets/Tevatron/msmarco-passage-corpus) on Hugging Face.
 
@@ -654,10 +662,10 @@ The index was created using the e5-base-v2 retriever on our uploaded wiki18_100w
 
 ## :raised_hands: Additional FAQs
 
-- [How should I set different experimental parameters?](./docs/configuration.md)
-- [How to build my own corpus, such as a specific segmented Wikipedia?](./docs/process-wiki.md)
-- [How to index my own corpus?](./docs/building-index.md)
-- [How to reproduce supporting methods?](./docs/reproduce_experiment.md)
+- [How should I set different experimental parameters?](./docs/original_docs/configuration.md)
+- [How to build my own corpus, such as a specific segmented Wikipedia?](./docs/original_docs/process-wiki.md)
+- [How to index my own corpus?](./docs/original_docs/building-index.md)
+- [How to reproduce supporting methods?](./docs/original_docs/reproduce_experiment.md)
 
 ## :bookmark: License
 

{flashrag_dev-0.1.4.dev20250310 → flashrag_dev-0.1.4.dev20250321}/flashrag/config/basic_config.yaml

@@ -66,6 +66,8 @@ retrieval_cache_path: ~ # path to the retrieval cache
 retrieval_pooling_method: ~ # set automatically if not provided
 bm25_backend: bm25s # pyserini, bm25s
 use_sentence_transformer: False
+silent_retrieval: True # whether to silent the retrieval process
+
 
 use_reranker: False # whether to use reranker
 rerank_model_name: ~ # same as retrieval_method

{flashrag_dev-0.1.4.dev20250310 → flashrag_dev-0.1.4.dev20250321}/flashrag/dataset/dataset.py

@@ -48,6 +48,13 @@ class Item:
                 except AttributeError:
                     raise AttributeError(f"Attribute `{attr_name}` not found")
 
+    def __setattr__(self, attr_name: str, value: Any) -> None:
+        predefined_attrs = ["id", "question", "golden_answers", "metadata", "output", "choices", 'data']
+        if attr_name in predefined_attrs:
+            super().__setattr__(attr_name, value)
+        else:
+            self.update_output(attr_name, value)
+
     def to_dict(self) -> Dict[str, Any]:
         """Convert all information within the data sample into a dict. Information generated
         during the inference will be saved into output field.

{flashrag_dev-0.1.4.dev20250310 → flashrag_dev-0.1.4.dev20250321}/flashrag/generator/openai_generator.py

@@ -35,7 +35,7 @@ class OpenaiGenerator:
         try:
             self.tokenizer = tiktoken.encoding_for_model(self.model_name)
         except Exception as e:
-            print("Error: ", e)
+            print("Warning: ", e)
             warnings.warn("This model is not supported by tiktoken. Use gpt-3.5-turbo instead.")
             self.tokenizer = tiktoken.encoding_for_model('gpt-3.5-turbo')
     @property

{flashrag_dev-0.1.4.dev20250310 → flashrag_dev-0.1.4.dev20250321}/flashrag/pipeline/__init__.py

@@ -1,4 +1,5 @@
 from flashrag.pipeline.mm_pipeline import *
 from flashrag.pipeline.pipeline import *
 from flashrag.pipeline.branching_pipeline import REPLUGPipeline, SuRePipeline
-from flashrag.pipeline.active_pipeline import IterativePipeline, SelfRAGPipeline, FLAREPipeline, SelfAskPipeline, IRCOTPipeline, RQRAGPipeline
+from flashrag.pipeline.active_pipeline import IterativePipeline, SelfRAGPipeline, FLAREPipeline, SelfAskPipeline, IRCOTPipeline, RQRAGPipeline
+from flashrag.pipeline.reasoning_pipeline import ReasoningPipeline

flashrag_dev-0.1.4.dev20250321/flashrag/pipeline/reasoning_pipeline.py

@@ -0,0 +1,154 @@
+import re
+from tqdm import tqdm
+from typing import List, Tuple, Dict
+import math
+import numpy as np
+import copy
+from transformers import AutoTokenizer, PreTrainedTokenizer, PreTrainedTokenizerFast
+from flashrag.utils import get_retriever, get_generator, selfask_pred_parse, ircot_pred_parse
+from flashrag.pipeline import BasicPipeline
+from flashrag.dataset.utils import get_batch_dataset, merge_batch_dataset
+from flashrag.prompt import PromptTemplate
+from flashrag.utils.utils import extract_between
+
+class ReasoningPipeline(BasicPipeline):
+    system_prompt=""
+    user_prompt=(
+        "The User asks a question, and the Assistant solves it.\n"
+        "The Assistant first thinks about the reasoning process in the mind and then provides the User with the final answer.\n"
+        "The output format of reasoning process and final answer are enclosed within <think> </think> and <answer> </answer> tags, "
+        "respectively, i.e., \"<think> reasoning process here </think>\\n\\n<answer> final answer here </answer>\".\n"
+        "During the thinking process, the Assistant can perform searching for uncertain knowledge if necessary with the format of "
+        "\"<|begin_of_query|> search query (only keywords) here <|end_of_query|>\". **A query must involve only a single triple**.\n"
+        "Then, the system will provide the Assistant with helpful information with the format of "
+        "\"<|begin_of_documents|> ...search results... <|end_of_documents|>\".\n\n"
+        "User:{question}\n"
+        "Assistant: <think>"
+    )
+
+
+    def __init__(self, config, 
+        prompt_template=None, 
+        max_retrieval_num=5, 
+        begin_of_query_token="<|begin_of_query|>",
+        end_of_query_token="<|end_of_query|>",
+        begin_of_documents_token="<|begin_of_documents|>",
+        end_of_documents_token="<|end_of_documents|>",
+        begin_of_answer_token="<answer>",
+        end_of_answer_token='</answer>',
+        retriever=None, 
+        generator=None
+    ):
+        if prompt_template is None:
+            prompt_template = PromptTemplate(
+                config=config,
+                system_prompt=self.system_prompt,
+                user_prompt=self.user_prompt
+            )
+        super().__init__(config, prompt_template)
+
+        if generator is None:
+            self.generator = get_generator(config)
+        else:
+            self.generator = generator
+        if retriever is None:
+            self.retriever = get_retriever(config)
+        else:
+            self.retriever = retriever
+
+        self.max_retrieval_num = max_retrieval_num
+
+        self.begin_of_query_token = begin_of_query_token
+        self.end_of_query_token = end_of_query_token
+        self.begin_of_documents_token = begin_of_documents_token
+        self.end_of_documents_token = end_of_documents_token
+        self.begin_of_answer_token = begin_of_answer_token
+        self.end_of_answer_token = end_of_answer_token
+
+        self.stop_tokens = ["<|im_end|>", "<|endoftext|>", self.end_of_answer_token, self.end_of_query_token]
+    
+    def _retrieved_docs_to_string(self, retrieved_docs: List[Dict]):
+        format_doc_string = ""
+        for idx, doc in enumerate(retrieved_docs):
+            contents = doc['contents']
+            title = contents.split('\n')[0]
+            text = '\n'.join(contents.split('\n')[1:])
+            doc_string = f"Title: {title} Text: {text}"
+            doc_string = re.sub(r'^\d+\s+', '', doc_string)
+            format_doc_string += f'({idx+1}){doc_string}\n'
+        format_doc_string = f'\n\n{self.begin_of_documents_token}\n{format_doc_string}\n{self.end_of_documents_token}\n\n'
+        return format_doc_string
+
+    def run(self, dataset, do_eval=True, pred_process_fun=None):
+        prompts = [self.prompt_template.get_string(question=question) for question in dataset.question]
+        dataset.update_output('prompt', prompts)
+        dataset.update_output('finish_flag', [False] * len(prompts))
+        dataset.update_output('retrieval_results', [{}] * len(prompts))
+        dataset.update_output('retrieved_times', [0] * len(prompts))
+
+        # Logic of reasoning
+        for current_step_idx in range(self.max_retrieval_num + 1):
+            exist_items = [item for item in dataset if item.finish_flag == False]
+            exist_prompts = [item.prompt for item in exist_items]
+            
+            print(f"Current step: {current_step_idx}, exist_items: {len(exist_items)}")
+
+            if len(exist_items) == 0:
+                print("All prompts are finished")
+                break
+            if current_step_idx == self.max_retrieval_num:
+                print("Max retrieval number reached")
+                for item in exist_items:
+                    item.pred = 'No valid answer found'
+                    item.finish_flag = True
+                    item.finish_reason = 'Reach max retrieval number'
+                break
+
+            step_outputs = self.generator.generate(exist_prompts, stop=self.stop_tokens)
+            step_query_list = [] # store generated queries for retrieval
+
+            # parse each sample's step output
+            for item, step_output in zip(exist_items, step_outputs):
+                item.prompt = item.prompt + step_output.strip()
+                if self.end_of_answer_token in step_output and step_output.endswith(self.end_of_answer_token):
+                    item.pred = str(extract_between(step_output, self.begin_of_answer_token, self.end_of_answer_token))
+                    item.finish_flag = True
+                    item.finish_reason = "Finished"
+                
+                elif self.begin_of_query_token in step_output and step_output.endswith(self.end_of_query_token):
+                    query = extract_between(step_output, self.begin_of_query_token, self.end_of_query_token)
+                    if query is not None:
+                        step_query_list.append({'item': item, 'query': query})
+                    else:
+                        item.pred = 'No valid answer found'
+                        item.finish_flag = True
+                        item.finish_reason = 'Query instruction error'
+
+                else:
+                    item.pred = step_output.strip()
+                    item.finish_flag = True
+                    item.finish_reason = 'Normal finish without answer pattern'
+                
+            # do retrieval and add retrieved docs to prompt
+            if len(step_query_list) > 0:
+                retrieved_docs = self.retriever.batch_search([it['query'] for it in step_query_list])
+                for it, item_retrieved_docs in zip(step_query_list, retrieved_docs):
+                    item = it['item']
+                    query = it['query']
+                    item.retrieval_results[item.retrieved_times] = {'query': query,'docs': copy.copy(item_retrieved_docs)}
+                    #item.retrieved_docs += [item_retrieved_docs]
+                    format_doc_string = self._retrieved_docs_to_string(item_retrieved_docs)
+                    item.prompt += format_doc_string
+                    item.retrieved_times += 1
+
+        dataset = self.evaluate(dataset, do_eval=do_eval, pred_process_fun=pred_process_fun)
+        return dataset
+
+            
+                
+                        
+
+                
+            
+        
+            

{flashrag_dev-0.1.4.dev20250310 → flashrag_dev-0.1.4.dev20250321}/flashrag/retriever/encoder.py

@@ -24,13 +24,14 @@ class Encoder:
             Encodes a list of queries into embeddings.
     """
 
-    def __init__(self, model_name, model_path, pooling_method, max_length, use_fp16, instruction):
+    def __init__(self, model_name, model_path, pooling_method, max_length, use_fp16, instruction, silent=False):
         self.model_name = model_name
         self.model_path = model_path
         self.pooling_method = pooling_method
         self.max_length = max_length
         self.use_fp16 = use_fp16
         self.instruction = instruction
+        self.silent = silent
         self.gpu_num = torch.cuda.device_count()
         self.model, self.tokenizer = load_model(model_path=model_path, use_fp16=use_fp16)
 
@@ -67,7 +68,7 @@ class Encoder:
     @torch.inference_mode()
     def encode(self, query_list: List[str], batch_size=64, is_query=True) -> np.ndarray:
         query_emb = []
-        for i in tqdm(range(0, len(query_list), batch_size), desc="Encoding process: "):
+        for i in tqdm(range(0, len(query_list), batch_size), desc="Encoding process: ", disable=self.silent):
             query_emb.append(self.single_batch_encode(query_list[i : i + batch_size], is_query))
         query_emb = np.concatenate(query_emb, axis=0)
         return query_emb
@@ -98,7 +99,7 @@ class STEncoder:
             Encodes a list of queries into embeddings using multiple GPUs.
     """
 
-    def __init__(self, model_name, model_path, max_length, use_fp16, instruction):
+    def __init__(self, model_name, model_path, max_length, use_fp16, instruction, silent=False):
         import torch
         from sentence_transformers import SentenceTransformer
 
@@ -107,6 +108,7 @@ class STEncoder:
         self.max_length = max_length
         self.use_fp16 = use_fp16
         self.instruction = instruction
+        self.silent = silent
         self.model = SentenceTransformer(
             model_path, trust_remote_code=True, model_kwargs={"torch_dtype": torch.float16 if use_fp16 else torch.float}
         )
@@ -115,7 +117,7 @@ class STEncoder:
     def encode(self, query_list: Union[List[str], str], batch_size=64, is_query=True) -> np.ndarray:
         query_list = parse_query(self.model_name, query_list, self.instruction, is_query)
         query_emb = self.model.encode(
-            query_list, batch_size=batch_size, convert_to_numpy=True, normalize_embeddings=True, show_progress_bar=True
+            query_list, batch_size=batch_size, convert_to_numpy=True, normalize_embeddings=True, show_progress_bar=not self.silent
         )
         query_emb = query_emb.astype(np.float32, order="C")
 
@@ -130,7 +132,7 @@ class STEncoder:
             pool,
             normalize_embeddings=True,
             batch_size=batch_size,
-            show_progress_bar=True,
+            show_progress_bar=not self.silent,
         )
         self.model.stop_multi_process_pool(pool)
         query_emb = query_emb.astype(np.float32, order="C")
@@ -190,7 +192,7 @@ class ClipEncoder:
         if not isinstance(query_list, list):
             query_list = [query_list]
         query_emb = []
-        for i in tqdm(range(0, len(query_list), batch_size), desc="Encoding process: "):
+        for i in tqdm(range(0, len(query_list), batch_size), desc="Encoding process: ", disable=self.silent):
             query_emb.append(self.single_batch_encode(query_list[i : i + batch_size], modal))
         query_emb = np.concatenate(query_emb, axis=0)
         return query_emb