flashrag-dev 0.1.4.dev20250104__tar.gz → 0.1.4.dev20250107__tar.gz

{flashrag_dev-0.1.4.dev20250104 → flashrag_dev-0.1.4.dev20250107}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: flashrag_dev
-Version: 0.1.4.dev20250104
+Version: 0.1.4.dev20250107
 Summary: A library for efficient Retrieval-Augmented Generation research
 Home-page: https://github.com/RUC-NLPIR/FlashRAG
 Author: Jiajie Jin, Yutao Zhu, Chenghao Zhang, Xinyu Yang, Zhicheng Dou
@@ -11,12 +11,12 @@ Description-Content-Type: text/markdown
 Provides-Extra: core
 Provides-Extra: retriever
 Provides-Extra: generator
+Provides-Extra: multimodal
 Provides-Extra: full
 License-File: LICENSE
 
 # <div align="center">⚡FlashRAG: A Python Toolkit for Efficient RAG Research<div>
-
-
+\[ English | [中文](README_zh.md) \]
 <div align="center">
 <a href="https://arxiv.org/abs/2405.13576" target="_blank"><img src=https://img.shields.io/badge/arXiv-b5212f.svg?logo=arxiv></a>
 <a href="https://huggingface.co/datasets/RUC-NLPIR/FlashRAG_datasets/" target="_blank"><img src=https://img.shields.io/badge/%F0%9F%A4%97%20HuggingFace%20Datasets-27b3b4.svg></a>
@@ -30,25 +30,38 @@ License-File: LICENSE
 <p>
 <a href="#wrench-installation">Installation</a> |
 <a href="#sparkles-features">Features</a> |
-<a href="#running-quick-start">Quick-Start</a> |
+<a href="#rocket-quick-start">Quick-Start</a> |
 <a href="#gear-components"> Components</a> |
 <a href="#robot-supporting-methods"> Supporting Methods</a> |
-<a href="#notebook-supporting-datasets"> Supporting Datasets</a> |
+<a href="#notebook-supporting-datasets--document-corpus"> Supporting Datasets</a> |
 <a href="#raised_hands-additional-faqs"> FAQs</a>
 </p>
 
 </h4>
-FlashRAG is a Python toolkit for the reproduction and development of Retrieval Augmented Generation (RAG) research. Our toolkit includes 36 pre-processed benchmark RAG datasets and 15 state-of-the-art RAG algorithms. 
+FlashRAG is a Python toolkit for the reproduction and development of Retrieval Augmented Generation (RAG) research. Our toolkit includes 36 pre-processed benchmark RAG datasets and 15 state-of-the-art RAG algorithms.
 
 <p align="center">
 <img src="asset/framework.jpg">
 </p>
 
 With FlashRAG and provided resources, you can effortlessly reproduce existing SOTA works in the RAG domain or implement your custom RAG processes and components.
+
 <p>
 <a href="https://trendshift.io/repositories/10454" target="_blank"><img src="https://trendshift.io/api/badge/repositories/10454" alt="RUC-NLPIR%2FFlashRAG | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
 </p>
 
+## :link: Navigation
+- [Features](#sparkles-features)
+- [Roadmap](#mag_right-roadmap)
+- [Changelog](#page_with_curl-changelog)
+- [Installation](#wrench-installation)
+- [Quick Start](#rocket-quick-start)
+- [Components](#gear-components)
+- [Supporting Methods](#robot-supporting-methods)
+- [Supporting Datasets & Document Corpus](#notebook-supporting-datasets--document-corpus)
+- [Additional FAQs](#raised_hands-additional-faqs)
+- [License](#bookmark-license)
+- [Citation](#star2-citation)
 
 ## :sparkles: Features
 
@@ -62,6 +75,8 @@ With FlashRAG and provided resources, you can effortlessly reproduce existing SO
 
 - **Optimized Execution**: The library's efficiency is enhanced with tools like vLLM, FastChat for LLM inference acceleration, and Faiss for vector index management.
 
+- **Easy to Use UI** : We have developed a very easy to use UI to easily and quickly configure and experience the RAG baselines we have implemented, as well as run evaluation scripts on a visual interface.
+
 ## :mag_right: Roadmap
 
 FlashRAG is still under development and there are many issues and room for improvement. We will continue to update. And we also sincerely welcome contributions on this open-source toolkit.
@@ -73,8 +88,12 @@ FlashRAG is still under development and there are many issues and room for impro
 - [ ] Add more evaluation metrics (e.g., Unieval, name-entity F1) and benchmarks (e.g., RGB benchmark)
 - [ ] Enhance code adaptability and readability
 
-
 ## :page_with_curl: Changelog
+
+[25/01/07] We have currently support the aggregation of multiple retrievers, see it in [**multi retriever usage**](https://github.com/RUC-NLPIR/FlashRAG/blob/main/docs/multi_retriever_usage.md).
+
+[25/01/07] We have integrated a very flexible and lightweight corpus chunking library [**Chunkie**](https://github.com/chonkie-ai/chonkie?tab=readme-ov-file#usage), which supports various custom chunking methods (tokens, sentences, semantic, etc.). Use it in [<u>chunking doc corpus</u>](docs/chunk-doc-corpus.md).
+
 [24/10/21] We have released a version based on the Paddle framework that supports Chinese hardware platforms. Please refer to [FlashRAG Paddle](https://github.com/RUC-NLPIR/FlashRAG-Paddle) for details.
 
 [24/10/13] A new in-domain dataset and corpus - [DomainRAG](https://arxiv.org/pdf/2406.05654) have been added to the dataset. The dataset is based on the internal enrollment data of Renmin University of China, covering seven types of tasks, which can be used for conducting domain-specific RAG testing.
@@ -87,13 +106,13 @@ FlashRAG is still under development and there are many issues and room for impro
 
 [24/08/02] We add support for a new method [<u>Spring</u>](https://arxiv.org/abs/2405.19670), significantly improve the performance of LLM by adding only a few token embeddings. See it result in [<u>result table</u>](#robot-supporting-methods).
 
+<details>
+<summary>Show more</summary>
+
 [24/07/17] Due to some unknown issues with HuggingFace, our original dataset link has been invalid. We have updated it. Please check the [new link](https://huggingface.co/datasets/RUC-NLPIR/FlashRAG_datasets/) if you encounter any problems.
 
 [24/07/06] We add support for a new method: [<u>Trace</u>](https://arxiv.org/abs/2406.11460), which refine text by constructing a knowledge graph. See it [<u>results</u>](#robot-supporting-methods) and [<u>details</u>](./docs/baseline_details.md).
 
-<details>
-<summary>Show more</summary>
-
 [24/06/19] We add support for a new method: [<u>IRCoT</u>](https://arxiv.org/abs/2212.10509), and update the [<u>result table</u>](#robot-supporting-methods).
 
 [24/06/15] We provide a [<u>demo</u>](./examples/quick_start/demo_en.py) to perform the RAG process using our toolkit.
@@ -108,15 +127,15 @@ FlashRAG is still under development and there are many issues and room for impro
 
 </details>
 
-
-## :wrench: Installation 
+## :wrench: Installation
 
 To get started with FlashRAG, you can simply install it with pip:
+
 ```base
 pip install flashrag-dev --pre
 ```
 
-Or you can clone it from Github and install (requires Python 3.9+): 
+Or you can clone it from Github and install (requires Python 3.10+):
 
 ```bash
 git clone https://github.com/RUC-NLPIR/FlashRAG.git
@@ -125,6 +144,7 @@ pip install -e .
 ```
 
 If you want to use vllm, sentence-transformers or pyserini, you can install the optional dependencies:
+
 ```bash
 # Install all extra dependencies
 pip install flashrag[full]
@@ -140,6 +160,7 @@ pip install pyserini
 ```
 
 Due to the incompatibility when installing `faiss` using `pip`, it is necessary to use the following conda command for installation.
+
 ```bash
 # CPU-only version
 conda install -c pytorch faiss-cpu=1.8.0
@@ -157,68 +178,106 @@ From the official Faiss repository ([source](https://github.com/facebookresearch
 
 ## :rocket: Quick Start
 
+### Corpus Construction
+To build an index, you first need to save your corpus as a `jsonl` file with each line representing a document.
 
-### Toy Example
+```jsonl
+{"id": "0", "contents": "content"}
+{"id": "1", "contents": "content"}
+...
+```
 
-For beginners, we provide a [<u>an introduction to flashrag</u>](./docs/introduction_for_beginners_en.md) ([<u>中文版</u>](./docs/introduction_for_beginners_zh.md) [<u>한국어</u>](./docs/introduction_for_beginners_kr.md)) to help you familiarize yourself with our toolkit. Alternatively, you can directly refer to the code below.
+If you want to use Wikipedia as your corpus, you can refer to our documentation [Processing Wikipedia](./docs/process-wiki.md) to convert it into an indexable format.
 
-#### Demo 
+### Index Construction
 
-We provide a toy demo to implement a simple RAG process. You can freely change the corpus and model you want to use. The English demo uses [general knowledge](https://huggingface.co/datasets/MuskumPillerum/General-Knowledge) as the corpus, `e5-base-v2` as the retriever, and `Llama3-8B-instruct` as generator. The Chinese demo uses data crawled from the official website of Remin University of China as the corpus, `bge-large-zh-v1.5` as the retriever, and qwen1.5-14B as the generator. Please fill in the corresponding path in the file.
+You can use the following code to build your own index.
 
-<div style="display: flex; justify-content: space-around;">
-  <div style="text-align: center;">
-    <img src="./asset/demo_en.gif" style="width: 100%;">
-  </div>
-</div>
+* For **dense retrieval methods**, especially popular embedding models, we use `faiss` to build the index.
+
+* For **sparse retrieval methods (BM25)**, we use `Pyserini` or `bm25s` to build the corpus into a Lucene inverted index. The built index contains the original documents.
+
+#### For Dense Retrieval Methods
 
-To run the demo:
+Modify the parameters in the following code to your own.
 
 ```bash
-cd examples/quick_start
+python -m flashrag.retriever.index_builder \
+  --retrieval_method e5 \
+  --model_path /model/e5-base-v2/ \
+  --corpus_path indexes/sample_corpus.jsonl \
+  --save_dir indexes/ \
+  --use_fp16 \
+  --max_length 512 \
+  --batch_size 256 \
+  --pooling_method mean \
+  --faiss_type Flat 
+```
+
+* ```--pooling_method```: If this parameter is not specified, we will automatically select it based on the model name and model file. However, since different embedding models use different pooling methods, **we may not have fully implemented them**. To ensure accuracy, you can **specify the pooling method corresponding to the retrieval model you are using** (`mean`, `pooler`, or `cls`).
 
-# copy the config file here, otherwise, streamlit will complain that file s
-cp ../methods/my_config.yaml .
+* ```---instruction```: Some embedding models require additional instructions to be concatenated to the query before encoding, which can be specified here. Currently, we will automatically fill in the instructions for **E5** and **BGE** models, while other models need to be supplemented manually.
 
-# run english demo
-streamlit run demo_en.py
+If the retrieval model supports the `sentence transformers` library, you can use the following code to build the index (**without considering the pooling method**).
 
-# run chinese demo
-streamlit run demo_zh.py
+```bash
+python -m flashrag.retriever.index_builder \
+  --retrieval_method e5 \
+  --model_path /model/e5-base-v2/ \
+  --corpus_path indexes/sample_corpus.jsonl \
+  --save_dir indexes/ \
+  --use_fp16 \
+  --max_length 512 \
+  --batch_size 256 \
+  --pooling_method mean \
+  --sentence_transformer \
+  --faiss_type Flat 
 ```
 
+#### For Sparse Retrieval Methods (BM25)
 
-#### Pipeline
+If building a bm25 index, there is no need to specify `model_path`.
 
-We also provide an example to use our framework for pipeline execution.
-Run the following code to implement a naive RAG pipeline using provided toy datasets.
-The default retriever is `e5-base-v2` and default generator is `Llama3-8B-instruct`. You need to fill in the corresponding model path in the following command. If you wish to use other models, please refer to the detailed instructions below.
+##### Building Index with BM25s
 
 ```bash
-cd examples/quick_start
-python simple_pipeline.py \
-    --model_path <Llama-3-8B-instruct-PATH> \
-    --retriever_path <E5-PATH>
+python -m flashrag.retriever.index_builder \
+  --retrieval_method bm25 \
+  --corpus_path indexes/sample_corpus.jsonl \
+  --bm25_backend bm25s \
+  --save_dir indexes/ 
 ```
 
-After the code is completed, you can view the intermediate results of the run and the final evaluation score in the output folder under the corresponding path.
+##### Building Index with Pyserini
+
+```bash
+python -m flashrag.retriever.index_builder \
+  --retrieval_method bm25 \
+  --corpus_path indexes/sample_corpus.jsonl \
+  --bm25_backend pyserini \
+  --save_dir indexes/ 
+```
 
 ### Using the ready-made pipeline
 
 You can use the pipeline class we have already built (as shown in [<u>pipelines</u>](#pipelines)) to implement the RAG process inside. In this case, you just need to configure the config and load the corresponding pipeline.
 
-Firstly, load the entire process's config, which records various hyperparameters required in the RAG process. You can input yaml files as parameters or directly as variables. The priority of variables as input is higher than that of files.
+Firstly, load the entire process's config, which records various hyperparameters required in the RAG process. You can input yaml files as parameters or directly as variables.
+
+Please note that **variables as input take precedence over files**.
 
 ```python
 from flashrag.config import Config
 
+# hybrid load configs
 config_dict = {'data_dir': 'dataset/'}
-my_config = Config(config_file_path = 'my_config.yaml',
-                config_dict = config_dict)
+my_config = Config(
+    config_file_path = 'my_config.yaml',
+    config_dict = config_dict
 ```
 
 We provide comprehensive guidance on how to set configurations, you can see our [<u>configuration guidance</u>](./docs/configuration.md).
-You can also refer to the [<u>basic yaml file</u>](./flashrag/config/basic_config.yaml) we provide to set your own parameters. 
+You can also refer to the [<u>basic yaml file</u>](./flashrag/config/basic_config.yaml) we provide to set your own parameters.
 
 Next, load the corresponding dataset and initialize the pipeline. The components in the pipeline will be automatically loaded.
 
@@ -229,8 +288,10 @@ from flashrag.prompt import PromptTemplate
 from flashrag.config import Config
 
 config_dict = {'data_dir': 'dataset/'}
-my_config = Config(config_file_path = 'my_config.yaml',
-                config_dict = config_dict)
+my_config = Config(
+    config_file_path = 'my_config.yaml',
+    config_dict = config_dict
+)
 all_split = get_dataset(my_config)
 test_data = all_split['test']
 
@@ -238,13 +299,17 @@ pipeline = SequentialPipeline(my_config)
 ```
 
 You can specify your own input prompt using `PromptTemplete`:
+
 ```python
 prompt_templete = PromptTemplate(
-    config, 
+    config,
     system_prompt = "Answer the question based on the given document. Only give me the answer and do not output any other words.\nThe following are given documents.\n\n{reference}",
     user_prompt = "Question: {question}\nAnswer:"
 )
-pipeline = SequentialPipeline(my_config, prompt_template=prompt_templete)
+pipeline = SequentialPipeline(
+  my_config,
+  prompt_template = prompt_templete
+)
 ```
 
 Finally, execute `pipeline.run` to obtain the final result.
@@ -252,10 +317,11 @@ Finally, execute `pipeline.run` to obtain the final result.
 ```python
 output_dataset = pipeline.run(test_data, do_eval=True)
 ```
+
 The `output_dataset` contains the intermediate results and metric scores for each item in the input dataset.
 Meanwhile, the dataset with intermediate results and the overall evaluation score will also be saved as a file (if `save_intermediate_data` and `save_metric_score` are specified).
 
-### Build your own pipeline
+### Build your own pipeline!
 
 Sometimes you may need to implement more complex RAG process, and you can build your own pipeline to implement it.
 You just need to inherit `BasicPipeline`, initialize the components you need, and complete the `run` function.
@@ -283,7 +349,6 @@ class ToyPipeline(BasicPipeline):
 
 Please first understand the input and output forms of the components you need to use from our [<u>documentation</u>](./docs/basic_usage.md).
 
-
 ### Just use components
 
 If you already have your own code and only want to use our components to embed the original code, you can refer to the [<u>basic introduction of the components</u>](./docs/basic_usage.md) to obtain the input and output formats of each component.
@@ -427,10 +492,10 @@ In each category, we have implemented corresponding common pipelines. Some pipel
     </tbody>
 </table>
 
-
 ## :robot: Supporting Methods
 
 We have implemented 15 works with a consistent setting of:
+
 - **Generator:** LLAMA3-8B-instruct with input length of 2048
 - **Retriever:** e5-base-v2 as embedding model, retrieve 5 docs per query
 - **Prompt:** A consistent default prompt, template can be found in the [<u>method details</u>](./docs/baseline_details.md).
@@ -441,35 +506,34 @@ For necessary settings and hyperparameters specific to some methods, we have doc
 
 It’s important to note that, to ensure consistency, we have utilized a uniform setting. However, this setting may differ from the original setting of the method, leading to variations in results compared to the original outcomes.
 
-
-| Method               | Type           | NQ (EM) | TriviaQA (EM) | Hotpotqa (F1) | 2Wiki (F1)| PopQA (F1)| WebQA(EM) | Specific setting                                                                  |
-|----------------------|----------------|---------|---------------|---------------|---------------|---------------|---------------|------------------------------------------------------------------------------------|
-| Naive Generation     | Sequential     | 22.6    | 55.7          | 28.4          |  33.9| 21.7| 18.8| |
-| Standard RAG         | Sequential     | 35.1    | 58.9          | 35.3          | 21.0 | 36.7|15.7| |
-| [AAR-contriever-kilt](https://aclanthology.org/2023.acl-long.136.pdf)  | Sequential     | 30.1    | 56.8          | 33.4          | 19.8 | 36.1  | 16.1| |
-| [LongLLMLingua](https://arxiv.org/abs/2310.06839)        | Sequential     | 32.2    | 59.2          | 37.5          |25.0| 38.7| 17.5| Compress Ratio=0.5 |
-| [RECOMP-abstractive](https://arxiv.org/pdf/2310.04408)   | Sequential     | 33.1    | 56.4          | 37.5          | 32.4 | 39.9| 20.2| |
-| [Selective-Context](https://arxiv.org/abs/2310.06201)    | Sequential     | 30.5    | 55.6          | 34.4          |18.5| 33.5| 17.3| Compress Ratio=0.5|
-| [Trace](https://arxiv.org/abs/2406.11460) | Sequential | 30.7 | 50.2 | 34.0 | 15.5 | 37.4 | 19.9 | |
-| [Spring](https://arxiv.org/abs/2405.19670) | Sequential | 37.9 | 64.6 |42.6 | 37.3 |54.8 |27.7 | Use Llama2-7B-chat with trained embedding table |
-| [SuRe](https://arxiv.org/abs/2404.13081)                 | Branching      | 37.1    | 53.2          | 33.4          |20.6|48.1|24.2| Use provided prompt|
-| [REPLUG](https://arxiv.org/abs/2301.12652)               | Branching      | 28.9    | 57.7          | 31.2          |21.1|27.8|20.2|  |
-| [SKR](https://aclanthology.org/2023.findings-emnlp.691.pdf)                  | Conditional    | 33.2   | 56.0          | 32.4          | 23.4 |31.7|17.0|Use infernece-time training data|
-|[Adaptive-RAG](https://aclanthology.org/2024.naacl-long.389.pdf) | Conditional | 35.1 | 56.6 | 39.1 | 28.4 | 40.4 | 16.0| |
-| [Ret-Robust](https://arxiv.org/abs/2310.01558)           | Loop     | 42.9    | 68.2          | 35.8          |43.4|57.2|33.7| Use LLAMA2-13B with trained lora|
-| [Self-RAG](https://arxiv.org/abs/2310.11511)             | Loop   | 36.4    | 38.2          | 29.6          | 25.1|32.7|21.9| Use trained selfrag-llama2-7B|
-| [FLARE](https://arxiv.org/abs/2305.06983)                | Loop   | 22.5    | 55.8          | 28.0          |33.9| 20.7| 20.2| |
-| [Iter-Retgen](https://arxiv.org/abs/2305.15294),      [ITRG](https://arxiv.org/abs/2310.05149)   | Loop | 36.8    | 60.1          | 38.3          | 21.6| 37.9| 18.2| |
-| [IRCoT](https://aclanthology.org/2023.acl-long.557.pdf) | Loop | 33.3| 56.9|41.5|32.4 |45.6 |20.7 | |
-
+| Method                                                                                    | Type        | NQ (EM) | TriviaQA (EM) | Hotpotqa (F1) | 2Wiki (F1) | PopQA (F1) | WebQA(EM) | Specific setting                                |
+| ----------------------------------------------------------------------------------------- | ----------- | ------- | ------------- | ------------- | ---------- | ---------- | --------- | ----------------------------------------------- |
+| Naive Generation                                                                          | Sequential  | 22.6    | 55.7          | 28.4          | 33.9       | 21.7       | 18.8      |                                                 |
+| Standard RAG                                                                              | Sequential  | 35.1    | 58.9          | 35.3          | 21.0       | 36.7       | 15.7      |                                                 |
+| [AAR-contriever-kilt](https://aclanthology.org/2023.acl-long.136.pdf)                     | Sequential  | 30.1    | 56.8          | 33.4          | 19.8       | 36.1       | 16.1      |                                                 |
+| [LongLLMLingua](https://arxiv.org/abs/2310.06839)                                         | Sequential  | 32.2    | 59.2          | 37.5          | 25.0       | 38.7       | 17.5      | Compress Ratio=0.5                              |
+| [RECOMP-abstractive](https://arxiv.org/pdf/2310.04408)                                    | Sequential  | 33.1    | 56.4          | 37.5          | 32.4       | 39.9       | 20.2      |                                                 |
+| [Selective-Context](https://arxiv.org/abs/2310.06201)                                     | Sequential  | 30.5    | 55.6          | 34.4          | 18.5       | 33.5       | 17.3      | Compress Ratio=0.5                              |
+| [Trace](https://arxiv.org/abs/2406.11460)                                                 | Sequential  | 30.7    | 50.2          | 34.0          | 15.5       | 37.4       | 19.9      |                                                 |
+| [Spring](https://arxiv.org/abs/2405.19670)                                                | Sequential  | 37.9    | 64.6          | 42.6          | 37.3       | 54.8       | 27.7      | Use Llama2-7B-chat with trained embedding table |
+| [SuRe](https://arxiv.org/abs/2404.13081)                                                  | Branching   | 37.1    | 53.2          | 33.4          | 20.6       | 48.1       | 24.2      | Use provided prompt                             |
+| [REPLUG](https://arxiv.org/abs/2301.12652)                                                | Branching   | 28.9    | 57.7          | 31.2          | 21.1       | 27.8       | 20.2      |                                                 |
+| [SKR](https://aclanthology.org/2023.findings-emnlp.691.pdf)                               | Conditional | 33.2    | 56.0          | 32.4          | 23.4       | 31.7       | 17.0      | Use infernece-time training data                |
+| [Adaptive-RAG](https://aclanthology.org/2024.naacl-long.389.pdf)                          | Conditional | 35.1    | 56.6          | 39.1          | 28.4       | 40.4       | 16.0      |                                                 |
+| [Ret-Robust](https://arxiv.org/abs/2310.01558)                                            | Loop        | 42.9    | 68.2          | 35.8          | 43.4       | 57.2       | 33.7      | Use LLAMA2-13B with trained lora                |
+| [Self-RAG](https://arxiv.org/abs/2310.11511)                                              | Loop        | 36.4    | 38.2          | 29.6          | 25.1       | 32.7       | 21.9      | Use trained selfrag-llama2-7B                   |
+| [FLARE](https://arxiv.org/abs/2305.06983)                                                 | Loop        | 22.5    | 55.8          | 28.0          | 33.9       | 20.7       | 20.2      |                                                 |
+| [Iter-Retgen](https://arxiv.org/abs/2305.15294), [ITRG](https://arxiv.org/abs/2310.05149) | Loop        | 36.8    | 60.1          | 38.3          | 21.6       | 37.9       | 18.2      |                                                 |
+| [IRCoT](https://aclanthology.org/2023.acl-long.557.pdf)                                   | Loop        | 33.3    | 56.9          | 41.5          | 32.4       | 45.6       | 20.7      |                                                 |
 
 ## :notebook: Supporting Datasets & Document Corpus
 
 ### Datasets
 
-We have collected and processed 36 datasets widely used in RAG research, pre-processing them to ensure a consistent format for ease of use. For certain datasets (such as Wiki-asp), we have adapted them to fit the requirements of RAG tasks according to the methods commonly used within the community. All datasets are available at [<u>Huggingface datasets</u>](https://huggingface.co/datasets/RUC-NLPIR/FlashRAG_datasets). 
+We have collected and processed 36 datasets widely used in RAG research, pre-processing them to ensure a consistent format for ease of use. For certain datasets (such as Wiki-asp), we have adapted them to fit the requirements of RAG tasks according to the methods commonly used within the community. All datasets are available at [<u>Huggingface datasets</u>](https://huggingface.co/datasets/RUC-NLPIR/FlashRAG_datasets).
 
 For each dataset, we save each split as a `jsonl` file, and each line is a dict as follows:
+
 ```python
 {
   'id': str,
@@ -479,11 +543,10 @@ For each dataset, we save each split as a `jsonl` file, and each line is a dict
 }
 ```
 
-
 Below is the list of datasets along with the corresponding sample sizes:
 
 | Task                      | Dataset Name    | Knowledge Source | # Train   | # Dev   | # Test |
-|---------------------------|-----------------|------------------|-----------|---------|--------|
+| ------------------------- | --------------- | ---------------- | --------- | ------- | ------ |
 | QA                        | NQ              | wiki             | 79,168    | 8,757   | 3,610  |
 | QA                        | TriviaQA        | wiki & web       | 78,785    | 8,837   | 11,313 |
 | QA                        | PopQA           | wiki             | /         | /       | 14,267 |
@@ -494,7 +557,7 @@ Below is the list of datasets along with the corresponding sample sizes:
 | QA                        | WebQuestions    | Google Freebase  | 3,778     | /       | 2,032  |
 | QA                        | AmbigQA         | wiki             | 10,036    | 2,002   | /      |
 | QA                        | SIQA            | -                | 33,410    | 1,954   | /      |
-| QA                        | CommonSenseQA      | -                | 9,741     | 1,221   | /      |
+| QA                        | CommonSenseQA   | -                | 9,741     | 1,221   | /      |
 | QA                        | BoolQ           | wiki             | 9,427     | 3,270   | /      |
 | QA                        | PIQA            | -                | 16,113    | 1,838   | /      |
 | QA                        | Fermi           | wiki             | 8,000     | 1,000   | 1,000  |
@@ -502,24 +565,24 @@ Below is the list of datasets along with the corresponding sample sizes:
 | multi-hop QA              | 2WikiMultiHopQA | wiki             | 15,000    | 12,576  | /      |
 | multi-hop QA              | Musique         | wiki             | 19,938    | 2,417   | /      |
 | multi-hop QA              | Bamboogle       | wiki             | /         | /       | 125    |
-| multi-hop QA              | StrategyQA      | wiki             | 2290      | /       | /
+| multi-hop QA              | StrategyQA      | wiki             | 2290      | /       | /      |
 | Long-form QA              | ASQA            | wiki             | 4,353     | 948     | /      |
 | Long-form QA              | ELI5            | Reddit           | 272,634   | 1,507   | /      |
-| Long-form QA              | WikiPassageQA            | wiki             | 3,332     | 417    |  416      |
+| Long-form QA              | WikiPassageQA   | wiki             | 3,332     | 417     | 416    |
 | Open-Domain Summarization | WikiASP         | wiki             | 300,636   | 37,046  | 37,368 |
 | multiple-choice           | MMLU            | -                | 99,842    | 1,531   | 14,042 |
 | multiple-choice           | TruthfulQA      | wiki             | /         | 817     | /      |
 | multiple-choice           | HellaSWAG       | ActivityNet      | 39,905    | 10,042  | /      |
 | multiple-choice           | ARC             | -                | 3,370     | 869     | 3,548  |
 | multiple-choice           | OpenBookQA      | -                | 4,957     | 500     | 500    |
-| multiple-choice           | QuaRTz      | -                | 2696     | 384     | 784    |
+| multiple-choice           | QuaRTz          | -                | 2696      | 384     | 784    |
 | Fact Verification         | FEVER           | wiki             | 104,966   | 10,444  | /      |
 | Dialog Generation         | WOW             | wiki             | 63,734    | 3,054   | /      |
 | Entity Linking            | AIDA CoNll-yago | Freebase & wiki  | 18,395    | 4,784   | /      |
 | Entity Linking            | WNED            | Wiki             | /         | 8,995   | /      |
 | Slot Filling              | T-REx           | DBPedia          | 2,284,168 | 5,000   | /      |
 | Slot Filling              | Zero-shot RE    | wiki             | 147,909   | 3,724   | /      |
-| In-domain QA| DomainRAG | Web pages of RUC| / | / | 485|
+| In-domain QA              | DomainRAG       | Web pages of RUC | /         | /       | 485    |
 
 ### Document Corpus
 
@@ -529,23 +592,23 @@ Our toolkit supports jsonl format for retrieval document collections, with the f
 {"id":"0", "contents": "...."}
 {"id":"1", "contents": "..."}
 ```
+
 The `contents` key is essential for building the index. For documents that include both text and title, we recommend setting the value of `contents` to `{title}\n{text}`. The corpus file can also contain other keys to record additional characteristics of the documents.
 
 In the academic research, Wikipedia and MS MARCO are the most commonly used retrieval document collections. For Wikipedia, we provide a [<u>comprehensive script</u>](./docs/process-wiki.md) to process any Wikipedia dump into a clean corpus. Additionally, various processed versions of the Wikipedia corpus are available in many works, and we have listed some reference links.
 
-
 For MS MARCO, it is already processed upon release and can be directly downloaded from its [<u>hosting link</u>](https://huggingface.co/datasets/Tevatron/msmarco-passage-corpus) on Hugging Face.
 
 ### Index
 
-To facilitate easier replication of the experiments, we now provide a preprocessed index available in the ModelScope dataset page: [FlashRAG_Dataset/retrieval_corpus/wiki18_100w_e5_index.zip](https://www.modelscope.cn/datasets/hhjinjiajie/FlashRAG_Dataset/file/view/master?id=47985&status=2&fileName=retrieval_corpus%252Fwiki18_100w_e5_index.zip). 
+To facilitate easier replication of the experiments, we now provide a preprocessed index available in the ModelScope dataset page: [FlashRAG_Dataset/retrieval_corpus/wiki18_100w_e5_index.zip](https://www.modelscope.cn/datasets/hhjinjiajie/FlashRAG_Dataset/file/view/master?id=47985&status=2&fileName=retrieval_corpus%252Fwiki18_100w_e5_index.zip).
 
 The index was created using the e5-base-v2 retriever on our uploaded wiki18_100w dataset, which is consistent with the index used in our experiments.
 
 ## :raised_hands: Additional FAQs
 
 - [How should I set different experimental parameters?](./docs/configuration.md)
-- [How to build my own corpus, such as a specific segmented Wikipedia?](./docs/process-wiki.md) 
+- [How to build my own corpus, such as a specific segmented Wikipedia?](./docs/process-wiki.md)
 - [How to index my own corpus?](./docs/building-index.md)
 - [How to reproduce supporting methods?](./docs/reproduce_experiment.md)
 
@@ -553,9 +616,10 @@ The index was created using the e5-base-v2 retriever on our uploaded wiki18_100w
 
 FlashRAG is licensed under the [<u>MIT License</u>](./LICENSE).
 
-
 ## :star2: Citation
+
 Please kindly cite our paper if helps your research:
+
 ```BibTex
 @article{FlashRAG,
     author={Jiajie Jin and