AutoRAG 0.2.13__tar.gz → 0.2.14__tar.gz

autorag-0.2.14/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,35 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: "[BUG] "
+labels: bug
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Full Error log**
+If applicable, add full error log to help explain your problem.
+
+**Code that bug is happened**
+If applicable, add the code that bug is happened.
+(Especially, your AutoRAG YAML file or python codes that you wrote)
+
+**Desktop (please complete the following information):**
+ - OS: [e.g. Windows, Linux, MacOS]
+ - Python version [e.g. 3.10]
+
+**Additional context**
+Add any other context about the problem here.

autorag-0.2.14/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: "[Feature Request]"
+labels: enhancement
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

{autorag-0.2.13 → autorag-0.2.14}/AutoRAG.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: AutoRAG
-Version: 0.2.13
+Version: 0.2.14
 Summary: Automatically Evaluate RAG pipelines with your own data. Find optimal structure for new RAG product.
 Author-email: Marker-Inc <vkehfdl1@gmail.com>
 License:                                  Apache License
@@ -221,13 +221,12 @@ Requires-Python: >=3.8
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: numpy<2.0.0
-Requires-Dist: pandas
+Requires-Dist: pandas>=2.1.0
 Requires-Dist: tqdm
 Requires-Dist: tiktoken>=0.7.0
 Requires-Dist: openai>=1.0.0
 Requires-Dist: rank_bm25
 Requires-Dist: transformers
-Requires-Dist: swifter
 Requires-Dist: pyyaml
 Requires-Dist: pyarrow
 Requires-Dist: fastparquet
@@ -242,7 +241,7 @@ Requires-Dist: uvicorn
 Requires-Dist: torch
 Requires-Dist: sentencepiece
 Requires-Dist: guidance
-Requires-Dist: cohere<=5.6.2
+Requires-Dist: cohere>=5.8.0
 Requires-Dist: tokenlog>=0.0.2
 Requires-Dist: aiohttp
 Requires-Dist: bert_score
@@ -250,6 +249,7 @@ Requires-Dist: sentence-transformers
 Requires-Dist: FlagEmbedding
 Requires-Dist: ragas
 Requires-Dist: llmlingua
+Requires-Dist: peft
 Requires-Dist: llama-index>=0.10.1
 Requires-Dist: llama-index-core>=0.10.1
 Requires-Dist: llama-index-readers-file

{autorag-0.2.13 → autorag-0.2.14}/AutoRAG.egg-info/SOURCES.txt

@@ -7,6 +7,8 @@ pyproject.toml
 requirements.txt
 .github/FUNDING.yml
 .github/dependabot.yml
+.github/ISSUE_TEMPLATE/bug_report.md
+.github/ISSUE_TEMPLATE/feature_request.md
 .github/workflows/publish.yml
 .github/workflows/sphinx.yml
 .github/workflows/test.yml

{autorag-0.2.13 → autorag-0.2.14}/AutoRAG.egg-info/requires.txt

@@ -1,11 +1,10 @@
 numpy<2.0.0
-pandas
+pandas>=2.1.0
 tqdm
 tiktoken>=0.7.0
 openai>=1.0.0
 rank_bm25
 transformers
-swifter
 pyyaml
 pyarrow
 fastparquet
@@ -20,7 +19,7 @@ uvicorn
 torch
 sentencepiece
 guidance
-cohere<=5.6.2
+cohere>=5.8.0
 tokenlog>=0.0.2
 aiohttp
 bert_score
@@ -28,6 +27,7 @@ sentence-transformers
 FlagEmbedding
 ragas
 llmlingua
+peft
 llama-index>=0.10.1
 llama-index-core>=0.10.1
 llama-index-readers-file

{autorag-0.2.13 → autorag-0.2.14}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: AutoRAG
-Version: 0.2.13
+Version: 0.2.14
 Summary: Automatically Evaluate RAG pipelines with your own data. Find optimal structure for new RAG product.
 Author-email: Marker-Inc <vkehfdl1@gmail.com>
 License:                                  Apache License
@@ -221,13 +221,12 @@ Requires-Python: >=3.8
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: numpy<2.0.0
-Requires-Dist: pandas
+Requires-Dist: pandas>=2.1.0
 Requires-Dist: tqdm
 Requires-Dist: tiktoken>=0.7.0
 Requires-Dist: openai>=1.0.0
 Requires-Dist: rank_bm25
 Requires-Dist: transformers
-Requires-Dist: swifter
 Requires-Dist: pyyaml
 Requires-Dist: pyarrow
 Requires-Dist: fastparquet
@@ -242,7 +241,7 @@ Requires-Dist: uvicorn
 Requires-Dist: torch
 Requires-Dist: sentencepiece
 Requires-Dist: guidance
-Requires-Dist: cohere<=5.6.2
+Requires-Dist: cohere>=5.8.0
 Requires-Dist: tokenlog>=0.0.2
 Requires-Dist: aiohttp
 Requires-Dist: bert_score
@@ -250,6 +249,7 @@ Requires-Dist: sentence-transformers
 Requires-Dist: FlagEmbedding
 Requires-Dist: ragas
 Requires-Dist: llmlingua
+Requires-Dist: peft
 Requires-Dist: llama-index>=0.10.1
 Requires-Dist: llama-index-core>=0.10.1
 Requires-Dist: llama-index-readers-file

autorag-0.2.14/autorag/VERSION

@@ -0,0 +1 @@
+0.2.14

{autorag-0.2.13 → autorag-0.2.14}/autorag/__init__.py

@@ -9,19 +9,16 @@ from llama_index.embeddings.huggingface import HuggingFaceEmbedding
 from llama_index.embeddings.openai import OpenAIEmbedding
 from llama_index.embeddings.openai import OpenAIEmbeddingModelType
 from llama_index.llms.huggingface import HuggingFaceLLM
-from llama_index.llms.openai import OpenAI
 from llama_index.llms.ollama import Ollama
+from llama_index.llms.openai import OpenAI
 from llama_index.llms.openai_like import OpenAILike
 from rich.logging import RichHandler
-from swifter import set_defaults
 
 version_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'VERSION')
 
 with open(version_path, 'r') as f:
     __version__ = f.read().strip()
 
-set_defaults(allow_dask_on_strings=True)
-
 
 class LazyInit:
     def __init__(self, factory, *args, **kwargs):
@@ -50,7 +47,8 @@ embedding_models = {
     'huggingface_cointegrated_rubert_tiny2': LazyInit(HuggingFaceEmbedding, model_name="cointegrated/rubert-tiny2"),
     'huggingface_all_mpnet_base_v2': LazyInit(HuggingFaceEmbedding,
                                               model_name="sentence-transformers/all-mpnet-base-v2",
-                                              max_length=512, )
+                                              max_length=512, ),
+    'huggingface_bge_m3': LazyInit(HuggingFaceEmbedding, model_name="BAAI/bge-m3"),
 }
 
 generator_models = {

autorag-0.2.14/autorag/data/qacreation/__init__.py

@@ -0,0 +1,2 @@
+from .base import make_single_content_qa, make_qa_with_existing_queries
+from .llama_index import generate_qa_llama_index, generate_answers, generate_qa_llama_index_by_ratio

autorag-0.2.14/autorag/data/qacreation/base.py

@@ -0,0 +1,168 @@
+import logging
+import uuid
+from typing import Callable, Optional, List
+
+import chromadb
+import pandas as pd
+from tqdm import tqdm
+
+import autorag
+from autorag.nodes.retrieval.vectordb import vectordb_ingest, vectordb
+from autorag.utils.util import save_parquet_safe, fetch_contents
+
+logger = logging.getLogger("AutoRAG")
+
+
+def make_single_content_qa(corpus_df: pd.DataFrame,
+                           content_size: int,
+                           qa_creation_func: Callable,
+                           output_filepath: Optional[str] = None,
+                           upsert: bool = False,
+                           random_state: int = 42,
+                           cache_batch: int = 32,
+                           **kwargs) -> pd.DataFrame:
+    """
+    Make single content (single-hop, single-document) QA dataset using given qa_creation_func.
+    It generates a single content QA dataset, which means its retrieval ground truth will be only one.
+    It is the most basic form of QA dataset.
+
+    :param corpus_df: The corpus dataframe to make QA dataset from.
+    :param content_size: This function will generate QA dataset for the given number of contents.
+    :param qa_creation_func: The function to create QA pairs.
+        You can use like `generate_qa_llama_index` or `generate_qa_llama_index_by_ratio`.
+        The input func must have `contents` parameter for the list of content string.
+    :param output_filepath: Optional filepath to save the parquet file.
+        If None, the function will return the processed_data as pd.DataFrame, but do not save as parquet.
+        File directory must exist. File extension must be .parquet
+    :param upsert: If true, the function will overwrite the existing file if it exists.
+        Default is False.
+    :param random_state: The random state for sampling corpus from the given corpus_df.
+    :param cache_batch: The number of batches to use for caching the generated QA dataset.
+        When the cache_batch size data is generated, the dataset will save to the designated output_filepath.
+        If the cache_batch size is too small, the process time will be longer.
+    :param kwargs: The keyword arguments for qa_creation_func.
+    :return: QA dataset dataframe.
+        You can save this as parquet file to use at AutoRAG.
+    """
+    assert content_size > 0, "content_size must be greater than 0."
+    if content_size > len(corpus_df):
+        logger.warning(f"content_size {content_size} is larger than the corpus size {len(corpus_df)}. "
+                       "Setting content_size to the corpus size.")
+        content_size = len(corpus_df)
+    sampled_corpus = corpus_df.sample(n=content_size, random_state=random_state)
+    sampled_corpus = sampled_corpus.reset_index(drop=True)
+
+    def make_query_generation_gt(row):
+        return row['qa']['query'], row['qa']['generation_gt']
+
+    qa_data = pd.DataFrame()
+    for idx, i in tqdm(enumerate(range(0, len(sampled_corpus), cache_batch))):
+        qa = qa_creation_func(contents=sampled_corpus['contents'].tolist()[i:i + cache_batch], **kwargs)
+
+        temp_qa_data = pd.DataFrame({
+            'qa': qa,
+            'retrieval_gt': sampled_corpus['doc_id'].tolist()[i:i + cache_batch],
+        })
+        temp_qa_data = temp_qa_data.explode('qa', ignore_index=True)
+        temp_qa_data['qid'] = [str(uuid.uuid4()) for _ in range(len(temp_qa_data))]
+        temp_qa_data[['query', 'generation_gt']] = temp_qa_data.apply(make_query_generation_gt, axis=1,
+                                                                      result_type='expand')
+        temp_qa_data = temp_qa_data.drop(columns=['qa'])
+
+        temp_qa_data['retrieval_gt'] = temp_qa_data['retrieval_gt'].apply(lambda x: [[x]])
+        temp_qa_data['generation_gt'] = temp_qa_data['generation_gt'].apply(lambda x: [x])
+
+        if idx == 0:
+            qa_data = temp_qa_data
+        else:
+            qa_data = pd.concat([qa_data, temp_qa_data], ignore_index=True)
+        if output_filepath is not None:
+            save_parquet_safe(qa_data, output_filepath, upsert=upsert)
+
+    return qa_data
+
+
+def make_qa_with_existing_queries(
+        corpus_df: pd.DataFrame,
+        existing_query_df: pd.DataFrame,
+        content_size: int,
+        answer_creation_func: Callable,
+        output_filepath: Optional[str] = None,
+        embedding_model: str = 'openai_embed_3_large',
+        collection: Optional[chromadb.Collection] = None,
+        upsert: bool = False,
+        random_state: int = 42,
+        cache_batch: int = 32,
+        top_k: int = 3,
+        **kwargs
+) -> pd.DataFrame:
+    """
+    Make single-hop QA dataset using given qa_creation_func and existing queries.
+    
+    :param corpus_df: The corpus dataframe to make QA dataset from.
+    :param existing_query_df: Dataframe containing existing queries to use for QA pair creation.
+    :param content_size: This function will generate QA dataset for the given number of contents.
+    :param answer_creation_func: The function to create answer with input query.
+    :param output_filepath: Optional filepath to save the parquet file.
+    :param embedding_model: The embedding model to use for vectorization.
+        You can add your own embedding model in the autorag.embedding_models.
+        Please refer to how to add an embedding model in this doc: https://docs.auto-rag.com/local_model.html
+        The default is 'openai_embed_3_large'.
+    :param collection: The chromadb collection to use for vector DB.
+        You can make any chromadb collection and use it here.
+        If you already ingested the corpus_df to the collection, the embedding process will not be repeated.
+        The default is None. If None, it makes a temporary collection.
+    :param upsert: If true, the function will overwrite the existing file if it exists.
+    :param random_state: The random state for sampling corpus from the given corpus_df.
+    :param cache_batch: The number of batches to use for caching the generated QA dataset.
+    :param top_k: The number of sources to refer by model.
+        Default is 3.
+    :param kwargs: The keyword arguments for qa_creation_func.
+    :return: QA dataset dataframe.
+    """
+    assert 'query' in existing_query_df.columns, "existing_query_df must have 'query' column."
+    assert content_size > 0, "content_size must be greater than 0."
+    if content_size > len(corpus_df):
+        logger.warning(f"content_size {content_size} is larger than the corpus size {len(corpus_df)}. "
+                       "Setting content_size to the corpus size.")
+        content_size = len(corpus_df)
+
+    logger.info("Loading local embedding model...")
+    embeddings = autorag.embedding_models[embedding_model]
+
+    # Vector DB creation
+    if collection is None:
+        chroma_client = chromadb.Client()
+        collection_name = "auto-rag"
+        collection = chroma_client.get_or_create_collection(collection_name)
+
+    # embed corpus_df
+    vectordb_ingest(collection, corpus_df, embeddings)
+    vectordb_func = vectordb.__wrapped__
+    retrieved_ids, retrieve_scores = vectordb_func(existing_query_df['query'].tolist(), top_k, collection, embeddings)
+
+    retrieved_contents: List[List[str]] = fetch_contents(corpus_df, retrieved_ids)
+    input_passage_strs: List[str] = list(map(
+        lambda x: '\n'.join([f"Document {i + 1}\n{content}" for i, content in enumerate(x)]),
+        retrieved_contents))
+    retrieved_qa_df = pd.DataFrame({
+        'qid': [str(uuid.uuid4()) for _ in range(len(existing_query_df))],
+        'query': existing_query_df['query'].tolist(),
+        'retrieval_gt': list(map(lambda x: [x], retrieved_ids)),
+        'input_passage_str': input_passage_strs,
+    })
+
+    sample_qa_df = retrieved_qa_df.sample(n=min(content_size, len(retrieved_qa_df)), random_state=random_state)
+
+    generation_gt = answer_creation_func(contents=sample_qa_df['input_passage_str'].tolist(),
+                                         queries=sample_qa_df['query'].tolist(),
+                                         batch=cache_batch,
+                                         **kwargs)
+    qa_df = sample_qa_df.copy(deep=True)
+    qa_df.drop(columns=['input_passage_str'], inplace=True)
+    qa_df['generation_gt'] = generation_gt
+
+    if output_filepath is not None:
+        save_parquet_safe(qa_df, output_filepath, upsert=upsert)
+
+    return qa_df

{autorag-0.2.13 → autorag-0.2.14}/autorag/data/qacreation/llama_index.py

@@ -4,6 +4,7 @@ import random
 from typing import Optional, List, Dict, Any
 
 import pandas as pd
+from llama_index.core.base.llms.types import ChatMessage, MessageRole
 from llama_index.core.service_context_elements.llm_predictor import LLMPredictorType
 
 from autorag.utils.util import process_batch
@@ -53,6 +54,31 @@ def generate_qa_llama_index(
     return results
 
 
+def generate_answers(
+    llm: LLMPredictorType,
+    contents: List[str],
+    queries: List[str],
+    batch: int = 4,
+) -> List[List[Dict]]:
+    """
+    Generate qa sets from the list of contents using existing queries.
+
+    :param llm: Llama index model
+    :param contents: List of content strings.
+    :param queries: List of existing queries.
+    :param batch: The batch size to process asynchronously.
+    :return: 2-d list of dictionaries containing the query and generation_gt.
+    """
+
+    tasks = [
+        generate_basic_answer(llm, content, query)
+        for content, query in zip(contents, queries)
+    ]
+    loops = asyncio.get_event_loop()
+    results = loops.run_until_complete(process_batch(tasks, batch))
+    return results
+
+
 def generate_qa_llama_index_by_ratio(
         llm: LLMPredictorType,
         contents: List[str],
@@ -147,6 +173,21 @@ async def async_qa_gen_llama_index(
     return await generate(content, llm)
 
 
+async def generate_basic_answer(llm: LLMPredictorType, passage_str: str, query: str) -> str:
+    basic_answer_system_prompt = """You are an AI assistant to answer the given question in the provide evidence text.
+    You can find the evidence from the given text about question, and you have to write a proper answer to the given question.
+    You have to preserve the question's language at the answer.
+    For example, if the input question is Korean, the output answer must be in Korean.
+    """
+    user_prompt = f"Text:\n<|text_start|>\n{passage_str}\n<|text_end|>\n\nQuestion:\n{query}\n\nAnswer:"
+
+    response = await llm.achat(messages=[
+        ChatMessage(role=MessageRole.SYSTEM, content=basic_answer_system_prompt),
+        ChatMessage(role=MessageRole.USER, content=user_prompt)
+    ], temperature=1.0)
+    return response.message.content
+
+
 def validate_llama_index_prompt(prompt: str) -> bool:
     """
     Validate the prompt for the llama index model.

{autorag-0.2.13 → autorag-0.2.14}/autorag/deploy.py

@@ -4,6 +4,7 @@ import uuid
 from copy import deepcopy
 from typing import Optional, Dict, List
 
+import nest_asyncio
 import pandas as pd
 import uvicorn
 import yaml
@@ -65,7 +66,7 @@ def summary_df_to_yaml(summary_df: pd.DataFrame, config_dict: Dict) -> Dict:
     summary_df['categorical_node_line_name'] = pd.Categorical(summary_df['node_line_name'], categories=node_line_names,
                                                               ordered=True)
     summary_df = summary_df.sort_values(by='categorical_node_line_name')
-    grouped = summary_df.groupby('categorical_node_line_name')
+    grouped = summary_df.groupby('categorical_node_line_name', observed=False)
 
     node_lines = [
         {
@@ -207,7 +208,7 @@ class Runner:
 
             {
                 "Query": "your query",
-                "result_column": "answer"
+                "result_column": "generated_texts"
             }
 
         And it returns json response like below:
@@ -222,10 +223,11 @@ class Runner:
         :param port: The port of the api server.
         :param kwargs: Other arguments for uvicorn.run.
         """
+        nest_asyncio.apply()
         logger.info(f"Run api server at {host}:{port}")
-        uvicorn.run(self.app, host=host, port=port, **kwargs)
+        uvicorn.run(self.app, host=host, port=port, loop="asyncio", **kwargs)
 
 
 class RunnerInput(BaseModel):
     query: str
-    result_column: str = "answer"
+    result_column: str = "generated_texts"

{autorag-0.2.13 → autorag-0.2.14}/autorag/evaluation/metric/generation.py

@@ -5,6 +5,7 @@ import os
 from typing import List, Optional
 
 import evaluate
+import nltk
 import pandas as pd
 import torch
 from llama_index.core.embeddings import BaseEmbedding
@@ -109,6 +110,7 @@ def meteor(generation_gt: List[List[str]], generations: List[str],
         Default is 0.5.
     :return: A list of computed metric scores.
     """
+    nltk.download('punkt_tab')
     meteor_instance = evaluate.load("meteor")
     result = huggingface_evaluate(meteor_instance, 'meteor', generation_gt, generations,
                                   alpha=alpha, beta=beta, gamma=gamma)

{autorag-0.2.13 → autorag-0.2.14}/autorag/nodes/passageaugmenter/base.py

@@ -44,7 +44,8 @@ def passage_augmenter_node(func):
         if func.__name__ == 'prev_next_augmenter':
             corpus_df = cast_corpus_dataset(corpus_df)
             slim_corpus_df = corpus_df[["doc_id", "metadata"]]
-            slim_corpus_df['metadata'] = slim_corpus_df['metadata'].apply(filter_dict_keys, keys=['prev_id', 'next_id'])
+            slim_corpus_df.loc[:, 'metadata'] = slim_corpus_df['metadata'].apply(
+                filter_dict_keys, keys=['prev_id', 'next_id'])
 
             mode = kwargs.pop("mode", 'both')
             num_passages = kwargs.pop("num_passages", 1)

{autorag-0.2.13 → autorag-0.2.14}/autorag/nodes/passagefilter/threshold_cutoff.py

@@ -1,8 +1,7 @@
 from typing import List, Tuple
 
-import numpy as np
-
 from autorag.nodes.passagefilter.base import passage_filter_node
+from autorag.utils.util import convert_inputs_to_list
 
 
 @passage_filter_node
@@ -32,6 +31,7 @@ def threshold_cutoff(queries: List[str], contents_list: List[List[str]],
     return remain_content_list, remain_ids_list, remain_scores_list
 
 
+@convert_inputs_to_list
 def threshold_cutoff_pure(scores_list: List[float],
                           threshold: float,
                           reverse: bool = False) -> List[int]:
@@ -45,20 +45,13 @@ def threshold_cutoff_pure(scores_list: List[float],
         Default is False.
     :return: Indices to remain at the contents
     """
-    if isinstance(scores_list, np.ndarray):
-        if reverse:
-            remain_indices = np.where(scores_list <= threshold)[0]
-            default_index = np.argmin(scores_list)
-        else:
-            remain_indices = np.where(scores_list >= threshold)[0]
-            default_index = np.argmax(scores_list)
+    assert isinstance(scores_list, list), "scores_list must be a list."
+
+    if reverse:
+        remain_indices = [i for i, score in enumerate(scores_list) if score <= threshold]
+        default_index = scores_list.index(min(scores_list))
     else:
-        if reverse:
-            remain_indices = [i for i, score in enumerate(scores_list) if score <= threshold]
-            default_index = scores_list.index(min(scores_list))
-        else:
-            remain_indices = [i for i, score in enumerate(scores_list) if score >= threshold]
-            default_index = scores_list.index(max(scores_list))
+        remain_indices = [i for i, score in enumerate(scores_list) if score >= threshold]
+        default_index = scores_list.index(max(scores_list))
 
-    return remain_indices.tolist() if isinstance(remain_indices, np.ndarray) and remain_indices.size > 0 else \
-        remain_indices if remain_indices else [default_index]
+    return remain_indices if remain_indices else [default_index]

{autorag-0.2.13 → autorag-0.2.14}/autorag/nodes/retrieval/base.py

@@ -55,7 +55,7 @@ def retrieval_node(func):
         assert "query" in previous_result.columns, "previous_result must have query column."
         if "queries" not in previous_result.columns:
             previous_result["queries"] = previous_result["query"]
-        previous_result["queries"] = previous_result["queries"].apply(cast_queries)
+        previous_result.loc[:, "queries"] = previous_result["queries"].apply(cast_queries)
         queries = previous_result["queries"].tolist()
 
         # run retrieval function

{autorag-0.2.13 → autorag-0.2.14}/autorag/strategy.py

@@ -127,7 +127,7 @@ def select_best_rr(results: List[pd.DataFrame], columns: Iterable[str],
     results, columns, metadatas = validate_strategy_inputs(results, columns, metadatas)
     each_average_df = pd.DataFrame([df[columns].mean(axis=0).to_dict() for df in results])
     rank_df = each_average_df.rank(ascending=False)
-    rr_df = rank_df.applymap(lambda x: 1 / x)
+    rr_df = rank_df.map(lambda x: 1 / x)
     best_index = np.array(rr_df.sum(axis=1)).argmax()
     return results[best_index], metadatas[best_index]
 

{autorag-0.2.13 → autorag-0.2.14}/autorag/utils/util.py

@@ -96,7 +96,7 @@ def load_summary_file(summary_path: str,
                 raise ValueError(f"Malformed dict received : {elem}\nCan't convert to dict properly")
             return {'threshold': date_object}
 
-    summary_df[dict_columns] = summary_df[dict_columns].applymap(convert_dict)
+    summary_df[dict_columns] = summary_df[dict_columns].map(convert_dict)
     return summary_df
 
 

{autorag-0.2.13 → autorag-0.2.14}/docs/source/data_creation/tutorial.md

@@ -14,6 +14,7 @@ myst:
 3. [Corpus data to QA data](#make-qa-data-from-corpus-data)
 4. [Use custom prompt](#use-custom-prompt)
 5. [Use multiple prompts](#use-multiple-prompts)
+6. [If there are existing queries](#when-you-have-existing-queries)
 
 
 ## Overview
@@ -167,3 +168,48 @@ qa_df = make_single_content_qa(corpus_df, content_size=50, qa_creation_func=gene
 ```{warning}
 Remeber all prompts must have the placeholders `{{text}}` and `{{num_questions}}`.
 ```
+
+## When you have existing queries
+
+When you have existing queries, you can use it for AutoRAG.
+The real user's question is valuable data, so it is always great to use it prior to generating synthetic data.
+
+But you have to make retrieval_gt for existing queries from your corpus data.
+The process to find the retrieval_gt at the corpus is hard, but must be accurate.
+For making it less hard, we use an embedding model and vectordb for finding relevant passages.
+After that, you have to clarify the retrieval_gt is right.
+If retrieval_gt is not relevant, you have to remove it on the dataset.
+
+```python
+import pandas as pd
+from llama_index.llms.openai import OpenAI
+from autorag.data.qacreation import make_qa_with_existing_queries, generate_answers
+
+corpus_df = pd.read_parquet('path/to/corpus.parquet')
+existing_qa_df = pd.read_parquet('path/to/existing_qa.parquet')  # It have to contain 'query' column
+llm = OpenAI(model='gpt-3.5-turbo', temperature=1.0)
+qa_df = make_qa_with_existing_queries(corpus_df, existing_qa_df, content_size=50,
+                                      answer_creation_func=generate_answers,
+                                      llm=llm, output_filepath='path/to/qa.parquet', cache_batch=64,
+                                      embedding_model='openai_embed_3_large', top_k=5)
+```
+
+You can use `PersistentClient` for saving corpus embeddings locally as well.
+
+```python
+import pandas as pd
+import chromadb
+from llama_index.llms.openai import OpenAI
+from autorag.data.qacreation import make_qa_with_existing_queries, generate_answers
+
+client = chromadb.PersistentClient('path/to/chromadb')
+collection = client.get_or_create_collection('auto-rag')
+
+corpus_df = pd.read_parquet('path/to/corpus.parquet')
+existing_qa_df = pd.read_parquet('path/to/existing_qa.parquet')  # It have to contain 'query' column
+llm = OpenAI(model='gpt-3.5-turbo', temperature=1.0)
+qa_df = make_qa_with_existing_queries(corpus_df, existing_qa_df, content_size=50,
+                                      answer_creation_func=generate_answers, collection=collection,
+                                      llm=llm, output_filepath='path/to/qa.parquet', cache_batch=64,
+                                      embedding_model='openai_embed_3_large', top_k=5)
+```

{autorag-0.2.13 → autorag-0.2.14}/docs/source/local_model.md

@@ -117,6 +117,7 @@ To change the embedding model, you can change the `embedding_model` parameter to
 |                  [BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5)                  |      huggingface_baai_bge_small       |
 |               [cointegrated/rubert-tiny2](https://huggingface.co/cointegrated/rubert-tiny2)               | huggingface_cointegrated_rubert_tiny2 |
 | [sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) |     huggingface_all_mpnet_base_v2     |
+|                             [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3)                             |          huggingface_bge_m3           |
 
 For example, if you want to use OpenAI text embedding large model, you can set `embedding_model` parameter
 to `openai_embed_3_large`.