AutoRAG 0.2.12__tar.gz → 0.2.13__tar.gz

{autorag-0.2.12 → autorag-0.2.13}/.github/workflows/test.yml

@@ -16,6 +16,10 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
+      - uses: actions/setup-java@v4
+        with:
+          distribution: 'zulu'
+          java-version: '17'
       - name: Upgrade pip
         run: |
           python3 -m pip install --upgrade pip
@@ -24,7 +28,7 @@ jobs:
           sudo apt-get install gcc
       - name: Install AutoRAG
         run: |
-          pip install -e .
+          pip install -e '.[all]'
       - name: Install dependencies
         run: |
           pip install -r tests/requirements.txt

{autorag-0.2.12 → autorag-0.2.13}/AutoRAG.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: AutoRAG
-Version: 0.2.12
+Version: 0.2.13
 Summary: Automatically Evaluate RAG pipelines with your own data. Find optimal structure for new RAG product.
 Author-email: Marker-Inc <vkehfdl1@gmail.com>
 License:                                  Apache License
@@ -249,7 +249,6 @@ Requires-Dist: bert_score
 Requires-Dist: sentence-transformers
 Requires-Dist: FlagEmbedding
 Requires-Dist: ragas
-Requires-Dist: kiwipiepy
 Requires-Dist: llmlingua
 Requires-Dist: llama-index>=0.10.1
 Requires-Dist: llama-index-core>=0.10.1
@@ -268,6 +267,11 @@ Requires-Dist: seaborn
 Requires-Dist: ipykernel
 Requires-Dist: ipywidgets
 Requires-Dist: ipywidgets_bokeh
+Provides-Extra: ko
+Requires-Dist: kiwipiepy; extra == "ko"
+Requires-Dist: konlpy; extra == "ko"
+Provides-Extra: all
+Requires-Dist: AutoRAG[ko]; extra == "all"
 
 # AutoRAG
 
@@ -282,7 +286,6 @@ Plus, join our 📞 [Discord](https://discord.gg/P4DYXfmSAs) Community.
 ### 💪 Colab Tutorial
 
 - [Step 1: Basic of AutoRAG | Optimizing your RAG pipeline](https://colab.research.google.com/drive/19OEQXO_pHN6gnn2WdfPd4hjnS-4GurVd?usp=sharing)
-- [Step 2: Create evaluation dataset](https://colab.research.google.com/drive/1HXjVHCLTaX7mkmZp3IKlEPt0B3jVeHvP#scrollTo=cgFUCuaUZvTr)
 
 ---
 
@@ -330,7 +333,7 @@ Try now and find the best RAG pipeline for your own use-case.
 
 ## ❗Supporting Nodes & modules
 
-![module_1](https://github.com/Marker-Inc-Korea/AutoRAG/assets/96727832/828d8752-13f2-4918-8ca4-2605a86596cb)
+![module_1](https://github.com/user-attachments/assets/d49cc7b4-56a6-4b5f-bb3d-2fc5b962996b)
 ![module_2](https://github.com/Marker-Inc-Korea/AutoRAG/assets/96727832/4db3d83f-3178-4b81-bad1-b751dcd31a13)
 You can check our all supporting Nodes & modules
 at [here](https://edai.notion.site/Supporting-Nodes-modules-0ebc7810649f4e41aead472a92976be4?pvs=4)

{autorag-0.2.12 → autorag-0.2.13}/AutoRAG.egg-info/SOURCES.txt

@@ -25,6 +25,7 @@ autorag/evaluator.py
 autorag/node_line.py
 autorag/strategy.py
 autorag/support.py
+autorag/validator.py
 autorag/web.py
 autorag/data/__init__.py
 autorag/data/corpus/__init__.py
@@ -264,6 +265,7 @@ tests/autorag/test_deploy.py
 tests/autorag/test_evaluator.py
 tests/autorag/test_strategy.py
 tests/autorag/test_support.py
+tests/autorag/test_validator.py
 tests/autorag/test_web.py
 tests/autorag/data/corpus/test_base.py
 tests/autorag/data/corpus/test_langchain.py

{autorag-0.2.12 → autorag-0.2.13}/AutoRAG.egg-info/requires.txt

@@ -27,7 +27,6 @@ bert_score
 sentence-transformers
 FlagEmbedding
 ragas
-kiwipiepy
 llmlingua
 llama-index>=0.10.1
 llama-index-core>=0.10.1
@@ -46,3 +45,10 @@ seaborn
 ipykernel
 ipywidgets
 ipywidgets_bokeh
+
+[all]
+AutoRAG[ko]
+
+[ko]
+kiwipiepy
+konlpy

{autorag-0.2.12 → autorag-0.2.13}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: AutoRAG
-Version: 0.2.12
+Version: 0.2.13
 Summary: Automatically Evaluate RAG pipelines with your own data. Find optimal structure for new RAG product.
 Author-email: Marker-Inc <vkehfdl1@gmail.com>
 License:                                  Apache License
@@ -249,7 +249,6 @@ Requires-Dist: bert_score
 Requires-Dist: sentence-transformers
 Requires-Dist: FlagEmbedding
 Requires-Dist: ragas
-Requires-Dist: kiwipiepy
 Requires-Dist: llmlingua
 Requires-Dist: llama-index>=0.10.1
 Requires-Dist: llama-index-core>=0.10.1
@@ -268,6 +267,11 @@ Requires-Dist: seaborn
 Requires-Dist: ipykernel
 Requires-Dist: ipywidgets
 Requires-Dist: ipywidgets_bokeh
+Provides-Extra: ko
+Requires-Dist: kiwipiepy; extra == "ko"
+Requires-Dist: konlpy; extra == "ko"
+Provides-Extra: all
+Requires-Dist: AutoRAG[ko]; extra == "all"
 
 # AutoRAG
 
@@ -282,7 +286,6 @@ Plus, join our 📞 [Discord](https://discord.gg/P4DYXfmSAs) Community.
 ### 💪 Colab Tutorial
 
 - [Step 1: Basic of AutoRAG | Optimizing your RAG pipeline](https://colab.research.google.com/drive/19OEQXO_pHN6gnn2WdfPd4hjnS-4GurVd?usp=sharing)
-- [Step 2: Create evaluation dataset](https://colab.research.google.com/drive/1HXjVHCLTaX7mkmZp3IKlEPt0B3jVeHvP#scrollTo=cgFUCuaUZvTr)
 
 ---
 
@@ -330,7 +333,7 @@ Try now and find the best RAG pipeline for your own use-case.
 
 ## ❗Supporting Nodes & modules
 
-![module_1](https://github.com/Marker-Inc-Korea/AutoRAG/assets/96727832/828d8752-13f2-4918-8ca4-2605a86596cb)
+![module_1](https://github.com/user-attachments/assets/d49cc7b4-56a6-4b5f-bb3d-2fc5b962996b)
 ![module_2](https://github.com/Marker-Inc-Korea/AutoRAG/assets/96727832/4db3d83f-3178-4b81-bad1-b751dcd31a13)
 You can check our all supporting Nodes & modules
 at [here](https://edai.notion.site/Supporting-Nodes-modules-0ebc7810649f4e41aead472a92976be4?pvs=4)

{autorag-0.2.12 → autorag-0.2.13}/README.md

@@ -11,7 +11,6 @@ Plus, join our 📞 [Discord](https://discord.gg/P4DYXfmSAs) Community.
 ### 💪 Colab Tutorial
 
 - [Step 1: Basic of AutoRAG | Optimizing your RAG pipeline](https://colab.research.google.com/drive/19OEQXO_pHN6gnn2WdfPd4hjnS-4GurVd?usp=sharing)
-- [Step 2: Create evaluation dataset](https://colab.research.google.com/drive/1HXjVHCLTaX7mkmZp3IKlEPt0B3jVeHvP#scrollTo=cgFUCuaUZvTr)
 
 ---
 
@@ -59,7 +58,7 @@ Try now and find the best RAG pipeline for your own use-case.
 
 ## ❗Supporting Nodes & modules
 
-![module_1](https://github.com/Marker-Inc-Korea/AutoRAG/assets/96727832/828d8752-13f2-4918-8ca4-2605a86596cb)
+![module_1](https://github.com/user-attachments/assets/d49cc7b4-56a6-4b5f-bb3d-2fc5b962996b)
 ![module_2](https://github.com/Marker-Inc-Korea/AutoRAG/assets/96727832/4db3d83f-3178-4b81-bad1-b751dcd31a13)
 You can check our all supporting Nodes & modules
 at [here](https://edai.notion.site/Supporting-Nodes-modules-0ebc7810649f4e41aead472a92976be4?pvs=4)

autorag-0.2.13/autorag/VERSION

@@ -0,0 +1 @@
+0.2.13

{autorag-0.2.12 → autorag-0.2.13}/autorag/cli.py

@@ -12,6 +12,7 @@ from autorag import dashboard
 from autorag.deploy import Runner
 from autorag.deploy import extract_best_config as original_extract_best_config
 from autorag.evaluator import Evaluator
+from autorag.validator import Validator
 
 logger = logging.getLogger("AutoRAG")
 
@@ -33,7 +34,6 @@ def evaluate(config, qa_data_path, corpus_data_path, project_dir):
         raise ValueError(f"Config file {config} does not exist.")
     evaluator = Evaluator(qa_data_path, corpus_data_path, project_dir=project_dir)
     evaluator.start_trial(config)
-    logger.info('Evaluation complete.')
 
 
 @click.command()
@@ -95,7 +95,19 @@ def restart_evaluate(trial_path):
     corpus_data_path = os.path.join(project_dir, 'data', 'corpus.parquet')
     evaluator = Evaluator(qa_data_path, corpus_data_path, project_dir)
     evaluator.restart_trial(trial_path)
-    logger.info('Evaluation complete.')
+
+
+@click.command()
+@click.option('--config', '-c', help='Path to config yaml file. Must be yaml or yml file.', type=str)
+@click.option('--qa_data_path', help='Path to QA dataset. Must be parquet file.', type=str)
+@click.option('--corpus_data_path', help='Path to corpus dataset. Must be parquet file.', type=str)
+def validate(config, qa_data_path, corpus_data_path):
+    if not config.endswith('.yaml') and not config.endswith('.yml'):
+        raise ValueError(f"Config file {config} is not a parquet file.")
+    if not os.path.exists(config):
+        raise ValueError(f"Config file {config} does not exist.")
+    validator = Validator(qa_data_path=qa_data_path, corpus_data_path=corpus_data_path)
+    validator.validate(config)
 
 
 cli.add_command(evaluate, 'evaluate')
@@ -104,6 +116,7 @@ cli.add_command(run_web, 'run_web')
 cli.add_command(run_dashboard, 'dashboard')
 cli.add_command(extract_best_config, 'extract_best_config')
 cli.add_command(restart_evaluate, 'restart_evaluate')
+cli.add_command(validate, 'validate')
 
 if __name__ == '__main__':
     cli()

{autorag-0.2.12 → autorag-0.2.13}/autorag/evaluator.py

@@ -101,6 +101,8 @@ class Evaluator:
 
         trial_summary_df.to_csv(os.path.join(self.project_dir, trial_name, 'summary.csv'), index=False)
 
+        logger.info('Evaluation complete.')
+
     def __embed(self, node_lines: Dict[str, List[Node]]):
         if any(list(map(lambda nodes: module_type_exists(nodes, 'bm25'), node_lines.values()))):
             # ingest BM25 corpus
@@ -265,6 +267,8 @@ class Evaluator:
                 trial_summary_df = self._append_node_line_summary(node_line_name, node_line_dir, trial_summary_df)
         trial_summary_df.to_csv(os.path.join(trial_path, 'summary.csv'), index=False)
 
+        logger.info('Evaluation complete.')
+
     def __find_conflict_point(self, trial_path: str, node_line_names: List[str],
                               node_lines: Dict[str, List[Node]]) -> tuple[str, str]:
         for node_line_name in node_line_names:

{autorag-0.2.12 → autorag-0.2.13}/autorag/nodes/retrieval/bm25.py

@@ -6,7 +6,6 @@ from typing import List, Dict, Tuple, Callable, Union, Iterable, Optional
 
 import numpy as np
 import pandas as pd
-from kiwipiepy import Kiwi, Token
 from llama_index.core.indices.keyword_table.utils import simple_extract_keywords
 from nltk import PorterStemmer
 from rank_bm25 import BM25Okapi
@@ -18,12 +17,42 @@ from autorag.utils.util import normalize_string
 
 
 def tokenize_ko_kiwi(texts: List[str]) -> List[List[str]]:
+    try:
+        from kiwipiepy import Kiwi, Token
+    except ImportError:
+        raise ImportError("You need to install kiwipiepy to use 'ko_kiwi' tokenizer. "
+                          "Please install kiwipiepy by running 'pip install kiwipiepy'. "
+                          "Or install Korean version of AutoRAG by running 'pip install AutoRAG[ko]'.")
     texts = list(map(lambda x: x.strip().lower(), texts))
     kiwi = Kiwi()
     tokenized_list: Iterable[List[Token]] = kiwi.tokenize(texts)
     return [list(map(lambda x: x.form, token_list)) for token_list in tokenized_list]
 
 
+def tokenize_ko_kkma(texts: List[str]) -> List[List[str]]:
+    try:
+        from konlpy.tag import Kkma
+    except ImportError:
+        raise ImportError("You need to install konlpy to use 'ko_kkma' tokenizer. "
+                          "Please install konlpy by running 'pip install konlpy'. "
+                          "Or install Korean version of AutoRAG by running 'pip install AutoRAG[ko]'.")
+    tokenizer = Kkma()
+    tokenized_list: List[List[str]] = list(map(lambda x: tokenizer.morphs(x), texts))
+    return tokenized_list
+
+
+def tokenize_ko_okt(texts: List[str]) -> List[List[str]]:
+    try:
+        from konlpy.tag import Okt
+    except ImportError:
+        raise ImportError("You need to install konlpy to use 'ko_kkma' tokenizer. "
+                          "Please install konlpy by running 'pip install konlpy'. "
+                          "Or install Korean version of AutoRAG by running 'pip install AutoRAG[ko]'.")
+    tokenizer = Okt()
+    tokenized_list: List[List[str]] = list(map(lambda x: tokenizer.morphs(x), texts))
+    return tokenized_list
+
+
 def tokenize_porter_stemmer(texts: List[str]) -> List[List[str]]:
     def tokenize_remove_stopword(text: str, stemmer) -> List[str]:
         text = text.lower()
@@ -47,6 +76,8 @@ BM25_TOKENIZER = {
     'porter_stemmer': tokenize_porter_stemmer,
     'ko_kiwi': tokenize_ko_kiwi,
     'space': tokenize_space,
+    'ko_kkma': tokenize_ko_kkma,
+    'ko_okt': tokenize_ko_okt,
 }
 
 

autorag-0.2.13/autorag/validator.py

@@ -0,0 +1,66 @@
+import itertools
+import logging
+import os
+import tempfile
+
+import pandas as pd
+
+from autorag.evaluator import Evaluator
+from autorag.utils import cast_qa_dataset, cast_corpus_dataset, validate_qa_from_corpus_dataset
+
+logger = logging.getLogger("AutoRAG")
+
+
+class Validator:
+    def __init__(self, qa_data_path: str, corpus_data_path: str):
+        """
+        Initialize a Validator object.
+
+        :param qa_data_path: The path to the QA dataset.
+            Must be parquet file.
+        :param corpus_data_path: The path to the corpus dataset.
+            Must be parquet file.
+        """
+        # validate data paths
+        if not os.path.exists(qa_data_path):
+            raise ValueError(f"QA data path {qa_data_path} does not exist.")
+        if not os.path.exists(corpus_data_path):
+            raise ValueError(f"Corpus data path {corpus_data_path} does not exist.")
+        if not qa_data_path.endswith('.parquet'):
+            raise ValueError(f"QA data path {qa_data_path} is not a parquet file.")
+        if not corpus_data_path.endswith('.parquet'):
+            raise ValueError(f"Corpus data path {corpus_data_path} is not a parquet file.")
+        self.qa_data = pd.read_parquet(qa_data_path, engine='pyarrow')
+        self.corpus_data = pd.read_parquet(corpus_data_path, engine='pyarrow')
+        self.qa_data = cast_qa_dataset(self.qa_data)
+        self.corpus_data = cast_corpus_dataset(self.corpus_data)
+
+    def validate(self, yaml_path: str, qa_cnt: int = 5, random_state: int = 42):
+        # sample QA data
+        sample_qa_df = self.qa_data.sample(qa_cnt, random_state=random_state)
+        sample_qa_df.reset_index(drop=True, inplace=True)
+
+        # get doc_id
+        temp_qa_df = sample_qa_df.copy(deep=True)
+        flatten_retrieval_gts = temp_qa_df['retrieval_gt'].apply(
+            lambda x: list(itertools.chain.from_iterable(x))).tolist()
+        target_doc_ids = list(itertools.chain.from_iterable(flatten_retrieval_gts))
+
+        # make sample corpus data
+        sample_corpus_df = self.corpus_data.loc[self.corpus_data['doc_id'].isin(target_doc_ids)]
+        sample_corpus_df.reset_index(drop=True, inplace=True)
+
+        validate_qa_from_corpus_dataset(sample_qa_df, sample_corpus_df)
+
+        # start Evaluate at temp project directory
+        with tempfile.NamedTemporaryFile(suffix='.parquet') as qa_path, \
+                tempfile.NamedTemporaryFile(suffix='.parquet') as corpus_path, \
+                tempfile.TemporaryDirectory() as temp_project_dir:
+            sample_qa_df.to_parquet(qa_path.name, index=False)
+            sample_corpus_df.to_parquet(corpus_path.name, index=False)
+
+            evaluator = Evaluator(qa_data_path=qa_path.name, corpus_data_path=corpus_path.name,
+                                  project_dir=temp_project_dir)
+            evaluator.start_trial(yaml_path)
+
+        logger.info("Validation complete.")

{autorag-0.2.12 → autorag-0.2.13}/docs/source/api_spec/autorag.rst

@@ -72,6 +72,14 @@ autorag.support module
    :undoc-members:
    :show-inheritance:
 
+autorag.validator module
+------------------------
+
+.. automodule:: autorag.validator
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
 autorag.web module
 ------------------
 

{autorag-0.2.12 → autorag-0.2.13}/docs/source/data_creation/data_format.md

@@ -76,6 +76,14 @@ it is okay to save it as 1-d list or just string.
 If you save it as 1-d list, it treats as 'and' operation.
 ```
 
+```{attention}
+The ids of retrieval_gt must be included in the corpus dataset as `doc_id`.
+
+When AutoRAG starts to evaluate, it checks the existence of the retrieval ids in the corpus dataset.
+
+You MUST match the retrieval_gt ids with the corpus dataset.
+```
+
 This column is crucial because AutoRAG evaluate retrieval performance with this column.
 It can affect hugely to optimization performance or nodes like retrieval, query expansion or passage reranker.
 
@@ -111,6 +119,8 @@ A unique identifier for each passage. Its type is `string`.
 
 ```{warning}
 Do not make a duplicate doc id. It can occur unexpected behavior.
+
+Plus, we suggest you to double-check that retrieval_gt ids in the QA dataset are included in the corpus dataset as `doc_id`.
 ```
 
 ### contents

{autorag-0.2.12 → autorag-0.2.13}/docs/source/install.md

@@ -18,6 +18,20 @@ Do you have any trouble with installation?
 First, you can check out the [troubleshooting](troubleshooting.md) page.
 ```
 
+### Installation for Korean 🇰🇷
+
+You can install optional dependencies for Korean language.
+
+```bash
+pip install AutoRAG[ko]
+```
+
+And after that, you have to install **jdk 17** for using `konlpy`.
+Plus, don't forget to set environment PATH for jdk. (JAVA_HOME and PATH)
+
+The instruction for mac users
+is [here](https://velog.io/@yoonsy/M1%EC%B9%A9-Mac%EC%97%90-konlpy-%EC%84%A4%EC%B9%98%ED%95%98%EA%B8%B0).
+
 ## Setup OPENAI API KEY
 To use LLM and embedding models, it is common to use OpenAI models.
 If you want to use other models, check out [here](local_model.md)

{autorag-0.2.12 → autorag-0.2.13}/docs/source/local_model.md

@@ -66,7 +66,7 @@ This is the parameter for the LLM model.
 You can set the model parameter for LlamaIndex LLM initialization.
 The most frequently used parameters are `model`, `max_token`, and `temperature`.
 Please check what you can set for the model parameter
-at [LlamaIndex LLM](https://docs.llamaindex.ai/en/latest/api_reference/llms.html).
+at [LlamaIndex LLM](https://docs.llamaindex.ai/en/stable/module_guides/models/llms/).
 
 ### Add more LLM models
 

{autorag-0.2.12 → autorag-0.2.13}/docs/source/nodes/generator/llama_index_llm.md

@@ -7,7 +7,9 @@ myst:
 ---
 # llama_index LLM
 
-The `llama_index_llm` module is generator based on [llama_index](https://docs.llamaindex.ai/en/stable/api_reference/llms.html). It gets the LLM instance from llama index, and returns generated text by the input prompt. 
+The `llama_index_llm` module is generator based
+on [llama_index](https://docs.llamaindex.ai/en/stable/module_guides/models/llms/). It gets the LLM instance from llama
+index, and returns generated text by the input prompt.
 It does not generate log probs.
 
 ## **Module Parameters**

{autorag-0.2.12 → autorag-0.2.13}/docs/source/nodes/prompt_maker/prompt_maker.md

@@ -25,18 +25,30 @@ Please refer to the parameter of [Generator Node](../generator/generator.md) for
 
 #### **Strategy Parameters**:
 
-1. **Metrics**: Metrics such as `bleu`,`meteor`, and `rouge` are used to evaluate the performance of the prompt maker process through its impact on generator (llm) outcomes.
-2. **Speed Threshold**: `speed_threshold` is applied across all nodes, ensuring that any method exceeding the average processing time for a query is not utilized.
-3. **Token Threshold**: `token_threshold` ensuring that output prompt average token length does not exceed the
+1. **Metrics**: (Essential) Metrics such as `bleu`,`meteor`, and `rouge` are used to evaluate the performance of the
+   prompt maker process through its impact on generator (llm) outcomes.
+2. **Speed Threshold**: (Optional) `speed_threshold` is applied across all nodes, ensuring that any method exceeding the
+   average processing time for a query is not utilized.
+3. **Token Threshold**: (Optional) `token_threshold` ensuring that output prompt average token length does not exceed
+   the
    threshold.
-4. **tokenizer**: Since you don't know what LLM model you will use in the next nodes, you can specify the tokenizer name
+4. **tokenizer**: (Optional) Since you don't know what LLM model you will use in the next nodes, you can specify the
+   tokenizer name
    to use in `token_threshold` strategy.
    You can use OpenAI model names or Huggingface model names that support `AutoTokenizer`.
    It will automatically find the tokenizer for the model name you specify.
    Default is 'gpt2'.
-5. **Generator Modules**: The prompt maker node can use all modules and module parameters from the generator node,
+5. **Generator Modules**: (Optional, but recommended to set) The prompt maker node can use all modules and module
+   parameters from the generator node,
    including:
    - [llama_index_llm](../generator/llama_index_llm.md): with `llm` and additional llm parameters
+   - [openai_llm](../generator/openai_llm.md): with `llm` and additional openai `AsyncOpenAI` parameters
+   - [vllm](../generator/vllm.md): with `llm` and additional vllm parameters
+
+   And the default model of generator module for evaluating prompt maker is openai gpt-3.5-turbo model.
+
+   Plus, the evaluation of prompt maker will skip when there are only one combination of prompt maker modules and
+   options.
 
 ### Example config.yaml file
 ```yaml
@@ -69,4 +81,4 @@ maxdepth: 1
 fstring.md
 long_context_reorder.md
 window_replacement.md
-```
+```

{autorag-0.2.12 → autorag-0.2.13}/docs/source/nodes/retrieval/bm25.md

@@ -13,7 +13,8 @@ The `BM25` is the most popular TF-IDF method for retrieval, which reflects how i
 
 - **bm25_tokenizer**: You can select which tokenize method you use for bm25.
   The default method is 'porter_stemmer'.
-  And you can choose between 'ko_kiwi', 'space', and huggingface AutoTokenizer name.
+  And you can choose between 'space', and huggingface AutoTokenizer name.
+  Plus, you can choose Korean tokenizer such as 'ko_kiwi', 'ko_kkma', and 'ko_okt'.
 
 ### porter_stemmer
 
@@ -21,12 +22,6 @@ The default method to tokenize. It is optimized for English. It divides sentence
 
 It means, stemmer can change 'studying', 'studies' to 'study'.
 
-### ko_kiwi
-
-It uses kiwi tokenizer for Korean language.
-We highly recommend to use it for Korean documents.
-You can check more information about kiwi at [here](https://github.com/bab2min/Kiwi).
-
 ### space
 
 It is simple method to divide words into just space.
@@ -37,9 +32,30 @@ It is simple, but it can be a great choice for multilingual documents.
 You can use any `AutoTokenizer` from huggingface, like gpt2 or mistralai/Mistral-7B-Instruct-v0.2.
 Just type huggingface repo path, and you can use the tokenizer.
 
+### ko_kiwi (For Korean 🇰🇷)
+
+It uses kiwi tokenizer for Korean language.
+We highly recommend to use it for Korean documents.
+You can check more information about kiwi at [here](https://github.com/bab2min/Kiwi).
+
+### ko_kkma (For Korean 🇰🇷)
+
+It uses okt tokenizer for Korean. You have to install `konlpy` to use this tokenizer.
+You can check more information about kkma at [here](https://konlpy.org/ko/latest/api/konlpy.tag/#konlpy.tag._kkma.Kkma).
+
+### ko_okt (For Korean 🇰🇷)
+
+It uses okt tokenizer for Korean. You have to install `konlpy` to use this tokenizer.
+You can check more information about okt at [here](https://konlpy.org/ko/latest/api/konlpy.tag/#konlpy.tag._okt.Okt).
+
+```{admonition} Any trouble to use Korean tokenizer?
+You need to install extra dependencies to properly use Korean tokenizer.
+Please go to [here](https://docs.auto-rag.com/install.html) to look at the installation guide.
+```
+
 ## **Example config.yaml**
 ```yaml
 modules:
   - module_type: bm25
-    bm25_tokenizer: [ porter_stemmer, ko_kiwi, space, gpt2 ]
+    bm25_tokenizer: [ porter_stemmer, ko_kiwi, space, gpt2, ko_kkma, ko_okt ]
 ```

{autorag-0.2.12 → autorag-0.2.13}/docs/source/optimization/optimization.md

@@ -12,11 +12,17 @@ In this documentation, you can learn about how AutoRAG works under the hood.
 
 ## Swapping modules in Node
 
-![Advanced RAG](../_static/roadmap/advanced_RAG.png)
+![Advanced RAG](https://github.com/Marker-Inc-Korea/AutoRAG/assets/96727832/79dda7ba-e9d8-4552-9e7b-6a5f9edc4c1a)
 
 Here is the diagram of the overall AutoRAG pipeline.
 Each node represents a node, and each node's result passed to the next node.
 
+```{admonition} Do I need to use all nodes?
+No. The essential node for the 'working' RAG pipeline is `retrieval`, `prompt maker` and `generator`.
+
+The other nodes are optional, so you can add it for the better performance.
+```
+
 But remember, you can set multiple modules and multiple parameters in each node. 
 And you get the best result among them.