AutoRAG 0.2.18__tar.gz → 0.3.0__tar.gz

{autorag-0.2.18 → autorag-0.3.0}/AutoRAG.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: AutoRAG
-Version: 0.2.18
+Version: 0.3.0
 Summary: Automatically Evaluate RAG pipelines with your own data. Find optimal structure for new RAG product.
 Author-email: Marker-Inc <vkehfdl1@gmail.com>
 License:                                  Apache License
@@ -210,14 +210,13 @@ Keywords: RAG,AutoRAG,autorag,rag-evaluation,evaluation,rag-auto,AutoML,AutoML-R
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Information Technology
 Classifier: Intended Audience :: Science/Research
-Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
 Classifier: Topic :: Software Development :: Libraries
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
-Requires-Python: >=3.8
+Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: numpy<2.0.0
@@ -293,19 +292,27 @@ Requires-Dist: AutoRAG[parse]; extra == "all"
 
 RAG AutoML tool for automatically finds an optimal RAG pipeline for your data.
 
-Explore our 📖 [Document](https://docs.auto-rag.com)!!
+![Thumbnail](https://github.com/user-attachments/assets/6bab243d-a4b3-431a-8ac0-fe17336ab4de)
 
-Plus, join our 📞 [Discord](https://discord.gg/P4DYXfmSAs) Community.
+There are many RAG pipelines and modules out there,
+but you don’t know what pipeline is great for “your own data” and "your own use-case."
+Making and evaluating all RAG modules is very time-consuming and hard to do.
+But without it, you will never know which RAG pipeline is the best for your own use-case.
 
----
+AutoRAG is a tool for finding optimal RAG pipeline for “your data.”
+You can evaluate various RAG modules automatically with your own evaluation data,
+and find the best RAG pipeline for your own use-case.
 
-### 💪 Colab Tutorial
+AutoRAG supports a simple way to evaluate many RAG module combinations.
+Try now and find the best RAG pipeline for your own use-case.
 
-- [Step 1: Basic of AutoRAG | Optimizing your RAG pipeline](https://colab.research.google.com/drive/19OEQXO_pHN6gnn2WdfPd4hjnS-4GurVd?usp=sharing)
+Explore our 📖 [Document](https://docs.auto-rag.com)!!
+
+Plus, join our 📞 [Discord](https://discord.gg/P4DYXfmSAs) Community.
 
 ---
 
-### 🚨 YouTube Tutorial
+## YouTube Tutorial
 
 https://github.com/Marker-Inc-Korea/AutoRAG/assets/96727832/c0d23896-40c0-479f-a17b-aa2ec3183a26
 
@@ -313,85 +320,168 @@ _Muted by default, enable sound for voice-over_
 
 You can see on [YouTube](https://youtu.be/2ojK8xjyXAU?feature=shared)
 
----
+## Colab Tutorial
 
-### ☎️ FaQ
+- [Step 1: Basic of AutoRAG | Optimizing your RAG pipeline](https://colab.research.google.com/drive/19OEQXO_pHN6gnn2WdfPd4hjnS-4GurVd?usp=sharing)
+- [Step 2: Data Creation | Create your own Data for RAG Optimization](https://colab.research.google.com/drive/1BOdzMndYgMY_iqhwKcCCS7ezHbZ4Oz5X?usp=sharing)
+- [Step 3: Use Custom LLM & Embedding Model | Use Custom Model](https://colab.research.google.com/drive/12VpWcSTSOsLSyW0BKb-kPoEzK22ACxvS?usp=sharing)
+
+# Index
+
+- [Quick Install](#quick-install)
+- [Data Creation](#data-creation)
+  - [Parsing](#1-parsing)
+  - [Chunking](#2-chunking)
+  - [QA Creation](#3-qa-creation)
+- [RAG Optimization](#rag-optimization)
+    - [Set YAML File](#1-set-yaml-file)
+    - [Run AutoRAG](#2-run-autorag)
+    - [Run Dashboard](#3-run-dashboard)
+    - [Deploy your optimal RAG pipeline (for testing)](#4-deploy-your-optimal-rag-pipeline-for-testing)
+- [Supporting Data Creation Modules](#-supporting-data-creation-modules)
+- [Supporting RAG Optimization Nodes & modules](#supporting-rag-optimization-nodes--modules)
+- [Supporting Evaluation Metrics](#supporting-evaluation-metrics)
+- [FaQ](#-faq)
+
+# Quick Install
+
+We recommend using Python version 3.10 or higher for AutoRAG.
 
-🛣️ [Support plans & Roadmap](https://edai.notion.site/Support-plans-Roadmap-02ca7c97376340c393885855e2d99630?pvs=4)
+```bash
+pip install AutoRAG
+```
 
-💻 [Hardware Specs](https://edai.notion.site/Hardware-specs-28cefcf2a26246ffadc91e2f3dc3d61c?pvs=4)
+# Data Creation
 
-⭐ [Running AutoRAG](https://edai.notion.site/About-running-AutoRAG-44a8058307af42068fc218a073ee480b?pvs=4)
+![image](https://github.com/user-attachments/assets/6079f696-207c-4221-8d28-5561a203dfe2)
 
-🍯 [Tips/Tricks](https://edai.notion.site/Tips-Tricks-10708a0e36ff461cb8a5d4fb3279ff15?pvs=4)
+RAG Optimization requires two types of data: QA dataset and Corpus dataset.
 
-☎️ [TroubleShooting](https://medium.com/@autorag/autorag-troubleshooting-5cf872b100e3)
+1. **QA** dataset file (qa.parquet)
+2. **Corpus** dataset file (corpus.parquet)
 
----
+**QA** dataset is important for accurate and reliable evaluation and optimization.
 
-# 😁 Introduction
+**Corpus** dataset is critical to the performance of RAGs.
+This is because RAG uses the corpus to retrieve documents and generate answers using it.
 
-There are many RAG pipelines and modules out there,
-but you don’t know what pipeline is great for “your own data” and "your own use-case."
-Making and evaluating all RAG modules is very time-consuming and hard to do.
-But without it, you will never know which RAG pipeline is the best for your own use-case.
+## Quick Start
 
-AutoRAG is a tool for finding optimal RAG pipeline for “your data.”
-You can evaluate various RAG modules automatically with your own evaluation data,
-and find the best RAG pipeline for your own use-case.
+### 1. Parsing
 
-AutoRAG supports a simple way to evaluate many RAG module combinations.
-Try now and find the best RAG pipeline for your own use-case.
+#### Set YAML File
 
-# 📌AutoRAG Structure
+```yaml
+modules:
+  - module_type: langchain_parse
+    parse_method: pdfminer
+```
 
-![Node](https://github.com/Marker-Inc-Korea/AutoRAG/assets/96727832/79dda7ba-e9d8-4552-9e7b-6a5f9edc4c1a)
+You can also use multiple Parse modules at once.
+However, in this case, you'll need to return a new process for each parsed result.
 
-## ❗Supporting Nodes & modules
+#### Start Parsing
 
-![module_1](https://github.com/user-attachments/assets/d49cc7b4-56a6-4b5f-bb3d-2fc5b962996b)
-![module_2](https://github.com/Marker-Inc-Korea/AutoRAG/assets/96727832/4db3d83f-3178-4b81-bad1-b751dcd31a13)
-You can check our all supporting Nodes & modules
-at [here](https://edai.notion.site/Supporting-Nodes-modules-0ebc7810649f4e41aead472a92976be4?pvs=4)
+You can parse your raw documents with just a few lines of code.
 
-## ❗Supporting Evaluation Metrics
+```python
+from autorag.parser import Parser
 
-![Metrics](https://github.com/Marker-Inc-Korea/AutoRAG/assets/96727832/9f090e3c-d762-41cb-9f1d-cd8d7da137dd)
+parser = Parser(data_path_glob="your/data/path/*")
+parser.start_parsing("your/path/to/parse_config.yaml")
+```
 
-You can check our all supporting Evaluation Metrics
-at [here](https://edai.notion.site/Supporting-metrics-867d71caefd7401c9264dd91ba406043?pvs=4)
+### 2. Chunking
 
-- [Retrieval Metrics](https://edai.notion.site/Retrieval-Metrics-dde3d9fa1d9547cdb8b31b94060d21e7?pvs=4)
-- [Retrieval Token Metrics](https://edai.notion.site/Retrieval-Token-Metrics-c3e2d83358e04510a34b80429ebb543f?pvs=4)
-- [Generation Metrics](https://edai.notion.site/Retrieval-Token-Metrics-c3e2d83358e04510a34b80429ebb543f?pvs=4)
+#### Set YAML File
+
+```yaml
+modules:
+  - module_type: llama_index_chunk
+    chunk_method: Token
+    chunk_size: 1024
+    chunk_overlap: 24
+    add_file_name: english
+```
 
-# ⚡ Quick Install
+You can also use multiple Chunk modules at once.
+In this case, you need to use one corpus to create QA, and then map the rest of the corpus to QA Data.
+If the chunk method is different, the retrieval_gt will be different, so we need to remap it to the QA dataset.
 
-We recommend using Python version 3.9 or higher for AutoRAG.
+#### Start Chunking
 
-```bash
-pip install AutoRAG
+You can chunk your parsed results with just a few lines of code.
+
+```python
+from autorag.chunker import Chunker
+
+chunker = Chunker.from_parquet(parsed_data_path="your/parsed/data/path")
+chunker.start_chunking("your/path/to/chunk_config.yaml")
 ```
 
-# ⚡ QuickStart
+### 3. QA Creation
 
-### 1. Prepare your evaluation data
+You can create QA dataset with just a few lines of code.
 
-For evaluation, you need to prepare just three files.
+```python
+import pandas as pd
+from llama_index.llms.openai import OpenAI
+
+from autorag.data.beta.filter.dontknow import dontknow_filter_rule_based
+from autorag.data.beta.generation_gt.llama_index_gen_gt import (
+    make_basic_gen_gt,
+    make_concise_gen_gt,
+)
+from autorag.data.beta.schema import Raw, Corpus
+from autorag.data.beta.query.llama_gen_query import factoid_query_gen
+from autorag.data.beta.sample import random_single_hop
+
+llm = OpenAI()
+raw_df = pd.read_parquet("your/path/to/corpus.parquet")
+raw_instance = Raw(raw_df)
+
+corpus_df = pd.read_parquet("your/path/to/corpus.parquet")
+corpus_instance = Corpus(corpus_df, raw_instance)
+
+initial_qa = (
+    corpus_instance.sample(random_single_hop, n=3)
+    .map(
+        lambda df: df.reset_index(drop=True),
+    )
+    .make_retrieval_gt_contents()
+    .batch_apply(
+        factoid_query_gen,  # query generation
+        llm=llm,
+    )
+    .batch_apply(
+        make_basic_gen_gt,  # answer generation (basic)
+        llm=llm,
+    )
+    .batch_apply(
+        make_concise_gen_gt,  # answer generation (concise)
+        llm=llm,
+    )
+    .filter(
+        dontknow_filter_rule_based,  # filter don't know
+        lang="en",
+    )
+)
+
+initial_qa.to_parquet('./qa.parquet', './corpus.parquet')
+```
 
-1. **QA** dataset file (qa.parquet)
-2. **Corpus** dataset file (corpus.parquet)
-3. **Config yaml file** (config.yaml)
+# RAG Optimization
+![rag](https://github.com/user-attachments/assets/214d842e-fc67-4113-9c24-c94158b00c23)
+
+### How AutoRAG optimizes RAG pipeline?
 
-There is a template for your evaluation data for using AutoRAG.
+![rag_opt_gif](https://github.com/user-attachments/assets/55bd09cd-8420-4f6d-bc7d-0a66af288317)
 
-- Check out how to make evaluation data
-  at [here](https://docs.auto-rag.com/data_creation/tutorial.html).
-- Check out the evaluation data rule
-  at [here](https://docs.auto-rag.com/data_creation/data_format.html).
-- Plus, you can get example datasets for testing AutoRAG at [here](./sample_dataset).
+## Quick Start
 
-### 2. Evaluate your data to various RAG modules
+### 1. Set YAML File
+
+First, you need to set the config yaml file for your RAG optimization.
 
 You can get various config yaml files at [here](./sample_config).
 We highly recommend using pre-made config yaml files for starter.
@@ -399,6 +489,49 @@ We highly recommend using pre-made config yaml files for starter.
 If you want to make your own config yaml files, check out the [Config yaml file](#-create-your-own-config-yaml-file)
 section.
 
+Here is an example of the config yaml file to use `retrieval`, `prompt_maker`, and `generator` nodes.
+
+```yaml
+node_lines:
+- node_line_name: retrieve_node_line  # Set Node Line (Arbitrary Name)
+  nodes:
+    - node_type: retrieval  # Set Retrieval Node
+      strategy:
+        metrics: [retrieval_f1, retrieval_recall, retrieval_ndcg, retrieval_mrr]  # Set Retrieval Metrics
+      top_k: 3
+      modules:
+        - module_type: vectordb
+          embedding_model: openai
+        - module_type: bm25
+        - module_type: hybrid_rrf
+          weight_range: (4,80)
+- node_line_name: post_retrieve_node_line  # Set Node Line (Arbitrary Name)
+  nodes:
+    - node_type: prompt_maker  # Set Prompt Maker Node
+      strategy:
+        metrics:   # Set Generation Metrics
+          - metric_name: meteor
+          - metric_name: rouge
+          - metric_name: sem_score
+            embedding_model: openai
+      modules:
+        - module_type: fstring
+          prompt: "Read the passages and answer the given question. \n Question: {query} \n Passage: {retrieved_contents} \n Answer : "
+    - node_type: generator  # Set Generator Node
+      strategy:
+        metrics:  # Set Generation Metrics
+          - metric_name: meteor
+          - metric_name: rouge
+          - metric_name: sem_score
+            embedding_model: openai
+      modules:
+        - module_type: openai_llm
+          llm: gpt-4o-mini
+          batch: 16
+```
+
+### 2. Run AutoRAG
+
 You can evaluate your RAG pipeline with just a few lines of code.
 
 ```python
@@ -421,40 +554,7 @@ you can check `summary.csv` file that summarizes the evaluation results and the
 For more details, you can check out how the folder structure looks like
 at [here](https://docs.auto-rag.com/optimization/folder_structure.html).
 
-### 3. Use a found optimal RAG pipeline
-
-You can use a found optimal RAG pipeline right away.
-It needs just a few lines of code, and you are ready to use!
-
-First, you need to build pipeline yaml file from your evaluated trial folder.
-You can find the trial folder in your current directory.
-Just looking folder like '0' or other numbers.
-
-```python
-from autorag.deploy import Runner
-
-runner = Runner.from_trial_folder('your/path/to/trial_folder')
-runner.run('your question')
-```
-
-Or, you can run this pipeline as api server.
-You can use python code or CLI command.
-Check out API endpoint at [here](https://docs.auto-rag.com/deploy/api_endpoint.html).
-
-```python
-from autorag.deploy import Runner
-
-runner = Runner.from_trial_folder('your/path/to/trial_folder')
-runner.run_api_server()
-```
-
-You can run api server with CLI command.
-
-```bash
-autorag run_api --config_path your/path/to/pipeline.yaml --host 0.0.0.0 --port 8000
-```
-
-### 4. Run Dashboard
+### 3. Run Dashboard
 
 You can run dashboard to easily see the result.
 
@@ -462,27 +562,13 @@ You can run dashboard to easily see the result.
 autorag dashboard --trial_dir /your/path/to/trial_dir
 ```
 
-- sample dashboard:
+#### sample dashboard
 
 ![dashboard](https://github.com/Marker-Inc-Korea/AutoRAG/assets/96727832/3798827d-31d7-4c4e-a9b1-54340b964e53)
 
-### 5. Share your RAG pipeline
-
-You can use your RAG pipeline from extracted pipeline yaml file.
-This extracted pipeline is great for sharing your RAG pipeline to others.
-
-You must run this at project folder, which contains datas in data folder, and ingested corpus for retrieval at resources
-folder.
-
-```python
-from autorag.deploy import extract_best_config
-
-pipeline_dict = extract_best_config(trial_path='your/path/to/trial_folder', output_path='your/path/to/pipeline.yaml')
-```
-
-### 6. Deploy your optimal RAG pipeline (for testing)
+### 4. Deploy your optimal RAG pipeline (for testing)
 
-### 6-1. Run as a CLI
+### 4-1. Run as a CLI
 
 You can use a found optimal RAG pipeline right away with extracted yaml file.
 
@@ -493,7 +579,7 @@ runner = Runner.from_yaml('your/path/to/pipeline.yaml')
 runner.run('your question')
 ```
 
-### 6-2. Run as an API server
+### 4-2. Run as an API server
 
 You can run this pipeline as an API server.
 
@@ -510,7 +596,7 @@ runner.run_api_server()
 autorag run_api --config_path your/path/to/pipeline.yaml --host 0.0.0.0 --port 8000
 ```
 
-### 6-3. Run as a Web Interface
+### 4-3. Run as a Web Interface
 
 you can run this pipeline as a web interface.
 
@@ -520,13 +606,51 @@ Check out web interface at [here](deploy/web.md).
 autorag run_web --trial_path your/path/to/trial_path
 ```
 
-- sample web interface:
+#### sample web interface
 
 <img width="1491" alt="web_interface" src="https://github.com/Marker-Inc-Korea/AutoRAG/assets/96727832/f6b00353-f6bb-4d8f-8740-1c264c0acbb8">
 
-# ⭐ Star History ⭐
+## 📌 Supporting Data Creation Modules
+![Data Creation](https://github.com/user-attachments/assets/383876cc-0d02-4e5f-9528-87f36489640f)
+
+- You can check our all Parsing Modules at [here](https://edai.notion.site/Supporting-Parse-Modules-e0b7579c7c0e4fb2963e408eeccddd75?pvs=4)
+- You can check our all Chunk Modules at [here](https://edai.notion.site/Supporting-Chunk-Modules-8db803dba2ec4cd0a8789659106e86a3?pvs=4)
+
+## ❗Supporting RAG Optimization Nodes & modules
+
+![module_1](https://github.com/user-attachments/assets/c2e4c7d2-0f46-4b13-bb21-0ffc19cc9492)
+![module_2](https://github.com/user-attachments/assets/e013f04e-f69d-4dd7-96da-06e6b5921c3d)
+![module_3](https://github.com/user-attachments/assets/72490b97-81a1-4620-b9bd-5c5e0ead79a7)
+![module_4](https://github.com/user-attachments/assets/7a8ee260-9c60-4a27-b708-5286d1c37851)
+
+You can check our all supporting Nodes & modules
+at [here](https://edai.notion.site/Supporting-Nodes-modules-0ebc7810649f4e41aead472a92976be4?pvs=4)
+
+## ❗Supporting Evaluation Metrics
+
+![Metrics](https://github.com/user-attachments/assets/fab551cd-8892-4bda-acae-eff22ece94b3)
+
+You can check our all supporting Evaluation Metrics
+at [here](https://edai.notion.site/Supporting-metrics-867d71caefd7401c9264dd91ba406043?pvs=4)
+
+- [Retrieval Metrics](https://edai.notion.site/Retrieval-Metrics-dde3d9fa1d9547cdb8b31b94060d21e7?pvs=4)
+- [Retrieval Token Metrics](https://edai.notion.site/Retrieval-Token-Metrics-c3e2d83358e04510a34b80429ebb543f?pvs=4)
+- [Generation Metrics](https://github.com/user-attachments/assets/7d4a3069-9186-4854-885d-ca0f7bcc17e8)
+
+
+## ☎️ FaQ
+
+🛣️ [Support plans & Roadmap](https://edai.notion.site/Support-plans-Roadmap-02ca7c97376340c393885855e2d99630?pvs=4)
+
+💻 [Hardware Specs](https://edai.notion.site/Hardware-specs-28cefcf2a26246ffadc91e2f3dc3d61c?pvs=4)
 
-[![Star History Chart](https://api.star-history.com/svg?repos=Marker-Inc-Korea/AutoRAG&type=Date)](https://star-history.com/#Marker-Inc-Korea/AutoRAG&Date)
+⭐ [Running AutoRAG](https://edai.notion.site/About-running-AutoRAG-44a8058307af42068fc218a073ee480b?pvs=4)
+
+🍯 [Tips/Tricks](https://edai.notion.site/Tips-Tricks-10708a0e36ff461cb8a5d4fb3279ff15?pvs=4)
+
+☎️ [TroubleShooting](https://medium.com/@autorag/autorag-troubleshooting-5cf872b100e3)
+
+---
 
 # ✨ Contributors ✨