AutoRAG 0.0.0__tar.gz

autorag-0.0.0/.github/FUNDING.yml

@@ -0,0 +1,2 @@
+ko_fi: autorag
+polar: Marker-Inc-Korea

autorag-0.0.0/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,35 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: "[BUG] "
+labels: bug
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Full Error log**
+If applicable, add full error log to help explain your problem.
+
+**Code that bug is happened**
+If applicable, add the code that bug is happened.
+(Especially, your AutoRAG YAML file or python codes that you wrote)
+
+**Desktop (please complete the following information):**
+ - OS: [e.g. Windows, Linux, MacOS]
+ - Python version [e.g. 3.10]
+
+**Additional context**
+Add any other context about the problem here.

autorag-0.0.0/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: "[Feature Request]"
+labels: enhancement
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

autorag-0.0.0/.github/copilot-instructions.md

@@ -0,0 +1,317 @@
+# AutoRAG - AI-Powered RAG Pipeline Optimization
+
+AutoRAG is a Python package for automatically finding optimal RAG (Retrieval-Augmented Generation) pipelines for your data using AutoML techniques. The system evaluates various RAG module combinations to find the best configuration for your specific use case.
+
+**ALWAYS reference these instructions first and fallback to search or bash commands only when you encounter unexpected information that does not match the info here.**
+
+## Working Effectively
+
+### Prerequisites and System Setup
+- **CRITICAL**: Use Python 3.10 or higher (Python 3.12 recommended)
+- Install UV package manager for faster dependency management: `curl -LsSf https://astral.sh/uv/install.sh | sh`
+- For systems without UV, fallback to pip installation
+- Install system dependencies first (see Platform Dependencies section)
+
+### Platform Dependencies
+Install these system packages before building AutoRAG:
+
+**Ubuntu/Debian:**
+```bash
+sudo apt-get update
+sudo apt-get install -y build-essential gcc poppler-utils tesseract-ocr libssl-dev
+```
+
+**Java (required for some features):**
+```bash
+# Install Java 17 (required)
+sudo apt-get install openjdk-17-jdk
+```
+
+### Bootstrap, Build, and Test - COMPLETE WORKFLOW
+
+**TIMING EXPECTATIONS:**
+- **NEVER CANCEL builds or tests** - they can take 15-45+ minutes
+- Initial dependency installation: 15-30 minutes
+- Full test suite: 15-20 minutes
+- Documentation build: 5-10 minutes
+- **ALWAYS set timeouts to 60+ minutes for builds and 30+ minutes for tests**
+
+**1. Clone and Environment Setup:**
+```bash
+git clone https://github.com/Marker-Inc-Korea/AutoRAG.git
+cd AutoRAG/autorag
+```
+
+**2. Create Virtual Environment and Install (Primary Method - UV):**
+```bash
+# Install UV if not available
+curl -LsSf https://astral.sh/uv/install.sh | sh
+source ~/.bashrc  # Reload shell
+
+# Create virtual environment and install
+uv venv
+source .venv/bin/activate  # On Windows: .venv\Scripts\activate
+
+# Install AutoRAG with all extras - TAKES 15-30 MINUTES, NEVER CANCEL
+uv pip install -r pyproject.toml --all-extras -e . --timeout 3600
+
+# Install development dependencies - TAKES 5-10 MINUTES
+uv pip install --group dev --timeout 1800
+```
+
+**3. Fallback Installation Method (Pip):**
+```bash
+# If UV fails, use pip
+python3 -m venv venv
+source venv/bin/activate
+
+# Install AutoRAG - TAKES 20-45 MINUTES, NEVER CANCEL
+pip install -e '.[all]' --timeout 3600
+
+# Install dev dependencies
+pip install ruff pre-commit pytest pytest-env pytest-xdist pytest-asyncio pytest-mock aioresponses asyncstdlib
+```
+
+**4. Additional Required Dependencies:**
+```bash
+# Upgrade SSL and install NLTK - REQUIRED
+pip install --upgrade pyOpenSSL
+pip install nltk
+python -c "import nltk; nltk.download('punkt_tab')"
+python -c "import nltk; nltk.download('averaged_perceptron_tagger_eng')"
+```
+
+**5. Setup Testing Environment:**
+Create `pytest.ini` file in the repository root:
+```bash
+cat > pytest.ini << 'EOF'
+[pytest]
+env =
+    OPENAI_API_KEY=sk-your-api-key-here
+
+log_cli=true
+log_cli_level=INFO
+EOF
+```
+
+**6. Run Tests - CRITICAL TIMING:**
+```bash
+# Delete conflicting test packages first
+python tests/delete_tests.py
+
+# Run tests - TAKES 15-20 MINUTES, NEVER CANCEL
+python -m pytest -o log_cli=true --log-cli-level=INFO -n auto tests/autorag --timeout 1800
+```
+
+**7. Setup Pre-commit Hooks:**
+```bash
+pre-commit install
+```
+
+### Daily Development Commands
+
+**Run the CLI (after installation):**
+```bash
+# Check available commands
+autorag --help
+
+# Main commands:
+autorag evaluate --config config.yaml --qa_data_path qa.parquet --corpus_data_path corpus.parquet --project_dir ./project
+autorag validate --config config.yaml --qa_data_path qa.parquet --corpus_data_path corpus.parquet
+autorag run_api --trial_path ./trial --host 0.0.0.0 --port 8000
+autorag dashboard --trial_dir ./trial --port 7690
+autorag run_web --trial_path ./trial --host 0.0.0.0 --port 8501
+```
+
+**Build Documentation:**
+```bash
+cd docs/
+# Install docs dependencies - TAKES 3-5 MINUTES
+pip install -r requirements.txt
+
+# Build documentation - TAKES 5-10 MINUTES, NEVER CANCEL
+sphinx-build -b html source build/html --timeout 600
+```
+
+**Code Quality and Linting:**
+```bash
+# Format and lint code (run before committing)
+ruff check --fix
+ruff format
+
+# These commands are FAST (< 30 seconds each)
+```
+
+## Validation and Testing
+
+### Manual Validation Scenarios
+**ALWAYS run these validation scenarios after making changes:**
+
+1. **CLI Basic Validation:**
+   ```bash
+   # Test CLI help works
+   autorag --help
+
+   # Test configuration validation (use actual existing files)
+   autorag validate --config autorag/sample_config/rag/full.yaml \
+     --qa_data_path projects/tutorial_1/qa/qa_8c42b9e6-490d-4971-bb9e-705b36b7a3a2.parquet \
+     --corpus_data_path projects/tutorial_1/parse/parse_8c42b9e6-490d-4971-bb9e-705b36b7a3a2/0.parquet
+   ```
+
+2. **API Server Test:**
+   ```bash
+   # Start API server (runs in background)
+   autorag run_api --trial_path ./projects/tutorial_1 --host 0.0.0.0 --port 8000 &
+
+   # Test API endpoint
+   curl http://localhost:8000/health
+
+   # Stop server
+   pkill -f "autorag run_api"
+   ```
+
+3. **Dashboard Test:**
+   ```bash
+   # Start dashboard (runs in background)
+   autorag dashboard --trial_dir ./projects/tutorial_1 --port 7690 &
+
+   # Verify it's running
+   curl http://localhost:7690
+
+   # Stop dashboard
+   pkill -f "autorag dashboard"
+   ```
+
+4. **Full Evaluation Pipeline Test:**
+   ```bash
+   # WARNING: This test requires OpenAI API key and may cost ~$0.30
+   # Use sample data from projects/tutorial_1/ (actual files have UUIDs in names)
+   autorag evaluate \
+     --config projects/tutorial_1/configs/parse_config_8c42b9e6-490d-4971-bb9e-705b36b7a3a2.yaml \
+     --qa_data_path projects/tutorial_1/qa/qa_8c42b9e6-490d-4971-bb9e-705b36b7a3a2.parquet \
+     --corpus_data_path projects/tutorial_1/parse/parse_8c42b9e6-490d-4971-bb9e-705b36b7a3a2/0.parquet \
+     --project_dir projects/tutorial_1/
+   ```
+
+### Testing Warnings and Notes
+- **OpenAI API Key Required**: Many tests require `OPENAI_API_KEY` environment variable
+- **GitHub Actions**: Tests may fail in CI due to missing API keys - this is expected
+- **GPU Tests**: Some tests require CUDA - use `@pytest.mark.skipif(is_github_action())` to skip in CI
+- **Test Data**: Tests must not leave extra files behind
+- **Mock Usage**: Heavy GPU models and external API calls must use mocks and pytest fixtures
+
+## Common Issues and Troubleshooting
+
+### Installation Issues
+- **Network timeouts**: Use `--timeout 3600` flag with pip/uv commands
+- **Missing system packages**: Install platform dependencies first
+- **Python version**: Ensure Python 3.10+ is being used
+- **Virtual environment**: Always use a virtual environment to avoid conflicts
+
+### Build Failures
+- **Java not found**: Install openjdk-17-jdk
+- **SSL errors**: Run `pip install --upgrade pyOpenSSL`
+- **NLTK missing**: Download required NLTK data as shown above
+- **Permission errors**: Use virtual environment, avoid system-wide installs
+
+### Test Failures
+- **API key missing**: Set OPENAI_API_KEY in pytest.ini or environment
+- **Import errors**: Run `python tests/delete_tests.py` to clean conflicting packages
+- **Timeout errors**: Tests can take 15-20 minutes, never cancel early
+
+## Project Structure and Key Locations
+
+### Repository Layout
+```
+/
+├── .github/                 # GitHub workflows and CI/CD
+├── autorag/                 # Main Python package directory
+│   ├── autorag/            # Core AutoRAG package
+│   │   ├── cli.py          # Command-line interface
+│   │   ├── evaluator.py    # Pipeline evaluation logic
+│   │   ├── dashboard.py    # Web dashboard
+│   │   ├── nodes/          # RAG pipeline nodes
+│   │   ├── vectordb/       # Vector database integrations
+│   │   └── web.py          # Web interface
+│   ├── sample_config/      # Sample configuration files
+│   │   ├── rag/            # RAG pipeline configs
+│   │   ├── chunk/          # Chunking configs
+│   │   └── parse/          # Parsing configs
+│   └── pyproject.toml      # Main project configuration
+├── tests/                  # Test suite
+├── docs/                   # Sphinx documentation
+├── projects/               # Sample projects and datasets
+└── api/                    # API server implementation
+```
+
+### Key Files to Check After Changes
+- **Configuration Changes**: Always validate with `autorag validate`
+- **CLI Changes**: Test with `autorag --help` and specific commands
+- **API Changes**: Check `autorag/autorag/deploy/api.py` and test endpoints
+- **Node Changes**: Validate in `autorag/autorag/nodes/` and run related tests
+- **Config Changes**: Update sample configs in `autorag/sample_config/`
+
+### Configuration Files
+- **Sample Configs**: `autorag/sample_config/rag/full.yaml` - comprehensive example
+- **Docker Config**: `autorag/Dockerfile.base` - multi-stage build configuration
+- **Test Config**: `projects/tutorial_1/` - working example with test data
+- **Dependencies**: `autorag/pyproject.toml` - all package dependencies and extras
+
+## Docker Usage (Alternative Deployment)
+
+**Pre-built Images Available:**
+- `autoraghq/autorag:api-latest` - API server
+- `autoraghq/autorag:all` - Full installation
+- `autoraghq/autorag:gpu` - GPU-enabled version
+
+**Quick Docker Test:**
+```bash
+# Download sample data first, then run evaluation
+docker run --rm \
+  -v ~/.cache/huggingface:/root/.cache/huggingface \
+  -v $(pwd)/projects:/usr/src/app/projects \
+  -e OPENAI_API_KEY=${OPENAI_API_KEY} \
+  autoraghq/autorag:api-latest evaluate \
+  --config /usr/src/app/projects/tutorial_1/configs/parse_config_8c42b9e6-490d-4971-bb9e-705b36b7a3a2.yaml \
+  --qa_data_path /usr/src/app/projects/tutorial_1/qa/qa_8c42b9e6-490d-4971-bb9e-705b36b7a3a2.parquet \
+  --corpus_data_path /usr/src/app/projects/tutorial_1/parse/parse_8c42b9e6-490d-4971-bb9e-705b36b7a3a2/0.parquet \
+  --project_dir /usr/src/app/projects/tutorial_1/
+```
+
+## Development Workflow
+
+### Before Making Changes
+1. **ALWAYS** run the bootstrap and test sequence above
+2. Create feature branch: `git checkout -b feature/your-feature`
+3. Set up pre-commit hooks: `pre-commit install`
+
+### Making Changes
+1. **Make minimal changes** - change as few lines as possible
+2. **Follow existing patterns** in the codebase
+3. **Add tests** for new functionality in `tests/autorag/`
+4. **Update configs** if adding new nodes or features
+
+### Before Committing
+1. **Run formatting**: `ruff check --fix && ruff format`
+2. **Run affected tests**: `python -m pytest tests/autorag/path/to/relevant/tests`
+3. **Validate CLI still works**: `autorag --help`
+4. **Test your changes manually** using the validation scenarios above
+
+### Submitting Changes
+1. **Ensure all linting passes**: CI will fail if ruff checks fail
+2. **Include test coverage**: Add unit tests for new features
+3. **Document breaking changes**: Update relevant config samples
+4. **Test with sample data**: Ensure tutorial examples still work
+
+## Timing Expectations Summary
+
+| Operation | Expected Time | Timeout Setting | Notes |
+|-----------|---------------|-----------------|--------|
+| Initial install | 15-30 min | 60+ min | NEVER CANCEL |
+| Full test suite | 15-20 min | 30+ min | NEVER CANCEL |
+| Documentation build | 5-10 min | 15+ min | Usually fast |
+| Code formatting | < 30 sec | 2 min | Very fast |
+| Single test file | 1-5 min | 10 min | Varies by test |
+| API evaluation | 5-15 min | 20+ min | Depends on data size |
+
+**CRITICAL**: Always wait for builds and tests to complete. The CI pipeline expects these timings and builds may appear to hang but are actually processing large dependency trees.

autorag-0.0.0/.github/dependabot.yml

@@ -0,0 +1,11 @@
+# To get started with Dependabot version updates, you'll need to specify which
+# package ecosystems to update and where the package manifests are located.
+# Please see the documentation for all configuration options:
+# https://docs.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
+
+version: 2
+updates:
+  - package-ecosystem: "pip" # See documentation for possible values
+    directory: "/" # Location of package manifests
+    schedule:
+      interval: "daily"

autorag-0.0.0/.github/workflows/publish.yml

@@ -0,0 +1,63 @@
+name: Publish Python 🐍 distributions 📦 to PyPI
+permissions:
+  contents: read
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  set-version:
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Export tag
+        id: vars
+        run: echo tag=${GITHUB_REF#refs/*/} >> $GITHUB_OUTPUT
+        if: ${{ github.event_name == 'release' }}
+
+      - name: Update project version
+        run: |
+          sed -i "s/^version = \".*\"/version = \"$RELEASE_VERSION\"/" pyproject.toml
+        env:
+          RELEASE_VERSION: ${{ steps.vars.outputs.tag }}
+        if: ${{ github.event_name == 'release' }}
+
+      - name: Upload updated pyproject.toml
+        uses: actions/upload-artifact@v4
+        with:
+          name: pyproject-toml
+          path: pyproject.toml
+
+  publish:
+    runs-on: ubuntu-latest
+    needs: [set-version]
+    steps:
+      - name: Check out
+        uses: actions/checkout@v4
+
+      - name: Set up the python environment
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+      - name: Install uv
+        uses: astral-sh/setup-uv@v6
+        with:
+          enable-cache: 'true'
+          cache-suffix: '3.11'
+      - name: Download updated pyproject.toml
+        uses: actions/download-artifact@v4
+        with:
+          name: pyproject-toml
+      - name: Install Python dependencies
+        run: uv sync --frozen --all-extras
+        shell: bash
+
+      - name: Build package
+        run: uv build
+
+      - name: Publish package
+        run: uv publish
+        env:
+          UV_PUBLISH_TOKEN: ${{ secrets.PYPI_TOKEN }}

autorag-0.0.0/.github/workflows/sphinx.yml

@@ -0,0 +1,41 @@
+name: Sphinx build
+
+on:
+  push:
+    branches:
+      - main
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install gcc
+      run: |
+        sudo apt-get install gcc
+    - name: Install uv
+      uses: astral-sh/setup-uv@v5
+    - name: Install Python 3.10
+      run: uv python install 3.10
+    - name: Install Venv
+      run: uv venv
+    - name: Install dependencies
+      run: |
+        uv pip install -r docs/requirements.txt
+    - name: Build HTML
+      run: |
+        uv run sphinx-build -b html docs/source docs/build/html
+    - name: Copy CNAME
+      run: |
+        cp docs/source/CNAME docs/build/html/CNAME
+    - name: Upload artifacts
+      uses: actions/upload-artifact@v4
+      with:
+        name: html-docs
+        path: docs/build/html/
+    - name: Deploy
+      uses: peaceiris/actions-gh-pages@v4
+      if: github.ref == 'refs/heads/main'
+      with:
+        github_token: ${{ secrets.GITHUB_TOKEN }}
+        publish_dir: docs/build/html

autorag-0.0.0/.github/workflows/test.yml

@@ -0,0 +1,55 @@
+name: Unit Test
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+    branches:
+      - main
+
+env:
+  OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-java@v4
+        with:
+          distribution: 'zulu'
+          java-version: '17'
+      - name: Update apt-get
+        run: |
+          sudo apt-get update
+      - name: Install gcc
+        run: |
+          sudo apt-get install gcc
+      - name: Install poppler-utils
+        run: |
+          sudo apt-get install poppler-utils
+      - name: Install tesseract
+        run: |
+          sudo apt-get install tesseract-ocr
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+      - name: Install Python 3.12
+        run: uv python install 3.12
+      - name: Install Venv
+        run: uv venv
+      - name: Install AutoRAG
+        run: uv sync --all-extras
+      - name: Upgrade pyOpenSSL
+        run: |
+          uv pip install --upgrade pyOpenSSL
+      - name: Install NLTK and download model
+        run: |
+          uv pip install nltk
+          uv run python -c "import nltk; nltk.download('punkt_tab')"
+          uv run python -c "import nltk; nltk.download('averaged_perceptron_tagger_eng')"
+      - name: delete tests package
+        run: uv run python tests/delete_tests.py
+      - name: Run AutoRAG tests
+        run: |
+          uv run python -m pytest -o log_cli=true --log-cli-level=INFO -n auto tests/autorag