visual-rag-toolkit 0.1.1__tar.gz → 0.1.3__tar.gz

{visual_rag_toolkit-0.1.1 → visual_rag_toolkit-0.1.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: visual-rag-toolkit
-Version: 0.1.1
+Version: 0.1.3
 Summary: End-to-end visual document retrieval with ColPali, featuring two-stage pooling for scalable search
 Project-URL: Homepage, https://github.com/Ara-Yeroyan/visual-rag-toolkit
 Project-URL: Documentation, https://github.com/Ara-Yeroyan/visual-rag-toolkit#readme
@@ -85,10 +85,9 @@ Description-Content-Type: text/markdown
 
 # Visual RAG Toolkit
 
-[![PyPI version](https://badge.fury.io/py/visual-rag-toolkit.svg)](https://badge.fury.io/py/visual-rag-toolkit)
-[![CI](https://github.com/Ara-Yeroyan/visual-rag-toolkit/actions/workflows/ci.yaml/badge.svg)](https://github.com/Ara-Yeroyan/visual-rag-toolkit/actions/workflows/ci.yaml)
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
+[![PyPI](https://img.shields.io/pypi/v/visual-rag-toolkit)](https://pypi.org/project/visual-rag-toolkit/)
+[![Python](https://img.shields.io/pypi/pyversions/visual-rag-toolkit)](https://pypi.org/project/visual-rag-toolkit/)
+[![License](https://img.shields.io/pypi/l/visual-rag-toolkit)](LICENSE)
 
 End-to-end visual document retrieval toolkit featuring **fast multi-stage retrieval** (prefetch with pooled vectors + exact MaxSim reranking).
 
@@ -112,11 +111,10 @@ This repo contains:
 pip install visual-rag-toolkit
 
 # With specific features
-pip install visual-rag-toolkit[embedding]    # ColSmol/ColPali embedding support
-pip install visual-rag-toolkit[pdf]          # PDF processing
+pip install visual-rag-toolkit[ui]           # Streamlit demo dependencies
 pip install visual-rag-toolkit[qdrant]       # Vector database
+pip install visual-rag-toolkit[embedding]    # ColSmol/ColPali embedding support
 pip install visual-rag-toolkit[cloudinary]   # Image CDN
-pip install visual-rag-toolkit[ui]           # Streamlit demo dependencies
 
 # All dependencies
 pip install visual-rag-toolkit[all]
@@ -157,6 +155,95 @@ for r in results[:3]:
     print(r["id"], r["score_final"])
 ```
 
+### End-to-end: ingest PDFs (with cropping) → index in Qdrant
+
+This is the "SDK-style" pipeline: PDF → images → optional crop → embed → store vectors + payload in Qdrant.
+
+```python
+import os
+from pathlib import Path
+
+import numpy as np
+import torch
+
+from visual_rag import VisualEmbedder
+from visual_rag.indexing import ProcessingPipeline, QdrantIndexer
+
+QDRANT_URL = os.environ["QDRANT_URL"]
+QDRANT_KEY = os.getenv("QDRANT_API_KEY", "")
+
+collection = "my_visual_docs"
+
+embedder = VisualEmbedder(
+    model_name="vidore/colSmol-500M",
+    torch_dtype=torch.float16,
+    output_dtype=np.float16,
+    batch_size=8,
+)
+
+indexer = QdrantIndexer(
+    url=QDRANT_URL,
+    api_key=QDRANT_KEY,
+    collection_name=collection,
+    prefer_grpc=True,
+    vector_datatype="float16",
+)
+
+# Creates collection + required payload indexes (e.g., "filename" for skip_existing)
+indexer.create_collection(force_recreate=False)
+
+pipeline = ProcessingPipeline(
+    embedder=embedder,
+    indexer=indexer,
+    embedding_strategy="all",  # store full tokens + pooled vectors in one pass
+    crop_empty=True,
+    crop_empty_percentage_to_remove=0.99,  # kept for traceability
+    crop_empty_remove_page_number=True,
+    crop_empty_preserve_border_px=1,
+    crop_empty_uniform_rowcol_std_threshold=3.0,
+)
+
+pdfs = [Path("docs/a.pdf"), Path("docs/b.pdf")]
+for pdf_path in pdfs:
+    result = pipeline.process_pdf(
+        pdf_path,
+        skip_existing=True,  # Skip pages already in Qdrant (uses filename index)
+        upload_to_cloudinary=False,
+        upload_to_qdrant=True,
+    )
+    # Logs automatically shown:
+    # [10:23:45] 📚 Processing PDF: a.pdf
+    # [10:23:45] 🖼️ Converting PDF to images...
+    # [10:23:46]    ✅ Converted 12 pages
+    # [10:23:46] 📦 Processing pages 1-8/12
+    # [10:23:46] 🤖 Generating embeddings for 8 pages...
+    # [10:23:48] 📤 Uploading batch of 8 pages...
+    # [10:23:48]    ✅ Uploaded 8 points to Qdrant
+    # [10:23:48] 📦 Processing pages 9-12/12
+    # [10:23:48] 🤖 Generating embeddings for 4 pages...
+    # [10:23:50] 📤 Uploading batch of 4 pages...
+    # [10:23:50]    ✅ Uploaded 4 points to Qdrant
+    # [10:23:50] ✅ Completed a.pdf: 12 uploaded, 0 skipped, 0 failed
+```
+
+CLI equivalent:
+
+```bash
+export QDRANT_URL="https://YOUR_QDRANT"
+export QDRANT_API_KEY="YOUR_KEY"
+
+visual-rag process \
+  --reports-dir ./docs \
+  --collection my_visual_docs \
+  --model vidore/colSmol-500M \
+  --strategy all \
+  --batch-size 8 \
+  --qdrant-vector-dtype float16 \
+  --prefer-grpc \
+  --crop-empty \
+  --crop-empty-remove-page-number
+```
+
 ### Process a PDF into images (no embedding, no vector DB)
 
 ```python
@@ -186,7 +273,7 @@ Stage 2: Exact MaxSim reranking on candidates
          └── Return top-k results (e.g., 10)
 ```
 
-Three-stage extends this with an additional “cheap prefetch” stage before stage 2.
+Three-stage extends this with an additional "cheap prefetch" stage before stage 2.
 
 ## 📁 Package Structure
 
@@ -209,16 +296,11 @@ visual-rag-toolkit/
 Configure via environment variables or YAML:
 
 ```bash
-# Qdrant credentials (preferred names used by the demo + scripts)
-export SIGIR_QDRANT_URL="https://your-cluster.qdrant.io"
-export SIGIR_QDRANT_KEY="your-api-key"
 
-# Backwards-compatible fallbacks (also supported)
+# Qdrant credentials (preferred names used by the demo + scripts)
 export QDRANT_URL="https://your-cluster.qdrant.io"
 export QDRANT_API_KEY="your-api-key"
 
-export VISUALRAG_MODEL="vidore/colSmol-500M"
-
 # Special token handling (default: filter them out)
 export VISUALRAG_INCLUDE_SPECIAL_TOKENS=true  # Include special tokens
 ```
@@ -269,7 +351,7 @@ python -m benchmarks.vidore_beir_qdrant.run_qdrant_beir \
 ```
 
 More commands (including multi-stage variants and cropping configs) live in:
-- `benchmarks/vidore_tatdqa_test/COMMANDS.md`
+- `examples/COMMANDS.md`
 
 ## 🔧 Development
 
@@ -302,4 +384,3 @@ MIT License - see [LICENSE](LICENSE) for details.
 - [Qdrant](https://qdrant.tech/) - Vector database with multi-vector support
 - [ColPali](https://github.com/illuin-tech/colpali) - Visual document retrieval models
 - [ViDoRe](https://huggingface.co/spaces/vidore/vidore-leaderboard) - Benchmark dataset
-

{visual_rag_toolkit-0.1.1 → visual_rag_toolkit-0.1.3}/README.md

@@ -1,9 +1,8 @@
 # Visual RAG Toolkit
 
-[![PyPI version](https://badge.fury.io/py/visual-rag-toolkit.svg)](https://badge.fury.io/py/visual-rag-toolkit)
-[![CI](https://github.com/Ara-Yeroyan/visual-rag-toolkit/actions/workflows/ci.yaml/badge.svg)](https://github.com/Ara-Yeroyan/visual-rag-toolkit/actions/workflows/ci.yaml)
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
+[![PyPI](https://img.shields.io/pypi/v/visual-rag-toolkit)](https://pypi.org/project/visual-rag-toolkit/)
+[![Python](https://img.shields.io/pypi/pyversions/visual-rag-toolkit)](https://pypi.org/project/visual-rag-toolkit/)
+[![License](https://img.shields.io/pypi/l/visual-rag-toolkit)](LICENSE)
 
 End-to-end visual document retrieval toolkit featuring **fast multi-stage retrieval** (prefetch with pooled vectors + exact MaxSim reranking).
 
@@ -27,11 +26,10 @@ This repo contains:
 pip install visual-rag-toolkit
 
 # With specific features
-pip install visual-rag-toolkit[embedding]    # ColSmol/ColPali embedding support
-pip install visual-rag-toolkit[pdf]          # PDF processing
+pip install visual-rag-toolkit[ui]           # Streamlit demo dependencies
 pip install visual-rag-toolkit[qdrant]       # Vector database
+pip install visual-rag-toolkit[embedding]    # ColSmol/ColPali embedding support
 pip install visual-rag-toolkit[cloudinary]   # Image CDN
-pip install visual-rag-toolkit[ui]           # Streamlit demo dependencies
 
 # All dependencies
 pip install visual-rag-toolkit[all]
@@ -72,6 +70,95 @@ for r in results[:3]:
     print(r["id"], r["score_final"])
 ```
 
+### End-to-end: ingest PDFs (with cropping) → index in Qdrant
+
+This is the "SDK-style" pipeline: PDF → images → optional crop → embed → store vectors + payload in Qdrant.
+
+```python
+import os
+from pathlib import Path
+
+import numpy as np
+import torch
+
+from visual_rag import VisualEmbedder
+from visual_rag.indexing import ProcessingPipeline, QdrantIndexer
+
+QDRANT_URL = os.environ["QDRANT_URL"]
+QDRANT_KEY = os.getenv("QDRANT_API_KEY", "")
+
+collection = "my_visual_docs"
+
+embedder = VisualEmbedder(
+    model_name="vidore/colSmol-500M",
+    torch_dtype=torch.float16,
+    output_dtype=np.float16,
+    batch_size=8,
+)
+
+indexer = QdrantIndexer(
+    url=QDRANT_URL,
+    api_key=QDRANT_KEY,
+    collection_name=collection,
+    prefer_grpc=True,
+    vector_datatype="float16",
+)
+
+# Creates collection + required payload indexes (e.g., "filename" for skip_existing)
+indexer.create_collection(force_recreate=False)
+
+pipeline = ProcessingPipeline(
+    embedder=embedder,
+    indexer=indexer,
+    embedding_strategy="all",  # store full tokens + pooled vectors in one pass
+    crop_empty=True,
+    crop_empty_percentage_to_remove=0.99,  # kept for traceability
+    crop_empty_remove_page_number=True,
+    crop_empty_preserve_border_px=1,
+    crop_empty_uniform_rowcol_std_threshold=3.0,
+)
+
+pdfs = [Path("docs/a.pdf"), Path("docs/b.pdf")]
+for pdf_path in pdfs:
+    result = pipeline.process_pdf(
+        pdf_path,
+        skip_existing=True,  # Skip pages already in Qdrant (uses filename index)
+        upload_to_cloudinary=False,
+        upload_to_qdrant=True,
+    )
+    # Logs automatically shown:
+    # [10:23:45] 📚 Processing PDF: a.pdf
+    # [10:23:45] 🖼️ Converting PDF to images...
+    # [10:23:46]    ✅ Converted 12 pages
+    # [10:23:46] 📦 Processing pages 1-8/12
+    # [10:23:46] 🤖 Generating embeddings for 8 pages...
+    # [10:23:48] 📤 Uploading batch of 8 pages...
+    # [10:23:48]    ✅ Uploaded 8 points to Qdrant
+    # [10:23:48] 📦 Processing pages 9-12/12
+    # [10:23:48] 🤖 Generating embeddings for 4 pages...
+    # [10:23:50] 📤 Uploading batch of 4 pages...
+    # [10:23:50]    ✅ Uploaded 4 points to Qdrant
+    # [10:23:50] ✅ Completed a.pdf: 12 uploaded, 0 skipped, 0 failed
+```
+
+CLI equivalent:
+
+```bash
+export QDRANT_URL="https://YOUR_QDRANT"
+export QDRANT_API_KEY="YOUR_KEY"
+
+visual-rag process \
+  --reports-dir ./docs \
+  --collection my_visual_docs \
+  --model vidore/colSmol-500M \
+  --strategy all \
+  --batch-size 8 \
+  --qdrant-vector-dtype float16 \
+  --prefer-grpc \
+  --crop-empty \
+  --crop-empty-remove-page-number
+```
+
 ### Process a PDF into images (no embedding, no vector DB)
 
 ```python
@@ -101,7 +188,7 @@ Stage 2: Exact MaxSim reranking on candidates
          └── Return top-k results (e.g., 10)
 ```
 
-Three-stage extends this with an additional “cheap prefetch” stage before stage 2.
+Three-stage extends this with an additional "cheap prefetch" stage before stage 2.
 
 ## 📁 Package Structure
 
@@ -124,16 +211,11 @@ visual-rag-toolkit/
 Configure via environment variables or YAML:
 
 ```bash
-# Qdrant credentials (preferred names used by the demo + scripts)
-export SIGIR_QDRANT_URL="https://your-cluster.qdrant.io"
-export SIGIR_QDRANT_KEY="your-api-key"
 
-# Backwards-compatible fallbacks (also supported)
+# Qdrant credentials (preferred names used by the demo + scripts)
 export QDRANT_URL="https://your-cluster.qdrant.io"
 export QDRANT_API_KEY="your-api-key"
 
-export VISUALRAG_MODEL="vidore/colSmol-500M"
-
 # Special token handling (default: filter them out)
 export VISUALRAG_INCLUDE_SPECIAL_TOKENS=true  # Include special tokens
 ```
@@ -184,7 +266,7 @@ python -m benchmarks.vidore_beir_qdrant.run_qdrant_beir \
 ```
 
 More commands (including multi-stage variants and cropping configs) live in:
-- `benchmarks/vidore_tatdqa_test/COMMANDS.md`
+- `examples/COMMANDS.md`
 
 ## 🔧 Development
 
@@ -217,4 +299,3 @@ MIT License - see [LICENSE](LICENSE) for details.
 - [Qdrant](https://qdrant.tech/) - Vector database with multi-vector support
 - [ColPali](https://github.com/illuin-tech/colpali) - Visual document retrieval models
 - [ViDoRe](https://huggingface.co/spaces/vidore/vidore-leaderboard) - Benchmark dataset
-

{visual_rag_toolkit-0.1.1 → visual_rag_toolkit-0.1.3}/demo/app.py

@@ -1,13 +1,23 @@
 """Main entry point for the Visual RAG Toolkit demo application."""
 
+import os
 import sys
 from pathlib import Path
 
-ROOT_DIR = Path(__file__).parent.parent
-sys.path.insert(0, str(ROOT_DIR))
+# Ensure repo root is in sys.path for local development
+# (In HF Space / Docker, PYTHONPATH is already set correctly)
+_app_dir = Path(__file__).resolve().parent
+_repo_root = _app_dir.parent
+if str(_repo_root) not in sys.path:
+    sys.path.insert(0, str(_repo_root))
 
 from dotenv import load_dotenv
-load_dotenv(ROOT_DIR / ".env")
+
+# Load .env from the repo root (works both locally and in Docker)
+if (_repo_root / ".env").exists():
+    load_dotenv(_repo_root / ".env")
+if (_app_dir / ".env").exists():
+    load_dotenv(_app_dir / ".env")
 
 import streamlit as st
 
@@ -28,15 +38,17 @@ from demo.ui.benchmark import render_benchmark_tab
 def main():
     render_header()
     render_sidebar()
-    
-    tab_upload, tab_playground, tab_benchmark = st.tabs(["📤 Upload", "🎮 Playground", "📊 Benchmarking"])
-    
+
+    tab_upload, tab_playground, tab_benchmark = st.tabs(
+        ["📤 Upload", "🎮 Playground", "📊 Benchmarking"]
+    )
+
     with tab_upload:
         render_upload_tab()
-    
+
     with tab_playground:
         render_playground_tab()
-    
+
     with tab_benchmark:
         render_benchmark_tab()
 

{visual_rag_toolkit-0.1.1 → visual_rag_toolkit-0.1.3}/demo/evaluation.py

@@ -1,20 +1,23 @@
 """Evaluation runner with UI updates."""
 
 import hashlib
-import importlib.util
 import json
 import logging
 import time
 import traceback
 from datetime import datetime
-from pathlib import Path
 from typing import Any, Dict, List, Optional
 
 import numpy as np
 import streamlit as st
 import torch
+from qdrant_client.models import FieldCondition, Filter, MatchValue
 
 from visual_rag import VisualEmbedder
+from visual_rag.retrieval import MultiVectorRetriever
+from benchmarks.vidore_tatdqa_test.dataset_loader import load_vidore_beir_dataset
+from benchmarks.vidore_tatdqa_test.metrics import ndcg_at_k, mrr_at_k, recall_at_k
+from demo.qdrant_utils import get_qdrant_credentials
 
 
 TORCH_DTYPE_MAP = {
@@ -22,49 +25,6 @@ TORCH_DTYPE_MAP = {
     "float32": torch.float32,
     "bfloat16": torch.bfloat16,
 }
-from qdrant_client.models import Filter, FieldCondition, MatchValue
-
-from visual_rag.retrieval import MultiVectorRetriever
-
-
-def _load_local_benchmark_module(module_filename: str):
-    """
-    Load `benchmarks/vidore_tatdqa_test/<module_filename>` via file path.
-
-    Motivation:
-    - Some environments (notably containers / Spaces) can have a third-party
-      `benchmarks` package installed, causing `import benchmarks...` to resolve
-      to the wrong module.
-    - This fallback guarantees we load the repo's benchmark utilities.
-    """
-    root = Path(__file__).resolve().parents[1]  # demo/.. = repo root
-    target = root / "benchmarks" / "vidore_tatdqa_test" / module_filename
-    if not target.exists():
-        raise ModuleNotFoundError(f"Missing local benchmark module file: {target}")
-
-    name = f"_visual_rag_toolkit_local_{target.stem}"
-    spec = importlib.util.spec_from_file_location(name, str(target))
-    if spec is None or spec.loader is None:
-        raise ModuleNotFoundError(f"Could not load module spec for: {target}")
-    mod = importlib.util.module_from_spec(spec)
-    spec.loader.exec_module(mod)  # type: ignore[attr-defined]
-    return mod
-
-
-try:
-    # Preferred: normal import
-    from benchmarks.vidore_tatdqa_test.dataset_loader import load_vidore_beir_dataset
-    from benchmarks.vidore_tatdqa_test.metrics import ndcg_at_k, mrr_at_k, recall_at_k
-except ModuleNotFoundError:
-    # Robust fallback: load from local file paths
-    _dl = _load_local_benchmark_module("dataset_loader.py")
-    _mx = _load_local_benchmark_module("metrics.py")
-    load_vidore_beir_dataset = _dl.load_vidore_beir_dataset
-    ndcg_at_k = _mx.ndcg_at_k
-    mrr_at_k = _mx.mrr_at_k
-    recall_at_k = _mx.recall_at_k
-
-from demo.qdrant_utils import get_qdrant_credentials
 
 logger = logging.getLogger(__name__)
 logging.basicConfig(level=logging.INFO, format="[%(levelname)s] %(message)s")