visual-rag-toolkit 0.1.2__tar.gz → 0.1.4__tar.gz

{visual_rag_toolkit-0.1.2 → visual_rag_toolkit-0.1.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: visual-rag-toolkit
-Version: 0.1.2
+Version: 0.1.4
 Summary: End-to-end visual document retrieval with ColPali, featuring two-stage pooling for scalable search
 Project-URL: Homepage, https://github.com/Ara-Yeroyan/visual-rag-toolkit
 Project-URL: Documentation, https://github.com/Ara-Yeroyan/visual-rag-toolkit#readme
@@ -88,14 +88,12 @@ Description-Content-Type: text/markdown
 [![PyPI](https://img.shields.io/pypi/v/visual-rag-toolkit)](https://pypi.org/project/visual-rag-toolkit/)
 [![Python](https://img.shields.io/pypi/pyversions/visual-rag-toolkit)](https://pypi.org/project/visual-rag-toolkit/)
 [![License](https://img.shields.io/pypi/l/visual-rag-toolkit)](LICENSE)
-[![CI](https://img.shields.io/github/actions/workflow/status/Ara-Yeroyan/visual-rag-toolkit/ci.yaml?branch=main)](https://github.com/Ara-Yeroyan/visual-rag-toolkit/actions/workflows/ci.yaml)
-
-Note:
-- The **PyPI badge** shows “not found” until the first release is published.
-- The **CI badge** requires the GitHub repo to be **public** (GitHub does not serve Actions badges for private repos).
+[![Demo](https://img.shields.io/badge/Demo-Hugging%20Face-yellow)](https://huggingface.co/spaces/Yeroyan/visual-rag-toolkit)
 
 End-to-end visual document retrieval toolkit featuring **fast multi-stage retrieval** (prefetch with pooled vectors + exact MaxSim reranking).
 
+**[Try the Live Demo](https://huggingface.co/spaces/Yeroyan/visual-rag-toolkit)** - Upload PDFs, index to Qdrant, and query with visual retrieval.
+
 This repo contains:
 - a **Python package** (`visual_rag`)
 - a **Streamlit demo app** (`demo/`)
@@ -162,7 +160,7 @@ for r in results[:3]:
 
 ### End-to-end: ingest PDFs (with cropping) → index in Qdrant
 
-This is the “SDK-style” pipeline: PDF → images → optional crop → embed → store vectors + payload in Qdrant.
+This is the "SDK-style" pipeline: PDF → images → optional crop → embed → store vectors + payload in Qdrant.
 
 ```python
 import os
@@ -174,8 +172,8 @@ import torch
 from visual_rag import VisualEmbedder
 from visual_rag.indexing import ProcessingPipeline, QdrantIndexer
 
-QDRANT_URL = os.environ["SIGIR_QDRANT_URL"]  # or QDRANT_URL
-QDRANT_KEY = os.getenv("SIGIR_QDRANT_KEY", "")  # or QDRANT_API_KEY
+QDRANT_URL = os.environ["QDRANT_URL"]
+QDRANT_KEY = os.getenv("QDRANT_API_KEY", "")
 
 collection = "my_visual_docs"
 
@@ -193,6 +191,8 @@ indexer = QdrantIndexer(
     prefer_grpc=True,
     vector_datatype="float16",
 )
+
+# Creates collection + required payload indexes (e.g., "filename" for skip_existing)
 indexer.create_collection(force_recreate=False)
 
 pipeline = ProcessingPipeline(
@@ -208,19 +208,32 @@ pipeline = ProcessingPipeline(
 
 pdfs = [Path("docs/a.pdf"), Path("docs/b.pdf")]
 for pdf_path in pdfs:
-    pipeline.process_pdf(
+    result = pipeline.process_pdf(
         pdf_path,
-        skip_existing=True,
+        skip_existing=True,  # Skip pages already in Qdrant (uses filename index)
         upload_to_cloudinary=False,
         upload_to_qdrant=True,
     )
+    # Logs automatically shown:
+    # [10:23:45] 📚 Processing PDF: a.pdf
+    # [10:23:45] 🖼️ Converting PDF to images...
+    # [10:23:46]    ✅ Converted 12 pages
+    # [10:23:46] 📦 Processing pages 1-8/12
+    # [10:23:46] 🤖 Generating embeddings for 8 pages...
+    # [10:23:48] 📤 Uploading batch of 8 pages...
+    # [10:23:48]    ✅ Uploaded 8 points to Qdrant
+    # [10:23:48] 📦 Processing pages 9-12/12
+    # [10:23:48] 🤖 Generating embeddings for 4 pages...
+    # [10:23:50] 📤 Uploading batch of 4 pages...
+    # [10:23:50]    ✅ Uploaded 4 points to Qdrant
+    # [10:23:50] ✅ Completed a.pdf: 12 uploaded, 0 skipped, 0 failed
 ```
 
 CLI equivalent:
 
 ```bash
-export SIGIR_QDRANT_URL="https://YOUR_QDRANT"
-export SIGIR_QDRANT_KEY="YOUR_KEY"
+export QDRANT_URL="https://YOUR_QDRANT"
+export QDRANT_API_KEY="YOUR_KEY"
 
 visual-rag process \
   --reports-dir ./docs \
@@ -263,7 +276,7 @@ Stage 2: Exact MaxSim reranking on candidates
          └── Return top-k results (e.g., 10)
 ```
 
-Three-stage extends this with an additional “cheap prefetch” stage before stage 2.
+Three-stage extends this with an additional "cheap prefetch" stage before stage 2.
 
 ## 📁 Package Structure
 
@@ -358,7 +371,7 @@ If you use this toolkit in your research, please cite:
 
 ```bibtex
 @software{visual_rag_toolkit,
-  title = {Visual RAG Toolkit: Scalable Visual Document Retrieval with Two-Stage Pooling},
+  title = {Visual RAG Toolkit: Scalable Visual Document Retrieval with 1D Convolutional Pooling},
   author = {Ara Yeroyan},
   year = {2026},
   url = {https://github.com/Ara-Yeroyan/visual-rag-toolkit}
@@ -374,4 +387,3 @@ MIT License - see [LICENSE](LICENSE) for details.
 - [Qdrant](https://qdrant.tech/) - Vector database with multi-vector support
 - [ColPali](https://github.com/illuin-tech/colpali) - Visual document retrieval models
 - [ViDoRe](https://huggingface.co/spaces/vidore/vidore-leaderboard) - Benchmark dataset
-

{visual_rag_toolkit-0.1.2 → visual_rag_toolkit-0.1.4}/README.md

@@ -3,14 +3,12 @@
 [![PyPI](https://img.shields.io/pypi/v/visual-rag-toolkit)](https://pypi.org/project/visual-rag-toolkit/)
 [![Python](https://img.shields.io/pypi/pyversions/visual-rag-toolkit)](https://pypi.org/project/visual-rag-toolkit/)
 [![License](https://img.shields.io/pypi/l/visual-rag-toolkit)](LICENSE)
-[![CI](https://img.shields.io/github/actions/workflow/status/Ara-Yeroyan/visual-rag-toolkit/ci.yaml?branch=main)](https://github.com/Ara-Yeroyan/visual-rag-toolkit/actions/workflows/ci.yaml)
-
-Note:
-- The **PyPI badge** shows “not found” until the first release is published.
-- The **CI badge** requires the GitHub repo to be **public** (GitHub does not serve Actions badges for private repos).
+[![Demo](https://img.shields.io/badge/Demo-Hugging%20Face-yellow)](https://huggingface.co/spaces/Yeroyan/visual-rag-toolkit)
 
 End-to-end visual document retrieval toolkit featuring **fast multi-stage retrieval** (prefetch with pooled vectors + exact MaxSim reranking).
 
+**[Try the Live Demo](https://huggingface.co/spaces/Yeroyan/visual-rag-toolkit)** - Upload PDFs, index to Qdrant, and query with visual retrieval.
+
 This repo contains:
 - a **Python package** (`visual_rag`)
 - a **Streamlit demo app** (`demo/`)
@@ -77,7 +75,7 @@ for r in results[:3]:
 
 ### End-to-end: ingest PDFs (with cropping) → index in Qdrant
 
-This is the “SDK-style” pipeline: PDF → images → optional crop → embed → store vectors + payload in Qdrant.
+This is the "SDK-style" pipeline: PDF → images → optional crop → embed → store vectors + payload in Qdrant.
 
 ```python
 import os
@@ -89,8 +87,8 @@ import torch
 from visual_rag import VisualEmbedder
 from visual_rag.indexing import ProcessingPipeline, QdrantIndexer
 
-QDRANT_URL = os.environ["SIGIR_QDRANT_URL"]  # or QDRANT_URL
-QDRANT_KEY = os.getenv("SIGIR_QDRANT_KEY", "")  # or QDRANT_API_KEY
+QDRANT_URL = os.environ["QDRANT_URL"]
+QDRANT_KEY = os.getenv("QDRANT_API_KEY", "")
 
 collection = "my_visual_docs"
 
@@ -108,6 +106,8 @@ indexer = QdrantIndexer(
     prefer_grpc=True,
     vector_datatype="float16",
 )
+
+# Creates collection + required payload indexes (e.g., "filename" for skip_existing)
 indexer.create_collection(force_recreate=False)
 
 pipeline = ProcessingPipeline(
@@ -123,19 +123,32 @@ pipeline = ProcessingPipeline(
 
 pdfs = [Path("docs/a.pdf"), Path("docs/b.pdf")]
 for pdf_path in pdfs:
-    pipeline.process_pdf(
+    result = pipeline.process_pdf(
         pdf_path,
-        skip_existing=True,
+        skip_existing=True,  # Skip pages already in Qdrant (uses filename index)
         upload_to_cloudinary=False,
         upload_to_qdrant=True,
     )
+    # Logs automatically shown:
+    # [10:23:45] 📚 Processing PDF: a.pdf
+    # [10:23:45] 🖼️ Converting PDF to images...
+    # [10:23:46]    ✅ Converted 12 pages
+    # [10:23:46] 📦 Processing pages 1-8/12
+    # [10:23:46] 🤖 Generating embeddings for 8 pages...
+    # [10:23:48] 📤 Uploading batch of 8 pages...
+    # [10:23:48]    ✅ Uploaded 8 points to Qdrant
+    # [10:23:48] 📦 Processing pages 9-12/12
+    # [10:23:48] 🤖 Generating embeddings for 4 pages...
+    # [10:23:50] 📤 Uploading batch of 4 pages...
+    # [10:23:50]    ✅ Uploaded 4 points to Qdrant
+    # [10:23:50] ✅ Completed a.pdf: 12 uploaded, 0 skipped, 0 failed
 ```
 
 CLI equivalent:
 
 ```bash
-export SIGIR_QDRANT_URL="https://YOUR_QDRANT"
-export SIGIR_QDRANT_KEY="YOUR_KEY"
+export QDRANT_URL="https://YOUR_QDRANT"
+export QDRANT_API_KEY="YOUR_KEY"
 
 visual-rag process \
   --reports-dir ./docs \
@@ -178,7 +191,7 @@ Stage 2: Exact MaxSim reranking on candidates
          └── Return top-k results (e.g., 10)
 ```
 
-Three-stage extends this with an additional “cheap prefetch” stage before stage 2.
+Three-stage extends this with an additional "cheap prefetch" stage before stage 2.
 
 ## 📁 Package Structure
 
@@ -273,7 +286,7 @@ If you use this toolkit in your research, please cite:
 
 ```bibtex
 @software{visual_rag_toolkit,
-  title = {Visual RAG Toolkit: Scalable Visual Document Retrieval with Two-Stage Pooling},
+  title = {Visual RAG Toolkit: Scalable Visual Document Retrieval with 1D Convolutional Pooling},
   author = {Ara Yeroyan},
   year = {2026},
   url = {https://github.com/Ara-Yeroyan/visual-rag-toolkit}
@@ -289,4 +302,3 @@ MIT License - see [LICENSE](LICENSE) for details.
 - [Qdrant](https://qdrant.tech/) - Vector database with multi-vector support
 - [ColPali](https://github.com/illuin-tech/colpali) - Visual document retrieval models
 - [ViDoRe](https://huggingface.co/spaces/vidore/vidore-leaderboard) - Benchmark dataset
-

{visual_rag_toolkit-0.1.2 → visual_rag_toolkit-0.1.4}/demo/__init__.py

@@ -7,4 +7,4 @@ A Streamlit-based UI for:
 - Interactive playground for visual search
 """
 
-__version__ = "0.1.0"
+__version__ = "0.1.4"

{visual_rag_toolkit-0.1.2 → visual_rag_toolkit-0.1.4}/demo/app.py

@@ -1,13 +1,23 @@
 """Main entry point for the Visual RAG Toolkit demo application."""
 
+import os
 import sys
 from pathlib import Path
 
-ROOT_DIR = Path(__file__).parent.parent
-sys.path.insert(0, str(ROOT_DIR))
+# Ensure repo root is in sys.path for local development
+# (In HF Space / Docker, PYTHONPATH is already set correctly)
+_app_dir = Path(__file__).resolve().parent
+_repo_root = _app_dir.parent
+if str(_repo_root) not in sys.path:
+    sys.path.insert(0, str(_repo_root))
 
 from dotenv import load_dotenv
-load_dotenv(ROOT_DIR / ".env")
+
+# Load .env from the repo root (works both locally and in Docker)
+if (_repo_root / ".env").exists():
+    load_dotenv(_repo_root / ".env")
+if (_app_dir / ".env").exists():
+    load_dotenv(_app_dir / ".env")
 
 import streamlit as st
 
@@ -28,15 +38,17 @@ from demo.ui.benchmark import render_benchmark_tab
 def main():
     render_header()
     render_sidebar()
-    
-    tab_upload, tab_playground, tab_benchmark = st.tabs(["📤 Upload", "🎮 Playground", "📊 Benchmarking"])
-    
+
+    tab_upload, tab_playground, tab_benchmark = st.tabs(
+        ["📤 Upload", "🎮 Playground", "📊 Benchmarking"]
+    )
+
     with tab_upload:
         render_upload_tab()
-    
+
     with tab_playground:
         render_playground_tab()
-    
+
     with tab_benchmark:
         render_benchmark_tab()
 

{visual_rag_toolkit-0.1.2 → visual_rag_toolkit-0.1.4}/demo/evaluation.py

@@ -1,20 +1,23 @@
 """Evaluation runner with UI updates."""
 
 import hashlib
-import importlib.util
 import json
 import logging
 import time
 import traceback
 from datetime import datetime
-from pathlib import Path
 from typing import Any, Dict, List, Optional
 
 import numpy as np
 import streamlit as st
 import torch
+from qdrant_client.models import FieldCondition, Filter, MatchValue
 
 from visual_rag import VisualEmbedder
+from visual_rag.retrieval import MultiVectorRetriever
+from benchmarks.vidore_tatdqa_test.dataset_loader import load_vidore_beir_dataset
+from benchmarks.vidore_tatdqa_test.metrics import ndcg_at_k, mrr_at_k, recall_at_k
+from demo.qdrant_utils import get_qdrant_credentials
 
 
 TORCH_DTYPE_MAP = {
@@ -22,49 +25,6 @@ TORCH_DTYPE_MAP = {
     "float32": torch.float32,
     "bfloat16": torch.bfloat16,
 }
-from qdrant_client.models import Filter, FieldCondition, MatchValue
-
-from visual_rag.retrieval import MultiVectorRetriever
-
-
-def _load_local_benchmark_module(module_filename: str):
-    """
-    Load `benchmarks/vidore_tatdqa_test/<module_filename>` via file path.
-
-    Motivation:
-    - Some environments (notably containers / Spaces) can have a third-party
-      `benchmarks` package installed, causing `import benchmarks...` to resolve
-      to the wrong module.
-    - This fallback guarantees we load the repo's benchmark utilities.
-    """
-    root = Path(__file__).resolve().parents[1]  # demo/.. = repo root
-    target = root / "benchmarks" / "vidore_tatdqa_test" / module_filename
-    if not target.exists():
-        raise ModuleNotFoundError(f"Missing local benchmark module file: {target}")
-
-    name = f"_visual_rag_toolkit_local_{target.stem}"
-    spec = importlib.util.spec_from_file_location(name, str(target))
-    if spec is None or spec.loader is None:
-        raise ModuleNotFoundError(f"Could not load module spec for: {target}")
-    mod = importlib.util.module_from_spec(spec)
-    spec.loader.exec_module(mod)  # type: ignore[attr-defined]
-    return mod
-
-
-try:
-    # Preferred: normal import
-    from benchmarks.vidore_tatdqa_test.dataset_loader import load_vidore_beir_dataset
-    from benchmarks.vidore_tatdqa_test.metrics import ndcg_at_k, mrr_at_k, recall_at_k
-except ModuleNotFoundError:
-    # Robust fallback: load from local file paths
-    _dl = _load_local_benchmark_module("dataset_loader.py")
-    _mx = _load_local_benchmark_module("metrics.py")
-    load_vidore_beir_dataset = _dl.load_vidore_beir_dataset
-    ndcg_at_k = _mx.ndcg_at_k
-    mrr_at_k = _mx.mrr_at_k
-    recall_at_k = _mx.recall_at_k
-
-from demo.qdrant_utils import get_qdrant_credentials
 
 logger = logging.getLogger(__name__)
 logging.basicConfig(level=logging.INFO, format="[%(levelname)s] %(message)s")

visual_rag_toolkit-0.1.4/demo/indexing.py

@@ -0,0 +1,274 @@
+"""Indexing runner with UI updates."""
+
+import hashlib
+import json
+import time
+import traceback
+from datetime import datetime
+from typing import Any, Dict, Optional
+
+import numpy as np
+import streamlit as st
+import torch
+
+from visual_rag import VisualEmbedder
+from visual_rag.embedding.pooling import tile_level_mean_pooling
+from visual_rag.indexing.qdrant_indexer import QdrantIndexer
+from benchmarks.vidore_tatdqa_test.dataset_loader import load_vidore_beir_dataset
+from demo.qdrant_utils import get_qdrant_credentials
+
+
+TORCH_DTYPE_MAP = {
+    "float16": torch.float16,
+    "float32": torch.float32,
+    "bfloat16": torch.bfloat16,
+}
+
+
+def _stable_uuid(text: str) -> str:
+    """Generate a stable UUID from text (same as benchmark script)."""
+    hex_str = hashlib.sha256(text.encode("utf-8")).hexdigest()[:32]
+    return f"{hex_str[:8]}-{hex_str[8:12]}-{hex_str[12:16]}-{hex_str[16:20]}-{hex_str[20:32]}"
+
+
+def _union_point_id(
+    *, dataset_name: str, source_doc_id: str, union_namespace: Optional[str]
+) -> str:
+    """Generate union point ID (same as benchmark script)."""
+    ns = f"{union_namespace}::{dataset_name}" if union_namespace else dataset_name
+    return _stable_uuid(f"{ns}::{source_doc_id}")
+
+
+def run_indexing_with_ui(config: Dict[str, Any]):
+    st.divider()
+
+    print("=" * 60)
+    print("[INDEX] Starting indexing via UI")
+    print("=" * 60)
+
+    url, api_key = get_qdrant_credentials()
+    if not url:
+        st.error("QDRANT_URL not configured")
+        return
+
+    datasets = config.get("datasets", [])
+    collection = config["collection"]
+    model = config.get("model", "vidore/colpali-v1.3")
+    recreate = config.get("recreate", False)
+    torch_dtype = config.get("torch_dtype", "float16")
+    qdrant_vector_dtype = config.get("qdrant_vector_dtype", "float16")
+    prefer_grpc = config.get("prefer_grpc", True)
+    batch_size = config.get("batch_size", 4)
+    max_docs = config.get("max_docs")
+
+    print(f"[INDEX] Config: collection={collection}, model={model}")
+    print(f"[INDEX] Datasets: {datasets}")
+    print(
+        f"[INDEX] max_docs={max_docs}, batch_size={batch_size}, recreate={recreate}"
+    )
+    print(
+        f"[INDEX] torch_dtype={torch_dtype}, qdrant_dtype={qdrant_vector_dtype}, grpc={prefer_grpc}"
+    )
+
+    phase1_container = st.container()
+    phase2_container = st.container()
+    phase3_container = st.container()
+    results_container = st.container()
+
+    try:
+        with phase1_container:
+            st.markdown("##### 🤖 Phase 1: Loading Model")
+            model_status = st.empty()
+            model_status.info(f"Loading `{model.split('/')[-1]}`...")
+
+            print(f"[INDEX] Loading embedder: {model}")
+            torch_dtype_obj = TORCH_DTYPE_MAP.get(torch_dtype, torch.float16)
+            output_dtype_obj = (
+                np.float16 if qdrant_vector_dtype == "float16" else np.float32
+            )
+            embedder = VisualEmbedder(
+                model_name=model,
+                torch_dtype=torch_dtype_obj,
+                output_dtype=output_dtype_obj,
+            )
+            embedder._load_model()
+            print(
+                f"[INDEX] Embedder loaded (torch_dtype={torch_dtype}, output_dtype={qdrant_vector_dtype})"
+            )
+            model_status.success(f"✅ Model `{model.split('/')[-1]}` loaded")
+
+        with phase2_container:
+            st.markdown("##### 📦 Phase 2: Setting Up Collection")
+
+            indexer_status = st.empty()
+            indexer_status.info("Connecting to Qdrant...")
+
+            print("[INDEX] Connecting to Qdrant...")
+            indexer = QdrantIndexer(
+                url=url,
+                api_key=api_key,
+                collection_name=collection,
+                prefer_grpc=prefer_grpc,
+                vector_datatype=qdrant_vector_dtype,
+            )
+            print("[INDEX] Connected to Qdrant")
+            indexer_status.success("✅ Connected to Qdrant")
+
+            coll_status = st.empty()
+            action = "Recreating" if recreate else "Creating/verifying"
+            coll_status.info(f"{action} collection `{collection}`...")
+
+            print(f"[INDEX] {action} collection: {collection}")
+            indexer.create_collection(force_recreate=recreate)
+            indexer.create_payload_indexes(
+                fields=[
+                    {"field": "dataset", "type": "keyword"},
+                    {"field": "doc_id", "type": "keyword"},
+                    {"field": "source_doc_id", "type": "keyword"},
+                ]
+            )
+            print("[INDEX] Collection ready")
+            coll_status.success(f"✅ Collection `{collection}` ready")
+
+        with phase3_container:
+            st.markdown("##### 📊 Phase 3: Processing Datasets")
+
+            all_results = []
+
+            for ds_idx, dataset_name in enumerate(datasets):
+                ds_short = dataset_name.split("/")[-1]
+                ds_container = st.container()
+
+                with ds_container:
+                    st.markdown(
+                        f"**Dataset {ds_idx + 1}/{len(datasets)}: `{ds_short}`**"
+                    )
+
+                    load_status = st.empty()
+                    load_status.info(f"Loading dataset `{ds_short}`...")
+
+                    print(f"[INDEX] Loading dataset: {dataset_name}")
+                    corpus, queries, qrels = load_vidore_beir_dataset(dataset_name)
+                    total_docs = len(corpus)
+                    print(f"[INDEX] Dataset loaded: {total_docs} docs")
+                    load_status.success(f"✅ Loaded {total_docs:,} documents")
+
+                    if max_docs and max_docs < total_docs:
+                        corpus = corpus[:max_docs]
+                        print(f"[INDEX] Limiting to {max_docs} docs")
+
+                    progress_bar = st.progress(0)
+                    status_text = st.empty()
+
+                    uploaded = 0
+                    failed = 0
+                    total = len(corpus)
+
+                    for i, doc in enumerate(corpus):
+                        try:
+                            doc_id = str(doc.doc_id)
+                            image = doc.image
+                            if image is None:
+                                failed += 1
+                                continue
+
+                            status_text.text(
+                                f"Processing {i + 1}/{total}: {doc_id[:30]}..."
+                            )
+
+                            embeddings, token_infos = embedder.embed_images(
+                                [image],
+                                return_token_info=True,
+                                show_progress=False,
+                            )
+                            emb = embeddings[0]
+                            token_info = token_infos[0] if token_infos else {}
+
+                            if hasattr(emb, "cpu"):
+                                emb = emb.cpu()
+                            emb_np = np.asarray(emb, dtype=output_dtype_obj)
+
+                            initial = emb_np.tolist()
+                            global_pool = emb_np.mean(axis=0).tolist()
+
+                            num_tiles = token_info.get("num_tiles")
+                            mean_pooling = None
+                            experimental_pooling = None
+
+                            if num_tiles and num_tiles > 0:
+                                try:
+                                    mean_pooling = tile_level_mean_pooling(
+                                        emb_np, num_tiles=num_tiles, patches_per_tile=64
+                                    ).tolist()
+                                except Exception:
+                                    pass
+
+                                try:
+                                    exp_pool = embedder.experimental_pool_visual_embedding(
+                                        emb_np, num_tiles=num_tiles
+                                    )
+                                    if exp_pool is not None:
+                                        experimental_pooling = exp_pool.tolist()
+                                except Exception:
+                                    pass
+
+                            union_doc_id = _union_point_id(
+                                dataset_name=dataset_name,
+                                source_doc_id=doc_id,
+                                union_namespace=collection,
+                            )
+
+                            payload = {
+                                "dataset": dataset_name,
+                                "doc_id": doc_id,
+                                "source_doc_id": doc_id,
+                                "union_doc_id": union_doc_id,
+                                "num_tiles": num_tiles,
+                                "num_visual_tokens": token_info.get("num_visual_tokens"),
+                            }
+
+                            vectors = {"initial": initial, "global_pooling": global_pool}
+                            if mean_pooling:
+                                vectors["mean_pooling"] = mean_pooling
+                            if experimental_pooling:
+                                vectors["experimental_pooling"] = experimental_pooling
+
+                            indexer.upsert_point(
+                                point_id=union_doc_id,
+                                vectors=vectors,
+                                payload=payload,
+                            )
+
+                            uploaded += 1
+
+                        except Exception as e:
+                            print(f"[INDEX] Error on doc {i}: {e}")
+                            failed += 1
+
+                        progress_bar.progress((i + 1) / total)
+
+                    status_text.text(f"✅ Done: {uploaded} uploaded, {failed} failed")
+                    all_results.append(
+                        {
+                            "dataset": dataset_name,
+                            "total": total,
+                            "uploaded": uploaded,
+                            "failed": failed,
+                        }
+                    )
+
+        with results_container:
+            st.markdown("##### 📋 Results Summary")
+
+            for r in all_results:
+                st.write(
+                    f"**{r['dataset'].split('/')[-1]}**: {r['uploaded']:,} uploaded, {r['failed']:,} failed"
+                )
+
+            st.success("✅ Indexing complete!")
+
+    except Exception as e:
+        st.error(f"Indexing error: {e}")
+        st.code(traceback.format_exc())
+        print(f"[INDEX] ERROR: {e}")
+        traceback.print_exc()

{visual_rag_toolkit-0.1.2 → visual_rag_toolkit-0.1.4}/demo/qdrant_utils.py

@@ -8,12 +8,19 @@ import streamlit as st
 
 
 def get_qdrant_credentials() -> Tuple[Optional[str], Optional[str]]:
-    url = st.session_state.get("qdrant_url_input") or os.getenv("SIGIR_QDRANT_URL") or os.getenv("DEST_QDRANT_URL") or os.getenv("QDRANT_URL")
-    api_key = st.session_state.get("qdrant_key_input") or (
-        os.getenv("SIGIR_QDRANT_KEY")
-        or os.getenv("SIGIR_QDRANT_API_KEY")
-        or os.getenv("DEST_QDRANT_API_KEY")
+    """Get Qdrant credentials from session state or environment variables.
+    
+    Priority: session_state > QDRANT_URL/QDRANT_API_KEY > legacy env vars
+    """
+    url = (
+        st.session_state.get("qdrant_url_input")
+        or os.getenv("QDRANT_URL")
+        or os.getenv("SIGIR_QDRANT_URL")  # legacy
+    )
+    api_key = (
+        st.session_state.get("qdrant_key_input")
         or os.getenv("QDRANT_API_KEY")
+        or os.getenv("SIGIR_QDRANT_KEY")  # legacy
     )
     return url, api_key
 

{visual_rag_toolkit-0.1.2 → visual_rag_toolkit-0.1.4}/demo/ui/playground.py

@@ -9,6 +9,7 @@ from demo.qdrant_utils import (
     sample_points_cached,
     search_collection,
 )
+from visual_rag.retrieval import MultiVectorRetriever
 
 
 def render_playground_tab():
@@ -46,7 +47,6 @@ def render_playground_tab():
         if not st.session_state.get("model_loaded"):
             with st.spinner(f"Loading {model_short}..."):
                 try:
-                    from visual_rag.retrieval import MultiVectorRetriever
                     _ = MultiVectorRetriever(collection_name=active_collection, model_name=model_name)
                     st.session_state["model_loaded"] = True
                     st.session_state["loaded_model_key"] = cache_key