haiku.rag 0.7.0__tar.gz → 0.7.2__tar.gz

{haiku_rag-0.7.0 → haiku_rag-0.7.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: haiku.rag
-Version: 0.7.0
+Version: 0.7.2
 Summary: Retrieval Augmented Generation (RAG) with LanceDB
 Author-email: Yiorgis Gozadinos <ggozadinos@gmail.com>
 License: MIT
@@ -47,10 +47,10 @@ Retrieval-Augmented Generation (RAG) library built on LanceDB.
 ## Features
 
 - **Local LanceDB**: No external servers required, supports also LanceDB cloud storage, S3, Google Cloud & Azure
-- **Multiple embedding providers**: Ollama, VoyageAI, OpenAI
+- **Multiple embedding providers**: Ollama, VoyageAI, OpenAI, vLLM
 - **Multiple QA providers**: Any provider/model supported by Pydantic AI
 - **Native hybrid search**: Vector + full-text search with native LanceDB RRF reranking
-- **Reranking**: Default search result reranking with MixedBread AI or Cohere
+- **Reranking**: Default search result reranking with MixedBread AI, Cohere, or vLLM
 - **Question answering**: Built-in QA agents on your documents
 - **File monitoring**: Auto-index files when run as server
 - **40+ file formats**: PDF, DOCX, HTML, Markdown, code files, URLs

{haiku_rag-0.7.0 → haiku_rag-0.7.2}/README.md

@@ -9,10 +9,10 @@ Retrieval-Augmented Generation (RAG) library built on LanceDB.
 ## Features
 
 - **Local LanceDB**: No external servers required, supports also LanceDB cloud storage, S3, Google Cloud & Azure
-- **Multiple embedding providers**: Ollama, VoyageAI, OpenAI
+- **Multiple embedding providers**: Ollama, VoyageAI, OpenAI, vLLM
 - **Multiple QA providers**: Any provider/model supported by Pydantic AI
 - **Native hybrid search**: Vector + full-text search with native LanceDB RRF reranking
-- **Reranking**: Default search result reranking with MixedBread AI or Cohere
+- **Reranking**: Default search result reranking with MixedBread AI, Cohere, or vLLM
 - **Question answering**: Built-in QA agents on your documents
 - **File monitoring**: Auto-index files when run as server
 - **40+ file formats**: PDF, DOCX, HTML, Markdown, code files, URLs

{haiku_rag-0.7.0 → haiku_rag-0.7.2}/docs/configuration.md

@@ -53,6 +53,18 @@ EMBEDDINGS_VECTOR_DIM=1536
 OPENAI_API_KEY="your-api-key"
 ```
 
+### vLLM
+For high-performance local inference, you can use vLLM to serve embedding models with OpenAI-compatible APIs:
+
+```bash
+EMBEDDINGS_PROVIDER="vllm"
+EMBEDDINGS_MODEL="mixedbread-ai/mxbai-embed-large-v1"  # Any embedding model supported by vLLM
+EMBEDDINGS_VECTOR_DIM=512  # Dimension depends on the model
+VLLM_EMBEDDINGS_BASE_URL="http://localhost:8000"  # vLLM server URL
+```
+
+**Note:** You need to run a vLLM server separately with an embedding model loaded.
+
 ## Question Answering Providers
 
 Configure which LLM provider to use for question answering. Any provider and model supported by [Pydantic AI](https://ai.pydantic.dev/models/) can be used.
@@ -85,6 +97,18 @@ QA_MODEL="claude-3-5-haiku-20241022"  # or claude-3-5-sonnet-20241022, etc.
 ANTHROPIC_API_KEY="your-api-key"
 ```
 
+### vLLM
+
+For high-performance local inference, you can use vLLM to serve models with OpenAI-compatible APIs:
+
+```bash
+QA_PROVIDER="vllm"
+QA_MODEL="Qwen/Qwen3-4B"  # Any model with tool support in vLLM
+VLLM_QA_BASE_URL="http://localhost:8002"  # vLLM server URL
+```
+
+**Note:** You need to run a vLLM server separately with a model that supports tool calling loaded. Consult the specific model's documentation for proper vLLM serving configuration.
+
 ### Other Providers
 
 Any provider supported by Pydantic AI can be used. Examples include:
@@ -136,6 +160,18 @@ RERANK_MODEL="rerank-v3.5"
 COHERE_API_KEY="your-api-key"
 ```
 
+### vLLM
+
+For high-performance local reranking using dedicated reranking models:
+
+```bash
+RERANK_PROVIDER="vllm"
+RERANK_MODEL="mixedbread-ai/mxbai-rerank-base-v2"  # Any reranking model supported by vLLM
+VLLM_RERANK_BASE_URL="http://localhost:8001"  # vLLM server URL
+```
+
+**Note:** vLLM reranking uses the `/rerank` API endpoint. You need to run a vLLM server separately with a reranking model loaded. Consult the specific model's documentation for proper vLLM serving configuration.
+
 ## Other Settings
 
 ### Database and Storage

haiku_rag-0.7.2/docs/installation.md

@@ -0,0 +1,74 @@
+# Installation
+
+## Basic Installation
+
+```bash
+uv pip install haiku.rag
+```
+
+This includes support for:
+- **Ollama** (default embedding provider using `mxbai-embed-large`)
+- **OpenAI** (GPT models for QA and embeddings)
+- **Anthropic** (Claude models for QA)
+- **Cohere** (reranking models)
+- **vLLM** (high-performance local inference for embeddings, QA, and reranking)
+
+## Provider-Specific Installation
+
+For additional embedding providers, install with extras:
+
+### VoyageAI
+
+```bash
+uv pip install haiku.rag[voyageai]
+```
+
+### MixedBread AI Reranking
+
+```bash
+uv pip install haiku.rag[mxbai]
+```
+
+### vLLM Setup
+
+vLLM requires no additional installation - it works with the base haiku.rag package. However, you need to run vLLM servers separately:
+
+```bash
+# Install vLLM
+pip install vllm
+
+# Serve an embedding model
+vllm serve mixedbread-ai/mxbai-embed-large-v1 --port 8000
+
+# Serve a model for QA (requires tool calling support)
+vllm serve Qwen/Qwen3-4B --port 8002 --enable-auto-tool-choice --tool-call-parser hermes
+
+# Serve a model for reranking
+vllm serve mixedbread-ai/mxbai-rerank-base-v2 --hf_overrides '{"architectures": ["Qwen2ForSequenceClassification"],"classifier_from_token": ["0", "1"], "method": "from_2_way_softmax"}' --port 8001
+```
+
+Then configure haiku.rag to use the vLLM servers:
+
+```bash
+# Embeddings
+EMBEDDINGS_PROVIDER="vllm"
+EMBEDDINGS_MODEL="mixedbread-ai/mxbai-embed-large-v1"
+EMBEDDINGS_VECTOR_DIM=512
+VLLM_EMBEDDINGS_BASE_URL="http://localhost:8000"
+
+# QA (optional)
+QA_PROVIDER="vllm"
+QA_MODEL="Qwen/Qwen3-4B"
+VLLM_QA_BASE_URL="http://localhost:8002"
+
+# Reranking (optional)
+RERANK_PROVIDER="vllm"
+RERANK_MODEL="mixedbread-ai/mxbai-rerank-base-v2"
+VLLM_RERANK_BASE_URL="http://localhost:8001"
+```
+
+## Requirements
+
+- Python 3.10+
+- Ollama (for default embeddings)
+- vLLM server (for vLLM provider)

{haiku_rag-0.7.0 → haiku_rag-0.7.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "haiku.rag"
-version = "0.7.0"
+version = "0.7.2"
 description = "Retrieval Augmented Generation (RAG) with LanceDB"
 authors = [{ name = "Yiorgis Gozadinos", email = "ggozadinos@gmail.com" }]
 license = { text = "MIT" }

{haiku_rag-0.7.0 → haiku_rag-0.7.2}/src/haiku/rag/config.py

@@ -33,6 +33,9 @@ class AppConfig(BaseModel):
     CONTEXT_CHUNK_RADIUS: int = 0
 
     OLLAMA_BASE_URL: str = "http://localhost:11434"
+    VLLM_EMBEDDINGS_BASE_URL: str = ""
+    VLLM_RERANK_BASE_URL: str = ""
+    VLLM_QA_BASE_URL: str = ""
 
     # Provider keys
     VOYAGE_API_KEY: str = ""

haiku_rag-0.7.2/src/haiku/rag/embeddings/vllm.py

@@ -0,0 +1,16 @@
+from openai import AsyncOpenAI
+
+from haiku.rag.config import Config
+from haiku.rag.embeddings.base import EmbedderBase
+
+
+class Embedder(EmbedderBase):
+    async def embed(self, text: str) -> list[float]:
+        client = AsyncOpenAI(
+            base_url=f"{Config.VLLM_EMBEDDINGS_BASE_URL}/v1", api_key="dummy"
+        )
+        response = await client.embeddings.create(
+            model=self._model,
+            input=text,
+        )
+        return response.data[0].embedding

{haiku_rag-0.7.0 → haiku_rag-0.7.2}/src/haiku/rag/logging.py

@@ -6,6 +6,7 @@ from rich.logging import RichHandler
 logging.basicConfig(level=logging.DEBUG)
 logging.getLogger("httpx").setLevel(logging.WARNING)
 logging.getLogger("httpcore").setLevel(logging.WARNING)
+logging.getLogger("docling").setLevel(logging.WARNING)
 
 
 def get_logger() -> logging.Logger:

{haiku_rag-0.7.0 → haiku_rag-0.7.2}/src/haiku/rag/qa/agent.py

@@ -1,7 +1,8 @@
 from pydantic import BaseModel, Field
 from pydantic_ai import Agent, RunContext
-from pydantic_ai.models.openai import OpenAIModel
+from pydantic_ai.models.openai import OpenAIChatModel
 from pydantic_ai.providers.ollama import OllamaProvider
+from pydantic_ai.providers.openai import OpenAIProvider
 
 from haiku.rag.client import HaikuRAG
 from haiku.rag.config import Config
@@ -61,10 +62,17 @@ class QuestionAnswerAgent:
     def _get_model(self, provider: str, model: str):
         """Get the appropriate model object for the provider."""
         if provider == "ollama":
-            return OpenAIModel(
+            return OpenAIChatModel(
                 model_name=model,
                 provider=OllamaProvider(base_url=f"{Config.OLLAMA_BASE_URL}/v1"),
             )
+        elif provider == "vllm":
+            return OpenAIChatModel(
+                model_name=model,
+                provider=OpenAIProvider(
+                    base_url=f"{Config.VLLM_QA_BASE_URL}/v1", api_key="none"
+                ),
+            )
         else:
             # For all other providers, use the provider:model format
             return f"{provider}:{model}"

haiku_rag-0.7.2/src/haiku/rag/reranking/vllm.py

@@ -0,0 +1,44 @@
+import httpx
+
+from haiku.rag.config import Config
+from haiku.rag.reranking.base import RerankerBase
+from haiku.rag.store.models.chunk import Chunk
+
+
+class VLLMReranker(RerankerBase):
+    def __init__(self, model: str):
+        self._model = model
+        self._base_url = Config.VLLM_RERANK_BASE_URL
+
+    async def rerank(
+        self, query: str, chunks: list[Chunk], top_n: int = 10
+    ) -> list[tuple[Chunk, float]]:
+        if not chunks:
+            return []
+
+        # Prepare documents for reranking
+        documents = [chunk.content for chunk in chunks]
+
+        async with httpx.AsyncClient() as client:
+            response = await client.post(
+                f"{self._base_url}/v1/rerank",
+                json={"model": self._model, "query": query, "documents": documents},
+                headers={
+                    "accept": "application/json",
+                    "Content-Type": "application/json",
+                },
+            )
+            response.raise_for_status()
+
+            result = response.json()
+
+            # Extract scores and pair with chunks
+            scored_chunks = []
+            for item in result.get("results", []):
+                index = item["index"]
+                score = item["relevance_score"]
+                scored_chunks.append((chunks[index], score))
+
+            # Sort by score (descending) and return top_n
+            scored_chunks.sort(key=lambda x: x[1], reverse=True)
+            return scored_chunks[:top_n]

{haiku_rag-0.7.0 → haiku_rag-0.7.2}/tests/generate_benchmark_db.py

@@ -6,6 +6,7 @@ from llm_judge import LLMJudge
 from rich.console import Console
 from rich.progress import Progress
 
+from haiku.rag import logging  # noqa
 from haiku.rag.client import HaikuRAG
 from haiku.rag.qa import get_qa_agent
 

{haiku_rag-0.7.0 → haiku_rag-0.7.2}/tests/llm_judge.py

@@ -1,6 +1,6 @@
 from pydantic import BaseModel
 from pydantic_ai import Agent
-from pydantic_ai.models.openai import OpenAIModel
+from pydantic_ai.models.openai import OpenAIChatModel
 from pydantic_ai.providers.ollama import OllamaProvider
 
 from haiku.rag.config import Config
@@ -37,9 +37,9 @@ class LLMJudgeResponseSchema(BaseModel):
 class LLMJudge:
     """LLM-as-judge for evaluating answer equivalence using Pydantic AI."""
 
-    def __init__(self, model: str = Config.QA_MODEL):
+    def __init__(self, model: str = "qwen3"):
         # Create Ollama model
-        ollama_model = OpenAIModel(
+        ollama_model = OpenAIChatModel(
             model_name=model,
             provider=OllamaProvider(base_url=f"{Config.OLLAMA_BASE_URL}/v1"),
         )

{haiku_rag-0.7.0 → haiku_rag-0.7.2}/tests/test_embedder.py

@@ -4,9 +4,11 @@ import pytest
 from haiku.rag.config import Config
 from haiku.rag.embeddings.ollama import Embedder as OllamaEmbedder
 from haiku.rag.embeddings.openai import Embedder as OpenAIEmbedder
+from haiku.rag.embeddings.vllm import Embedder as VLLMEmbedder
 
 OPENAI_AVAILABLE = bool(Config.OPENAI_API_KEY)
 VOYAGEAI_AVAILABLE = bool(Config.VOYAGE_API_KEY)
+VLLM_EMBEDDINGS_AVAILABLE = bool(Config.VLLM_EMBEDDINGS_BASE_URL)
 
 
 # Calculate cosine similarity
@@ -111,3 +113,35 @@ async def test_voyageai_embedder():
 
     except ImportError:
         pytest.skip("VoyageAI package not installed")
+
+
+@pytest.mark.asyncio
+@pytest.mark.skipif(
+    not VLLM_EMBEDDINGS_AVAILABLE, reason="vLLM embeddings server not configured"
+)
+async def test_vllm_embedder():
+    embedder = VLLMEmbedder("mixedbread-ai/mxbai-embed-large-v1", 512)
+    phrases = [
+        "I enjoy eating great food.",
+        "Python is my favorite programming language.",
+        "I love to travel and see new places.",
+    ]
+    embeddings = [np.array(await embedder.embed(phrase)) for phrase in phrases]
+
+    test_phrase = "I am going for a camping trip."
+    test_embedding = await embedder.embed(test_phrase)
+
+    sims = similarities(embeddings, test_embedding)
+    assert max(sims) == sims[2]
+
+    test_phrase = "When is dinner ready?"
+    test_embedding = await embedder.embed(test_phrase)
+
+    sims = similarities(embeddings, test_embedding)
+    assert max(sims) == sims[0]
+
+    test_phrase = "I work as a software developer."
+    test_embedding = await embedder.embed(test_phrase)
+
+    sims = similarities(embeddings, test_embedding)
+    assert max(sims) == sims[1]

{haiku_rag-0.7.0 → haiku_rag-0.7.2}/tests/test_qa.py

@@ -9,6 +9,7 @@ from .llm_judge import LLMJudge
 
 OPENAI_AVAILABLE = bool(Config.OPENAI_API_KEY)
 ANTHROPIC_AVAILABLE = bool(Config.ANTHROPIC_API_KEY)
+VLLM_QA_AVAILABLE = bool(Config.VLLM_QA_BASE_URL)
 
 
 @pytest.mark.asyncio
@@ -80,3 +81,26 @@ async def test_qa_anthropic(qa_corpus: Dataset, temp_db_path):
     assert is_equivalent, (
         f"Generated answer not equivalent to expected answer.\nQuestion: {question}\nGenerated: {answer}\nExpected: {expected_answer}"
     )
+
+
+@pytest.mark.asyncio
+@pytest.mark.skipif(not VLLM_QA_AVAILABLE, reason="vLLM QA server not configured")
+async def test_qa_vllm(qa_corpus: Dataset, temp_db_path):
+    """Test vLLM QA with LLM judge."""
+    client = HaikuRAG(temp_db_path)
+    qa = QuestionAnswerAgent(client, "vllm", "Qwen/Qwen3-4B")
+    llm_judge = LLMJudge()
+
+    doc = qa_corpus[1]
+    await client.create_document(
+        content=doc["document_extracted"], uri=doc["document_id"]
+    )
+
+    question = doc["question"]
+    expected_answer = doc["answer"]
+    answer = await qa.answer(question)
+    is_equivalent = await llm_judge.judge_answers(question, answer, expected_answer)
+
+    assert is_equivalent, (
+        f"Generated answer not equivalent to expected answer.\nQuestion: {question}\nGenerated: {answer}\nExpected: {expected_answer}"
+    )

{haiku_rag-0.7.0 → haiku_rag-0.7.2}/tests/test_reranker.py

@@ -2,9 +2,11 @@ import pytest
 
 from haiku.rag.config import Config
 from haiku.rag.reranking.base import RerankerBase
+from haiku.rag.reranking.vllm import VLLMReranker
 from haiku.rag.store.models.chunk import Chunk
 
 COHERE_AVAILABLE = bool(Config.COHERE_API_KEY)
+VLLM_RERANK_AVAILABLE = bool(Config.VLLM_RERANK_BASE_URL)
 
 chunks = [
     Chunk(content=content, document_id=str(i))
@@ -66,3 +68,22 @@ async def test_cohere_reranker():
 
     except ImportError:
         pytest.skip("Cohere package not installed")
+
+
+@pytest.mark.asyncio
+@pytest.mark.skipif(
+    not VLLM_RERANK_AVAILABLE, reason="vLLM rerank server not configured"
+)
+async def test_vllm_reranker():
+    try:
+        reranker = VLLMReranker("mixedbread-ai/mxbai-rerank-base-v2")
+
+        reranked = await reranker.rerank(
+            "Who wrote 'To Kill a Mockingbird'?", chunks, top_n=2
+        )
+        assert [chunk.document_id for chunk, score in reranked] == ["0", "2"]
+        assert all(isinstance(score, float) for chunk, score in reranked)
+
+    except Exception:
+        # Skip test if vLLM rerank server is not available
+        pytest.skip("vLLM rerank server not available")

{haiku_rag-0.7.0 → haiku_rag-0.7.2}/uv.lock

@@ -951,7 +951,7 @@ wheels = [
 
 [[package]]
 name = "haiku-rag"
-version = "0.7.0"
+version = "0.7.2"
 source = { editable = "." }
 dependencies = [
     { name = "docling" },

haiku_rag-0.7.0/docs/installation.md

@@ -1,34 +0,0 @@
-# Installation
-
-## Basic Installation
-
-```bash
-uv pip install haiku.rag
-```
-
-This includes support for:
-- **Ollama** (default embedding provider using `mxbai-embed-large`)
-- **OpenAI** (GPT models for QA and embeddings)
-- **Anthropic** (Claude models for QA)
-- **Cohere** (reranking models)
-
-## Provider-Specific Installation
-
-For additional embedding providers, install with extras:
-
-### VoyageAI
-
-```bash
-uv pip install haiku.rag[voyageai]
-```
-
-### MixedBread AI Reranking
-
-```bash
-uv pip install haiku.rag[mxbai]
-```
-
-## Requirements
-
-- Python 3.10+
-- Ollama (for default embeddings)