haiku.rag 0.1.0__tar.gz → 0.3.0__tar.gz

{haiku_rag-0.1.0 → haiku_rag-0.3.0}/.pre-commit-config.yaml

@@ -20,3 +20,13 @@ repos:
     rev: v1.1.399
     hooks:
       - id: pyright
+
+  - repo: https://github.com/RodrigoGonzalez/check-mkdocs
+    rev: v1.2.0
+    hooks:
+      - id: check-mkdocs
+        name: check-mkdocs
+        args: ["--config", "mkdocs.yml"] # Optional, mkdocs.yml is the default
+        # If you have additional plugins or libraries that are not included in
+        # check-mkdocs, add them here
+        additional_dependencies: ["mkdocs-material"]

haiku_rag-0.3.0/BENCHMARKS.md

@@ -0,0 +1,13 @@
+# `haiku.rag` benchmarks
+
+We use [repliqa](https://huggingface.co/datasets/ServiceNow/repliqa) for the evaluation of `haiku.rag`
+
+* Recall
+
+We load the `News Stories` from `repliqa_3` which is 1035 documents, using `tests/generate_benchmark_db.py`, using the `mxbai-embed-large` Ollama embeddings.
+
+Subsequently, we run a search over the `question` for each row of the dataset and check whether we match the document that answers the question. The recall obtained is ~0.75 for matching in the top result, raising to ~0.75 for the top 3 results.
+
+* Question/Answer evaluation
+
+We use the  `News Stories` from `repliqa_3` using the `mxbai-embed-large` Ollama embeddings, with a QA agent also using Ollama with the `qwen3` model (8b). For each story we ask the `question` and use an LLM judge (also `qwen3`) to evaluate whether the answer is correct or not. Thus we obtain accuracy of ~0.54.

haiku_rag-0.3.0/PKG-INFO

@@ -0,0 +1,112 @@
+Metadata-Version: 2.4
+Name: haiku.rag
+Version: 0.3.0
+Summary: Retrieval Augmented Generation (RAG) with SQLite
+Author-email: Yiorgis Gozadinos <ggozadinos@gmail.com>
+License: MIT
+License-File: LICENSE
+Keywords: RAG,mcp,ml,sqlite,sqlite-vec
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: MacOS
+Classifier: Operating System :: Microsoft :: Windows :: Windows 10
+Classifier: Operating System :: Microsoft :: Windows :: Windows 11
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: fastmcp>=2.8.1
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: markitdown[audio-transcription,docx,pdf,pptx,xlsx]>=0.1.2
+Requires-Dist: ollama>=0.5.1
+Requires-Dist: pydantic>=2.11.7
+Requires-Dist: python-dotenv>=1.1.0
+Requires-Dist: rich>=14.0.0
+Requires-Dist: sqlite-vec>=0.1.6
+Requires-Dist: tiktoken>=0.9.0
+Requires-Dist: typer>=0.16.0
+Requires-Dist: watchfiles>=1.1.0
+Provides-Extra: openai
+Requires-Dist: openai>=1.0.0; extra == 'openai'
+Provides-Extra: voyageai
+Requires-Dist: voyageai>=0.3.2; extra == 'voyageai'
+Description-Content-Type: text/markdown
+
+# Haiku SQLite RAG
+
+Retrieval-Augmented Generation (RAG) library on SQLite.
+
+`haiku.rag` is a Retrieval-Augmented Generation (RAG) library built to work on SQLite alone without the need for external vector databases. It uses [sqlite-vec](https://github.com/asg017/sqlite-vec) for storing the embeddings and performs semantic (vector) search as well as full-text search combined through Reciprocal Rank Fusion. Both open-source (Ollama) as well as commercial (OpenAI, VoyageAI) embedding providers are supported.
+
+## Features
+
+- **Local SQLite**: No external servers required
+- **Multiple embedding providers**: Ollama, VoyageAI, OpenAI
+- **Hybrid search**: Vector + full-text search with Reciprocal Rank Fusion
+- **Question answering**: Built-in QA agents on your documents
+- **File monitoring**: Auto-index files when run as server
+- **40+ file formats**: PDF, DOCX, HTML, Markdown, audio, URLs
+- **MCP server**: Expose as tools for AI assistants
+- **CLI & Python API**: Use from command line or Python
+
+## Quick Start
+
+```bash
+# Install
+uv pip install haiku.rag
+
+# Add documents
+haiku-rag add "Your content here"
+haiku-rag add-src document.pdf
+
+# Search
+haiku-rag search "query"
+
+# Ask questions
+haiku-rag ask "Who is the author of haiku.rag?"
+
+# Start server with file monitoring
+export MONITOR_DIRECTORIES="/path/to/docs"
+haiku-rag serve
+```
+
+## Python Usage
+
+```python
+from haiku.rag.client import HaikuRAG
+
+async with HaikuRAG("database.db") as client:
+    # Add document
+    doc = await client.create_document("Your content")
+
+    # Search
+    results = await client.search("query")
+    for chunk, score in results:
+        print(f"{score:.3f}: {chunk.content}")
+
+    # Ask questions
+    answer = await client.ask("Who is the author of haiku.rag?")
+    print(answer)
+```
+
+## MCP Server
+
+Use with AI assistants like Claude Desktop:
+
+```bash
+haiku-rag serve --stdio
+```
+
+Provides tools for document management and search directly in your AI assistant.
+
+## Documentation
+
+Full documentation at: https://ggozad.github.io/haiku.rag/
+
+- [Installation](https://ggozad.github.io/haiku.rag/installation/) - Provider setup
+- [Configuration](https://ggozad.github.io/haiku.rag/configuration/) - Environment variables
+- [CLI](https://ggozad.github.io/haiku.rag/cli/) - Command reference
+- [Python API](https://ggozad.github.io/haiku.rag/python/) - Complete API docs

haiku_rag-0.3.0/README.md

@@ -0,0 +1,75 @@
+# Haiku SQLite RAG
+
+Retrieval-Augmented Generation (RAG) library on SQLite.
+
+`haiku.rag` is a Retrieval-Augmented Generation (RAG) library built to work on SQLite alone without the need for external vector databases. It uses [sqlite-vec](https://github.com/asg017/sqlite-vec) for storing the embeddings and performs semantic (vector) search as well as full-text search combined through Reciprocal Rank Fusion. Both open-source (Ollama) as well as commercial (OpenAI, VoyageAI) embedding providers are supported.
+
+## Features
+
+- **Local SQLite**: No external servers required
+- **Multiple embedding providers**: Ollama, VoyageAI, OpenAI
+- **Hybrid search**: Vector + full-text search with Reciprocal Rank Fusion
+- **Question answering**: Built-in QA agents on your documents
+- **File monitoring**: Auto-index files when run as server
+- **40+ file formats**: PDF, DOCX, HTML, Markdown, audio, URLs
+- **MCP server**: Expose as tools for AI assistants
+- **CLI & Python API**: Use from command line or Python
+
+## Quick Start
+
+```bash
+# Install
+uv pip install haiku.rag
+
+# Add documents
+haiku-rag add "Your content here"
+haiku-rag add-src document.pdf
+
+# Search
+haiku-rag search "query"
+
+# Ask questions
+haiku-rag ask "Who is the author of haiku.rag?"
+
+# Start server with file monitoring
+export MONITOR_DIRECTORIES="/path/to/docs"
+haiku-rag serve
+```
+
+## Python Usage
+
+```python
+from haiku.rag.client import HaikuRAG
+
+async with HaikuRAG("database.db") as client:
+    # Add document
+    doc = await client.create_document("Your content")
+
+    # Search
+    results = await client.search("query")
+    for chunk, score in results:
+        print(f"{score:.3f}: {chunk.content}")
+
+    # Ask questions
+    answer = await client.ask("Who is the author of haiku.rag?")
+    print(answer)
+```
+
+## MCP Server
+
+Use with AI assistants like Claude Desktop:
+
+```bash
+haiku-rag serve --stdio
+```
+
+Provides tools for document management and search directly in your AI assistant.
+
+## Documentation
+
+Full documentation at: https://ggozad.github.io/haiku.rag/
+
+- [Installation](https://ggozad.github.io/haiku.rag/installation/) - Provider setup
+- [Configuration](https://ggozad.github.io/haiku.rag/configuration/) - Environment variables
+- [CLI](https://ggozad.github.io/haiku.rag/cli/) - Command reference
+- [Python API](https://ggozad.github.io/haiku.rag/python/) - Complete API docs

haiku_rag-0.3.0/docs/cli.md

@@ -0,0 +1,83 @@
+# Command Line Interface
+
+The `haiku-rag` CLI provides complete document management functionality.
+
+## Document Management
+
+### List Documents
+
+```bash
+haiku-rag list
+```
+
+### Add Documents
+
+From text:
+```bash
+haiku-rag add "Your document content here"
+```
+
+From file or URL:
+```bash
+haiku-rag add-src /path/to/document.pdf
+haiku-rag add-src https://example.com/article.html
+```
+
+### Get Document
+
+```bash
+haiku-rag get 1
+```
+
+### Delete Document
+
+```bash
+haiku-rag delete 1
+```
+
+## Search
+
+Basic search:
+```bash
+haiku-rag search "machine learning"
+```
+
+With options:
+```bash
+haiku-rag search "python programming" --limit 10 --k 100
+```
+
+## Question Answering
+
+Ask questions about your documents:
+```bash
+haiku-rag ask "Who is the author of haiku.rag?"
+```
+
+The QA agent will search your documents for relevant information and provide a comprehensive answer.
+
+## Server
+
+Start the MCP server:
+```bash
+# HTTP transport (default)
+haiku-rag serve
+
+# stdio transport
+haiku-rag serve --stdio
+
+# SSE transport
+haiku-rag serve --sse
+```
+
+## Options
+
+All commands support:
+- `--db` - Specify custom database path
+- `-h` - Show help for specific command
+
+Example:
+```bash
+haiku-rag list --db /path/to/custom.db
+haiku-rag add -h
+```

haiku_rag-0.3.0/docs/configuration.md

@@ -0,0 +1,104 @@
+# Configuration
+
+Configuration is done through the use of environment variables.
+
+## File Monitoring
+
+Set directories to monitor for automatic indexing:
+
+```bash
+# Monitor single directory
+MONITOR_DIRECTORIES="/path/to/documents"
+
+# Monitor multiple directories
+MONITOR_DIRECTORIES="/path/to/documents,/another_path/to/documents"
+```
+
+## Embedding Providers
+
+If you use Ollama, you can use any pulled model that supports embeddings.
+
+### Ollama (Default)
+
+```bash
+EMBEDDINGS_PROVIDER="ollama"
+EMBEDDINGS_MODEL="mxbai-embed-large"
+EMBEDDINGS_VECTOR_DIM=1024
+```
+
+### VoyageAI
+If you want to use VoyageAI embeddings you will need to install `haiku.rag` with the VoyageAI extras,
+
+```bash
+uv pip install haiku.rag --extra voyageai
+```
+
+```bash
+EMBEDDINGS_PROVIDER="voyageai"
+EMBEDDINGS_MODEL="voyage-3.5"
+EMBEDDINGS_VECTOR_DIM=1024
+VOYAGE_API_KEY="your-api-key"
+```
+
+### OpenAI
+If you want to use OpenAI embeddings you will need to install `haiku.rag` with the VoyageAI extras,
+
+```bash
+uv pip install haiku.rag --extra openai
+```
+
+and set environment variables.
+
+```bash
+EMBEDDINGS_PROVIDER="openai"
+EMBEDDINGS_MODEL="text-embedding-3-small"  # or text-embedding-3-large
+EMBEDDINGS_VECTOR_DIM=1536
+OPENAI_API_KEY="your-api-key"
+```
+
+## Question Answering Providers
+
+Configure which LLM provider to use for question answering.
+
+### Ollama (Default)
+
+```bash
+QA_PROVIDER="ollama"
+QA_MODEL="qwen3"
+OLLAMA_BASE_URL="http://localhost:11434"
+```
+
+### OpenAI
+
+For OpenAI QA, you need to install haiku.rag with OpenAI extras:
+
+```bash
+uv pip install haiku.rag --extra openai
+```
+
+Then configure:
+
+```bash
+QA_PROVIDER="openai"
+QA_MODEL="gpt-4o-mini"  # or gpt-4, gpt-3.5-turbo, etc.
+OPENAI_API_KEY="your-api-key"
+```
+
+## Other Settings
+
+### Database and Storage
+
+```bash
+# Default data directory (where SQLite database is stored)
+DEFAULT_DATA_DIR="/path/to/data"
+```
+
+### Document Processing
+
+```bash
+# Chunk size for document processing
+CHUNK_SIZE=256
+
+# Chunk overlap for better context
+CHUNK_OVERLAP=32
+```

haiku_rag-0.3.0/docs/index.md

@@ -0,0 +1,59 @@
+# haiku.rag
+
+`haiku.rag` is a Retrieval-Augmented Generation (RAG) library built to work on SQLite alone without the need for external vector databases. It uses [sqlite-vec](https://github.com/asg017/sqlite-vec) for storing the embeddings and performs semantic (vector) search as well as full-text search combined through Reciprocal Rank Fusion. Both open-source (Ollama) as well as commercial (OpenAI, VoyageAI) embedding providers are supported.
+
+
+## Features
+
+- **Local SQLite**: No need to run additional servers
+- **Support for various embedding providers**: Ollama, VoyageAI, OpenAI or add your own
+- **Hybrid Search**: Vector search using `sqlite-vec` combined with full-text search `FTS5`, using Reciprocal Rank Fusion
+- **Question Answering**: Built-in QA agents using Ollama or OpenAI.
+- **File monitoring**: Automatically index files when run as a server
+- **Extended file format support**: Parse 40+ file formats including PDF, DOCX, HTML, Markdown, audio and more. Or add a URL!
+- **MCP server**: Exposes functionality as MCP tools
+- **CLI commands**: Access all functionality from your terminal
+- **Python client**: Call `haiku.rag` from your own python applications
+
+## Quick Start
+
+Install haiku.rag:
+```bash
+uv pip install haiku.rag
+```
+
+Use from Python:
+```python
+from haiku.rag.client import HaikuRAG
+
+async with HaikuRAG("database.db") as client:
+    # Add a document
+    doc = await client.create_document("Your content here")
+
+    # Search documents
+    results = await client.search("query")
+
+    # Ask questions
+    answer = await client.ask("Who is the author of haiku.rag?")
+```
+
+Or use the CLI:
+```bash
+haiku-rag add "Your document content"
+haiku-rag search "query"
+haiku-rag ask "Who is the author of haiku.rag?"
+```
+
+## Documentation
+
+- [Installation](installation.md) - Install haiku.rag with different providers
+- [Configuration](configuration.md) - Environment variables and settings
+- [CLI](cli.md) - Command line interface usage
+- [Question Answering](qa.md) - QA agents and natural language queries
+- [Server](server.md) - File monitoring and server mode
+- [MCP](mcp.md) - Model Context Protocol integration
+- [Python](python.md) - Python API reference
+
+## License
+
+This project is licensed under the [MIT License](https://raw.githubusercontent.com/ggozad/haiku.rag/main/LICENSE).

haiku_rag-0.3.0/docs/installation.md

@@ -0,0 +1,31 @@
+# Installation
+
+## Basic Installation
+
+```bash
+uv pip install haiku.rag
+```
+
+By default, Ollama (with the `mxbai-embed-large` model) is used for embeddings.
+
+## Provider-Specific Installation
+
+For other embedding providers, install with extras:
+
+### VoyageAI
+
+```bash
+uv pip install haiku.rag --extra voyageai
+```
+
+### OpenAI
+
+```bash
+uv pip install haiku.rag --extra openai
+```
+
+## Requirements
+
+- Python 3.10+
+- SQLite 3.38+
+- Ollama (for default embeddings)

haiku_rag-0.3.0/docs/mcp.md

@@ -0,0 +1,33 @@
+# Model Context Protocol (MCP)
+
+The MCP server exposes `haiku.rag` as MCP tools for compatible MCP clients.
+
+## Available Tools
+
+### Document Management
+
+- `add_document_from_file` - Add documents from local file paths
+- `add_document_from_url` - Add documents from URLs
+- `add_document_from_text` - Add documents from raw text content
+- `get_document` - Retrieve specific documents by ID
+- `list_documents` - List all documents with pagination
+- `delete_document` - Delete documents by ID
+
+### Search
+
+- `search_documents` - Search documents using hybrid search (vector + full-text)
+
+## Starting MCP Server
+
+The MCP server starts automatically with the serve command and supports `Streamable HTTP`, `stdio` and `SSE` transports:
+
+```bash
+# Default HTTP transport
+haiku-rag serve
+
+# stdio transport (for Claude Desktop)
+haiku-rag serve --stdio
+
+# SSE transport
+haiku-rag serve --sse
+```

haiku_rag-0.3.0/docs/python.md

@@ -0,0 +1,109 @@
+# Python API
+
+Use `haiku.rag` directly in your Python applications.
+
+## Basic Usage
+
+```python
+from pathlib import Path
+from haiku.rag.client import HaikuRAG
+
+# Use as async context manager (recommended)
+async with HaikuRAG("path/to/database.db") as client:
+    # Your code here
+    pass
+```
+
+## Document Management
+
+### Creating Documents
+
+From text:
+```python
+doc = await client.create_document(
+    content="Your document content here",
+    uri="doc://example",
+    metadata={"source": "manual", "topic": "example"}
+)
+```
+
+From file:
+```python
+doc = await client.create_document_from_source("path/to/document.pdf")
+```
+
+From URL:
+```python
+doc = await client.create_document_from_source("https://example.com/article.html")
+```
+
+### Retrieving Documents
+
+By ID:
+```python
+doc = await client.get_document_by_id(1)
+```
+
+By URI:
+```python
+doc = await client.get_document_by_uri("file:///path/to/document.pdf")
+```
+
+List all documents:
+```python
+docs = await client.list_documents(limit=10, offset=0)
+```
+
+### Updating Documents
+
+```python
+doc.content = "Updated content"
+await client.update_document(doc)
+```
+
+### Deleting Documents
+
+```python
+await client.delete_document(doc.id)
+```
+
+## Searching Documents
+
+Basic search:
+```python
+results = await client.search("machine learning algorithms", limit=5)
+for chunk, score in results:
+    print(f"Score: {score:.3f}")
+    print(f"Content: {chunk.content}")
+    print(f"Document ID: {chunk.document_id}")
+```
+
+With options:
+```python
+results = await client.search(
+    query="machine learning",
+    limit=5,  # Maximum results to return
+    k=60      # RRF parameter for reciprocal rank fusion
+)
+
+# Process results
+for chunk, relevance_score in results:
+    print(f"Relevance: {relevance_score:.3f}")
+    print(f"Content: {chunk.content}")
+    print(f"From document: {chunk.document_id}")
+    print(f"Document URI: {chunk.document_uri}")
+    print(f"Document metadata: {chunk.document_meta}")
+```
+
+## Question Answering
+
+Ask questions about your documents:
+
+```python
+answer = await client.ask("Who is the author of haiku.rag?")
+print(answer)
+```
+
+The QA agent will search your documents for relevant information and use the configured LLM to generate a comprehensive answer.
+
+The QA provider and model can be configured via environment variables (see [Configuration](configuration.md)).

haiku_rag-0.3.0/docs/server.md

@@ -0,0 +1,41 @@
+# Server Mode
+
+The server provides automatic file monitoring and MCP functionality.
+
+## Starting the Server
+
+```bash
+haiku-rag serve
+```
+
+Transport options:
+- `--http` (default) - Streamable HTTP transport
+- `--stdio` - Standard input/output transport
+- `--sse` - Server-sent events transport
+
+## File Monitoring
+
+Set `MONITOR_DIRECTORIES` environment variable to enable automatic file monitoring:
+
+```bash
+export MONITOR_DIRECTORIES="/path/to/documents"
+haiku-rag serve
+```
+
+### Monitoring Features
+
+- **Startup**: Scans all monitored directories and adds new files
+- **File Added/Modified**: Automatically parses and updates documents
+- **File Deleted**: Removes corresponding documents from database
+
+### Supported Formats
+
+The server can parse 40+ file formats including:
+- PDF documents
+- Microsoft Office (DOCX, XLSX, PPTX)
+- HTML and Markdown
+- Plain text files
+- Audio files
+- And more...
+
+URLs are also supported for web content.