haiku.rag 0.7.5__tar.gz → 0.19.4__tar.gz

haiku_rag-0.19.4/.dockerignore

@@ -0,0 +1,66 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# Virtual environments (uv best practice)
+.venv/
+venv/
+env/
+
+# Node.js
+node_modules/
+.next/
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+
+# Data
+*.lancedb/
+data/
+
+# Docs
+mkdocs.yml
+docs/
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Git
+.git/
+.gitignore
+
+# Development
+tests/
+.pytest_cache/
+.coverage
+htmlcov/
+evaluations/evaluations/data/
+server.json
+# Examples
+examples/

haiku_rag-0.19.4/.gitignore

@@ -0,0 +1,31 @@
+# Python-generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+
+# Virtual environments
+.venv
+
+# tests
+.coverage*
+evaluations/evaluations/data/
+tests/data/
+.pytest_cache/
+.ruff_cache/
+
+# environment variables and config files
+.env
+haiku.rag.yaml
+TODO.md
+PLAN.md
+DEVNOTES.md
+
+# mcp registry
+.mcpregistry_github_token
+.mcpregistry_registry_token
+
+# MkDocs site directory when doing local docs builds
+site/

{haiku_rag-0.7.5 → haiku_rag-0.19.4}/.pre-commit-config.yaml

@@ -9,7 +9,7 @@ repos:
       - id: debug-statements
   - repo: https://github.com/astral-sh/ruff-pre-commit
     # Ruff version.
-    rev: v0.11.4
+    rev: v0.14.3
     hooks:
       # Run the linter.
       - id: ruff
@@ -17,16 +17,6 @@ repos:
       - id: ruff-format
 
   - repo: https://github.com/RobertCraigie/pyright-python
-    rev: v1.1.399
+    rev: v1.1.407
     hooks:
       - id: pyright
-
-  # - repo: https://github.com/RodrigoGonzalez/check-mkdocs
-  #   rev: v1.2.0
-  #   hooks:
-  #     - id: check-mkdocs
-  #       name: check-mkdocs
-  #       args: ["--config", "mkdocs.yml"] # Optional, mkdocs.yml is the default
-  #       # If you have additional plugins or libraries that are not included in
-  #       # check-mkdocs, add them here
-  #       additional_dependencies: ["mkdocs-material"]

haiku_rag-0.19.4/.python-version

@@ -0,0 +1 @@
+3.13

haiku_rag-0.19.4/CHANGELOG.md

@@ -0,0 +1,377 @@
+# Changelog
+## [Unreleased]
+
+## [0.19.4] - 2025-11-28
+
+### Added
+
+- **Rebuild Modes**: New options for `rebuild` command to control what gets rebuilt
+  - `--embed-only`: Only regenerate embeddings, keeping existing chunks (fastest option when changing embedding model)
+  - `--rechunk`: Re-chunk from existing document content without accessing source files
+  - Default (no flag): Full rebuild with source file re-conversion
+  - Python API: `rebuild_database(mode=RebuildMode.EMBED_ONLY | RECHUNK | FULL)`
+
+## [0.19.3] - 2025-11-27
+
+### Changed
+
+- **Async Chunker**: `DoclingServeChunker` now uses `httpx.AsyncClient` instead of sync `requests`
+
+### Fixed
+
+- **OCR Options**: Fixed `DoclingLocalConverter` using base `OcrOptions` class which docling's OCR factory doesn't recognize. Now uses `OcrAutoOptions` for automatic OCR engine selection.
+- **Dependencies**: Added `opencv-python-headless` to the `docling` optional dependency for table structure detection.
+
+## [0.19.2] - 2025-11-27
+
+### Changed
+
+- **Async Converters**: Made document converters fully async
+  - `BaseConverter.convert_file()` and `convert_text()` are now async methods
+  - `DoclingLocalConverter` wraps blocking Docling operations with `asyncio.to_thread()`
+  - `DoclingServeConverter` now uses `httpx.AsyncClient` instead of sync `requests`
+- **Async Model Prefetch**: `prefetch_models()` is now async
+  - Uses `httpx.AsyncClient` for Ollama model pulls
+  - Wraps blocking Docling and HuggingFace downloads with `asyncio.to_thread()`
+
+## [0.19.1] - 2025-11-26
+
+### Added
+
+- **LM Studio Provider**: Added support for LM Studio as a provider for embeddings and QA/research models
+  - Configure with `provider: lm_studio` in embeddings, QA, or research model settings
+  - Supports thinking control for reasoning models (gpt-oss, etc.)
+  - Default base URL: `http://localhost:1234`
+
+### Fixed
+
+- **Configuration**: Fixed `init-config` command generating invalid configuration files (#165)
+  - Refactored `generate_default_config()` to use Pydantic model serialization instead of manual dict construction
+  - Updated `qa`, `research`, and `reranking` sections to use new `ModelConfig` structure
+
+## [0.19.0] - 2025-11-25
+
+### Added
+
+- **Model Customization**: Added support for per-model configuration settings
+  - New `enable_thinking` parameter to control reasoning behavior (true/false/None)
+  - Support for `temperature` and `max_tokens` settings on QA and research models
+  - All settings apply to any provider that supports them
+- **Database Inspector**: New `inspect` CLI command launches interactive TUI for browsing documents and chunks & searching
+- **Evaluations**: Added `evaluations` CLI script for running benchmarks (replaces `python -m evaluations.benchmark`)
+- **Evaluations**: Added `--db` option to override evaluation database path
+  - Default database location moved to haiku.rag data directory:
+    - macOS: `~/Library/Application Support/haiku.rag/evaluations/dbs/`
+    - Linux: `~/.local/share/haiku.rag/evaluations/dbs/`
+    - Windows: `C:/Users/<USER>/AppData/Roaming/haiku.rag/evaluations/dbs/`
+  - Previously stored in `evaluations/data/` within the repository
+- **Evaluations**: Added comprehensive experiment metadata tracking for better reproducibility
+  - Records dataset name, test case count, and all model configurations
+  - Tracks embedder settings: provider, model, and vector dimensions
+  - Tracks QA model: provider and model name
+  - Tracks judge model: provider and model name for LLM evaluation
+  - Tracks processing parameters: `chunk_size` and `context_chunk_radius`
+  - Tracks retrieval configuration: `retrieval_limit` for number of chunks retrieved
+  - Tracks reranking configuration: `rerank_provider` and `rerank_model`
+  - Enables comparison of evaluation runs with different configurations in Logfire
+- **Evaluations**: Refactored retrieval evaluation to use pydantic-ai experiment framework
+  - New `evaluators` module with `MRREvaluator` (Mean Reciprocal Rank) and `MAPEvaluator` (Mean Average Precision)
+  - Retrieval benchmarks now use `Dataset.evaluate()` with full Logfire experiment tracking
+  - Dataset specifications now declare their retrieval evaluator (MRR for RepliQA, MAP for Wix)
+  - Replaced Recall@K and Success@K with industry-standard MRR and MAP metrics
+  - Unified evaluation framework for both retrieval and QA benchmarks
+- **AG-UI Events**: Enhanced ActivitySnapshot events with richer structured data
+  - Added `stepName` field to identify which graph node emitted each activity
+  - Added structured fields to activity content while preserving backward-compatible `message` field:
+    - **Planning**: `sub_questions` - list of sub-question strings
+    - **Searching**: `query` - the search query, `confidence` - answer confidence (on success), `error` - error message (on failure)
+    - **Analyzing** (research): `insights` - list of insight objects, `gaps` - list of gap objects, `resolved_gaps` - list of resolved gap strings
+    - **Evaluating** (research): `confidence` - confidence score, `is_sufficient` - sufficiency flag
+    - **Evaluating** (deep QA): `is_sufficient` - sufficiency flag, `iterations` - iteration count
+
+### Changed
+
+- **Evaluations**: Renamed `--qa-limit` CLI parameter to `--limit`, now applies to both retrieval and QA benchmarks
+- **Evaluations**: Retrieval evaluator selection moved from runtime logic to dataset configuration
+
+## [0.18.0] - 2025-11-21
+
+### Added
+
+- **Manual Vector Indexing**: New `create-index` CLI command for explicit vector index creation
+  - Creates IVF_PQ indexes
+  - Requires minimum 256 chunks (LanceDB training data requirement)
+  - New `search.vector_index_metric` config option: `cosine` (default), `l2`, or `dot`
+  - New `search.vector_refine_factor` config option (default: 30) for accuracy/speed tradeoff
+  - Indexes not created automatically during ingestion to avoid performance degradation
+  - Manual rebuilding required after adding significant new data
+- **Enhanced Info Command**: `haiku-rag info` now shows storage sizes and vector index statistics
+  - Displays storage size for documents and chunks tables in human-readable format
+  - Shows vector index status (exists/not created)
+  - Shows indexed and unindexed chunk counts for monitoring index staleness
+
+### Changed
+
+- **BREAKING: Default Embedding Model**: Changed default embedding model from `qwen3-embedding` to `qwen3-embedding:4b` with vector dimension 2560 (previously 4096)
+  - New installations will use the smaller, more efficient 4B parameter model by default
+  - **Action required**: Existing databases created with the old default will be incompatible. Users must either:
+    - Explicitly set `embeddings.model: "qwen3-embedding"` and `embeddings.vector_dim: 4096` in their config to maintain compatibility with existing databases
+    - Or run `haiku-rag rebuild` to re-embed all documents with the new default
+  - This change provides better performance for most use cases while reducing resource requirements
+- **Evaluations**: Improved evaluation dataset naming and simplified evaluator configuration
+  - `EvalDataset` now accepts dataset name for better organization in Logfire
+  - Added `--name` CLI parameter to override evaluation run names
+  - Removed `IsInstance` evaluator, using only `LLMJudge` for QA evaluation
+- **Search Accuracy**: Applied `refine_factor` to vector and hybrid searches for improved accuracy
+  - Retrieves `refine_factor * limit` candidates and re-ranks in memory
+  - Higher values increase accuracy but slow down queries
+
+### Fixed
+
+- **AG-UI Activity Events**: Activity events now correctly use structured dict content instead of strings
+- **Graph Configuration**: Graph builder functions now properly accept and use non-global config (#149)
+  - `build_research_graph()` and `build_deep_qa_graph()` now pass config to all agents and model creation
+  - `get_model()` utility function accepts `config` parameter (defaults to global Config)
+  - Allows creating multiple graphs with different configurations in the same application
+
+
+## [0.17.2] - 2025-11-19
+
+### Added
+
+- **Document Update API**: New `update_document_fields()` method for partial document updates
+  - Update individual fields (content, metadata, title, chunks) without fetching full document
+  - Support for custom chunks or auto-generation from content
+
+### Changed
+
+- **Chunk Creation**: `ChunkRepository.create()` now accepts both single chunks and lists for batch insertion
+  - Batch insertion reduces LanceDB version creation when adding multiple chunks with custom chunks
+  - Batch embedding generation for improved performance with multiple chunks
+- Updated core dependencies
+
+## [0.17.1] - 2025-11-18
+
+### Added
+
+- **Conversion Options**: Fine-grained control over document conversion for both local and remote converters
+  - New `conversion_options` config section in `ProcessingConfig`
+  - OCR settings: `do_ocr`, `force_ocr`, `ocr_lang` for controlling OCR behavior
+  - Table extraction: `do_table_structure`, `table_mode` (fast/accurate), `table_cell_matching`
+  - Image settings: `images_scale` to control image resolution
+  - Options work identically with both `docling-local` and `docling-serve` converters
+
+### Changed
+
+- Increase reranking candidate retrieval multiplier from 3x to 10x for improved result quality
+- **Docker Images**: Main `haiku.rag` image no longer automatically built and published
+- **Conversion Options**: Removed the legacy `pdf_backend` setting; docling now chooses the optimal backend automatically
+
+## [0.17.0] - 2025-11-17
+
+### Added
+
+- **Remote Processing**: Support for docling-serve as remote document processing and chunking service
+  - New `converter` config option: `docling-local` (default) or `docling-serve`
+  - New `chunker` config option: `docling-local` (default) or `docling-serve`
+  - New `providers.docling_serve` config section with `base_url`, `api_key`, and `timeout`
+  - Comprehensive error handling for connection, timeout, and authentication issues
+- **Chunking Strategies**: Support for both hybrid and hierarchical chunking
+  - New `chunker_type` config option: `hybrid` (default) or `hierarchical`
+  - Hybrid chunking: Structure-aware splitting that respects document boundaries
+  - Hierarchical chunking: Preserves document hierarchy for nested documents
+- **Table Serialization Control**: Configurable table representation in chunks
+  - New `chunking_use_markdown_tables` config option (default: `false`)
+  - `false`: Tables serialized as narrative text ("Value A, Column 2 = Value B")
+  - `true`: Tables preserved as markdown format with structure
+- **Chunking Configuration**: Additional chunking control options
+  - New `chunking_merge_peers` config option (default: `true`) to merge undersized successive chunks
+- **Docker Images**: Two Docker images for different deployment scenarios
+  - `haiku.rag`: Full image with all dependencies for self-contained deployments
+  - `haiku.rag-slim`: Minimal image designed for use with external docling-serve
+  - Multi-platform support (linux/amd64, linux/arm64)
+  - Docker Compose examples with docling-serve integration
+  - Automated CI/CD workflows for both images
+  - Build script (`scripts/build-docker-images.sh`) for local multi-platform builds
+
+### Changed
+
+- **BREAKING: Chunking Tokenizer**: Switched from tiktoken to HuggingFace tokenizers for consistency with docling-serve
+  - Default tokenizer changed from tiktoken "gpt-4o" to "Qwen/Qwen3-Embedding-0.6B"
+  - New `chunking_tokenizer` config option in `ProcessingConfig` for customization
+  - `download-models` CLI command now also downloads the configured HuggingFace tokenizer
+- **Docker Examples**: Updated examples to demonstrate remote processing
+  - `examples/docker` now uses slim image with docling-serve
+  - `examples/ag-ui-research` backend uses slim image with docling-serve
+  - Configuration examples include remote processing setup
+
+## [0.16.1] - 2025-11-14
+
+### Changed
+
+- **Evaluations**: Refactored QA benchmark to run entire dataset as single evaluation for better Logfire experiment tracking
+- **Evaluations**: Added `.env` file loading support via `python-dotenv` dependency
+
+## [0.16.0] - 2025-11-13
+
+### Added
+
+- **AG-UI Protocol Support**: Full AG-UI (Agent-UI) protocol implementation for graph execution with event streaming
+  - New `AGUIEmitter` class for emitting AG-UI events from graphs
+  - Support for all AG-UI event types: lifecycle events (`RUN_STARTED`, `RUN_FINISHED`, `RUN_ERROR`), step events (`STEP_STARTED`, `STEP_FINISHED`), state updates (`STATE_SNAPSHOT`, `STATE_DELTA`), activity narration (`ACTIVITY_SNAPSHOT`), and text messages (`TEXT_MESSAGE_CHUNK`)
+  - `AGUIConsoleRenderer` for rendering AG-UI event streams to terminal with Rich formatting
+  - `stream_graph()` utility function for executing graphs with AG-UI event emission
+  - State diff computation for efficient state synchronization
+  - **Delta State Updates**: AG-UI emitter now supports incremental state updates via JSON Patch operations (`STATE_DELTA` events) to reduce bandwidth, configurable via `use_deltas` parameter (enabled by default)
+- **AG-UI Server**: Starlette-based HTTP server for serving graphs via AG-UI protocol
+  - Server-Sent Events (SSE) streaming endpoint at `/v1/agent/stream`
+  - Health check endpoint at `/health`
+  - Full CORS support configurable via `agui` config section
+  - `create_agui_server()` function for programmatic server creation
+- **Deep QA AG-UI Support**: Deep QA graph now fully supports AG-UI event streaming
+  - Integration with `AGUIEmitter` for progress tracking
+  - Step-by-step execution visibility via AG-UI events
+- **CLI AG-UI Flag**: New `--agui` flag for `serve` command to start AG-UI server
+- **Graph Module**: New unified `haiku.rag.graph` module containing all graph-related functionality
+- **Common Graph Nodes**: New factory functions (`create_plan_node`, `create_search_node`) in `haiku.rag.graph.common.nodes` for reusable graph components
+- **AG-UI Research Example**: New full-stack example (`examples/ag-ui-research`) demonstrating agent+graph architecture with CopilotKit frontend
+  - Pydantic AI agent with research tool that invokes the research graph
+  - Custom AG-UI streaming endpoint with anyio memory streams
+  - React/Next.js frontend with split-pane UI showing live research state
+  - Real-time progress tracking of questions, answers, insights, and gaps
+  - Docker Compose setup for easy local development
+
+### Changed
+
+- **Vacuum Retention**: Default `vacuum_retention_seconds` increased from 60 seconds to 86400 seconds (1 day) for better version retention in typical workflows
+- **BREAKING**: Major refactoring of graph-related code into unified `haiku.rag.graph` module structure:
+  - `haiku.rag.research` → `haiku.rag.graph.research`
+  - `haiku.rag.qa.deep` → `haiku.rag.graph.deep_qa`
+  - `haiku.rag.agui` → `haiku.rag.graph.agui`
+  - `haiku.rag.graph_common` → `haiku.rag.graph.common`
+- **BREAKING**: Research and Deep QA graphs now use AG-UI event protocol instead of direct console logging
+  - Removed `console` and `stream` parameters from graph dependencies
+  - All progress updates now emit through `AGUIEmitter`
+- **BREAKING**: `ResearchState` converted from dataclass to Pydantic `BaseModel` for JSON serialization and AG-UI compatibility
+- Research and Deep QA graphs now emit detailed execution events for better observability
+- CLI research command now uses AG-UI event rendering for `--verbose` output
+- Improved graph execution visibility with step-by-step progress tracking
+- Updated all documentation to reflect new import paths and AG-UI usage
+- Updated examples (ag-ui-research, a2a-server) to use new import paths
+
+### Fixed
+
+- **Document Creation**: Optimized `create_document` to skip unnecessary DoclingDocument conversion when chunks are pre-provided
+- **FileReader**: Error messages now include both original exception details and file path for easier debugging
+- **Database Auto-creation**: Read operations (search, list, get, ask, research) no longer auto-create empty databases. Write operations (add, add-src, delete, rebuild) still create the database as needed. This prevents the confusing scenario where a search query creates an empty database. Fixes issue #137.
+
+### Removed
+
+- **BREAKING**: Removed `disable_autocreate` config option - the behavior is now automatic based on operation type
+- **BREAKING**: Removed legacy `ResearchStream` and `ResearchStreamEvent` classes (replaced by AG-UI event protocol)
+
+## [0.15.0] - 2025-11-07
+
+### Added
+
+- **File Monitor**: Orphan deletion feature - automatically removes documents from database when source files are deleted (enabled via `monitor.delete_orphans` config option, default: false)
+
+### Changed
+
+- **Configuration**: All CLI commands now properly support `--config` parameter for specifying custom configuration files
+- Configuration loading consolidated across CLI, app, and client with consistent resolution order
+- `HaikuRAGApp` and MCP server now accept `config` parameter for programmatic configuration
+- Updated CLI documentation to clarify global vs per-command options
+- **BREAKING**: Standardized configuration filename to `haiku.rag.yaml` in user directories (was incorrectly using `config.yaml`). Users with existing `config.yaml` in their user directory will need to rename it to `haiku.rag.yaml`
+
+### Fixed
+
+- **File Monitor**: Fixed incorrect "Updated document" logging for unchanged files - monitor now properly skips files when MD5 hash hasn't changed
+
+### Removed
+
+- **BREAKING**: A2A (Agent-to-Agent) protocol support has been moved to a separate self-contained package in `examples/a2a-server/`. The A2A server is no longer part of the main haiku.rag package. Users who need A2A functionality can install and run it from the examples directory with `cd examples/a2a-server && uv sync`.
+- **BREAKING**: Removed deprecated `.env`-based configuration system. The `haiku-rag init-config --from-env` command and `load_config_from_env()` function have been removed. All configuration must now be done via YAML files. Environment variables for API keys (e.g., `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`) and service URLs (e.g., `OLLAMA_BASE_URL`) are still supported and can be set via `.env` files.
+
+## [0.14.1] - 2025-11-06
+
+### Added
+
+- Migrated research and deep QA agents to use Pydantic Graph beta API for better graph execution
+- Automatic semaphore-based concurrency control for parallel sub-question processing
+- `max_concurrency` parameter for controlling parallel execution in research and deep QA (default: 1)
+
+### Changed
+
+- **BREAKING**: Research and Deep QA graphs now use `pydantic_graph.beta` instead of the class-based graph implementation
+- Refactored graph common patterns into `graph_common` module
+- Sub-questions now process using `.map()` for true parallel execution
+- Improved graph structure with cleaner node definitions and flow control
+- Pinned critical dependencies: `docling-core`, `lancedb`, `docling`
+
+## [0.14.0] - 2024-11-05
+
+### Added
+
+- New `haiku.rag-slim` package with minimal dependencies for users who want to install only what they need
+- Evaluations package (`haiku.rag-evals`) for internal benchmarking and testing
+- Improved search filtering performance by using pandas DataFrames for joins instead of SQL WHERE IN clauses
+
+### Changed
+
+- **BREAKING**: Restructured project into UV workspace with three packages:
+  - `haiku.rag-slim` - Core package with minimal dependencies
+  - `haiku.rag` - Full package with all extras (recommended for most users)
+  - `haiku.rag-evals` - Internal benchmarking and evaluation tools
+- Migrated from `pydantic-ai` to `pydantic-ai-slim` with extras system
+- Docling is now an optional dependency (install with `haiku.rag-slim[docling]`)
+- Package metadata checks now use `haiku.rag-slim` (always present) instead of `haiku.rag`
+- Docker image optimized: removed evaluations package, reducing installed packages from 307 to 259
+- Improved vector search performance through optimized score normalization
+
+### Fixed
+
+- ImportError now properly raised when optional docling dependency is missing
+
+## [0.13.3] - 2024-11-04
+
+### Added
+
+- Support for Zero Entropy reranker
+- Filter parameter to `search()` for filtering documents before search
+- Filter parameter to CLI `search` command
+- Filter parameter to CLI `list` command for filtering document listings
+- Config option to pass custom configuration files to evaluation commands
+- Document filtering now respects configured include/exclude patterns when using `add-src` with directories
+- Max retries to insight_agent when producing structured output
+
+### Fixed
+
+- CLI now loads `.env` files at startup
+- Info command no longer attempts to use deprecated `.env` settings
+- Documentation typos
+
+## [0.13.2] - 2024-11-04
+
+### Added
+
+- Gitignore-style pattern filtering for file monitoring using pathspec
+- Include/exclude pattern documentation for FileMonitor
+
+### Changed
+
+- Moved monitor configuration to its own section in config
+- Improved configuration documentation
+- Updated dependencies
+
+## [0.13.1] - 2024-11-03
+
+### Added
+
+- Initial version tracking
+
+[Unreleased]: https://github.com/ggozad/haiku.rag/compare/0.14.0...HEAD
+[0.14.0]: https://github.com/ggozad/haiku.rag/compare/0.13.3...0.14.0
+[0.13.3]: https://github.com/ggozad/haiku.rag/compare/0.13.2...0.13.3
+[0.13.2]: https://github.com/ggozad/haiku.rag/compare/0.13.1...0.13.2
+[0.13.1]: https://github.com/ggozad/haiku.rag/releases/tag/0.13.1