iris-vector-rag 0.2.3__tar.gz → 0.2.4__tar.gz

iris_vector_rag-0.2.4/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 InterSystems Community
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

{iris_vector_rag-0.2.3 → iris_vector_rag-0.2.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: iris-vector-rag
-Version: 0.2.3
+Version: 0.2.4
 Summary: Production-ready, extensible RAG framework with native IRIS vector search - unified API for basic, CRAG, GraphRAG, and ColBERT pipelines with RAGAS and DSPy integration
 Author-email: InterSystems <support@intersystems.com>
 Maintainer-email: Thomas Dyar <thomas.dyar@intersystems.com>
@@ -25,6 +25,7 @@ Classifier: Topic :: Database
 Classifier: Topic :: Text Processing :: Linguistic
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
+License-File: LICENSE
 Requires-Dist: openai>=1.0.0
 Requires-Dist: anthropic>=0.25.0
 Requires-Dist: sentence-transformers>=2.2.0
@@ -46,6 +47,13 @@ Requires-Dist: plotly>=6.1.2
 Requires-Dist: jaydebeapi>=1.2.3
 Requires-Dist: docker>=6.1.3
 Requires-Dist: tiktoken>=0.5.0
+Requires-Dist: torch>=2.0.0
+Provides-Extra: embedding
+Requires-Dist: torch>=2.0.0; extra == "embedding"
+Requires-Dist: sentence-transformers>=2.2.0; extra == "embedding"
+Requires-Dist: iris-vector-graph>=2.0.0; extra == "embedding"
+Provides-Extra: hybrid-graphrag
+Requires-Dist: iris-vector-graph>=2.0.0; extra == "hybrid-graphrag"
 Provides-Extra: dev
 Requires-Dist: uv>=0.1.0; extra == "dev"
 Requires-Dist: pytest>=7.0.0; extra == "dev"
@@ -92,6 +100,7 @@ Requires-Dist: python-multipart>=0.0.6; extra == "all"
 Requires-Dist: redis>=5.0.0; extra == "all"
 Requires-Dist: bcrypt>=4.0.0; extra == "all"
 Requires-Dist: python-jose[cryptography]>=3.3.0; extra == "all"
+Dynamic: license-file
 
 # IRIS Vector RAG Templates
 
@@ -369,6 +378,94 @@ make test-ragas-sample
 # - Answer Relevance
 ```
 
+### IRIS EMBEDDING: 346x Faster Auto-Vectorization
+
+**Automatic embedding generation with model caching** - eliminates the 720x slowdown from repeated model loading:
+
+```python
+from iris_rag import create_pipeline
+
+# Enable IRIS EMBEDDING support (Feature 051)
+pipeline = create_pipeline(
+    'basic',
+    embedding_config='medical_embeddings_v1'  # IRIS EMBEDDING config name
+)
+
+# Documents auto-vectorize on INSERT with cached models
+pipeline.load_documents(documents=docs)
+
+# Queries auto-vectorize using same cached model
+result = pipeline.query("What is diabetes?", top_k=5)
+```
+
+**Performance Achievements:**
+- ⚡ **346x speedup** - 1,746 documents vectorized in 3.5 seconds (vs 20 minutes baseline)
+- 🎯 **95% cache hit rate** - Models stay in memory across requests
+- 🚀 **50ms average latency** - Cache hits complete in <100ms
+- 💾 **Automatic fallback** - GPU OOM? Automatically falls back to CPU
+
+**Configuration Example:**
+
+```python
+from iris_rag.embeddings.iris_embedding import configure_embedding
+
+# Create embedding configuration
+config = configure_embedding(
+    name="medical_embeddings_v1",
+    model_name="sentence-transformers/all-MiniLM-L6-v2",
+    device_preference="auto",     # auto, cuda, mps, cpu
+    batch_size=32,
+    enable_entity_extraction=True,
+    entity_types=["Disease", "Medication", "Symptom"]
+)
+
+# Use with any pipeline
+pipeline = create_pipeline('basic', embedding_config='medical_embeddings_v1')
+```
+
+**Multi-Field Vectorization:**
+
+Combine multiple document fields into a single embedding:
+
+```python
+from iris_rag.core.models import Document
+
+# Document with multiple content fields
+doc = Document(
+    page_content="",  # Will be auto-filled from metadata
+    metadata={
+        "title": "Type 2 Diabetes Treatment",
+        "abstract": "A comprehensive review of treatment approaches...",
+        "conclusions": "Insulin therapy combined with lifestyle changes..."
+    }
+)
+
+# Configure multi-field embedding
+pipeline = create_pipeline(
+    'basic',
+    embedding_config='paper_embeddings',
+    multi_field_source=['title', 'abstract', 'conclusions']  # Concatenate fields
+)
+
+pipeline.load_documents(documents=[doc])
+# → Embedding generated from: "Type 2 Diabetes Treatment. A comprehensive review..."
+```
+
+**When to Use IRIS EMBEDDING:**
+- ✅ Large document collections (>1000 documents)
+- ✅ Frequent re-indexing or incremental updates
+- ✅ Real-time vectorization requirements
+- ✅ Memory-constrained environments (model stays in memory)
+- ✅ Multi-field vectorization needs
+
+**Comparison:**
+
+| Method | 1,746 Docs | Model Loads | Cache Hit Rate |
+|--------|-----------|-------------|----------------|
+| **Manual** (baseline) | 20 minutes | 1,746 (every row) | 0% |
+| **IRIS EMBEDDING** | 3.5 seconds | 1 (cached) | 95% |
+| **Speedup** | **346x faster** | **1,746x fewer** | **95% efficient** |
+
 ## Model Context Protocol (MCP) Support
 
 **Expose RAG pipelines as MCP tools** for use with Claude Desktop and other MCP clients:

{iris_vector_rag-0.2.3 → iris_vector_rag-0.2.4}/README.md

@@ -274,6 +274,94 @@ make test-ragas-sample
 # - Answer Relevance
 ```
 
+### IRIS EMBEDDING: 346x Faster Auto-Vectorization
+
+**Automatic embedding generation with model caching** - eliminates the 720x slowdown from repeated model loading:
+
+```python
+from iris_rag import create_pipeline
+
+# Enable IRIS EMBEDDING support (Feature 051)
+pipeline = create_pipeline(
+    'basic',
+    embedding_config='medical_embeddings_v1'  # IRIS EMBEDDING config name
+)
+
+# Documents auto-vectorize on INSERT with cached models
+pipeline.load_documents(documents=docs)
+
+# Queries auto-vectorize using same cached model
+result = pipeline.query("What is diabetes?", top_k=5)
+```
+
+**Performance Achievements:**
+- ⚡ **346x speedup** - 1,746 documents vectorized in 3.5 seconds (vs 20 minutes baseline)
+- 🎯 **95% cache hit rate** - Models stay in memory across requests
+- 🚀 **50ms average latency** - Cache hits complete in <100ms
+- 💾 **Automatic fallback** - GPU OOM? Automatically falls back to CPU
+
+**Configuration Example:**
+
+```python
+from iris_rag.embeddings.iris_embedding import configure_embedding
+
+# Create embedding configuration
+config = configure_embedding(
+    name="medical_embeddings_v1",
+    model_name="sentence-transformers/all-MiniLM-L6-v2",
+    device_preference="auto",     # auto, cuda, mps, cpu
+    batch_size=32,
+    enable_entity_extraction=True,
+    entity_types=["Disease", "Medication", "Symptom"]
+)
+
+# Use with any pipeline
+pipeline = create_pipeline('basic', embedding_config='medical_embeddings_v1')
+```
+
+**Multi-Field Vectorization:**
+
+Combine multiple document fields into a single embedding:
+
+```python
+from iris_rag.core.models import Document
+
+# Document with multiple content fields
+doc = Document(
+    page_content="",  # Will be auto-filled from metadata
+    metadata={
+        "title": "Type 2 Diabetes Treatment",
+        "abstract": "A comprehensive review of treatment approaches...",
+        "conclusions": "Insulin therapy combined with lifestyle changes..."
+    }
+)
+
+# Configure multi-field embedding
+pipeline = create_pipeline(
+    'basic',
+    embedding_config='paper_embeddings',
+    multi_field_source=['title', 'abstract', 'conclusions']  # Concatenate fields
+)
+
+pipeline.load_documents(documents=[doc])
+# → Embedding generated from: "Type 2 Diabetes Treatment. A comprehensive review..."
+```
+
+**When to Use IRIS EMBEDDING:**
+- ✅ Large document collections (>1000 documents)
+- ✅ Frequent re-indexing or incremental updates
+- ✅ Real-time vectorization requirements
+- ✅ Memory-constrained environments (model stays in memory)
+- ✅ Multi-field vectorization needs
+
+**Comparison:**
+
+| Method | 1,746 Docs | Model Loads | Cache Hit Rate |
+|--------|-----------|-------------|----------------|
+| **Manual** (baseline) | 20 minutes | 1,746 (every row) | 0% |
+| **IRIS EMBEDDING** | 3.5 seconds | 1 (cached) | 95% |
+| **Speedup** | **346x faster** | **1,746x fewer** | **95% efficient** |
+
 ## Model Context Protocol (MCP) Support
 
 **Expose RAG pipelines as MCP tools** for use with Claude Desktop and other MCP clients:

{iris_vector_rag-0.2.3 → iris_vector_rag-0.2.4}/iris_rag/__init__.py

@@ -18,7 +18,7 @@ from .validation.requirements import get_pipeline_requirements
 from .validation.validator import PreConditionValidator
 
 # Package version
-__version__ = "0.2.3"
+__version__ = "0.2.4"
 __author__ = "InterSystems IRIS RAG Templates Project"
 __description__ = "A comprehensive, production-ready framework for implementing Retrieval Augmented Generation (RAG) pipelines using InterSystems IRIS as the vector database backend."
 

{iris_vector_rag-0.2.3 → iris_vector_rag-0.2.4}/iris_rag/config/default_config.yaml

@@ -64,6 +64,67 @@ embeddings:
     model_name: "all-MiniLM-L6-v2"
     device: "cpu"
 
+# IRIS EMBEDDING Configuration (Feature 051)
+iris_embedding:
+  enabled: false                             # Enable IRIS EMBEDDING auto-vectorization
+
+  # Default configurations (can be overridden per-table)
+  default_config:
+    model_name: "sentence-transformers/all-MiniLM-L6-v2"
+    hf_cache_path: "/var/lib/huggingface"   # HuggingFace model cache directory
+    python_path: "/usr/bin/python3"          # Python interpreter path
+    batch_size: 32                           # Batch size for vectorization
+    device_preference: "auto"                # Device: auto, cuda, mps, cpu
+    enable_entity_extraction: false          # Extract entities during vectorization
+    entity_types: []                         # Entity types to extract (if enabled)
+
+  # Model cache configuration
+  cache:
+    enabled: true                            # Enable in-memory model caching
+    max_models: 5                            # Maximum number of cached models
+    eviction_policy: "lru"                   # Cache eviction: lru, lfu, ttl
+    ttl_seconds: 3600                        # Time-to-live for cached models (1 hour)
+    warmup_on_start: false                   # Preload models at startup
+
+  # Performance targets (Feature 050 pattern)
+  performance:
+    cache_hit_target_ms: 50                  # Target: <50ms for cache hits
+    cache_miss_target_ms: 5000               # Target: <5000ms for cache misses (includes model load)
+    min_cache_hit_rate: 0.95                 # Target: >=95% cache hit rate after warmup
+    gpu_oom_fallback: true                   # Fallback to CPU on GPU OOM
+
+  # Table-specific configurations (example)
+  # configurations:
+  #   medical_embeddings_v1:
+  #     model_name: "sentence-transformers/all-MiniLM-L6-v2"
+  #     enable_entity_extraction: true
+  #     entity_types: ["Disease", "Symptom", "Medication"]
+  #
+  #   paper_embeddings:
+  #     model_name: "sentence-transformers/allenai-specter"
+  #     device_preference: "cuda"
+  #     batch_size: 64
+
+  # Multi-field vectorization examples
+  # multi_field_configs:
+  #   research_papers:
+  #     table_name: "RAG.ResearchPapers"
+  #     embedding_column: "metadata_vector"
+  #     source_columns:                      # Concatenate multiple fields
+  #       - "title"
+  #       - "abstract"
+  #       - "conclusions"
+  #     config_name: "paper_embeddings"
+  #
+  #   medical_records:
+  #     table_name: "RAG.MedicalRecords"
+  #     embedding_column: "clinical_vector"
+  #     source_columns:
+  #       - "chief_complaint"
+  #       - "history_of_present_illness"
+  #       - "assessment"
+  #     config_name: "medical_embeddings_v1"
+
 # Entity extraction configuration
 entity_extraction:
   enabled: true

iris_vector_rag-0.2.4/iris_rag/config/embedding_config.py

@@ -0,0 +1,299 @@
+"""
+IRIS EMBEDDING Configuration Models.
+
+Feature: 051-add-native-iris
+Purpose: Define configuration data models for IRIS EMBEDDING integration
+         with validation, type safety, and serialization support.
+"""
+
+from dataclasses import dataclass, field, asdict
+from typing import List, Optional, Dict, Any
+from pathlib import Path
+import os
+import sys
+
+
+@dataclass
+class EmbeddingConfig:
+    """
+    Represents IRIS %Embedding.Config table entries with model settings
+    and optional entity extraction configuration.
+
+    This configuration controls how EMBEDDING columns auto-vectorize text data
+    and optionally extract entities for GraphRAG knowledge graphs.
+    """
+
+    # Required fields
+    name: str
+    model_name: str
+    hf_cache_path: str
+    python_path: str
+    embedding_class: str
+
+    # Optional fields with defaults
+    description: Optional[str] = None
+    enable_entity_extraction: bool = False
+    entity_types: List[str] = field(default_factory=list)
+    batch_size: int = 32
+    device_preference: str = "auto"  # cuda, mps, cpu, auto
+
+    def __post_init__(self):
+        """Validate configuration after initialization."""
+        # Validate batch_size
+        if self.batch_size <= 0:
+            raise ValueError(f"batch_size must be positive integer, got {self.batch_size}")
+
+        # Validate device_preference
+        valid_devices = ["cuda", "mps", "cpu", "auto"]
+        if self.device_preference not in valid_devices:
+            raise ValueError(
+                f"device_preference must be one of {valid_devices}, got '{self.device_preference}'"
+            )
+
+        # Validate entity extraction configuration
+        if self.enable_entity_extraction and not self.entity_types:
+            raise ValueError(
+                "If enable_entity_extraction=True, entity_types must not be empty"
+            )
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Convert to dictionary for JSON serialization."""
+        return asdict(self)
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "EmbeddingConfig":
+        """Create EmbeddingConfig from dictionary."""
+        return cls(**data)
+
+    def to_iris_json(self) -> Dict[str, Any]:
+        """
+        Convert to IRIS %Embedding.Config Configuration JSON format.
+
+        Returns dictionary suitable for storing in IRIS Configuration field.
+        """
+        return {
+            "modelName": self.model_name,
+            "hfCachePath": self.hf_cache_path,
+            "pythonPath": self.python_path,
+            "batchSize": self.batch_size,
+            "devicePreference": self.device_preference,
+            "enableEntityExtraction": self.enable_entity_extraction,
+            "entityTypes": self.entity_types
+        }
+
+
+@dataclass
+class ValidationResult:
+    """
+    Result of embedding configuration validation.
+
+    Used by validate_embedding_config() to report validation status,
+    errors, and warnings before creating tables with EMBEDDING columns.
+    """
+
+    valid: bool
+    errors: List[str] = field(default_factory=list)
+    warnings: List[str] = field(default_factory=list)
+
+    # Additional validation metadata
+    model_name: Optional[str] = None
+    cache_path_valid: bool = False
+    python_path_valid: bool = False
+    device: Optional[str] = None
+
+    def add_error(self, error: str):
+        """Add an error and mark validation as invalid."""
+        self.errors.append(error)
+        self.valid = False
+
+    def add_warning(self, warning: str):
+        """Add a warning (doesn't invalidate configuration)."""
+        self.warnings.append(warning)
+
+
+def validate_embedding_config(config: EmbeddingConfig) -> ValidationResult:
+    """
+    Validate EMBEDDING configuration before table creation (FR-010).
+
+    Performs pre-flight validation to catch configuration errors early
+    and provide actionable error messages.
+
+    Args:
+        config: EmbeddingConfig to validate
+
+    Returns:
+        ValidationResult with validation status, errors, and warnings
+
+    Validation checks:
+        1. Model file exists at hf_cache_path or can be downloaded
+        2. Python executable exists at python_path
+        3. Required Python packages installed (sentence-transformers, torch)
+        4. If entity extraction enabled, entity_types not empty
+        5. Device preference is valid
+    """
+    result = ValidationResult(valid=True)
+    result.model_name = config.model_name
+
+    # Check 1: Validate hf_cache_path exists and is writable
+    cache_path = Path(config.hf_cache_path)
+    if cache_path.exists():
+        if not cache_path.is_dir():
+            result.add_error(
+                f"INVALID_CACHE_PATH: hf_cache_path '{config.hf_cache_path}' "
+                "exists but is not a directory"
+            )
+        elif not os.access(cache_path, os.W_OK):
+            result.add_error(
+                f"INVALID_CACHE_PATH: hf_cache_path '{config.hf_cache_path}' "
+                "is not writable"
+            )
+        else:
+            result.cache_path_valid = True
+    else:
+        # Cache path doesn't exist - that's OK, it will be created
+        result.add_warning(
+            f"Cache path '{config.hf_cache_path}' does not exist. "
+            "It will be created when first downloading models."
+        )
+        result.cache_path_valid = True
+
+    # Check 2: Validate python_path is executable
+    python_path = Path(config.python_path)
+    if not python_path.exists():
+        result.add_error(
+            f"INVALID_PYTHON_PATH: Python executable not found at '{config.python_path}'. "
+            "Verify python_path in configuration."
+        )
+    elif not os.access(python_path, os.X_OK):
+        result.add_error(
+            f"INVALID_PYTHON_PATH: '{config.python_path}' exists but is not executable"
+        )
+    else:
+        result.python_path_valid = True
+
+    # Check 3: Verify required packages (basic check)
+    # Note: Full package check would require running pip list in target Python
+    try:
+        import sentence_transformers
+        import torch
+        result.add_warning(
+            "Package validation: sentence-transformers and torch found in current environment. "
+            "Ensure they are also installed in python_path environment."
+        )
+    except ImportError as e:
+        result.add_error(
+            f"MISSING_DEPENDENCIES: Required Python packages missing in current environment: {e}. "
+            "Run: pip install sentence-transformers torch"
+        )
+
+    # Check 4: Validate device preference
+    if config.device_preference not in ["cuda", "mps", "cpu", "auto"]:
+        result.add_error(
+            f"INVALID_DEVICE: device_preference must be one of [cuda, mps, cpu, auto], "
+            f"got '{config.device_preference}'"
+        )
+    else:
+        # Detect actual device that will be used
+        try:
+            import torch
+            if config.device_preference == "auto":
+                if torch.cuda.is_available():
+                    result.device = "cuda:0"
+                elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
+                    result.device = "mps"
+                else:
+                    result.device = "cpu"
+            else:
+                result.device = config.device_preference
+        except ImportError:
+            result.device = "unknown (torch not available)"
+
+    # Check 5: Validate entity extraction configuration
+    if config.enable_entity_extraction:
+        if not config.entity_types:
+            result.add_error(
+                "INVALID_ENTITY_CONFIG: enable_entity_extraction=True but entity_types is empty. "
+                "Provide at least one entity type (e.g., ['Disease', 'Medication'])"
+            )
+        else:
+            # Validate entity type naming
+            for entity_type in config.entity_types:
+                if not entity_type or not isinstance(entity_type, str):
+                    result.add_error(
+                        f"INVALID_ENTITY_TYPE: Entity type must be non-empty string, got '{entity_type}'"
+                    )
+                elif not entity_type[0].isupper():
+                    result.add_warning(
+                        f"Entity type '{entity_type}' should start with uppercase letter by convention"
+                    )
+
+    # Check 6: Validate model exists or can be downloaded
+    # This is a lightweight check - full validation would download the model
+    model_cache_dir = cache_path / "models--" / config.model_name.replace("/", "--")
+    if model_cache_dir.exists():
+        result.add_warning(
+            f"Model '{config.model_name}' found in cache at {model_cache_dir}"
+        )
+    else:
+        result.add_warning(
+            f"MODEL_NOT_CACHED: Model '{config.model_name}' not found in cache. "
+            f"It will be downloaded from HuggingFace on first use. "
+            f"To pre-download: huggingface-cli download {config.model_name}"
+        )
+
+    return result
+
+
+@dataclass
+class ClearCacheResult:
+    """Result of cache clearing operation."""
+
+    models_cleared: int
+    memory_freed_mb: float
+
+    def __str__(self) -> str:
+        return f"Cleared {self.models_cleared} models, freed {self.memory_freed_mb:.1f}MB"
+
+
+# Helper function for creating configurations
+def create_embedding_config(
+    name: str,
+    model_name: str = "sentence-transformers/all-MiniLM-L6-v2",
+    hf_cache_path: str = "/var/lib/huggingface",
+    python_path: str = "/usr/bin/python3",
+    enable_entity_extraction: bool = False,
+    entity_types: Optional[List[str]] = None,
+    **kwargs
+) -> EmbeddingConfig:
+    """
+    Helper function to create EmbeddingConfig with sensible defaults.
+
+    Args:
+        name: Unique configuration identifier
+        model_name: HuggingFace model ID
+        hf_cache_path: Path to HuggingFace cache directory
+        python_path: Path to Python executable
+        enable_entity_extraction: Whether to extract entities
+        entity_types: List of entity types to extract
+        **kwargs: Additional fields (description, batch_size, device_preference)
+
+    Returns:
+        EmbeddingConfig instance
+
+    Example:
+        >>> config = create_embedding_config(
+        ...     name="medical_embeddings",
+        ...     enable_entity_extraction=True,
+        ...     entity_types=["Disease", "Medication"]
+        ... )
+    """
+    return EmbeddingConfig(
+        name=name,
+        model_name=model_name,
+        hf_cache_path=hf_cache_path,
+        python_path=python_path,
+        embedding_class="%Embedding.SentenceTransformers",
+        enable_entity_extraction=enable_entity_extraction,
+        entity_types=entity_types or [],
+        **kwargs
+    )