mdb-engine 0.1.6__py3-none-any.whl

mdb_engine/embeddings/README.md

@@ -0,0 +1,184 @@
+# Embeddings Service Module
+
+Semantic text splitting and embedding generation for MDB_ENGINE applications.
+
+## Features
+
+- **Semantic Text Splitting**: Rust-based semantic-text-splitter for intelligent chunking
+- **OpenAI & AzureOpenAI Support**: Auto-detects provider from environment variables
+- **Token-Aware**: Never exceeds model token limits
+- **Batch Processing**: Efficient batch embedding generation
+- **MongoDB Integration**: Built-in support for storing embeddings with metadata
+
+## Installation
+
+```bash
+pip install semantic-text-splitter openai
+```
+
+## Configuration
+
+The embedding service auto-detects the provider from environment variables (same logic as mem0):
+
+- **OpenAI**: Requires `OPENAI_API_KEY`
+- **AzureOpenAI**: Requires `AZURE_OPENAI_API_KEY` and `AZURE_OPENAI_ENDPOINT`
+
+Enable embedding service in your `manifest.json`:
+
+```json
+{
+  "embedding_config": {
+    "enabled": true,
+    "default_embedding_model": "text-embedding-3-small",
+    "max_tokens_per_chunk": 1000,
+    "tokenizer_model": "gpt-3.5-turbo"
+  }
+}
+```
+
+## Usage
+
+### 1. Basic Usage (Auto-Detection)
+
+```python
+from mdb_engine.embeddings import EmbeddingService
+
+# Initialize - auto-detects OpenAI or AzureOpenAI from environment variables
+embedding_service = EmbeddingService()
+
+# Chunk text
+chunks = await embedding_service.chunk_text(
+    text_content="Your long document here...",
+    max_tokens=1000
+)
+
+# Generate embeddings
+vectors = await embedding_service.embed_chunks(chunks, model="text-embedding-3-small")
+```
+
+### 2. Process and Store in MongoDB
+
+```python
+from mdb_engine.embeddings import EmbeddingService
+
+embedding_service = EmbeddingService()
+
+# Process text and store in MongoDB
+result = await embedding_service.process_and_store(
+    text_content="Your long document here...",
+    source_id="doc_101",
+    collection=db.knowledge_base,
+    max_tokens=1000,
+    metadata={"source": "document.pdf", "page": 1}
+)
+
+print(f"Created {result['chunks_created']} chunks")
+```
+
+### 3. Explicit Provider
+
+```python
+from mdb_engine.embeddings import EmbeddingService, OpenAIEmbeddingProvider, EmbeddingProvider
+
+# Use OpenAI explicitly
+openai_provider = OpenAIEmbeddingProvider(default_model="text-embedding-3-small")
+provider = EmbeddingProvider(embedding_provider=openai_provider)
+embedding_service = EmbeddingService(embedding_provider=provider)
+```
+
+### 4. In FastAPI Routes
+
+```python
+from fastapi import FastAPI, Depends
+from mdb_engine.embeddings.dependencies import get_embedding_service_dependency
+from mdb_engine.embeddings import EmbeddingService
+
+app = FastAPI()
+
+# Set global engine during startup
+from mdb_engine.embeddings.dependencies import set_global_engine
+set_global_engine(engine, app_slug="my_app")
+
+@app.post("/embed")
+async def embed_endpoint(
+    embedding_service: EmbeddingService = Depends(get_embedding_service_dependency("my_app"))
+):
+    embeddings = await embedding_service.embed_chunks(["Hello world"])
+    return {"embeddings": embeddings}
+```
+
+## Environment Variables
+
+### OpenAI
+```bash
+export OPENAI_API_KEY="sk-..."
+```
+
+### AzureOpenAI
+```bash
+export AZURE_OPENAI_API_KEY="..."
+export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/"
+export AZURE_OPENAI_API_VERSION="2024-02-15-preview"  # Optional
+```
+
+## EmbeddingService Methods
+
+### `chunk_text(text_content, max_tokens=None, tokenizer_model=None)`
+
+Split text into semantic chunks.
+
+```python
+chunks = await service.chunk_text("Long document...", max_tokens=1000)
+```
+
+### `embed_chunks(chunks, model=None)`
+
+Generate embeddings for text chunks.
+
+```python
+vectors = await service.embed_chunks(chunks, model="text-embedding-3-small")
+```
+
+### `process_and_store(text_content, source_id, collection, ...)`
+
+Process text and store chunks with embeddings in MongoDB.
+
+```python
+result = await service.process_and_store(
+    text_content="Long document...",
+    source_id="doc_101",
+    collection=db.knowledge_base
+)
+```
+
+### `process_text(text_content, max_tokens=None, ...)`
+
+Process text and return chunks with embeddings (without storing).
+
+```python
+results = await service.process_text("Long document...")
+```
+
+## Supported Models
+
+- **OpenAI**: `text-embedding-3-small`, `text-embedding-3-large`, `text-embedding-ada-002`
+- **AzureOpenAI**: Any Azure OpenAI embedding deployment (e.g., `text-embedding-3-small`)
+
+## Error Handling
+
+All embedding operations raise `EmbeddingServiceError` on failure:
+
+```python
+from mdb_engine.embeddings import EmbeddingServiceError
+
+try:
+    vectors = await service.embed_chunks(["Hello"])
+except EmbeddingServiceError as e:
+    print(f"Embedding failed: {e}")
+```
+
+## Notes
+
+- The embedding service uses the same auto-detection logic as mem0 for consistency
+- LLM functionality (chat completions, structured extraction) should be implemented directly at the example level using the OpenAI SDK or your preferred provider
+- For memory functionality, use `mdb_engine.memory.Mem0MemoryService` which handles embeddings and LLM via environment variables

mdb_engine/embeddings/__init__.py

@@ -0,0 +1,62 @@
+"""
+Embeddings Service Module
+
+Provides EmbeddingService for semantic text splitting and embedding generation.
+Examples should implement their own LLM clients directly using the OpenAI SDK.
+
+For memory functionality, use mdb_engine.memory.Mem0MemoryService which
+handles embeddings and LLM via environment variables (.env).
+
+Example LLM implementation:
+    from openai import AzureOpenAI
+    from dotenv import load_dotenv
+    import os
+
+    load_dotenv()
+
+    client = AzureOpenAI(
+        api_version=os.getenv("AZURE_OPENAI_API_VERSION", "2024-02-01"),
+        azure_endpoint=os.getenv("AZURE_OPENAI_ENDPOINT"),
+        api_key=os.getenv("AZURE_OPENAI_API_KEY")
+    )
+
+    completion = client.chat.completions.create(
+        model=os.getenv("AZURE_OPENAI_DEPLOYMENT_NAME"),
+        messages=[...]
+    )
+
+Example EmbeddingService usage:
+    from mdb_engine.embeddings import EmbeddingService, get_embedding_service
+
+    # In FastAPI route
+    @app.post("/embed")
+    async def embed_text(embedding_service: EmbeddingService = Depends(get_embedding_service)):
+        embeddings = await embedding_service.embed_chunks(["Hello world"])
+        return {"embeddings": embeddings}
+"""
+
+from .dependencies import (create_embedding_dependency,
+                           get_embedding_service_dep,
+                           get_embedding_service_dependency,
+                           get_embedding_service_for_app, get_global_engine,
+                           set_global_engine)
+from .service import (AzureOpenAIEmbeddingProvider, BaseEmbeddingProvider,
+                      EmbeddingProvider, EmbeddingService,
+                      EmbeddingServiceError, OpenAIEmbeddingProvider,
+                      get_embedding_service)
+
+__all__ = [
+    "EmbeddingService",
+    "EmbeddingServiceError",
+    "BaseEmbeddingProvider",
+    "OpenAIEmbeddingProvider",
+    "AzureOpenAIEmbeddingProvider",
+    "EmbeddingProvider",
+    "get_embedding_service",
+    "get_embedding_service_for_app",
+    "create_embedding_dependency",
+    "set_global_engine",
+    "get_global_engine",
+    "get_embedding_service_dependency",
+    "get_embedding_service_dep",
+]

mdb_engine/embeddings/dependencies.py

@@ -0,0 +1,193 @@
+"""
+Embedding Service Dependency Injection for FastAPI
+
+This module provides FastAPI dependency functions to inject embedding services
+into route handlers. The embedding service is automatically initialized from
+the app's manifest.json configuration.
+"""
+
+from typing import Any, Optional
+
+# Optional FastAPI import (only needed if FastAPI is available)
+try:
+    from fastapi import Depends, HTTPException
+
+    FASTAPI_AVAILABLE = True
+except ImportError:
+    FASTAPI_AVAILABLE = False
+
+    # Stub for when FastAPI is not available
+    def Depends(*args, **kwargs):
+        return None
+
+    class HTTPException(Exception):
+        pass
+
+
+from .service import EmbeddingService, get_embedding_service
+
+# Global engine registry (for apps that don't pass engine explicitly)
+_global_engine: Optional[Any] = None
+_global_app_slug: Optional[str] = None
+
+
+def set_global_engine(engine: Any, app_slug: Optional[str] = None) -> None:
+    """
+    Set global MongoDBEngine instance for embedding dependency injection.
+
+    This is useful when you have a single engine instance that you want
+    to use across all apps. Call this during application startup.
+
+    Args:
+        engine: MongoDBEngine instance
+        app_slug: Optional app slug
+    """
+    global _global_engine, _global_app_slug
+    _global_engine = engine
+    _global_app_slug = app_slug
+
+
+def get_global_engine() -> Optional[Any]:
+    """
+    Get global MongoDBEngine instance.
+
+    Returns:
+        MongoDBEngine instance if set, None otherwise
+    """
+    return _global_engine
+
+
+def get_embedding_service_for_app(
+    app_slug: str, engine: Optional[Any] = None
+) -> Optional[EmbeddingService]:
+    """
+    Get embedding service for a specific app.
+
+    This is a helper function that can be used with FastAPI's Depends()
+    to inject the embedding service into route handlers.
+
+    Args:
+        app_slug: App slug (typically extracted from route context)
+        engine: MongoDBEngine instance (optional, will try to get from context)
+
+    Returns:
+        EmbeddingService instance if embedding is enabled for this app, None otherwise
+
+    Example:
+        ```python
+        from fastapi import Depends
+        from mdb_engine.embeddings.dependencies import get_embedding_service_for_app
+
+        @app.post("/embed")
+        async def embed_endpoint(
+            embedding_service = Depends(lambda: get_embedding_service_for_app("my_app"))
+        ):
+            if not embedding_service:
+                raise HTTPException(503, "Embedding service not available")
+            embeddings = await embedding_service.embed_chunks(["Hello world"])
+            return {"embeddings": embeddings}
+        ```
+    """
+    # Try to get engine from context if not provided
+    if engine is None:
+        engine = _global_engine
+
+    if engine is None:
+        return None
+
+    # Get app config to extract embedding_config
+    app_config = engine.get_app(app_slug)
+    if not app_config:
+        return None
+
+    embedding_config = app_config.get("embedding_config", {})
+    if not embedding_config.get("enabled", True):
+        return None
+
+    # Create embedding service with config
+    return get_embedding_service(config=embedding_config)
+
+
+def create_embedding_dependency(app_slug: str, engine: Optional[Any] = None):
+    """
+    Create a FastAPI dependency function for embedding service.
+
+    This creates a dependency function that can be used with Depends()
+    to inject the embedding service into route handlers.
+
+    Args:
+        app_slug: App slug
+        engine: MongoDBEngine instance (optional)
+
+    Returns:
+        Dependency function that returns EmbeddingService or raises HTTPException
+
+    Example:
+        ```python
+        from fastapi import Depends
+        from mdb_engine.embeddings.dependencies import create_embedding_dependency
+
+        embedding_dep = create_embedding_dependency("my_app", engine)
+
+        @app.post("/embed")
+        async def embed_endpoint(embedding_service = Depends(embedding_dep)):
+            embeddings = await embedding_service.embed_chunks(["Hello world"])
+            return {"embeddings": embeddings}
+        ```
+    """
+
+    def _get_embedding_service() -> EmbeddingService:
+        embedding_service = get_embedding_service_for_app(app_slug, engine)
+        if embedding_service is None:
+            if FASTAPI_AVAILABLE:
+                raise HTTPException(
+                    status_code=503,
+                    detail=f"Embedding service not available for app '{app_slug}'. "
+                    "Ensure 'embedding_config.enabled' is true in manifest.json and "
+                    "embedding dependencies are installed.",
+                )
+            else:
+                raise RuntimeError(
+                    f"Embedding service not available for app '{app_slug}'"
+                )
+        return embedding_service
+
+    return _get_embedding_service
+
+
+def get_embedding_service_dependency(app_slug: str):
+    """
+    Get embedding service dependency using global engine.
+
+    This is a convenience function that uses the global engine registry.
+    Set the engine with set_global_engine() during app startup.
+
+    Args:
+        app_slug: App slug
+
+    Returns:
+        Dependency function for FastAPI Depends()
+
+    Example:
+        ```python
+        from fastapi import FastAPI, Depends
+        from mdb_engine.embeddings.dependencies import (
+            set_global_engine, get_embedding_service_dependency
+        )
+
+        app = FastAPI()
+
+        # During startup
+        set_global_engine(engine, app_slug="my_app")
+
+        # In routes
+        @app.post("/embed")
+        async def embed(embedding_service = Depends(get_embedding_service_dependency("my_app"))):
+            return await embedding_service.embed_chunks(["Hello world"])
+        ```
+    """
+    return create_embedding_dependency(app_slug, _global_engine)
+
+
+# Alias for backward compatibility
+get_embedding_service_dep = get_embedding_service_dependency