remdb 0.3.7__py3-none-any.whl

rem/agentic/agents/README.md

@@ -0,0 +1,155 @@
+# REM Agents
+
+Built-in agents for REM system operations.
+
+## Overview
+
+This folder contains specialized agents that provide high-level interfaces for REM operations. These agents use LLMs to interpret natural language and convert it to structured REM queries.
+
+## REM Query Agent
+
+**File**: `rem_query_agent.py`
+
+Converts natural language questions into structured REM queries with PostgreSQL dialect awareness.
+
+### Features
+
+- **Query Type Selection**: Automatically chooses optimal query type (LOOKUP, FUZZY, SEARCH, SQL, TRAVERSE)
+- **PostgreSQL Dialect Aware**: Knows when to use KV_STORE vs primary tables
+- **Token Optimized**: Minimal output fields for fast generation and low cost
+- **Confidence Scoring**: Returns confidence (0-1) with reasoning for low scores
+- **Multi-Step Planning**: Can break complex queries into multiple REM calls
+
+### Usage
+
+#### Simple Query
+
+```python
+from rem.agentic.agents import ask_rem
+
+# Convert natural language to REM query
+result = await ask_rem("Show me Sarah Chen")
+
+print(result.query_type)  # QueryType.LOOKUP
+print(result.parameters)  # {"entity_key": "sarah-chen"}
+print(result.confidence)  # 1.0
+```
+
+#### With Custom Model
+
+```python
+# Use fast, cheap model for query generation
+result = await ask_rem(
+    "Find documents about databases",
+    llm_model="gpt-4o-mini"
+)
+
+print(result.query_type)  # QueryType.SEARCH
+print(result.parameters)
+# {
+#     "query_text": "database",
+#     "table_name": "resources",
+#     "field_name": "content",
+#     "limit": 10
+# }
+```
+
+#### Integration with RemService
+
+```python
+from rem.services.rem import RemService
+
+# RemService automatically uses REM Query Agent
+result = await rem_service.ask_rem(
+    natural_query="What does Sarah manage?",
+    tenant_id="acme-corp"
+)
+
+# Returns:
+# {
+#     "query_output": {
+#         "query_type": "TRAVERSE",
+#         "parameters": {"start_key": "sarah-chen", "max_depth": 1, "rel_type": "manages"},
+#         "confidence": 0.85,
+#         "reasoning": "TRAVERSE query to find entities Sarah manages via graph edges"
+#     },
+#     "results": [...],  # Executed query results (if confidence >= 0.7)
+#     "natural_query": "What does Sarah manage?"
+# }
+```
+
+### Query Types
+
+| Type | Description | When to Use | Example |
+|------|-------------|-------------|---------|
+| `LOOKUP` | O(1) entity lookup by natural key | User references specific entity by name | "Show me Sarah Chen" |
+| `FUZZY` | Trigram text similarity (pg_trgm) | Partial/misspelled names, approximate matches | "Find people named Sara" |
+| `SEARCH` | Semantic vector similarity | Conceptual questions, semantic similarity | "Documents about databases" |
+| `SQL` | Direct table queries with WHERE | Temporal, filtered, or aggregate queries | "Meetings in Q4 2024" |
+| `TRAVERSE` | Recursive graph traversal | Relationships, connections, "what's related" | "What does Sarah manage?" |
+
+### Configuration
+
+Set the model for REM Query Agent in your environment:
+
+```bash
+# .env
+LLM__QUERY_AGENT_MODEL=gpt-4o-mini  # Fast, cheap model recommended
+```
+
+If not set, uses `settings.llm.default_model`.
+
+### Output Schema
+
+```python
+class REMQueryOutput(BaseModel):
+    query_type: QueryType  # Selected query type
+    parameters: dict       # Query parameters
+    confidence: float      # 0.0-1.0 confidence score
+    reasoning: str | None  # Only if confidence < 0.7 or multi-step
+    multi_step: list[dict] | None  # For complex queries
+```
+
+### Design Philosophy
+
+1. **Token Efficiency**: Output is concise by design
+   - Reasoning only included when needed (low confidence or multi-step)
+   - Minimal fields to reduce generation time and cost
+
+2. **PostgreSQL Awareness**: Agent knows the database schema
+   - LOOKUP/FUZZY use UNLOGGED KV_STORE (fast cache)
+   - SEARCH joins KV_STORE + embeddings_<table>
+   - SQL queries primary tables directly
+   - TRAVERSE follows graph_edges JSONB field
+
+3. **Progressive Complexity**: Prefer simple queries over complex
+   - LOOKUP is fastest (O(1))
+   - FUZZY uses indexed trigrams
+   - SEARCH requires embedding generation
+   - SQL scans tables (filtered)
+   - TRAVERSE is recursive (most complex)
+
+4. **Confidence-Based Execution**: RemService auto-executes if confidence >= 0.7
+   - High confidence: Execute immediately
+   - Low confidence: Return query + reasoning for review
+
+### Testing
+
+See `tests/unit/agentic/agents/test_rem_query_agent.py` for unit tests.
+
+Tests cover:
+- Schema structure validation
+- Output model creation
+- Confidence validation
+- Multi-step query support
+
+Integration tests with actual LLM execution require API keys and are in `tests/integration/`.
+
+## Future Agents
+
+Additional agents can be added following the same pattern:
+
+- **Entity Summarization Agent**: Summarize entity relationships
+- **Query Explanation Agent**: Explain REM query results in natural language
+- **Schema Discovery Agent**: Discover available tables and fields
+- **Data Quality Agent**: Identify data quality issues in entities

rem/agentic/agents/__init__.py

@@ -0,0 +1,8 @@
+"""
+REM Agents - Specialized agents for REM operations.
+
+All agents are defined as YAML schemas in src/rem/schemas/agents/.
+Use create_agent_from_schema_file() to instantiate agents.
+"""
+
+__all__ = []

rem/agentic/context.py

@@ -0,0 +1,148 @@
+"""
+Agent execution context and configuration.
+
+Design pattern for session context that can be constructed from:
+- HTTP headers (X-User-Id, X-Session-Id, X-Model-Name)
+- Direct instantiation for testing/CLI
+
+Key Design Pattern 
+- AgentContext is passed to agent factory, not stored in agents
+- Enables session tracking across API, CLI, and test execution
+- Supports header-based configuration override (model, schema URI)
+- Clean separation: context (who/what) vs agent (how)
+"""
+
+from loguru import logger
+from pydantic import BaseModel, Field
+
+from ..settings import settings
+
+
+class AgentContext(BaseModel):
+    """
+    Session and configuration context for agent execution.
+
+    Provides session identifiers (user_id, tenant_id, session_id) and
+    configuration defaults (model) for agent factory and execution.
+
+    Design Pattern 
+    - Construct from HTTP headers via from_headers()
+    - Pass to agent factory, not stored in agent
+    - Enables header-based model/schema override
+    - Supports observability (user tracking, session continuity)
+
+    Example:
+        # From HTTP request
+        context = AgentContext.from_headers(request.headers)
+        agent = await create_agent(context)
+
+        # Direct construction for testing
+        context = AgentContext(user_id="test-user", tenant_id="test-tenant")
+        agent = await create_agent(context)
+    """
+
+    user_id: str | None = Field(
+        default=None,
+        description="User identifier for tracking and personalization",
+    )
+
+    tenant_id: str = Field(
+        default="default",
+        description="Tenant identifier for multi-tenancy isolation (REM requirement)",
+    )
+
+    session_id: str | None = Field(
+        default=None,
+        description="Session/conversation identifier for continuity",
+    )
+
+    default_model: str = Field(
+        default_factory=lambda: settings.llm.default_model,
+        description="Default LLM model (can be overridden via headers)",
+    )
+
+    agent_schema_uri: str | None = Field(
+        default=None,
+        description="Agent schema URI (e.g., 'rem-agents-query-agent')",
+    )
+
+    model_config = {"populate_by_name": True}
+
+    @staticmethod
+    def get_user_id_or_default(
+        user_id: str | None,
+        source: str = "context",
+        default: str = "default",
+    ) -> str:
+        """
+        Get user_id or fallback to default with logging.
+
+        Centralized helper for consistent user_id fallback behavior across
+        API endpoints, MCP tools, CLI commands, and services.
+
+        Args:
+            user_id: User identifier (may be None)
+            source: Source of the call (for logging clarity)
+            default: Default value to use (default: "default")
+
+        Returns:
+            user_id if provided, otherwise default
+
+        Example:
+            # In MCP tool
+            user_id = AgentContext.get_user_id_or_default(
+                user_id, source="ask_rem_agent"
+            )
+
+            # In API endpoint
+            user_id = AgentContext.get_user_id_or_default(
+                temp_context.user_id, source="chat_completions"
+            )
+
+            # In CLI command
+            user_id = AgentContext.get_user_id_or_default(
+                args.user_id, source="rem ask"
+            )
+        """
+        if user_id is None:
+            logger.debug(f"No user_id provided from {source}, using '{default}'")
+            return default
+        return user_id
+
+    @classmethod
+    def from_headers(cls, headers: dict[str, str]) -> "AgentContext":
+        """
+        Construct AgentContext from HTTP headers.
+
+        Reads standard headers:
+        - X-User-Id: User identifier
+        - X-Tenant-Id: Tenant identifier
+        - X-Session-Id: Session identifier
+        - X-Model-Name: Model override
+        - X-Agent-Schema: Agent schema URI
+
+        Args:
+            headers: Dictionary of HTTP headers (case-insensitive)
+
+        Returns:
+            AgentContext with values from headers
+
+        Example:
+            headers = {
+                "X-User-Id": "user123",
+                "X-Tenant-Id": "acme-corp",
+                "X-Session-Id": "sess-456",
+                "X-Model-Name": "anthropic:claude-opus-4-20250514"
+            }
+            context = AgentContext.from_headers(headers)
+        """
+        # Normalize header keys to lowercase for case-insensitive lookup
+        normalized = {k.lower(): v for k, v in headers.items()}
+
+        return cls(
+            user_id=normalized.get("x-user-id"),
+            tenant_id=normalized.get("x-tenant-id", "default"),
+            session_id=normalized.get("x-session-id"),
+            default_model=normalized.get("x-model-name") or settings.llm.default_model,
+            agent_schema_uri=normalized.get("x-agent-schema"),
+        )

rem/agentic/context_builder.py

@@ -0,0 +1,329 @@
+"""
+Centralized context builder for agent execution.
+
+Session History (ALWAYS loaded with compression):
+- Each chat request is a single message, so session history MUST be recovered
+- Uses SessionMessageStore with compression to keep context efficient
+- Long assistant responses include REM LOOKUP hints: "... [REM LOOKUP session-{id}-msg-{index}] ..."
+- Agent can retrieve full content on-demand using REM LOOKUP
+- Prevents context window bloat while maintaining conversation continuity
+
+User Context (on-demand by default):
+- System message includes REM LOOKUP hint for user profile
+- Agent decides whether to load profile based on query
+- More efficient for queries that don't need personalization
+- Example: "User ID: sarah@example.com. To load user profile: Use REM LOOKUP users/sarah@example.com"
+
+User Context (auto-inject when enabled):
+- Set CHAT__AUTO_INJECT_USER_CONTEXT=true
+- User profile automatically loaded from database and injected into system message
+- Simpler for basic chatbots that always need context
+
+Design Pattern:
+1. Extract AgentContext from headers (user_id, tenant_id, session_id)
+2. If auto-inject enabled: Load User/Session from database
+3. If auto-inject disabled: Provide REM LOOKUP hints in system message
+4. Construct system message with date + context (injected or hints)
+5. Return complete context ready for agent execution
+
+Integration Points:
+- API endpoints: build_from_headers() extracts user context from JWT/session headers
+- Tests: build_from_test() creates minimal test context without DB
+- Settings: CHAT__AUTO_INJECT_* controls auto-inject vs on-demand behavior
+
+Usage (on-demand, default):
+    # From FastAPI endpoint
+    context, messages = await ContextBuilder.build_from_headers(
+        headers=request.headers,
+        new_messages=[{"role": "user", "content": "What's next for the API migration?"}]
+    )
+
+    # Messages list structure (on-demand):
+    # [
+    #   {"role": "system", "content": "Today's date: 2025-11-22\nUser ID: sarah@example.com\nTo load user profile: Use REM LOOKUP users/sarah@example.com\nSession ID: sess-123\nTo load session history: Use REM LOOKUP messages?session_id=sess-123"},
+    #   {"role": "user", "content": "What's next for the API migration?"}
+    # ]
+
+    # Agent receives hints and can decide to load context if needed
+    agent = await create_agent(context=context, ...)
+    prompt = "\n".join(msg.content for msg in messages)
+    result = await agent.run(prompt)
+
+Usage (auto-inject, CHAT__AUTO_INJECT_USER_CONTEXT=true):
+    # Messages list structure (auto-inject):
+    # [
+    #   {"role": "system", "content": "Today's date: 2025-11-22\n\nUser Context (auto-injected):\nSummary: ...\nInterests: ...\n\nSession History (auto-injected, 5 messages):"},
+    #   {"role": "user", "content": "Previous message"},
+    #   {"role": "assistant", "content": "Previous response"},
+    #   {"role": "user", "content": "What's next for the API migration?"}
+    # ]
+
+Testing:
+    # From CLI/test (no database)
+    context, messages = await ContextBuilder.build_from_test(
+        user_id="test@rem.ai",
+        tenant_id="test-tenant",
+        message="Hello"
+    )
+"""
+
+from datetime import datetime, timezone
+from typing import Any
+
+from loguru import logger
+from pydantic import BaseModel
+
+from .context import AgentContext
+from ..models.entities.user import User
+from ..models.entities.message import Message
+from ..services.postgres.repository import Repository
+from ..services.postgres.service import PostgresService
+
+
+class ContextMessage(BaseModel):
+    """Standard message format for LLM conversations."""
+
+    role: str  # "system", "user", "assistant"
+    content: str
+
+
+class ContextBuilder:
+    """
+    Centralized builder for agent execution context.
+
+    Handles:
+    - User profile loading from database
+    - Session history recovery
+    - Context message construction
+    - Test context generation
+    """
+
+    @staticmethod
+    async def build_from_headers(
+        headers: dict[str, str],
+        new_messages: list[dict[str, str]] | None = None,
+        db: PostgresService | None = None,
+    ) -> tuple[AgentContext, list[ContextMessage]]:
+        """
+        Build complete context from HTTP headers.
+
+        Session History (ALWAYS loaded with compression):
+        - If session_id provided, session history is ALWAYS loaded using SessionMessageStore
+        - Compression keeps it efficient with REM LOOKUP hints for long messages
+        - Example: "... [Message truncated - REM LOOKUP session-{id}-msg-{index}] ..."
+        - Agent can retrieve full content on-demand using REM LOOKUP
+
+        User Context (on-demand by default):
+        - System message includes REM LOOKUP hint: "User ID: {user_id}. To load user profile: Use REM LOOKUP users/{user_id}"
+        - Agent decides whether to load profile based on query
+
+        User Context (auto-inject when enabled):
+        - Set CHAT__AUTO_INJECT_USER_CONTEXT=true
+        - User profile automatically loaded and injected into system message
+
+        Args:
+            headers: HTTP request headers (case-insensitive)
+            new_messages: New messages from current request
+            db: Optional PostgresService (creates if None)
+
+        Returns:
+            Tuple of (AgentContext, messages list)
+
+        Example:
+            headers = {"X-User-Id": "sarah@example.com", "X-Session-Id": "sess-123"}
+            context, messages = await ContextBuilder.build_from_headers(headers, new_messages)
+
+            # messages structure:
+            # [
+            #   {"role": "system", "content": "Today's date: 2025-11-22\nUser ID: sarah@example.com\nTo load user profile: Use REM LOOKUP users/sarah@example.com"},
+            #   {"role": "user", "content": "Previous message"},
+            #   {"role": "assistant", "content": "Start of long response... [REM LOOKUP session-123-msg-1] ...end"},
+            #   {"role": "user", "content": "New message"}
+            # ]
+        """
+        from ..settings import settings
+        from ..services.session.compression import SessionMessageStore
+
+        # Extract AgentContext from headers
+        context = AgentContext.from_headers(headers)
+
+        # Initialize DB if not provided and needed (for user context or session history)
+        close_db = False
+        if db is None and (settings.chat.auto_inject_user_context or context.session_id):
+            from ..services.postgres import get_postgres_service
+            db = get_postgres_service()
+            if db:
+                await db.connect()
+                close_db = True
+
+        try:
+            # Build messages list
+            messages: list[ContextMessage] = []
+
+            # Build context hint message
+            today = datetime.now(timezone.utc).strftime("%Y-%m-%d")
+            context_hint = f"Today's date: {today}."
+
+            # Add user context (auto-inject or on-demand hint)
+            if settings.chat.auto_inject_user_context and context.user_id and db:
+                # Auto-inject: Load and include user profile
+                user_context_content = await ContextBuilder._load_user_context(
+                    user_id=context.user_id,
+                    tenant_id=context.tenant_id,
+                    db=db,
+                )
+                if user_context_content:
+                    context_hint += f"\n\nUser Context (auto-injected):\n{user_context_content}"
+                else:
+                    context_hint += "\n\nNo user context available (anonymous or new user)."
+            elif context.user_id:
+                # On-demand: Provide hint to use REM LOOKUP
+                context_hint += f"\n\nUser ID: {context.user_id}"
+                context_hint += f"\nTo load user profile: Use REM LOOKUP users/{context.user_id}"
+
+            # Add system context hint
+            messages.append(ContextMessage(role="system", content=context_hint))
+
+            # ALWAYS load session history (if session_id provided) with compression
+            if context.session_id and settings.postgres.enabled:
+                store = SessionMessageStore(user_id=context.user_id or "default")
+                session_history = await store.load_session_messages(
+                    session_id=context.session_id,
+                    user_id=context.user_id,
+                    decompress=False,  # Use compressed versions with REM LOOKUP hints
+                )
+
+                # Convert to ContextMessage format
+                for msg_dict in session_history:
+                    messages.append(
+                        ContextMessage(
+                            role=msg_dict["role"],
+                            content=msg_dict["content"],
+                        )
+                    )
+
+                logger.debug(f"Loaded {len(session_history)} compressed messages for session {context.session_id}")
+
+            # Add new messages from request
+            if new_messages:
+                for msg in new_messages:
+                    messages.append(ContextMessage(**msg))
+
+            return context, messages
+
+        finally:
+            if close_db and db:
+                await db.disconnect()
+
+    @staticmethod
+    async def _load_user_context(
+        user_id: str | None,
+        tenant_id: str,
+        db: PostgresService,
+    ) -> str | None:
+        """
+        Load user profile from database and format as context.
+
+        Returns formatted string with:
+        - User summary (generated by dreaming worker)
+        - Current projects
+        - Technical interests
+        - Preferred topics
+
+        Returns None if user_id not provided or user not found.
+        """
+        if not user_id:
+            return None
+
+        try:
+            user_repo = Repository(User, "users", db=db)
+            user = await user_repo.get_by_id(user_id, tenant_id)
+
+            if not user:
+                logger.debug(f"User {user_id} not found in tenant {tenant_id}")
+                return None
+
+            # Build user context string
+            parts = []
+
+            if user.summary:
+                parts.append(f"Summary: {user.summary}")
+
+            if user.interests:
+                parts.append(f"Interests: {', '.join(user.interests[:5])}")
+
+            if user.preferred_topics:
+                parts.append(f"Topics: {', '.join(user.preferred_topics[:5])}")
+
+            # Add full profile from metadata if available
+            if user.metadata and "profile" in user.metadata:
+                profile = user.metadata["profile"]
+
+                if profile.get("current_projects"):
+                    projects = profile["current_projects"]
+                    project_names = [p.get("name", "Unnamed") for p in projects[:3]]
+                    parts.append(f"Current Projects: {', '.join(project_names)}")
+
+            if not parts:
+                return None
+
+            return "\n".join(parts)
+
+        except Exception as e:
+            logger.error(f"Failed to load user context: {e}")
+            return None
+
+
+    @staticmethod
+    async def build_from_test(
+        user_id: str = "test@rem.ai",
+        tenant_id: str = "test-tenant",
+        session_id: str | None = None,
+        message: str = "Hello",
+        model: str | None = None,
+    ) -> tuple[AgentContext, list[ContextMessage]]:
+        """
+        Build context for testing (no database lookup).
+
+        Creates minimal context with:
+        - Test user (test@rem.ai)
+        - Test tenant
+        - Context hint with date
+        - Single user message
+
+        Args:
+            user_id: Test user identifier (default: test@rem.ai)
+            tenant_id: Test tenant identifier
+            session_id: Optional session ID
+            message: User message content
+            model: Optional model override
+
+        Returns:
+            Tuple of (AgentContext, messages list)
+
+        Example:
+            context, messages = await ContextBuilder.build_from_test(
+                user_id="test@rem.ai",
+                message="What's the weather like?"
+            )
+        """
+        from ..settings import settings
+
+        # Create test context
+        context = AgentContext(
+            user_id=user_id,
+            tenant_id=tenant_id,
+            session_id=session_id,
+            default_model=model or settings.llm.default_model,
+        )
+
+        # Build minimal messages
+        today = datetime.now(timezone.utc).strftime("%Y-%m-%d")
+        context_hint = f"Today's date: {today}.\n\nTest user context: {user_id} (test mode, no profile loaded)."
+
+        messages = [
+            ContextMessage(role="system", content=context_hint),
+            ContextMessage(role="user", content=message),
+        ]
+
+        return context, messages