remdb 0.3.0__py3-none-any.whl → 0.3.114__py3-none-any.whl

rem/services/postgres/README.md

@@ -348,8 +348,27 @@ results = await service.vector_search(
 
 ### Initialize Service
 
+There are two ways to initialize the PostgresService:
+
+**Option 1: Factory function (recommended for apps using remdb as a library)**
+
+```python
+from rem.services.postgres import get_postgres_service
+
+# Uses POSTGRES__CONNECTION_STRING from environment
+pg = get_postgres_service()
+if pg is None:
+    raise RuntimeError("Database not configured - set POSTGRES__CONNECTION_STRING")
+
+await pg.connect()
+# ... use pg ...
+await pg.disconnect()
+```
+
+**Option 2: Direct instantiation**
+
 ```python
-from rem.services.postgres import PostgresService, Repository
+from rem.services.postgres import PostgresService
 
 service = PostgresService(
     connection_string="postgresql://user:pass@localhost/remdb",
@@ -359,6 +378,9 @@ service = PostgresService(
 await service.connect()
 ```
 
+> **Note**: `get_postgres_service()` returns the service directly. It does NOT support
+> `async with` context manager syntax. Always call `connect()` and `disconnect()` explicitly.
+
 ### Using Repository Pattern
 
 **Generic Repository** for simple CRUD operations:
@@ -514,34 +536,156 @@ results = await service.vector_search(
 - HNSW parameters: `m=16, ef_construction=64` (tunable)
 - Monitor shared_buffers and work_mem
 
-## Migrations
+## Schema Management
 
-Run migrations in order:
+REM uses a **code-as-source-of-truth** approach. Pydantic models define the schema, and the database is kept in sync via diff-based migrations.
 
-```bash
-psql -d remdb -f sql/migrations/001_setup_extensions.sql
-psql -d remdb -f sql/migrations/002_kv_store_cache.sql
-psql -d remdb -f sql/generated_schema.sql
+### File Structure
+
+```
+src/rem/sql/
+├── migrations/
+│   ├── 001_install.sql          # Core infrastructure (manual)
+│   └── 002_install_models.sql   # Entity tables (auto-generated)
+└── background_indexes.sql       # HNSW vector indexes (optional)
 ```
 
-Background indexes (after data load):
+**Key principle**: Only two migration files. No incremental `003_`, `004_` files.
+
+### CLI Commands
 
 ```bash
-psql -d remdb -f sql/background_indexes.sql
+# Apply migrations (installs extensions, core tables, entity tables)
+rem db migrate
+
+# Check migration status
+rem db status
+
+# Generate schema SQL from models (for remdb development)
+rem db schema generate --models src/rem/models/entities
+
+# Validate models for schema generation
+rem db schema validate --models src/rem/models/entities
 ```
 
-## CLI Usage
+### Model Registry
 
-Generate schema from models:
+Models are discovered via the registry:
 
-```bash
-rem schema generate --models src/rem/models/entities --output sql/schema.sql
+```python
+import rem
+from rem.models.core import CoreModel
+
+@rem.register_model
+class MyEntity(CoreModel):
+    name: str
+    description: str  # Auto-embeds
+```
+
+## Using REM as a Library (Downstream Apps)
+
+When building an application that **depends on remdb as a package** (e.g., `pip install remdb`),
+there are important differences from developing remdb itself.
+
+### What Works Out of the Box
+
+1. **All core entity tables** - Resources, Messages, Users, Sessions, etc.
+2. **PostgresService** - Full database access via `get_postgres_service()`
+3. **Repository pattern** - CRUD operations for core entities
+4. **Migrations** - `rem db migrate` applies the bundled SQL files
+
+```python
+# In your downstream app (e.g., myapp/main.py)
+from rem.services.postgres import get_postgres_service
+from rem.models.entities import Message, Resource
+
+pg = get_postgres_service()
+await pg.connect()
+
+# Use core entities - tables already exist
+messages = await pg.query(Message, {"session_id": "abc"})
 ```
 
-Validate models:
+### Custom Models in Downstream Apps
+
+The `@rem.register_model` decorator registers models in the **runtime registry**, which is useful for:
+- Schema introspection at runtime
+- Future tooling that reads the registry
+
+However, **`rem db migrate` only applies SQL files bundled in the remdb package**.
+Custom models from downstream apps do NOT automatically get tables created.
+
+**Options for custom model tables:**
+
+**Option A: Use core entities with metadata**
+
+Store custom data in the `metadata` JSONB field of existing entities:
+
+```python
+resource = Resource(
+    name="my-custom-thing",
+    content="...",
+    metadata={"custom_field": "value", "another": 123}
+)
+```
+
+**Option B: Create tables manually**
+
+Write and apply your own SQL:
+
+```sql
+-- myapp/sql/custom_tables.sql
+CREATE TABLE IF NOT EXISTS conversation_summaries (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    session_ref TEXT NOT NULL,
+    summary TEXT NOT NULL,
+    -- ... include CoreModel fields for compatibility
+    user_id VARCHAR(256),
+    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+);
+```
 
 ```bash
-rem schema validate --models src/rem/models/entities
+psql $DATABASE_URL -f myapp/sql/custom_tables.sql
+```
+
+**Option C: Contribute upstream**
+
+If your model is generally useful, contribute it to remdb so it's included in
+the next release and `rem db migrate` creates it automatically.
+
+### Example: Downstream App Structure
+
+```
+myapp/
+├── main.py              # Import models, start API
+├── models/
+│   └── __init__.py      # @rem.register_model decorators
+├── sql/
+│   └── custom.sql       # Manual migrations for custom tables
+├── .env                 # POSTGRES__CONNECTION_STRING, LLM keys
+└── pyproject.toml       # dependencies = ["remdb>=0.3.110"]
+```
+
+```python
+# myapp/models/__init__.py
+import rem
+from rem.models.core import CoreModel
+
+@rem.register_model
+class ConversationSummary(CoreModel):
+    """Registered for introspection, but table created via sql/custom.sql"""
+    session_ref: str
+    summary: str
+```
+
+```python
+# myapp/main.py
+import models  # Registers custom models
+
+from rem.api.main import app  # Use REM's FastAPI app
+# Or build your own app using rem.services
 ```
 
 ## Configuration

rem/services/postgres/__init__.py

@@ -2,6 +2,7 @@
 PostgreSQL service for CloudNativePG database operations.
 """
 
+from .diff_service import DiffService, SchemaDiff
 from .repository import Repository
 from .service import PostgresService
 
@@ -20,4 +21,4 @@ def get_postgres_service() -> PostgresService | None:
     return PostgresService()
 
 
-__all__ = ["PostgresService", "get_postgres_service", "Repository"]
+__all__ = ["PostgresService", "get_postgres_service", "Repository", "DiffService", "SchemaDiff"]

rem/services/postgres/diff_service.py

@@ -0,0 +1,426 @@
+"""
+Schema diff service for comparing Pydantic models against database.
+
+Uses Alembic autogenerate to detect differences between:
+- Target schema (derived from Pydantic models)
+- Current database schema
+
+This enables:
+1. Local development: See what would change before applying migrations
+2. CI validation: Detect drift between code and database (--check mode)
+3. Migration generation: Create incremental migration files
+"""
+
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Optional
+import io
+
+from alembic.autogenerate import produce_migrations, render_python_code
+from alembic.operations import ops
+from alembic.runtime.migration import MigrationContext
+from alembic.script import ScriptDirectory
+from loguru import logger
+from sqlalchemy import create_engine, text
+
+from ...settings import settings
+from .pydantic_to_sqlalchemy import get_target_metadata
+
+
+# Tables that are NOT managed by Pydantic models (infrastructure tables)
+# These are created by 001_install.sql and should be excluded from diff
+INFRASTRUCTURE_TABLES = {
+    "kv_store",
+    "rem_migrations",
+    "rate_limits",
+    "persons",  # Legacy table - to be removed from DB
+}
+
+# Prefixes for tables that should be included in diff
+# (embeddings tables are created alongside entity tables)
+EMBEDDINGS_PREFIX = "embeddings_"
+
+
+@dataclass
+class SchemaDiff:
+    """Result of schema comparison."""
+
+    has_changes: bool
+    summary: list[str] = field(default_factory=list)
+    sql: str = ""
+    upgrade_ops: Optional[ops.UpgradeOps] = None
+
+    @property
+    def change_count(self) -> int:
+        """Total number of detected changes."""
+        return len(self.summary)
+
+
+class DiffService:
+    """
+    Service for comparing Pydantic models against database schema.
+
+    Uses Alembic's autogenerate machinery without creating revision files.
+    """
+
+    def __init__(self, models_dir: Optional[Path] = None):
+        """
+        Initialize diff service.
+
+        Args:
+            models_dir: Directory containing Pydantic models.
+                       If None, uses default rem/models/entities location.
+        """
+        self.models_dir = models_dir
+        self._metadata = None
+
+    def get_connection_url(self) -> str:
+        """Build PostgreSQL connection URL from settings using psycopg (v3) driver."""
+        pg = settings.postgres
+        # Use postgresql+psycopg to use psycopg v3 (not psycopg2)
+        url = f"postgresql+psycopg://{pg.user}"
+        if pg.password:
+            url += f":{pg.password}"
+        url += f"@{pg.host}:{pg.port}/{pg.database}"
+        return url
+
+    def get_target_metadata(self):
+        """Get SQLAlchemy metadata from Pydantic models."""
+        if self._metadata is None:
+            if self.models_dir:
+                from .pydantic_to_sqlalchemy import build_sqlalchemy_metadata_from_pydantic
+                self._metadata = build_sqlalchemy_metadata_from_pydantic(self.models_dir)
+            else:
+                self._metadata = get_target_metadata()
+        return self._metadata
+
+    def _include_object(self, obj, name, type_, reflected, compare_to) -> bool:
+        """
+        Filter function for Alembic autogenerate.
+
+        Excludes infrastructure tables that are not managed by Pydantic models.
+
+        Args:
+            obj: The schema object (Table, Column, Index, etc.)
+            name: Object name
+            type_: Object type ("table", "column", "index", etc.)
+            reflected: True if object exists in database
+            compare_to: The object being compared to (if any)
+
+        Returns:
+            True to include in diff, False to exclude
+        """
+        if type_ == "table":
+            # Exclude infrastructure tables
+            if name in INFRASTRUCTURE_TABLES:
+                return False
+            # Include embeddings tables (they're part of the model schema)
+            # These are now generated in pydantic_to_sqlalchemy
+        return True
+
+    def compute_diff(self) -> SchemaDiff:
+        """
+        Compare Pydantic models against database and return differences.
+
+        Returns:
+            SchemaDiff with detected changes
+        """
+        url = self.get_connection_url()
+        engine = create_engine(url)
+        metadata = self.get_target_metadata()
+
+        summary = []
+
+        with engine.connect() as conn:
+            # Create migration context for comparison
+            context = MigrationContext.configure(
+                conn,
+                opts={
+                    "target_metadata": metadata,
+                    "compare_type": True,
+                    "compare_server_default": False,  # Avoid false positives
+                    "include_schemas": False,
+                    "include_object": self._include_object,
+                },
+            )
+
+            # Run autogenerate comparison
+            migration_script = produce_migrations(context, metadata)
+            upgrade_ops = migration_script.upgrade_ops
+
+            # Process detected operations
+            if upgrade_ops and upgrade_ops.ops:
+                for op in upgrade_ops.ops:
+                    summary.extend(self._describe_operation(op))
+
+        has_changes = len(summary) > 0
+
+        # Generate SQL if there are changes
+        sql = ""
+        if has_changes and upgrade_ops:
+            sql = self._render_sql(upgrade_ops, engine)
+
+        return SchemaDiff(
+            has_changes=has_changes,
+            summary=summary,
+            sql=sql,
+            upgrade_ops=upgrade_ops,
+        )
+
+    def _describe_operation(self, op: ops.MigrateOperation, prefix: str = "") -> list[str]:
+        """Convert Alembic operation to human-readable description."""
+        descriptions = []
+
+        if isinstance(op, ops.CreateTableOp):
+            descriptions.append(f"{prefix}+ CREATE TABLE {op.table_name}")
+            for col in op.columns:
+                if hasattr(col, 'name'):
+                    descriptions.append(f"{prefix}    + column {col.name}")
+
+        elif isinstance(op, ops.DropTableOp):
+            descriptions.append(f"{prefix}- DROP TABLE {op.table_name}")
+
+        elif isinstance(op, ops.AddColumnOp):
+            col_type = str(op.column.type) if op.column.type else "unknown"
+            descriptions.append(f"{prefix}+ ADD COLUMN {op.table_name}.{op.column.name} ({col_type})")
+
+        elif isinstance(op, ops.DropColumnOp):
+            descriptions.append(f"{prefix}- DROP COLUMN {op.table_name}.{op.column_name}")
+
+        elif isinstance(op, ops.AlterColumnOp):
+            changes = []
+            if op.modify_type is not None:
+                changes.append(f"type -> {op.modify_type}")
+            if op.modify_nullable is not None:
+                nullable = "NULL" if op.modify_nullable else "NOT NULL"
+                changes.append(f"nullable -> {nullable}")
+            if op.modify_server_default is not None:
+                changes.append(f"default -> {op.modify_server_default}")
+            change_str = ", ".join(changes) if changes else "modified"
+            descriptions.append(f"{prefix}~ ALTER COLUMN {op.table_name}.{op.column_name} ({change_str})")
+
+        elif isinstance(op, ops.CreateIndexOp):
+            # op.columns can be strings or Column objects
+            if op.columns:
+                cols = ", ".join(
+                    c if isinstance(c, str) else getattr(c, 'name', str(c))
+                    for c in op.columns
+                )
+            else:
+                cols = "?"
+            descriptions.append(f"{prefix}+ CREATE INDEX {op.index_name} ON {op.table_name} ({cols})")
+
+        elif isinstance(op, ops.DropIndexOp):
+            descriptions.append(f"{prefix}- DROP INDEX {op.index_name}")
+
+        elif isinstance(op, ops.CreateForeignKeyOp):
+            descriptions.append(f"{prefix}+ CREATE FK {op.constraint_name} ON {op.source_table}")
+
+        elif isinstance(op, ops.DropConstraintOp):
+            descriptions.append(f"{prefix}- DROP CONSTRAINT {op.constraint_name} ON {op.table_name}")
+
+        elif isinstance(op, ops.ModifyTableOps):
+            # Container for multiple operations on same table
+            descriptions.append(f"{prefix}Table: {op.table_name}")
+            for sub_op in op.ops:
+                descriptions.extend(self._describe_operation(sub_op, prefix + "  "))
+
+        else:
+            descriptions.append(f"{prefix}? {type(op).__name__}")
+
+        return descriptions
+
+    def _render_sql(self, upgrade_ops: ops.UpgradeOps, engine) -> str:
+        """Render upgrade operations as SQL statements."""
+        from alembic.runtime.migration import MigrationContext
+        from alembic.operations import Operations
+
+        sql_lines = []
+
+        # Use offline mode to generate SQL
+        buffer = io.StringIO()
+
+        def emit_sql(text, *args, **kwargs):
+            sql_lines.append(str(text))
+
+        with engine.connect() as conn:
+            context = MigrationContext.configure(
+                conn,
+                opts={
+                    "as_sql": True,
+                    "output_buffer": buffer,
+                    "target_metadata": self.get_target_metadata(),
+                },
+            )
+
+            with context.begin_transaction():
+                operations = Operations(context)
+                for op in upgrade_ops.ops:
+                    self._execute_op(operations, op)
+
+        return buffer.getvalue()
+
+    def _execute_op(self, operations: "Operations", op: ops.MigrateOperation):
+        """Execute a single operation via Operations proxy."""
+        from alembic.operations import Operations
+        from alembic.autogenerate import rewriter
+
+        if isinstance(op, ops.CreateTableOp):
+            operations.create_table(
+                op.table_name,
+                *op.columns,
+                schema=op.schema,
+                **op.kw,
+            )
+        elif isinstance(op, ops.DropTableOp):
+            operations.drop_table(op.table_name, schema=op.schema)
+        elif isinstance(op, ops.AddColumnOp):
+            operations.add_column(op.table_name, op.column, schema=op.schema)
+        elif isinstance(op, ops.DropColumnOp):
+            operations.drop_column(op.table_name, op.column_name, schema=op.schema)
+        elif isinstance(op, ops.AlterColumnOp):
+            operations.alter_column(
+                op.table_name,
+                op.column_name,
+                nullable=op.modify_nullable,
+                type_=op.modify_type,
+                server_default=op.modify_server_default,
+                schema=op.schema,
+            )
+        elif isinstance(op, ops.CreateIndexOp):
+            operations.create_index(
+                op.index_name,
+                op.table_name,
+                op.columns,
+                schema=op.schema,
+                unique=op.unique,
+                **op.kw,
+            )
+        elif isinstance(op, ops.DropIndexOp):
+            operations.drop_index(op.index_name, table_name=op.table_name, schema=op.schema)
+        elif isinstance(op, ops.ModifyTableOps):
+            for sub_op in op.ops:
+                self._execute_op(operations, sub_op)
+
+    def generate_migration_file(
+        self,
+        output_dir: Path,
+        message: str = "auto_migration",
+    ) -> Optional[Path]:
+        """
+        Generate a numbered migration file from the diff.
+
+        Args:
+            output_dir: Directory to write migration file
+            message: Migration description (used in filename)
+
+        Returns:
+            Path to generated file, or None if no changes
+        """
+        diff = self.compute_diff()
+
+        if not diff.has_changes:
+            logger.info("No schema changes detected")
+            return None
+
+        # Find next migration number
+        existing = sorted(output_dir.glob("*.sql"))
+        next_num = 1
+        for f in existing:
+            try:
+                num = int(f.stem.split("_")[0])
+                next_num = max(next_num, num + 1)
+            except (ValueError, IndexError):
+                pass
+
+        # Generate filename
+        safe_message = message.replace(" ", "_").replace("-", "_")[:40]
+        filename = f"{next_num:03d}_{safe_message}.sql"
+        output_path = output_dir / filename
+
+        # Write SQL
+        header = f"""-- Migration: {message}
+-- Generated by: rem db diff --generate
+-- Changes detected: {diff.change_count}
+--
+-- Review this file before applying!
+-- Apply with: rem db migrate
+--
+
+"""
+        # Build SQL from operations
+        sql_content = self._build_migration_sql(diff)
+
+        output_path.write_text(header + sql_content)
+        logger.info(f"Generated migration: {output_path}")
+
+        return output_path
+
+    def _build_migration_sql(self, diff: SchemaDiff) -> str:
+        """Build SQL from diff operations."""
+        if not diff.upgrade_ops or not diff.upgrade_ops.ops:
+            return "-- No changes\n"
+
+        lines = []
+        for op in diff.upgrade_ops.ops:
+            lines.extend(self._op_to_sql(op))
+
+        return "\n".join(lines) + "\n"
+
+    def _op_to_sql(self, op: ops.MigrateOperation) -> list[str]:
+        """Convert operation to SQL statements."""
+        lines = []
+
+        if isinstance(op, ops.CreateTableOp):
+            cols = []
+            for col in op.columns:
+                if hasattr(col, 'name') and hasattr(col, 'type'):
+                    nullable = "" if getattr(col, 'nullable', True) else " NOT NULL"
+                    cols.append(f"    {col.name} {col.type}{nullable}")
+            col_str = ",\n".join(cols)
+            lines.append(f"CREATE TABLE IF NOT EXISTS {op.table_name} (\n{col_str}\n);")
+
+        elif isinstance(op, ops.DropTableOp):
+            lines.append(f"DROP TABLE IF EXISTS {op.table_name};")
+
+        elif isinstance(op, ops.AddColumnOp):
+            col = op.column
+            nullable = "" if getattr(col, 'nullable', True) else " NOT NULL"
+            lines.append(f"ALTER TABLE {op.table_name} ADD COLUMN IF NOT EXISTS {col.name} {col.type}{nullable};")
+
+        elif isinstance(op, ops.DropColumnOp):
+            lines.append(f"ALTER TABLE {op.table_name} DROP COLUMN IF EXISTS {op.column_name};")
+
+        elif isinstance(op, ops.AlterColumnOp):
+            if op.modify_type is not None:
+                lines.append(f"ALTER TABLE {op.table_name} ALTER COLUMN {op.column_name} TYPE {op.modify_type};")
+            if op.modify_nullable is not None:
+                if op.modify_nullable:
+                    lines.append(f"ALTER TABLE {op.table_name} ALTER COLUMN {op.column_name} DROP NOT NULL;")
+                else:
+                    lines.append(f"ALTER TABLE {op.table_name} ALTER COLUMN {op.column_name} SET NOT NULL;")
+
+        elif isinstance(op, ops.CreateIndexOp):
+            # op.columns can be strings or Column objects
+            if op.columns:
+                cols = ", ".join(
+                    c if isinstance(c, str) else getattr(c, 'name', str(c))
+                    for c in op.columns
+                )
+            else:
+                cols = ""
+            unique = "UNIQUE " if op.unique else ""
+            lines.append(f"CREATE {unique}INDEX IF NOT EXISTS {op.index_name} ON {op.table_name} ({cols});")
+
+        elif isinstance(op, ops.DropIndexOp):
+            lines.append(f"DROP INDEX IF EXISTS {op.index_name};")
+
+        elif isinstance(op, ops.ModifyTableOps):
+            lines.append(f"-- Changes to table: {op.table_name}")
+            for sub_op in op.ops:
+                lines.extend(self._op_to_sql(sub_op))
+
+        else:
+            lines.append(f"-- Unsupported operation: {type(op).__name__}")
+
+        return lines