remdb 0.2.6__py3-none-any.whl

remdb-0.2.6.dist-info/METADATA

@@ -0,0 +1,1191 @@
+Metadata-Version: 2.4
+Name: remdb
+Version: 0.2.6
+Summary: Resources Entities Moments - Bio-inspired memory system for agentic AI workloads
+Project-URL: Homepage, https://github.com/mr-saoirse/remstack
+Project-URL: Documentation, https://github.com/mr-saoirse/remstack/blob/main/README.md
+Project-URL: Repository, https://github.com/mr-saoirse/remstack
+Project-URL: Issues, https://github.com/mr-saoirse/remstack/issues
+Author-email: mr-saoirse <amartey@gmail.com>
+License: MIT
+Keywords: agents,ai,mcp,memory,postgresql,vector-search
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.12
+Requires-Dist: aioboto3>=13.0.0
+Requires-Dist: arize-phoenix>=5.0.0
+Requires-Dist: asyncpg>=0.30.0
+Requires-Dist: boto3>=1.35.0
+Requires-Dist: click>=8.1.0
+Requires-Dist: fastapi>=0.115.0
+Requires-Dist: fastmcp>=0.5.0
+Requires-Dist: gitpython>=3.1.45
+Requires-Dist: gmft
+Requires-Dist: hypercorn>=0.17.0
+Requires-Dist: itsdangerous>=2.0.0
+Requires-Dist: json-schema-to-pydantic>=0.2.0
+Requires-Dist: kreuzberg[gmft]>=3.21.0
+Requires-Dist: loguru>=0.7.0
+Requires-Dist: openinference-instrumentation-pydantic-ai>=0.1.0
+Requires-Dist: opentelemetry-api>=1.28.0
+Requires-Dist: opentelemetry-exporter-otlp-proto-grpc>=1.28.0
+Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.28.0
+Requires-Dist: opentelemetry-exporter-otlp>=1.28.0
+Requires-Dist: opentelemetry-instrumentation-fastapi>=0.49b0
+Requires-Dist: opentelemetry-instrumentation>=0.49b0
+Requires-Dist: opentelemetry-sdk>=1.28.0
+Requires-Dist: psycopg[binary]>=3.2.0
+Requires-Dist: pydantic-ai>=0.0.14
+Requires-Dist: pydantic-settings>=2.6.0
+Requires-Dist: pydantic>=2.10.0
+Requires-Dist: pydub>=0.25.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: pyyaml>=6.0.0
+Requires-Dist: requests>=2.32.0
+Requires-Dist: semchunk>=2.2.0
+Requires-Dist: tenacity>=9.0.0
+Requires-Dist: tiktoken>=0.5.0
+Requires-Dist: uvicorn[standard]>=0.32.0
+Provides-Extra: all
+Requires-Dist: ipdb>=0.13.0; extra == 'all'
+Requires-Dist: ipython>=8.29.0; extra == 'all'
+Requires-Dist: json-schema-to-pydantic>=0.2.0; extra == 'all'
+Requires-Dist: mypy>=1.13.0; extra == 'all'
+Requires-Dist: pandas-stubs>=2.0.0; extra == 'all'
+Requires-Dist: pillow>=10.0.0; extra == 'all'
+Requires-Dist: polars>=1.0.0; extra == 'all'
+Requires-Dist: pydub>=0.25.0; extra == 'all'
+Requires-Dist: pytest-asyncio>=0.24.0; extra == 'all'
+Requires-Dist: pytest-cov>=6.0.0; extra == 'all'
+Requires-Dist: pytest-mock>=3.14.0; extra == 'all'
+Requires-Dist: pytest>=8.0.0; extra == 'all'
+Requires-Dist: ruff>=0.8.0; extra == 'all'
+Requires-Dist: types-pyyaml>=6.0.0; extra == 'all'
+Provides-Extra: audio
+Requires-Dist: pydub>=0.25.0; extra == 'audio'
+Provides-Extra: dev
+Requires-Dist: ipdb>=0.13.0; extra == 'dev'
+Requires-Dist: ipython>=8.29.0; extra == 'dev'
+Requires-Dist: mypy>=1.13.0; extra == 'dev'
+Requires-Dist: pandas-stubs>=2.0.0; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.24.0; extra == 'dev'
+Requires-Dist: pytest-cov>=6.0.0; extra == 'dev'
+Requires-Dist: pytest-mock>=3.14.0; extra == 'dev'
+Requires-Dist: pytest>=8.0.0; extra == 'dev'
+Requires-Dist: ruff>=0.8.0; extra == 'dev'
+Requires-Dist: types-pyyaml>=6.0.0; extra == 'dev'
+Provides-Extra: fs
+Requires-Dist: pillow>=10.0.0; extra == 'fs'
+Requires-Dist: polars>=1.0.0; extra == 'fs'
+Requires-Dist: pydub>=0.25.0; extra == 'fs'
+Provides-Extra: schema
+Requires-Dist: json-schema-to-pydantic>=0.2.0; extra == 'schema'
+Description-Content-Type: text/markdown
+
+# REM - Resources Entities Moments
+
+Cloud-native unified memory infrastructure for agentic AI systems built with Pydantic AI, FastAPI, and FastMCP.
+
+## Architecture Overview
+
+```mermaid
+graph TD
+    API[FastAPI<br/>Chat + MCP] --> AGENTS[JSON Schema<br/>Agents]
+    AGENTS --> TOOLS[MCP Tools<br/>5 Tools]
+
+    TOOLS --> QUERY[REM Query<br/>Dialect]
+    QUERY --> DB[(PostgreSQL<br/>+pgvector)]
+
+    FILES[File Processor] --> DREAM[Dreaming<br/>Workers]
+    DREAM --> DB
+
+    AGENTS --> OTEL[OpenTelemetry]
+    OTEL --> PHOENIX[Arize<br/>Phoenix]
+
+    EVAL[Evaluation<br/>Framework] --> PHOENIX
+
+    classDef api fill:#4A90E2,stroke:#2E5C8A,color:#fff
+    classDef agent fill:#7B68EE,stroke:#483D8B,color:#fff
+    classDef db fill:#50C878,stroke:#2E7D4E,color:#fff
+    classDef obs fill:#9B59B6,stroke:#6C3483,color:#fff
+
+    class API,TOOLS api
+    class AGENTS agent
+    class DB,QUERY db
+    class OTEL,PHOENIX,EVAL obs
+```
+
+**Key Components:**
+
+- **API Layer**: OpenAI-compatible chat completions + MCP server (not separate deployments)
+- **Agentic Framework**: JSON Schema-based agents with no-code configuration
+- **Database Layer**: PostgreSQL 18 with pgvector for multi-index memory (KV + Vector + Graph)
+- **REM Query Dialect**: Custom query language with O(1) lookups, semantic search, graph traversal
+- **Ingestion & Dreaming**: Background workers for content extraction and progressive index enrichment (0% → 100% answerable)
+- **Observability & Evals**: OpenTelemetry tracing + Arize Phoenix + LLM-as-a-Judge evaluation framework
+
+## Features
+
+| Feature | Description | Benefits |
+|---------|-------------|----------|
+| **OpenAI-Compatible Chat API** | Drop-in replacement for OpenAI chat completions API with streaming support | Use with existing OpenAI clients, switch models across providers (OpenAI, Anthropic, etc.) |
+| **Built-in MCP Server** | FastMCP server with 5 tools + 3 resources for memory operations | Export memory to Claude Desktop, Cursor, or any MCP-compatible host |
+| **REM Query Engine** | Multi-index query system (LOOKUP, FUZZY, SEARCH, SQL, TRAVERSE) with custom dialect | O(1) lookups, semantic search, graph traversal - all tenant-isolated |
+| **Dreaming Workers** | Background workers for entity extraction, moment generation, and affinity matching | Automatic knowledge graph construction from resources (0% → 100% query answerable) |
+| **PostgreSQL + pgvector** | CloudNativePG with PostgreSQL 18, pgvector extension, streaming replication | Production-ready vector search, no external vector DB needed |
+| **AWS EKS Recipe** | Complete infrastructure-as-code with Pulumi, Karpenter, ArgoCD | Deploy to production EKS in minutes with auto-scaling and GitOps |
+| **JSON Schema Agents** | Dynamic agent creation from YAML schemas via Pydantic AI factory | Define agents declaratively, version control schemas, load dynamically |
+| **Content Providers** | Audio transcription (Whisper), vision (GPT-4V, Claude), PDFs, DOCX, images | Multimodal ingestion out of the box with format detection |
+| **Configurable Embeddings** | Provider-agnostic embedding system (OpenAI, Cohere, Jina) | Switch embedding providers via env vars, no code changes |
+| **Multi-Tenancy** | Tenant isolation at database level with automatic scoping | SaaS-ready with complete data separation per tenant |
+| **Streaming Everything** | SSE for chat, background workers for embeddings, async throughout | Real-time responses, non-blocking operations, scalable |
+| **Zero Vendor Lock-in** | Raw HTTP clients (no OpenAI SDK), swappable providers, open standards | Not tied to any vendor, easy to migrate, full control |
+
+## Quick Start
+
+Choose your path:
+
+- **Option 1: Package Users** (Recommended for non-developers) - PyPI package + dockerized database
+- **Option 2: Developers** - Clone repo, local development with uv
+
+---
+
+## Option 1: Package Users (Recommended)
+
+**Best for**: Using REM as a service (API + CLI) without modifying code.
+
+### Step 1: Start Database and API with Docker Compose
+
+```bash
+# Create a project directory
+mkdir my-rem-project && cd my-rem-project
+
+# Download docker-compose file from public gist
+curl -O https://gist.githubusercontent.com/percolating-sirsh/d117b673bc0edfdef1a5068ccd3cf3e5/raw/docker-compose.prebuilt.yml
+
+# IMPORTANT: Export API keys BEFORE running docker compose
+# Docker Compose reads env vars at startup - exporting them after won't work!
+
+# Required: OpenAI for embeddings (text-embedding-3-small)
+export OPENAI_API_KEY="sk-..."
+
+# Recommended: At least one chat completion provider
+export ANTHROPIC_API_KEY="sk-ant-..."           # Claude Sonnet 4.5 (high quality)
+export CEREBRAS_API_KEY="csk-..."               # Cerebras (fast, cheap inference)
+
+# Start PostgreSQL + API
+docker compose -f docker-compose.prebuilt.yml up -d
+
+# Verify services are running
+curl http://localhost:8000/health
+```
+
+This starts:
+- **PostgreSQL** with pgvector on port **5051** (connection: `postgresql://rem:rem@localhost:5051/rem`)
+- **REM API** on port **8000** with OpenAI-compatible chat completions + MCP server
+- Uses pre-built Docker image from Docker Hub (no local build required)
+
+### Step 2: Install and Configure CLI (REQUIRED)
+
+**This step is required** before you can use REM - it installs the database schema and configures your LLM API keys.
+
+```bash
+# Install remdb package from PyPI
+pip install remdb[all]
+
+# Configure REM (defaults to port 5051 for package users)
+rem configure --install --claude-desktop
+```
+
+The interactive wizard will:
+1. **Configure PostgreSQL**: Defaults to `postgresql://rem:rem@localhost:5051/rem` (prebuilt docker-compose)
+   - Just press Enter to accept defaults
+   - Custom database: Enter your own host/port/credentials
+2. **Configure LLM providers**: Enter your OpenAI/Anthropic API keys
+3. **Install database tables**: Creates schema, functions, indexes (**required for CLI/API to work**)
+4. **Register with Claude Desktop**: Adds REM MCP server to Claude
+
+Configuration saved to `~/.rem/config.yaml` (can edit with `rem configure --edit`)
+
+**Port Guide:**
+- **5051**: Package users with `docker-compose.prebuilt.yml` (pre-built image)
+- **5050**: Developers with `docker-compose.yml` (local build)
+- **Custom**: Your own PostgreSQL database
+
+**Next Steps:**
+- See [CLI Reference](#cli-reference) for all available commands
+- See [REM Query Dialect](#rem-query-dialect) for query examples
+- See [API Endpoints](#api-endpoints) for OpenAI-compatible API usage
+
+### Step 3: Test the Stack
+
+```bash
+# Ingest a test file to populate your knowledge base
+echo "REM is a bio-inspired memory system for agentic AI workloads." > test-doc.txt
+rem process ingest test-doc.txt --user-id test-user --category documentation --tags rem,ai
+
+# Query your ingested data
+rem ask "What do you know about REM from my knowledge base?" --user-id test-user
+
+# Test with a general query (uses agent's built-in knowledge + your data)
+rem ask "What is REM?" --user-id test-user
+
+# Test the API
+curl -X POST http://localhost:8000/api/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -H "X-User-Id: test-user" \
+  -d '{
+    "model": "anthropic:claude-sonnet-4-5-20250929",
+    "messages": [{"role": "user", "content": "What is REM?"}],
+    "stream": false
+  }'
+```
+
+**File Ingestion Commands:**
+- `rem process ingest <file>` - Full ingestion pipeline (storage + parsing + embedding + database)
+- `rem process uri <file>` - READ-ONLY parsing (no database storage, useful for testing parsers)
+
+
+## See Also
+
+- [REM Query Dialect](#rem-query-dialect) - LOOKUP, SEARCH, TRAVERSE, SQL query types
+- [API Endpoints](#api-endpoints) - OpenAI-compatible chat completions, MCP server
+- [CLI Reference](#cli-reference) - Complete command-line interface documentation
+- [Production Deployment](#production-deployment) - AWS EKS with Kubernetes
+
+**Sample Data**: Test data with users, resources, and moments is at `tests/data/seed/test-user-data.yaml`
+
+---
+
+## REM Query Dialect
+
+REM provides a custom query language designed for **LLM-driven iterated retrieval** with performance guarantees.
+
+### Design Philosophy
+
+Unlike traditional single-shot SQL queries, the REM dialect is optimized for **multi-turn exploration** where LLMs participate in query planning:
+
+- **Iterated Queries**: Queries return partial results that LLMs use to refine subsequent queries
+- **Composable WITH Syntax**: Chain operations together (e.g., `TRAVERSE FROM ... WITH LOOKUP "..."`)
+- **Mixed Indexes**: Combines exact lookups (O(1)), semantic search (vector), and graph traversal
+- **Query Planner Participation**: Results include metadata for LLMs to decide next steps
+
+**Example Multi-Turn Flow**:
+```
+Turn 1: LOOKUP "sarah-chen" → Returns entity + available edge types
+Turn 2: TRAVERSE FROM "sarah-chen" TYPE "authored_by" DEPTH 1 → Returns connected documents
+Turn 3: SEARCH "architecture decisions" WITH TRAVERSE FROM "sarah-chen" → Combines semantic + graph
+```
+
+This enables LLMs to **progressively build context** rather than requiring perfect queries upfront.
+
+See [REM Query Dialect (AST)](#rem-query-dialect-ast) for complete grammar specification.
+
+### Query Types
+
+#### `LOOKUP` - O(1) Exact Label Lookup
+
+Fast exact match on entity labels (natural language identifiers, not UUIDs).
+
+```sql
+LOOKUP "sarah-chen" FROM resources
+LOOKUP "api-design-v2" FROM resources WHERE category = "projects"
+```
+
+**Performance**: O(1) - indexed on `label` column
+**Returns**: Single entity or null
+**Use case**: Fetch specific known entities by human-readable name
+
+#### `FUZZY` - Fuzzy Text Search
+
+Fuzzy matching for partial names or misspellings using PostgreSQL trigram similarity.
+
+```sql
+FUZZY "sara" FROM resources LIMIT 10
+FUZZY "api desgin" FROM resources THRESHOLD 0.3 LIMIT 5
+```
+
+**Performance**: O(n) with pg_trgm GIN index (fast for small-medium datasets)
+**Returns**: Ranked list by similarity score
+**Use case**: Handle typos, partial names, or when exact label is unknown
+
+#### `SEARCH` - Semantic Vector Search
+
+Semantic search using pgvector embeddings with cosine similarity.
+
+```sql
+SEARCH "machine learning architecture" FROM resources LIMIT 10
+SEARCH "contract disputes" FROM resources WHERE tags @> ARRAY['legal'] LIMIT 5
+```
+
+**Performance**: O(log n) with HNSW index
+**Returns**: Ranked list of semantically similar entities
+**Use case**: Find conceptually related content without exact keyword matches
+
+#### `TRAVERSE` - Recursive Graph Traversal
+
+Follow `graph_edges` relationships across the knowledge graph.
+
+```sql
+TRAVERSE FROM "sarah-chen" TYPE "authored_by" DEPTH 2
+TRAVERSE FROM "api-design-v2" TYPE "references,depends_on" DEPTH 3
+```
+
+**Features**:
+- **Polymorphic**: Seamlessly traverses `resources`, `moments`, `users` via `all_graph_edges` view
+- **Filtering**: Filter by one or multiple edge types (comma-separated)
+- **Depth Control**: Configurable recursion depth (default: 2)
+- **Data Model**: Requires `InlineEdge` JSON structure in `graph_edges` column
+
+**Returns**: Graph of connected entities with edge metadata
+**Use case**: Explore relationships, find connected entities, build context
+
+#### Direct SQL Queries
+
+Raw SQL for complex temporal, aggregation, or custom queries.
+
+```sql
+SELECT * FROM resources WHERE created_at > NOW() - INTERVAL '7 days' ORDER BY created_at DESC LIMIT 20
+SELECT category, COUNT(*) as count FROM resources GROUP BY category
+WITH recent AS (SELECT * FROM resources WHERE created_at > NOW() - INTERVAL '1 day') SELECT * FROM recent
+```
+
+**Performance**: Depends on query and indexes
+**Returns**: Raw query results
+**Use case**: Complex filtering, aggregations, temporal queries
+**Allowed**: SELECT, INSERT, UPDATE, WITH (read + data modifications)
+**Blocked**: DROP, DELETE, TRUNCATE, ALTER (destructive operations)
+**Note**: Can be used standalone or with `WITH` syntax for composition
+
+### Graph Edge Format
+
+Edges stored inline using `InlineEdge` pattern with human-readable destination labels.
+
+```json
+{
+  "dst": "sarah-chen",
+  "rel_type": "authored_by",
+  "weight": 1.0,
+  "properties": {
+    "dst_entity_type": "users:engineers/sarah-chen",
+    "created_at": "2025-01-15T10:30:00Z"
+  }
+}
+```
+
+**Destination Entity Type Convention** (`properties.dst_entity_type`):
+
+Format: `<table_schema>:<category>/<key>`
+
+Examples:
+- `"resources:managers/bob"` → Look up bob in resources table with category="managers"
+- `"users:engineers/sarah-chen"` → Look up sarah-chen in users table
+- `"moments:meetings/standup-2024-01"` → Look up in moments table
+- `"resources/api-design-v2"` → Look up in resources table (no category)
+- `"bob"` → Defaults to resources table, no category
+
+**Edge Type Format** (`rel_type`):
+- Use snake_case: `"authored_by"`, `"depends_on"`, `"references"`
+- Be specific but consistent
+- Use passive voice for bidirectional clarity
+
+### Multi-Turn Iterated Retrieval
+
+REM enables agents to conduct multi-turn database conversations:
+
+1. **Initial Query**: Agent runs SEARCH to find candidates
+2. **Refinement**: Agent analyzes results, runs LOOKUP on specific entities
+3. **Context Expansion**: Agent runs TRAVERSE to find related entities
+4. **Temporal Filter**: Agent runs SQL to filter by time range
+5. **Final Answer**: Agent synthesizes knowledge from all queries
+
+**Plan Memos**: Agents track query plans in scratchpad for iterative refinement.
+
+### Query Performance Contracts
+
+| Query Type | Complexity | Index | Use When |
+|------------|-----------|-------|----------|
+| `LOOKUP` | O(1) | B-tree on `label` | You know exact entity name |
+| `FUZZY` | O(n) | GIN on `label` (pg_trgm) | Handling typos/partial matches |
+| `SEARCH` | O(log n) | HNSW on `embedding` | Semantic similarity needed |
+| `TRAVERSE` | O(depth × edges) | B-tree on `graph_edges` | Exploring relationships |
+| `SQL` | Variable | Custom indexes | Complex filtering/aggregation |
+
+### Example: Multi-Query Session
+
+```python
+# Query 1: Find relevant documents
+SEARCH "API migration planning" FROM resources LIMIT 5
+
+# Query 2: Get specific document
+LOOKUP "tidb-migration-spec" FROM resources
+
+# Query 3: Find related people
+TRAVERSE FROM "tidb-migration-spec" TYPE "authored_by,reviewed_by" DEPTH 1
+
+# Query 4: Recent activity
+SELECT * FROM moments WHERE
+    'tidb-migration' = ANY(topic_tags) AND
+    start_time > NOW() - INTERVAL '30 days'
+```
+
+### Tenant Isolation
+
+All queries automatically scoped by `user_id` for complete data isolation:
+
+```sql
+-- Automatically filtered to user's data
+SEARCH "contracts" FROM resources LIMIT 10
+
+-- No cross-user data leakage
+TRAVERSE FROM "project-x" TYPE "references" DEPTH 3
+```
+
+## API Endpoints
+
+### Chat Completions (OpenAI-compatible)
+
+```bash
+POST /api/v1/chat/completions
+```
+
+**Headers**:
+- `X-Tenant-Id`: Tenant identifier (required for REM)
+- `X-User-Id`: User identifier
+- `X-Session-Id`: Session/conversation identifier
+- `X-Agent-Schema`: Agent schema URI to use
+
+**Body**:
+```json
+{
+  "model": "anthropic:claude-sonnet-4-5-20250929",
+  "messages": [
+    {"role": "user", "content": "Find all documents Sarah authored"}
+  ],
+  "stream": true,
+  "response_format": {"type": "text"}
+}
+```
+
+**Streaming Response** (SSE):
+```
+data: {"id": "chatcmpl-123", "choices": [{"delta": {"role": "assistant", "content": ""}}]}
+
+data: {"id": "chatcmpl-123", "choices": [{"delta": {"content": "[Calling: search_rem]"}}]}
+
+data: {"id": "chatcmpl-123", "choices": [{"delta": {"content": "Found 3 documents..."}}]}
+
+data: {"id": "chatcmpl-123", "choices": [{"delta": {}, "finish_reason": "stop"}]}
+
+data: [DONE]
+```
+
+### MCP Endpoint
+
+```bash
+# MCP HTTP transport
+POST /api/v1/mcp
+```
+
+Tools and resources for REM query execution, resource management, file operations.
+
+### Health Check
+
+```bash
+GET /health
+# {"status": "healthy", "version": "0.1.0"}
+```
+
+## CLI Reference
+
+REM provides a comprehensive command-line interface for all operations.
+
+### Configuration & Server
+
+#### `rem configure` - Interactive Setup Wizard
+
+Set up REM with PostgreSQL, LLM providers, and S3 storage. **Defaults to port 5051 (package users).**
+
+```bash
+# Complete setup (recommended for package users)
+rem configure --install --claude-desktop
+
+# This runs:
+# 1. Interactive wizard (creates ~/.rem/config.yaml)
+# 2. Installs database tables (rem db migrate)
+# 3. Registers REM MCP server with Claude Desktop
+
+# Other options:
+rem configure                  # Just run wizard
+rem configure --install        # Wizard + database install
+rem configure --show           # Show current configuration
+rem configure --edit           # Edit configuration in $EDITOR
+```
+
+**Default Configuration:**
+- **Package users**: `localhost:5051` (docker-compose.prebuilt.yml with Docker Hub image)
+- **Developers**: Change to `localhost:5050` during wizard (docker-compose.yml with local build)
+- **Custom database**: Enter your own host/port/credentials
+
+**Configuration File:** `~/.rem/config.yaml`
+
+```yaml
+postgres:
+  # Package users (prebuilt)
+  connection_string: postgresql://rem:rem@localhost:5051/rem
+  # OR Developers (local build)
+  # connection_string: postgresql://rem:rem@localhost:5050/rem
+  pool_min_size: 5
+  pool_max_size: 20
+
+llm:
+  default_model: anthropic:claude-sonnet-4-5-20250929
+  openai_api_key: sk-...
+  anthropic_api_key: sk-ant-...
+
+s3:
+  bucket_name: rem-storage
+  region: us-east-1
+```
+
+**Precedence:** Environment variables > Config file > Defaults
+
+**Port Guide:**
+- **5051**: Package users with `docker-compose.prebuilt.yml` (recommended)
+- **5050**: Developers with `docker-compose.yml` (local development)
+- **Custom**: Your own PostgreSQL instance
+
+#### `rem mcp` - Run MCP Server
+
+Run the FastMCP server for Claude Desktop integration.
+
+```bash
+# Stdio mode (for Claude Desktop)
+rem mcp
+
+# HTTP mode (for testing)
+rem mcp --http --port 8001
+```
+
+#### `rem serve` - Start API Server
+
+Start the FastAPI server with uvicorn.
+
+```bash
+# Use settings from config
+rem serve
+
+# Development mode (auto-reload)
+rem serve --reload
+
+# Production mode (4 workers)
+rem serve --workers 4
+
+# Bind to all interfaces
+rem serve --host 0.0.0.0 --port 8080
+
+# Override log level
+rem serve --log-level debug
+```
+
+### Database Management
+
+#### `rem db migrate` - Run Migrations
+
+Apply database migrations (install.sql and install_models.sql).
+
+```bash
+# Apply all migrations
+rem db migrate
+
+# Core infrastructure only (extensions, functions)
+rem db migrate --install
+
+# Entity tables only (Resource, Message, etc.)
+rem db migrate --models
+
+# Background indexes (HNSW for vectors)
+rem db migrate --background-indexes
+
+# Custom connection string
+rem db migrate --connection "postgresql://user:pass@host:5432/db"
+
+# Custom SQL directory
+rem db migrate --sql-dir /path/to/sql
+```
+
+#### `rem db status` - Migration Status
+
+Show applied migrations and execution times.
+
+```bash
+rem db status
+```
+
+#### `rem db rebuild-cache` - Rebuild KV Cache
+
+Rebuild KV_STORE cache from entity tables (after database restart or bulk imports).
+
+```bash
+rem db rebuild-cache
+```
+
+### Schema Management
+
+#### `rem db schema generate` - Generate SQL Schema
+
+Generate database schema from Pydantic models.
+
+```bash
+# Generate install_models.sql from entity models
+rem db schema generate \
+  --models src/rem/models/entities \
+  --output rem/src/rem/sql/install_models.sql
+
+# Generate migration file
+rem db schema generate \
+  --models src/rem/models/entities \
+  --output rem/src/rem/sql/migrations/003_add_fields.sql
+```
+
+#### `rem db schema indexes` - Generate Background Indexes
+
+Generate SQL for background index creation (HNSW for vectors).
+
+```bash
+# Generate background_indexes.sql
+rem db schema indexes \
+  --models src/rem/models/entities \
+  --output rem/src/rem/sql/background_indexes.sql
+```
+
+#### `rem db schema validate` - Validate Models
+
+Validate Pydantic models for schema generation.
+
+```bash
+rem db schema validate --models src/rem/models/entities
+```
+
+### File Processing
+
+#### `rem process files` - Process Files
+
+Process files with optional custom extractor (ontology extraction).
+
+```bash
+# Process all completed files for tenant
+rem process files \
+  --tenant-id acme-corp \
+  --status completed \
+  --limit 10
+
+# Process with custom extractor
+rem process files \
+  --tenant-id acme-corp \
+  --extractor cv-parser-v1 \
+  --limit 50
+
+# Process files from the last 7 days
+rem process files \
+  --tenant-id acme-corp \
+  --lookback-hours 168
+```
+
+#### `rem process ingest` - Ingest File into REM
+
+Ingest a file into REM with full pipeline (storage + parsing + embedding + database).
+
+```bash
+# Ingest local file
+rem process ingest /path/to/document.pdf \
+  --user-id user-123 \
+  --category legal \
+  --tags contract,2024
+
+# Ingest with minimal options
+rem process ingest ./meeting-notes.md --user-id user-123
+```
+
+#### `rem process uri` - Parse File (Read-Only)
+
+Parse a file and extract content **without** storing to database (useful for testing parsers).
+
+```bash
+# Parse local file (output to stdout)
+rem process uri /path/to/document.pdf
+
+# Parse and save extracted content to file
+rem process uri /path/to/document.pdf --save output.json
+
+# Parse S3 file
+rem process uri s3://bucket/key.docx --output text
+```
+
+### Memory & Knowledge Extraction (Dreaming)
+
+#### `rem dreaming full` - Complete Workflow
+
+Run full dreaming workflow: extractors → moments → affinity → user model.
+
+```bash
+# Full workflow for user
+rem dreaming full \
+  --user-id user-123 \
+  --tenant-id acme-corp
+
+# Skip ontology extractors
+rem dreaming full \
+  --user-id user-123 \
+  --tenant-id acme-corp \
+  --skip-extractors
+
+# Process last 24 hours only
+rem dreaming full \
+  --user-id user-123 \
+  --tenant-id acme-corp \
+  --lookback-hours 24
+
+# Limit resources processed
+rem dreaming full \
+  --user-id user-123 \
+  --tenant-id acme-corp \
+  --limit 100
+```
+
+#### `rem dreaming custom` - Custom Extractor
+
+Run specific ontology extractor on user's data.
+
+```bash
+# Run CV parser on user's files
+rem dreaming custom \
+  --user-id user-123 \
+  --tenant-id acme-corp \
+  --extractor cv-parser-v1
+
+# Process last week's files
+rem dreaming custom \
+  --user-id user-123 \
+  --tenant-id acme-corp \
+  --extractor contract-analyzer-v1 \
+  --lookback-hours 168 \
+  --limit 50
+```
+
+#### `rem dreaming moments` - Extract Moments
+
+Extract temporal narratives from resources.
+
+```bash
+# Generate moments for user
+rem dreaming moments \
+  --user-id user-123 \
+  --tenant-id acme-corp \
+  --limit 50
+
+# Process last 7 days
+rem dreaming moments \
+  --user-id user-123 \
+  --tenant-id acme-corp \
+  --lookback-hours 168
+```
+
+#### `rem dreaming affinity` - Build Relationships
+
+Build semantic relationships between resources using embeddings.
+
+```bash
+# Build affinity graph for user
+rem dreaming affinity \
+  --user-id user-123 \
+  --tenant-id acme-corp \
+  --limit 100
+
+# Process recent resources only
+rem dreaming affinity \
+  --user-id user-123 \
+  --tenant-id acme-corp \
+  --lookback-hours 24
+```
+
+#### `rem dreaming user-model` - Update User Model
+
+Update user model from recent activity (preferences, interests, patterns).
+
+```bash
+# Update user model
+rem dreaming user-model \
+  --user-id user-123 \
+  --tenant-id acme-corp
+```
+
+### Evaluation & Experiments
+
+#### `rem experiments` - Experiment Management
+
+Manage evaluation experiments with datasets, prompts, and traces.
+
+```bash
+# Create experiment configuration
+rem experiments create my-evaluation \
+  --agent ask_rem \
+  --evaluator rem-lookup-correctness \
+  --description "Baseline evaluation"
+
+# Run experiment
+rem experiments run my-evaluation
+
+# List experiments
+rem experiments list
+rem experiments show my-evaluation
+```
+
+#### `rem experiments dataset` - Dataset Management
+
+```bash
+# Create dataset from CSV
+rem experiments dataset create rem-lookup-golden \
+  --from-csv golden.csv \
+  --input-keys query \
+  --output-keys expected_label,expected_type
+
+# Add more examples
+rem experiments dataset add rem-lookup-golden \
+  --from-csv more-data.csv \
+  --input-keys query \
+  --output-keys expected_label,expected_type
+
+# List datasets
+rem experiments dataset list
+```
+
+#### `rem experiments prompt` - Prompt Management
+
+```bash
+# Create agent prompt
+rem experiments prompt create hello-world \
+  --system-prompt "You are a helpful assistant." \
+  --model-name gpt-4o
+
+# List prompts
+rem experiments prompt list
+```
+
+#### `rem experiments trace` - Trace Retrieval
+
+```bash
+# List recent traces
+rem experiments trace list --project rem-agents --days 7 --limit 50
+```
+
+#### `rem experiments` - Experiment Config
+
+Manage experiment configurations (A/B testing, parameter sweeps).
+
+```bash
+# Create experiment config
+rem experiments create \
+  --name cv-parser-test \
+  --description "Test CV parser with different models"
+
+# List experiments
+rem experiments list
+
+# Show experiment details
+rem experiments show cv-parser-test
+
+# Run experiment
+rem experiments run cv-parser-test
+```
+
+### Interactive Agent
+
+#### `rem ask` - Test Agent
+
+Test Pydantic AI agent with natural language queries.
+
+```bash
+# Ask a question
+rem ask "What documents did Sarah Chen author?"
+
+# With context headers
+rem ask "Find all resources about API design" \
+  --user-id user-123 \
+  --tenant-id acme-corp
+
+# Use specific agent schema
+rem ask "Analyze this contract" \
+  --agent-schema contract-analyzer-v1
+```
+
+### Global Options
+
+All commands support:
+
+```bash
+# Verbose logging
+rem --verbose <command>
+rem -v <command>
+
+# Version
+rem --version
+
+# Help
+rem --help
+rem <command> --help
+rem <command> <subcommand> --help
+```
+
+### Environment Variables
+
+Override any setting via environment variables:
+
+```bash
+# Database
+export POSTGRES__CONNECTION_STRING=postgresql://rem:rem@localhost:5432/rem
+export POSTGRES__POOL_MIN_SIZE=5
+
+# LLM
+export LLM__DEFAULT_MODEL=openai:gpt-4o
+export LLM__OPENAI_API_KEY=sk-...
+export LLM__ANTHROPIC_API_KEY=sk-ant-...
+
+# S3
+export S3__BUCKET_NAME=rem-storage
+export S3__REGION=us-east-1
+
+# Server
+export API__HOST=0.0.0.0
+export API__PORT=8000
+export API__RELOAD=true
+
+# Run command with overrides
+rem serve
+```
+
+## Development (For Contributors)
+
+**Best for**: Contributing to REM or customizing the codebase.
+
+### Step 1: Clone Repository
+
+```bash
+git clone https://github.com/mr-saoirse/remstack.git
+cd remstack/rem
+```
+
+### Step 2: Start PostgreSQL Only
+
+```bash
+# Start only PostgreSQL (port 5050 for developers, doesn't conflict with package users on 5051)
+docker compose up postgres -d
+
+# Verify connection
+psql -h localhost -p 5050 -U rem -d rem -c "SELECT version();"
+```
+
+### Step 3: Set Up Development Environment
+
+```bash
+# IMPORTANT: If you previously installed the package and ran `rem configure`,
+# delete the REM configuration directory to avoid conflicts:
+rm -rf ~/.rem/
+
+# Create virtual environment with uv
+uv venv
+source .venv/bin/activate  # On Windows: .venv\Scripts\activate
+
+# Install in editable mode with all dependencies
+uv pip install -e ".[all]"
+
+# Set LLM API keys
+export OPENAI_API_KEY="sk-..."
+export ANTHROPIC_API_KEY="sk-ant-..."
+export POSTGRES__CONNECTION_STRING="postgresql://rem:rem@localhost:5050/rem"
+
+# Verify CLI
+rem --version
+```
+
+### Step 4: Initialize Database
+
+```bash
+# Apply migrations
+rem db migrate
+
+# Verify tables
+psql -h localhost -p 5050 -U rem -d rem -c "\dt"
+```
+
+### Step 5: Run API Server (Optional)
+
+```bash
+# Start API server with hot reload
+uv run python -m rem.api.main
+
+# API runs on http://localhost:8000
+```
+
+### Step 6: Run Tests
+
+```bash
+# Run non-LLM tests (fast, no API costs)
+uv run pytest tests/integration/ -m "not llm" -v
+
+# Run all tests (uses API credits)
+uv run pytest tests/integration/ -v
+
+# Type check (saves report to .mypy/ folder)
+../scripts/run_mypy.sh
+```
+
+Type checking reports are saved to `.mypy/report_YYYYMMDD_HHMMSS.txt` (gitignored).
+Current status: 222 errors in 55 files (as of 2025-11-23).
+
+### Environment Variables
+
+All settings via environment variables with `__` delimiter:
+
+```bash
+# LLM
+LLM__DEFAULT_MODEL=anthropic:claude-sonnet-4-5-20250929
+LLM__DEFAULT_TEMPERATURE=0.5
+
+# Auth (disabled by default)
+AUTH__ENABLED=false
+AUTH__OIDC_ISSUER_URL=https://accounts.google.com
+
+# OTEL (disabled by default for local dev)
+OTEL__ENABLED=false
+OTEL__SERVICE_NAME=rem-api
+
+# Postgres
+POSTGRES__CONNECTION_STRING=postgresql://rem:rem@localhost:5050/rem
+
+# S3
+S3__BUCKET_NAME=rem-storage
+S3__REGION=us-east-1
+```
+
+### Production Deployment (Optional)
+
+For production deployment to AWS EKS with Kubernetes, see the main repository README:
+- **Infrastructure**: [../../manifests/infra/pulumi/eks-yaml/README.md](../../manifests/infra/pulumi/eks-yaml/README.md)
+- **Platform**: [../../manifests/platform/README.md](../../manifests/platform/README.md)
+- **Application**: [../../manifests/application/README.md](../../manifests/application/README.md)
+
+
+## REM Query Dialect (AST)
+
+REM queries follow a structured dialect with formal grammar specification.
+
+### Grammar
+
+```
+Query ::= LookupQuery | FuzzyQuery | SearchQuery | SqlQuery | TraverseQuery
+
+LookupQuery ::= LOOKUP <key:string|list[string]>
+  key         : Single entity name or list of entity names (natural language labels)
+  performance : O(1) per key
+  available   : Stage 1+
+  examples    :
+    - LOOKUP "Sarah"
+    - LOOKUP ["Sarah", "Mike", "Emily"]
+    - LOOKUP "Project Alpha"
+
+FuzzyQuery ::= FUZZY <text:string> [THRESHOLD <t:float>] [LIMIT <n:int>]
+  text        : Search text (partial/misspelled)
+  threshold   : Similarity score 0.0-1.0 (default: 0.5)
+  limit       : Max results (default: 5)
+  performance : Indexed (pg_trgm)
+  available   : Stage 1+
+  example     : FUZZY "sara" THRESHOLD 0.5 LIMIT 10
+
+SearchQuery ::= SEARCH <text:string> [TABLE <table:string>] [WHERE <clause:string>] [LIMIT <n:int>]
+  text        : Semantic query text
+  table       : Target table (default: "resources")
+  clause      : Optional PostgreSQL WHERE clause for hybrid filtering (combines vector + structured)
+  limit       : Max results (default: 10)
+  performance : Indexed (pgvector)
+  available   : Stage 3+
+  examples    :
+    - SEARCH "database migration" TABLE resources LIMIT 10
+    - SEARCH "team discussion" TABLE moments WHERE "moment_type='meeting'" LIMIT 5
+    - SEARCH "project updates" WHERE "created_at >= '2024-01-01'" LIMIT 20
+    - SEARCH "AI research" WHERE "tags @> ARRAY['machine-learning']" LIMIT 10
+
+  Hybrid Query Support: SEARCH combines semantic vector similarity with structured filtering.
+  Use WHERE clause to filter on system fields or entity-specific fields.
+
+SqlQuery ::= <raw_sql:string>
+           | SQL <table:string> [WHERE <clause:string>] [ORDER BY <order:string>] [LIMIT <n:int>]
+
+  Mode 1 (Raw SQL - Recommended):
+    Any query not starting with a REM keyword (LOOKUP, FUZZY, SEARCH, TRAVERSE) is treated as raw SQL.
+    Allowed: SELECT, INSERT, UPDATE, WITH (read + data modifications)
+    Blocked: DROP, DELETE, TRUNCATE, ALTER (destructive operations)
+
+  Mode 2 (Structured - Legacy):
+    SQL prefix with table + WHERE clause (automatic tenant isolation)
+
+  performance : O(n) with indexes
+  available   : Stage 1+
+  dialect     : PostgreSQL (full PostgreSQL syntax support)
+
+  examples    :
+    # Raw SQL (no prefix needed)
+    - SELECT * FROM resources WHERE created_at > NOW() - INTERVAL '7 days' LIMIT 20
+    - SELECT category, COUNT(*) as count FROM resources GROUP BY category
+    - WITH recent AS (SELECT * FROM resources WHERE created_at > NOW() - INTERVAL '1 day') SELECT * FROM recent
+
+    # Structured SQL (legacy, automatic tenant isolation)
+    - SQL moments WHERE "moment_type='meeting'" ORDER BY starts_timestamp DESC LIMIT 10
+    - SQL resources WHERE "metadata->>'status' = 'published'" LIMIT 20
+
+  PostgreSQL Dialect: Full support for:
+  - JSONB operators (->>, ->, @>, etc.)
+  - Array operators (&&, @>, <@, etc.)
+  - CTEs (WITH clauses)
+  - Advanced filtering and aggregations
+
+TraverseQuery ::= TRAVERSE [<edge_types:list>] WITH <initial_query:Query> [DEPTH <d:int>] [ORDER BY <order:string>] [LIMIT <n:int>]
+  edge_types    : Relationship types to follow (e.g., ["manages", "reports-to"], default: all)
+  initial_query : Starting query (typically LOOKUP)
+  depth         : Number of hops (0=PLAN mode, 1=single hop, N=multi-hop, default: 1)
+  order         : Order results (default: "edge.created_at DESC")
+  limit         : Max nodes (default: 9)
+  performance   : O(k) where k = visited nodes
+  available     : Stage 3+
+  examples      :
+    - TRAVERSE manages WITH LOOKUP "Sally" DEPTH 1
+    - TRAVERSE WITH LOOKUP "Sally" DEPTH 0  (PLAN mode: edge analysis only)
+    - TRAVERSE manages,reports-to WITH LOOKUP "Sarah" DEPTH 2 LIMIT 5
+```
+
+### Query Availability by Evolution Stage
+
+| Query Type | Stage 0 | Stage 1 | Stage 2 | Stage 3 | Stage 4 |
+|------------|---------|---------|---------|---------|---------|
+| LOOKUP     | ✗       | ✓       | ✓       | ✓       | ✓       |
+| FUZZY      | ✗       | ✓       | ✓       | ✓       | ✓       |
+| SEARCH     | ✗       | ✗       | ✗       | ✓       | ✓       |
+| SQL        | ✗       | ✓       | ✓       | ✓       | ✓       |
+| TRAVERSE   | ✗       | ✗       | ✗       | ✓       | ✓       |
+
+**Stage 0**: No data, all queries fail.
+
+**Stage 1** (20% answerable): Resources seeded with entity extraction. LOOKUP and FUZZY work for finding entities. SQL works for basic filtering.
+
+**Stage 2** (50% answerable): Moments extracted. SQL temporal queries work. LOOKUP includes moment entities.
+
+**Stage 3** (80% answerable): Affinity graph built. SEARCH and TRAVERSE become available. Multi-hop graph queries work.
+
+**Stage 4** (100% answerable): Mature graph with rich historical data. All query types fully functional with high-quality results.
+
+## License
+
+MIT