remdb 0.2.6__py3-none-any.whl → 0.3.103__py3-none-any.whl

{remdb-0.2.6.dist-info → remdb-0.3.103.dist-info}/METADATA

@@ -1,11 +1,11 @@
 Metadata-Version: 2.4
 Name: remdb
-Version: 0.2.6
+Version: 0.3.103
 Summary: Resources Entities Moments - Bio-inspired memory system for agentic AI workloads
-Project-URL: Homepage, https://github.com/mr-saoirse/remstack
-Project-URL: Documentation, https://github.com/mr-saoirse/remstack/blob/main/README.md
-Project-URL: Repository, https://github.com/mr-saoirse/remstack
-Project-URL: Issues, https://github.com/mr-saoirse/remstack/issues
+Project-URL: Homepage, https://github.com/Percolation-Labs/reminiscent
+Project-URL: Documentation, https://github.com/Percolation-Labs/reminiscent/blob/main/README.md
+Project-URL: Repository, https://github.com/Percolation-Labs/reminiscent
+Project-URL: Issues, https://github.com/Percolation-Labs/reminiscent/issues
 Author-email: mr-saoirse <amartey@gmail.com>
 License: MIT
 Keywords: agents,ai,mcp,memory,postgresql,vector-search
@@ -14,7 +14,7 @@ Classifier: Intended Audience :: Developers
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
-Requires-Python: >=3.12
+Requires-Python: <3.13,>=3.12
 Requires-Dist: aioboto3>=13.0.0
 Requires-Dist: arize-phoenix>=5.0.0
 Requires-Dist: asyncpg>=0.30.0
@@ -23,11 +23,10 @@ Requires-Dist: click>=8.1.0
 Requires-Dist: fastapi>=0.115.0
 Requires-Dist: fastmcp>=0.5.0
 Requires-Dist: gitpython>=3.1.45
-Requires-Dist: gmft
 Requires-Dist: hypercorn>=0.17.0
 Requires-Dist: itsdangerous>=2.0.0
 Requires-Dist: json-schema-to-pydantic>=0.2.0
-Requires-Dist: kreuzberg[gmft]>=3.21.0
+Requires-Dist: kreuzberg<4.0.0,>=3.21.0
 Requires-Dist: loguru>=0.7.0
 Requires-Dist: openinference-instrumentation-pydantic-ai>=0.1.0
 Requires-Dist: opentelemetry-api>=1.28.0
@@ -91,32 +90,9 @@ Cloud-native unified memory infrastructure for agentic AI systems built with Pyd
 
 ## Architecture Overview
 
-```mermaid
-graph TD
-    API[FastAPI<br/>Chat + MCP] --> AGENTS[JSON Schema<br/>Agents]
-    AGENTS --> TOOLS[MCP Tools<br/>5 Tools]
-
-    TOOLS --> QUERY[REM Query<br/>Dialect]
-    QUERY --> DB[(PostgreSQL<br/>+pgvector)]
-
-    FILES[File Processor] --> DREAM[Dreaming<br/>Workers]
-    DREAM --> DB
-
-    AGENTS --> OTEL[OpenTelemetry]
-    OTEL --> PHOENIX[Arize<br/>Phoenix]
-
-    EVAL[Evaluation<br/>Framework] --> PHOENIX
-
-    classDef api fill:#4A90E2,stroke:#2E5C8A,color:#fff
-    classDef agent fill:#7B68EE,stroke:#483D8B,color:#fff
-    classDef db fill:#50C878,stroke:#2E7D4E,color:#fff
-    classDef obs fill:#9B59B6,stroke:#6C3483,color:#fff
-
-    class API,TOOLS api
-    class AGENTS agent
-    class DB,QUERY db
-    class OTEL,PHOENIX,EVAL obs
-```
+<p align="center">
+  <img src="https://mermaid.ink/img/Z3JhcGggVEQKICAgIEFQSVtGYXN0QVBJPGJyLz5DaGF0ICsgTUNQXSAtLT4gQUdFTlRTW0pTT04gU2NoZW1hPGJyLz5BZ2VudHNdCiAgICBBR0VOVFMgLS0-IFRPT0xTW01DUCBUb29sczxici8-NSBUb29sc10KCiAgICBUT09MUyAtLT4gUVVFUllbUkVNIFF1ZXJ5PGJyLz5EaWFsZWN0XQogICAgUVVFUlkgLS0-IERCWyhQb3N0Z3JlU1FMPGJyLz4rcGd2ZWN0b3IpXQoKICAgIEZJTEVTW0ZpbGUgUHJvY2Vzc29yXSAtLT4gRFJFQU1bRHJlYW1pbmc8YnIvPldvcmtlcnNdCiAgICBEUkVBTSAtLT4gREIKCiAgICBBR0VOVFMgLS0-IE9URUxbT3BlblRlbGVtZXRyeV0KICAgIE9URUwgLS0-IFBIT0VOSVhbQXJpemU8YnIvPlBob2VuaXhdCgogICAgRVZBTFtFdmFsdWF0aW9uPGJyLz5GcmFtZXdvcmtdIC0tPiBQSE9FTklYCgogICAgY2xhc3NEZWYgYXBpIGZpbGw6IzRBOTBFMixzdHJva2U6IzJFNUM4QSxjb2xvcjojZmZmCiAgICBjbGFzc0RlZiBhZ2VudCBmaWxsOiM3QjY4RUUsc3Ryb2tlOiM0ODNEOEIsY29sb3I6I2ZmZgogICAgY2xhc3NEZWYgZGIgZmlsbDojNTBDODc4LHN0cm9rZTojMkU3RDRFLGNvbG9yOiNmZmYKICAgIGNsYXNzRGVmIG9icyBmaWxsOiM5QjU5QjYsc3Ryb2tlOiM2QzM0ODMsY29sb3I6I2ZmZgoKICAgIGNsYXNzIEFQSSxUT09MUyBhcGkKICAgIGNsYXNzIEFHRU5UUyBhZ2VudAogICAgY2xhc3MgREIsUVVFUlkgZGIKICAgIGNsYXNzIE9URUwsUEhPRU5JWCxFVkFMIG9icwo=" alt="REM Architecture" width="700">
+</p>
 
 **Key Components:**
 
@@ -125,37 +101,91 @@ graph TD
 - **Database Layer**: PostgreSQL 18 with pgvector for multi-index memory (KV + Vector + Graph)
 - **REM Query Dialect**: Custom query language with O(1) lookups, semantic search, graph traversal
 - **Ingestion & Dreaming**: Background workers for content extraction and progressive index enrichment (0% → 100% answerable)
-- **Observability & Evals**: OpenTelemetry tracing + Arize Phoenix + LLM-as-a-Judge evaluation framework
+- **Observability & Evals**: OpenTelemetry tracing supporting LLM-as-a-Judge evaluation frameworks
 
 ## Features
 
 | Feature | Description | Benefits |
 |---------|-------------|----------|
 | **OpenAI-Compatible Chat API** | Drop-in replacement for OpenAI chat completions API with streaming support | Use with existing OpenAI clients, switch models across providers (OpenAI, Anthropic, etc.) |
-| **Built-in MCP Server** | FastMCP server with 5 tools + 3 resources for memory operations | Export memory to Claude Desktop, Cursor, or any MCP-compatible host |
+| **Built-in MCP Server** | FastMCP server with 4 tools + 5 resources for memory operations | Export memory to Claude Desktop, Cursor, or any MCP-compatible host |
 | **REM Query Engine** | Multi-index query system (LOOKUP, FUZZY, SEARCH, SQL, TRAVERSE) with custom dialect | O(1) lookups, semantic search, graph traversal - all tenant-isolated |
 | **Dreaming Workers** | Background workers for entity extraction, moment generation, and affinity matching | Automatic knowledge graph construction from resources (0% → 100% query answerable) |
 | **PostgreSQL + pgvector** | CloudNativePG with PostgreSQL 18, pgvector extension, streaming replication | Production-ready vector search, no external vector DB needed |
 | **AWS EKS Recipe** | Complete infrastructure-as-code with Pulumi, Karpenter, ArgoCD | Deploy to production EKS in minutes with auto-scaling and GitOps |
 | **JSON Schema Agents** | Dynamic agent creation from YAML schemas via Pydantic AI factory | Define agents declaratively, version control schemas, load dynamically |
-| **Content Providers** | Audio transcription (Whisper), vision (GPT-4V, Claude), PDFs, DOCX, images | Multimodal ingestion out of the box with format detection |
-| **Configurable Embeddings** | Provider-agnostic embedding system (OpenAI, Cohere, Jina) | Switch embedding providers via env vars, no code changes |
+| **Content Providers** | Audio transcription (Whisper), vision (OpenAI, Anthropic, Gemini), PDFs, DOCX, PPTX, XLSX, images | Multimodal ingestion out of the box with format detection |
+| **Configurable Embeddings** | OpenAI embedding system (text-embedding-3-small) | Production-ready embeddings, additional providers planned |
 | **Multi-Tenancy** | Tenant isolation at database level with automatic scoping | SaaS-ready with complete data separation per tenant |
-| **Streaming Everything** | SSE for chat, background workers for embeddings, async throughout | Real-time responses, non-blocking operations, scalable |
 | **Zero Vendor Lock-in** | Raw HTTP clients (no OpenAI SDK), swappable providers, open standards | Not tied to any vendor, easy to migrate, full control |
 
 ## Quick Start
 
 Choose your path:
 
-- **Option 1: Package Users** (Recommended for non-developers) - PyPI package + dockerized database
-- **Option 2: Developers** - Clone repo, local development with uv
+- **Option 1: Package Users with Example Data** (Recommended for first-time users) - PyPI + example datasets
+- **Option 2: Package Users** (Recommended for non-developers) - PyPI package + dockerized database
+- **Option 3: Developers** - Clone repo, local development with uv
 
 ---
 
-## Option 1: Package Users (Recommended)
+## Option 1: Package Users with Example Data (Recommended)
+
+**Best for**: First-time users who want to explore REM with curated example datasets.
+
+```bash
+# Install system dependencies (tesseract for OCR)
+brew install tesseract  # macOS (Linux/Windows: see tesseract-ocr.github.io)
+
+# Install remdb
+pip install "remdb[all]"
 
-**Best for**: Using REM as a service (API + CLI) without modifying code.
+# Clone example datasets
+git clone https://github.com/Percolation-Labs/remstack-lab.git
+cd remstack-lab
+
+# Optional: Set default LLM provider via environment variable
+# export LLM__DEFAULT_MODEL="openai:gpt-4.1-nano"  # Fast and cheap
+# export LLM__DEFAULT_MODEL="anthropic:claude-sonnet-4-5-20250929"  # High quality (default)
+
+# Start PostgreSQL with docker-compose
+curl -O https://gist.githubusercontent.com/percolating-sirsh/d117b673bc0edfdef1a5068ccd3cf3e5/raw/docker-compose.prebuilt.yml
+docker compose -f docker-compose.prebuilt.yml up -d postgres
+
+# Configure REM (creates ~/.rem/config.yaml and installs database schema)
+# Add --claude-desktop to register with Claude Desktop app
+rem configure --install --claude-desktop
+
+# Load quickstart dataset (uses default user)
+rem db load datasets/quickstart/sample_data.yaml
+
+# Ask questions
+rem ask "What documents exist in the system?"
+rem ask "Show me meetings about API design"
+
+# Ingest files (PDF, DOCX, images, etc.) - note: requires remstack-lab
+rem process ingest datasets/formats/files/bitcoin_whitepaper.pdf --category research --tags bitcoin,whitepaper
+
+# Query ingested content
+rem ask "What is the Bitcoin whitepaper about?"
+
+# Try other datasets (use --user-id for multi-tenant scenarios)
+rem db load datasets/domains/recruitment/scenarios/candidate_pipeline/data.yaml --user-id acme-corp
+rem ask --user-id acme-corp "Show me candidates with Python experience"
+```
+
+**What you get:**
+- Quickstart: 3 users, 3 resources, 3 moments, 4 messages
+- Domain datasets: recruitment, legal, enterprise, misc
+- Format examples: engrams, documents, conversations, files
+
+**Learn more**: [remstack-lab repository](https://github.com/Percolation-Labs/remstack-lab)
+
+---
+
+## Option 2: Package Users (No Example Data)
+
+**Best for**: Using REM as a service (API + CLI) without modifying code, bringing your own data.
 
 ### Step 1: Start Database and API with Docker Compose
 
@@ -220,43 +250,277 @@ Configuration saved to `~/.rem/config.yaml` (can edit with `rem configure --edit
 - See [REM Query Dialect](#rem-query-dialect) for query examples
 - See [API Endpoints](#api-endpoints) for OpenAI-compatible API usage
 
-### Step 3: Test the Stack
+### Step 3: Load Sample Data (Optional but Recommended)
+
+**Option A: Clone example datasets** (Recommended - works with all README examples)
+
+```bash
+# Clone datasets repository
+git clone https://github.com/Percolation-Labs/remstack-lab.git
+
+# Load quickstart dataset (uses default user)
+rem db load --file remstack-lab/datasets/quickstart/sample_data.yaml
+
+# Test with sample queries
+rem ask "What documents exist in the system?"
+rem ask "Show me meetings about API design"
+rem ask "Who is Sarah Chen?"
+
+# Try domain-specific datasets (use --user-id for multi-tenant scenarios)
+rem db load --file remstack-lab/datasets/domains/recruitment/scenarios/candidate_pipeline/data.yaml --user-id acme-corp
+rem ask --user-id acme-corp "Show me candidates with Python experience"
+```
+
+**Option B: Bring your own data**
 
 ```bash
-# Ingest a test file to populate your knowledge base
+# Ingest your own files (uses default user)
 echo "REM is a bio-inspired memory system for agentic AI workloads." > test-doc.txt
-rem process ingest test-doc.txt --user-id test-user --category documentation --tags rem,ai
+rem process ingest test-doc.txt --category documentation --tags rem,ai
 
 # Query your ingested data
-rem ask "What do you know about REM from my knowledge base?" --user-id test-user
+rem ask "What do you know about REM from my knowledge base?"
+```
 
-# Test with a general query (uses agent's built-in knowledge + your data)
-rem ask "What is REM?" --user-id test-user
+### Step 4: Test the API
 
-# Test the API
+```bash
+# Test the OpenAI-compatible chat completions API
 curl -X POST http://localhost:8000/api/v1/chat/completions \
   -H "Content-Type: application/json" \
-  -H "X-User-Id: test-user" \
+  -H "X-User-Id: demo-user" \
   -d '{
     "model": "anthropic:claude-sonnet-4-5-20250929",
-    "messages": [{"role": "user", "content": "What is REM?"}],
+    "messages": [{"role": "user", "content": "What documents did Sarah Chen author?"}],
     "stream": false
   }'
 ```
 
-**File Ingestion Commands:**
+**Available Commands:**
+- `rem ask` - Natural language queries to REM
 - `rem process ingest <file>` - Full ingestion pipeline (storage + parsing + embedding + database)
 - `rem process uri <file>` - READ-ONLY parsing (no database storage, useful for testing parsers)
+- `rem db load --file <yaml>` - Load structured datasets directly
+
+## Example Datasets
+
+🎯 **Recommended**: Clone [remstack-lab](https://github.com/Percolation-Labs/remstack-lab) for curated datasets organized by domain and format.
+
+**What's included:**
+- **Quickstart**: Minimal dataset (3 users, 3 resources, 3 moments) - perfect for first-time users
+- **Domains**: Recruitment (CV parsing), Legal (contracts), Enterprise (team collaboration)
+- **Formats**: Engrams (voice memos), Documents (markdown/PDF), Conversations (chat logs)
+- **Evaluation**: Golden datasets for Phoenix-based agent testing
 
+**Working from remstack-lab:**
+```bash
+cd remstack-lab
+
+# Load any dataset (uses default user)
+rem db load --file datasets/quickstart/sample_data.yaml
+
+# Explore formats
+rem db load --file datasets/formats/engrams/scenarios/team_meeting/team_standup_meeting.yaml
+
+# Try domain-specific examples (use --user-id for multi-tenant scenarios)
+rem db load --file datasets/domains/recruitment/scenarios/candidate_pipeline/data.yaml --user-id acme-corp
+```
 
 ## See Also
 
 - [REM Query Dialect](#rem-query-dialect) - LOOKUP, SEARCH, TRAVERSE, SQL query types
 - [API Endpoints](#api-endpoints) - OpenAI-compatible chat completions, MCP server
 - [CLI Reference](#cli-reference) - Complete command-line interface documentation
+- [Bring Your Own Agent](#bring-your-own-agent) - Create custom agents with your own prompts and tools
 - [Production Deployment](#production-deployment) - AWS EKS with Kubernetes
+- [Example Datasets](https://github.com/Percolation-Labs/remstack-lab) - Curated datasets by domain and format
+
+---
+
+## Bring Your Own Agent
+
+REM allows you to create **custom agents** with your own system prompts, tools, and output schemas. Custom agents are stored in the database and dynamically loaded when referenced, enabling **no-code agent creation** without modifying the codebase.
+
+### How It Works
+
+1. **Define Agent Schema** - Create a YAML file with your agent's prompt, tools, and output structure
+2. **Ingest Schema** - Use `rem process ingest` to store the schema in the database
+3. **Use Your Agent** - Reference your agent by name with `rem ask <agent-name> "query"`
+
+When you run `rem ask my-agent "query"`, REM:
+1. Checks if `my-agent` exists in the filesystem (`schemas/agents/`)
+2. If not found, performs a **LOOKUP** query on the `schemas` table in the database
+3. Loads the schema dynamically and creates a Pydantic AI agent
+4. Runs your query with the custom agent
+
+### Expected Behavior
+
+**Schema Ingestion Flow** (`rem process ingest my-agent.yaml`):
+- Parse YAML file to extract JSON Schema content
+- Extract `json_schema_extra.kind` field → maps to `category` column
+- Extract `json_schema_extra.provider_configs` → stores provider configurations
+- Extract `json_schema_extra.embedding_fields` → stores semantic search fields
+- Create `Schema` entity in `schemas` table with `user_id` scoping
+- Schema is now queryable via `LOOKUP "my-agent" FROM schemas`
+
+**Agent Loading Flow** (`rem ask my-agent "query"`):
+1. `load_agent_schema("my-agent")` checks filesystem cache → miss
+2. Falls back to database: `LOOKUP "my-agent" FROM schemas WHERE user_id = '<user-id>'`
+3. Returns `Schema.spec` (JSON Schema dict) from database
+4. `create_agent()` factory creates Pydantic AI agent from schema
+5. Agent runs with tools specified in `json_schema_extra.tools`
+6. Returns structured output defined in `properties` field
+
+### Quick Example
+
+**Step 1: Create Agent Schema** (`my-research-assistant.yaml`)
+
+```yaml
+type: object
+description: |
+  You are a research assistant that helps users find and analyze documents.
+
+  Use the search_rem tool to find relevant documents, then analyze and summarize them.
+  Be concise and cite specific documents in your responses.
+
+properties:
+  summary:
+    type: string
+    description: A concise summary of findings
+  sources:
+    type: array
+    items:
+      type: string
+    description: List of document labels referenced
+
+required:
+  - summary
+  - sources
+
+json_schema_extra:
+  kind: agent
+  name: research-assistant
+  version: 1.0.0
+  tools:
+    - search_rem
+    - ask_rem_agent
+  resources: []
+```
+
+**For more examples**, see:
+- Simple agent (no tools): `src/rem/schemas/agents/examples/simple.yaml`
+- Agent with REM tools: `src/rem/schemas/agents/core/rem-query-agent.yaml`
+- Ontology extractor: `src/rem/schemas/agents/examples/cv-parser.yaml`
+
+**Step 2: Ingest Schema into Database**
+
+```bash
+# Ingest the schema (stores in database schemas table)
+rem process ingest my-research-assistant.yaml \
+  --category agents \
+  --tags custom,research
+
+# Verify schema is in database (should show schema details)
+rem ask "LOOKUP 'my-research-assistant' FROM schemas"
+```
+
+**Step 3: Use Your Custom Agent**
+
+```bash
+# Run a query with your custom agent
+rem ask research-assistant "Find documents about machine learning architecture"
+
+# With streaming
+rem ask research-assistant "Summarize recent API design documents" --stream
+
+# With session continuity
+rem ask research-assistant "What did we discuss about ML?" --session-id abc-123
+```
+
+### Agent Schema Structure
+
+Every agent schema must include:
+
+**Required Fields:**
+- `type: object` - JSON Schema type (always "object")
+- `description` - System prompt with instructions for the agent
+- `properties` - Output schema defining structured response fields
+
+**Optional Metadata** (`json_schema_extra`):
+- `kind` - Agent category ("agent", "evaluator", etc.) → maps to `Schema.category`
+- `name` - Agent identifier (used for LOOKUP)
+- `version` - Semantic version (e.g., "1.0.0")
+- `tools` - List of MCP tools to load (e.g., `["search_rem", "lookup_rem"]`)
+- `resources` - List of MCP resources to expose (e.g., `["user_profile"]`)
+- `provider_configs` - Multi-provider testing configurations (for ontology extractors)
+- `embedding_fields` - Fields to embed for semantic search (for ontology extractors)
+
+### Available MCP Tools
+
+REM provides **4 built-in MCP tools** your agents can use:
+
+| Tool | Purpose | Parameters |
+|------|---------|------------|
+| `search_rem` | Execute REM queries (LOOKUP, FUZZY, SEARCH, SQL, TRAVERSE) | `query_type`, `entity_key`, `query_text`, `table`, `sql_query`, `initial_query`, `edge_types`, `depth` |
+| `ask_rem_agent` | Natural language to REM query via agent-driven reasoning | `query`, `agent_schema`, `agent_version` |
+| `ingest_into_rem` | Full file ingestion pipeline (read → store → parse → chunk → embed) | `file_uri`, `category`, `tags`, `is_local_server` |
+| `read_resource` | Access MCP resources (schemas, status) for Claude Desktop | `uri` |
+
+**Tool Reference**: Tools are defined in `src/rem/api/mcp_router/tools.py`
+
+**Note**: `search_rem` is a unified tool that handles all REM query types via the `query_type` parameter:
+- `query_type="lookup"` - O(1) entity lookup by label
+- `query_type="fuzzy"` - Fuzzy text matching with similarity threshold
+- `query_type="search"` - Semantic vector search (table-specific)
+- `query_type="sql"` - Direct SQL queries (WHERE clause)
+- `query_type="traverse"` - Graph traversal with depth control
 
-**Sample Data**: Test data with users, resources, and moments is at `tests/data/seed/test-user-data.yaml`
+### Multi-User Isolation
+
+Custom agents are **scoped by `user_id`**, ensuring complete data isolation:
+
+```bash
+# User A creates a custom agent
+rem process ingest my-agent.yaml --user-id user-a --category agents
+
+# User B cannot see User A's agent
+rem ask my-agent "test" --user-id user-b
+# ❌ Error: Schema not found (LOOKUP returns no results for user-b)
+
+# User A can use their agent
+rem ask my-agent "test" --user-id user-a
+# ✅ Works - LOOKUP finds schema for user-a
+```
+
+### Advanced: Ontology Extractors
+
+Custom agents can also be used as **ontology extractors** to extract structured knowledge from files. See [CLAUDE.md](../CLAUDE.md#ontology-extraction-pattern) for details on:
+- Multi-provider testing (`provider_configs`)
+- Semantic search configuration (`embedding_fields`)
+- File matching rules (`OntologyConfig`)
+- Dreaming workflow integration
+
+### Troubleshooting
+
+**Schema not found error:**
+```bash
+# Check if schema was ingested correctly
+rem ask "SEARCH 'my-agent' FROM schemas"
+
+# List all schemas
+rem ask "SELECT name, category, created_at FROM schemas ORDER BY created_at DESC LIMIT 10"
+```
+
+**Agent not loading tools:**
+- Verify `json_schema_extra.tools` lists correct tool names
+- Valid tool names: `search_rem`, `ask_rem_agent`, `ingest_into_rem`, `read_resource`
+- Check MCP tool names in `src/rem/api/mcp_router/tools.py`
+- Tools are case-sensitive: use `search_rem`, not `Search_REM`
+
+**Agent not returning structured output:**
+- Ensure `properties` field defines all expected output fields
+- Use `required` field to mark mandatory fields
+- Check agent response with `--stream` disabled to see full JSON output
 
 ---
 
@@ -269,15 +533,15 @@ REM provides a custom query language designed for **LLM-driven iterated retrieva
 Unlike traditional single-shot SQL queries, the REM dialect is optimized for **multi-turn exploration** where LLMs participate in query planning:
 
 - **Iterated Queries**: Queries return partial results that LLMs use to refine subsequent queries
-- **Composable WITH Syntax**: Chain operations together (e.g., `TRAVERSE FROM ... WITH LOOKUP "..."`)
+- **Composable WITH Syntax**: Chain operations together (e.g., `TRAVERSE edge_type WITH LOOKUP "..."`)
 - **Mixed Indexes**: Combines exact lookups (O(1)), semantic search (vector), and graph traversal
 - **Query Planner Participation**: Results include metadata for LLMs to decide next steps
 
 **Example Multi-Turn Flow**:
 ```
 Turn 1: LOOKUP "sarah-chen" → Returns entity + available edge types
-Turn 2: TRAVERSE FROM "sarah-chen" TYPE "authored_by" DEPTH 1 → Returns connected documents
-Turn 3: SEARCH "architecture decisions" WITH TRAVERSE FROM "sarah-chen" → Combines semantic + graph
+Turn 2: TRAVERSE authored_by WITH LOOKUP "sarah-chen" DEPTH 1 → Returns connected documents
+Turn 3: SEARCH "architecture decisions" → Semantic search, then explore graph from results
 ```
 
 This enables LLMs to **progressively build context** rather than requiring perfect queries upfront.
@@ -330,8 +594,8 @@ SEARCH "contract disputes" FROM resources WHERE tags @> ARRAY['legal'] LIMIT 5
 Follow `graph_edges` relationships across the knowledge graph.
 
 ```sql
-TRAVERSE FROM "sarah-chen" TYPE "authored_by" DEPTH 2
-TRAVERSE FROM "api-design-v2" TYPE "references,depends_on" DEPTH 3
+TRAVERSE authored_by WITH LOOKUP "sarah-chen" DEPTH 2
+TRAVERSE references,depends_on WITH LOOKUP "api-design-v2" DEPTH 3
 ```
 
 **Features**:
@@ -424,7 +688,7 @@ SEARCH "API migration planning" FROM resources LIMIT 5
 LOOKUP "tidb-migration-spec" FROM resources
 
 # Query 3: Find related people
-TRAVERSE FROM "tidb-migration-spec" TYPE "authored_by,reviewed_by" DEPTH 1
+TRAVERSE authored_by,reviewed_by WITH LOOKUP "tidb-migration-spec" DEPTH 1
 
 # Query 4: Recent activity
 SELECT * FROM moments WHERE
@@ -441,7 +705,7 @@ All queries automatically scoped by `user_id` for complete data isolation:
 SEARCH "contracts" FROM resources LIMIT 10
 
 -- No cross-user data leakage
-TRAVERSE FROM "project-x" TYPE "references" DEPTH 3
+TRAVERSE references WITH LOOKUP "project-x" DEPTH 3
 ```
 
 ## API Endpoints
@@ -637,20 +901,21 @@ rem db rebuild-cache
 
 #### `rem db schema generate` - Generate SQL Schema
 
-Generate database schema from Pydantic models.
+Generate database schema from Pydantic models. Output goes directly to the migrations folder.
 
 ```bash
-# Generate install_models.sql from entity models
-rem db schema generate \
-  --models src/rem/models/entities \
-  --output rem/src/rem/sql/install_models.sql
+# Generate 002_install_models.sql from entity models (default)
+rem db schema generate --models src/rem/models/entities
 
-# Generate migration file
-rem db schema generate \
-  --models src/rem/models/entities \
-  --output rem/src/rem/sql/migrations/003_add_fields.sql
+# This writes to: src/rem/sql/migrations/002_install_models.sql
+# Then apply with: rem db migrate
 ```
 
+**Workflow for adding new models:**
+1. Add/modify models in `src/rem/models/entities/`
+2. Run `rem db schema generate -m src/rem/models/entities`
+3. Run `rem db migrate` to apply changes
+
 #### `rem db schema indexes` - Generate Background Indexes
 
 Generate SQL for background index creation (HNSW for vectors).
@@ -677,22 +942,14 @@ rem db schema validate --models src/rem/models/entities
 Process files with optional custom extractor (ontology extraction).
 
 ```bash
-# Process all completed files for tenant
-rem process files \
-  --tenant-id acme-corp \
-  --status completed \
-  --limit 10
+# Process all completed files
+rem process files --status completed --limit 10
 
 # Process with custom extractor
-rem process files \
-  --tenant-id acme-corp \
-  --extractor cv-parser-v1 \
-  --limit 50
+rem process files --extractor cv-parser-v1 --limit 50
 
-# Process files from the last 7 days
-rem process files \
-  --tenant-id acme-corp \
-  --lookback-hours 168
+# Process files for specific user
+rem process files --user-id user-123 --status completed
 ```
 
 #### `rem process ingest` - Ingest File into REM
@@ -700,14 +957,13 @@ rem process files \
 Ingest a file into REM with full pipeline (storage + parsing + embedding + database).
 
 ```bash
-# Ingest local file
+# Ingest local file with metadata
 rem process ingest /path/to/document.pdf \
-  --user-id user-123 \
   --category legal \
   --tags contract,2024
 
 # Ingest with minimal options
-rem process ingest ./meeting-notes.md --user-id user-123
+rem process ingest ./meeting-notes.md
 ```
 
 #### `rem process uri` - Parse File (Read-Only)
@@ -732,28 +988,17 @@ rem process uri s3://bucket/key.docx --output text
 Run full dreaming workflow: extractors → moments → affinity → user model.
 
 ```bash
-# Full workflow for user
-rem dreaming full \
-  --user-id user-123 \
-  --tenant-id acme-corp
+# Full workflow (uses default user from settings)
+rem dreaming full
 
 # Skip ontology extractors
-rem dreaming full \
-  --user-id user-123 \
-  --tenant-id acme-corp \
-  --skip-extractors
+rem dreaming full --skip-extractors
 
 # Process last 24 hours only
-rem dreaming full \
-  --user-id user-123 \
-  --tenant-id acme-corp \
-  --lookback-hours 24
+rem dreaming full --lookback-hours 24
 
-# Limit resources processed
-rem dreaming full \
-  --user-id user-123 \
-  --tenant-id acme-corp \
-  --limit 100
+# Limit resources processed for specific user
+rem dreaming full --user-id user-123 --limit 100
 ```
 
 #### `rem dreaming custom` - Custom Extractor
@@ -761,16 +1006,11 @@ rem dreaming full \
 Run specific ontology extractor on user's data.
 
 ```bash
-# Run CV parser on user's files
-rem dreaming custom \
-  --user-id user-123 \
-  --tenant-id acme-corp \
-  --extractor cv-parser-v1
+# Run CV parser on files
+rem dreaming custom --extractor cv-parser-v1
 
-# Process last week's files
+# Process last week's files with limit
 rem dreaming custom \
-  --user-id user-123 \
-  --tenant-id acme-corp \
   --extractor contract-analyzer-v1 \
   --lookback-hours 168 \
   --limit 50
@@ -781,17 +1021,11 @@ rem dreaming custom \
 Extract temporal narratives from resources.
 
 ```bash
-# Generate moments for user
-rem dreaming moments \
-  --user-id user-123 \
-  --tenant-id acme-corp \
-  --limit 50
+# Generate moments
+rem dreaming moments --limit 50
 
 # Process last 7 days
-rem dreaming moments \
-  --user-id user-123 \
-  --tenant-id acme-corp \
-  --lookback-hours 168
+rem dreaming moments --lookback-hours 168
 ```
 
 #### `rem dreaming affinity` - Build Relationships
@@ -799,17 +1033,11 @@ rem dreaming moments \
 Build semantic relationships between resources using embeddings.
 
 ```bash
-# Build affinity graph for user
-rem dreaming affinity \
-  --user-id user-123 \
-  --tenant-id acme-corp \
-  --limit 100
+# Build affinity graph
+rem dreaming affinity --limit 100
 
 # Process recent resources only
-rem dreaming affinity \
-  --user-id user-123 \
-  --tenant-id acme-corp \
-  --lookback-hours 24
+rem dreaming affinity --lookback-hours 24
 ```
 
 #### `rem dreaming user-model` - Update User Model
@@ -818,9 +1046,7 @@ Update user model from recent activity (preferences, interests, patterns).
 
 ```bash
 # Update user model
-rem dreaming user-model \
-  --user-id user-123 \
-  --tenant-id acme-corp
+rem dreaming user-model
 ```
 
 ### Evaluation & Experiments
@@ -1071,6 +1297,30 @@ S3__BUCKET_NAME=rem-storage
 S3__REGION=us-east-1
 ```
 
+### Building Docker Images
+
+We tag Docker images with three labels for traceability:
+1. `latest` - Always points to most recent build
+2. `<git-sha>` - Short commit hash for exact version tracing
+3. `<version>` - Semantic version from `pyproject.toml`
+
+```bash
+# Build and push multi-platform image to Docker Hub
+VERSION=$(grep '^version' pyproject.toml | cut -d'"' -f2) && \
+docker buildx build --platform linux/amd64,linux/arm64 \
+    -t percolationlabs/rem:latest \
+    -t percolationlabs/rem:$(git rev-parse --short HEAD) \
+    -t percolationlabs/rem:$VERSION \
+    --push \
+    -f Dockerfile .
+
+# Load locally for testing (single platform, no push)
+docker buildx build --platform linux/arm64 \
+    -t percolationlabs/rem:latest \
+    --load \
+    -f Dockerfile .
+```
+
 ### Production Deployment (Optional)
 
 For production deployment to AWS EKS with Kubernetes, see the main repository README:
@@ -1186,6 +1436,110 @@ TraverseQuery ::= TRAVERSE [<edge_types:list>] WITH <initial_query:Query> [DEPTH
 
 **Stage 4** (100% answerable): Mature graph with rich historical data. All query types fully functional with high-quality results.
 
+## Troubleshooting
+
+### Apple Silicon Mac: "Failed to build kreuzberg" Error
+
+**Problem**: Installation fails with `ERROR: Failed building wheel for kreuzberg` on Apple Silicon Macs.
+
+**Root Cause**: REM uses `kreuzberg>=4.0.0rc1` for document parsing with native ONNX/Rust table extraction. Kreuzberg 4.0.0rc1 provides pre-built wheels for ARM64 macOS (`macosx_14_0_arm64.whl`) but NOT for x86_64 (Intel) macOS. If you're using an x86_64 Python binary (running under Rosetta 2), pip cannot find a compatible wheel and attempts to build from source, which fails.
+
+**Solution**: Use ARM64 (native) Python instead of x86_64 Python.
+
+**Step 1: Verify your Python architecture**
+
+```bash
+python3 -c "import platform; print(f'Machine: {platform.machine()}')"
+```
+
+- **Correct**: `Machine: arm64` (native ARM Python)
+- **Wrong**: `Machine: x86_64` (Intel Python under Rosetta)
+
+**Step 2: Install ARM Python via Homebrew** (if not already installed)
+
+```bash
+# Install ARM Python
+brew install python@3.12
+
+# Verify it's ARM
+/opt/homebrew/bin/python3.12 -c "import platform; print(platform.machine())"
+# Should output: arm64
+```
+
+**Step 3: Create venv with ARM Python**
+
+```bash
+# Use full path to ARM Python
+/opt/homebrew/bin/python3.12 -m venv .venv
+
+# Activate and install
+source .venv/bin/activate
+pip install "remdb[all]"
+```
+
+**Why This Happens**: Some users have both Intel Homebrew (`/usr/local`) and ARM Homebrew (`/opt/homebrew`) installed. If your system `python3` points to the Intel version at `/usr/local/bin/python3`, you'll hit this issue. The fix is to explicitly use the ARM Python from `/opt/homebrew/bin/python3.12`.
+
+**Verification**: After successful installation, you should see:
+```
+Using cached kreuzberg-4.0.0rc1-cp310-abi3-macosx_14_0_arm64.whl (19.8 MB)
+Successfully installed ... kreuzberg-4.0.0rc1 ... remdb-0.3.10
+```
+
+## Using REM as a Library
+
+REM wraps FastAPI - extend it exactly as you would any FastAPI app.
+
+```python
+import rem
+from rem import create_app
+from rem.models.core import CoreModel
+
+# 1. Register models (for schema generation)
+rem.register_models(MyModel, AnotherModel)
+
+# 2. Register schema paths (for custom agents/evaluators)
+rem.register_schema_path("./schemas")
+
+# 3. Create app
+app = create_app()
+
+# 4. Extend like normal FastAPI
+app.include_router(my_router)
+
+@app.mcp_server.tool()
+async def my_tool(query: str) -> dict:
+    """Custom MCP tool."""
+    return {"result": query}
+```
+
+### Project Structure
+
+```
+my-rem-app/
+├── my_app/
+│   ├── main.py           # Entry point (create_app + extensions)
+│   ├── models.py         # Custom models (inherit CoreModel)
+│   └── routers/          # Custom FastAPI routers
+├── schemas/
+│   ├── agents/           # Custom agent YAML schemas
+│   └── evaluators/       # Custom evaluator schemas
+├── sql/migrations/       # Custom SQL migrations
+└── pyproject.toml
+```
+
+Generate this structure with: `rem scaffold my-app`
+
+### Extension Points
+
+| Extension | How |
+|-----------|-----|
+| **Routes** | `app.include_router(router)` or `@app.get()` |
+| **MCP Tools** | `@app.mcp_server.tool()` decorator or `app.mcp_server.add_tool(fn)` |
+| **MCP Resources** | `@app.mcp_server.resource("uri://...")` or `app.mcp_server.add_resource(fn)` |
+| **MCP Prompts** | `@app.mcp_server.prompt()` or `app.mcp_server.add_prompt(fn)` |
+| **Models** | `rem.register_models(Model)` then `rem db schema generate` |
+| **Agent Schemas** | `rem.register_schema_path("./schemas")` or `SCHEMA__PATHS` env var |
+
 ## License
 
 MIT