remdb 0.2.6__py3-none-any.whl

rem/services/content/README.md

@@ -0,0 +1,1269 @@
+# ContentService - File Ingestion & Processing System
+
+Centralized file ingestion pipeline with rich parsing state and pluggable content providers.
+
+## Overview
+
+ContentService provides **three distinct entry points** for file operations:
+
+1. **`ingest_file()`** - Complete ingestion pipeline (MCP tools, full workflow)
+   - Read from source (local/S3/HTTP)
+   - Store to internal tenant-scoped storage
+   - Parse and extract rich content (parsing state)
+   - Chunk and embed into searchable resources
+   - Create File entity + Resource chunks in database
+
+2. **`process_uri()`** - Read-only content extraction (CLI, testing)
+   - Extract content from any URI
+   - No storage, no database writes
+   - Returns: extracted text + metadata
+
+3. **`process_and_save()`** - Process existing stored files (workers)
+   - Process files already in internal storage
+   - Chunk and create Resource entities
+   - Used by SQS worker for async processing
+
+### Parsing State - The Innovation
+
+Files (PDF, WAV, DOCX, etc.) are converted to **rich parsing state**:
+- **Content**: Markdown-formatted text (preserves structure)
+- **Metadata**: File info, extraction details, timestamps
+- **Tables**: Structured data extracted as CSV
+- **Images**: Extracted images for multimodal RAG
+- **Provider Info**: Which parser was used, version, settings
+
+This enables agents to deeply understand documents beyond simple text.
+
+## File Processing Conventions
+
+### S3 Path Structure
+
+When files are uploaded to S3, they follow a strict convention for organization:
+
+```
+s3://bucket/uploads/path/to/document.pdf          # Original uploaded file
+s3://bucket/parsed/path/to/document-pdf/          # Parsed artifacts directory
+  ├── document.pdf.parsed.md                      # Structured markdown output
+  ├── document.pdf.meta.yaml                      # Extraction metadata
+  └── artifacts/                                  # Extracted artifacts
+      ├── images/                                 # Extracted images
+      │   ├── page-1-img-0.png
+      │   ├── page-2-img-1.jpg
+      │   └── ...
+      └── tables/                                 # Extracted tables
+          ├── table-0.csv
+          ├── table-0.png                         # Cropped table image
+          ├── table-1.csv
+          └── ...
+```
+
+### Path Mapping Convention
+
+The parsed path mirrors the upload path with two changes:
+1. Replace `/uploads/` prefix with `/parsed/`
+2. Use filename with extension as directory name (dots converted to hyphens)
+
+**Examples:**
+```
+uploads/docs/report.pdf          → parsed/docs/report-pdf/
+uploads/data/sheet.xlsx          → parsed/data/sheet-xlsx/
+uploads/user/123/invoice.pdf     → parsed/user/123/invoice-pdf/
+```
+
+### Output File Formats
+
+#### 1. Structured Markdown (`.parsed.md`)
+
+All documents are converted to structured markdown before chunking and embedding. This provides:
+- Consistent format for downstream processing
+- Preserves document structure (headings, lists, tables)
+- Human-readable intermediate representation
+- Easy to version control and diff
+
+**Example structure:**
+```markdown
+# Document Title
+
+## Metadata
+- **Source**: report.pdf
+- **Pages**: 42
+- **Extracted**: 2025-01-15T10:30:00Z
+- **Parser**: kreuzberg
+- **Language**: en (0.98 confidence)
+
+## Content
+
+Lorem ipsum dolor sit amet...
+
+### Section 1
+
+Content with **formatting** preserved.
+
+### Tables
+
+See [Table 1](artifacts/tables/table-0.csv) - Financial Summary
+
+### Images
+
+![Diagram 1](artifacts/images/page-5-img-0.png)
+```
+
+#### 2. Metadata YAML (`.meta.yaml`)
+
+Comprehensive extraction metadata in YAML format for downstream processing:
+
+```yaml
+# Source file metadata
+source:
+  uri: s3://bucket/uploads/docs/report.pdf
+  filename: report.pdf
+  size_bytes: 2456789
+  content_type: application/pdf
+  uploaded_at: "2025-01-15T10:25:00Z"
+  etag: "abc123..."
+
+# Extraction metadata
+extraction:
+  parser: kreuzberg
+  parser_version: "3.21.0"
+  extracted_at: "2025-01-15T10:30:00Z"
+  processing_time_seconds: 45.2
+  config:
+    extract_tables: true
+    extract_images: true
+    force_ocr: false
+
+# Document metadata
+document:
+  page_count: 42
+  word_count: 8542
+  char_count: 52341
+  detected_language: en
+  language_confidence: 0.98
+  document_type: report
+  document_type_confidence: 0.87
+
+# Extracted artifacts
+artifacts:
+  tables:
+    - id: table-0
+      page: 5
+      rows: 12
+      columns: 6
+      file: artifacts/tables/table-0.csv
+      image: artifacts/tables/table-0.png
+      confidence: 0.92
+    - id: table-1
+      page: 18
+      rows: 8
+      columns: 4
+      file: artifacts/tables/table-1.csv
+      image: artifacts/tables/table-1.png
+      confidence: 0.88
+
+  images:
+    - id: img-0
+      page: 3
+      file: artifacts/images/page-3-img-0.png
+      width: 1200
+      height: 800
+      format: png
+      ocr_applied: false
+    - id: img-1
+      page: 12
+      file: artifacts/images/page-12-img-1.jpg
+      width: 800
+      height: 600
+      format: jpeg
+      ocr_applied: true
+      ocr_text: "Extracted text from image..."
+
+# Quality metrics
+quality:
+  overall_score: 0.94
+  ocr_required: false
+  warnings: []
+
+# Chunking metadata (added after embedding)
+chunking:
+  strategy: semantic
+  chunk_count: 156
+  avg_chunk_size: 512
+  overlap_tokens: 50
+```
+
+### Processing Pipeline
+
+```
+1. Upload to S3
+   └─> uploads/docs/report.pdf
+
+2. S3 Event → SQS → Worker Pod
+   └─> ContentService.process_uri()
+
+3. Extract with Kreuzberg
+   ├─> Text content
+   ├─> Tables (CSV + images)
+   ├─> Images (PNG/JPEG)
+   └─> Metadata
+
+4. Generate structured outputs
+   ├─> Convert to markdown
+   ├─> Save artifacts to S3
+   └─> Generate meta.yaml
+
+5. Write to S3 parsed directory
+   └─> parsed/docs/report-pdf/
+       ├── report.pdf.parsed.md
+       ├── report.pdf.meta.yaml
+       └── artifacts/
+           ├── images/
+           └── tables/
+
+6. Chunk and embed markdown
+   └─> PostgreSQL + pgvector (as Resources)
+
+7. Create graph edges
+   └─> Link Resources to User, Files, Moments
+```
+
+### Why This Convention?
+
+**Separation of concerns:**
+- `/uploads/` - Raw user-uploaded files (immutable)
+- `/parsed/` - Processed, structured outputs (reproducible)
+
+**Artifact organization:**
+- Keeps all related files together in one directory
+- Easy to reference artifacts from markdown
+- Preserves relative paths for portability
+- Simple to delete all artifacts for a document
+
+**Structured markdown:**
+- Single format for all document types
+- Consistent chunking and embedding strategy
+- Easier to build RAG context
+- Human-readable for debugging
+
+**Metadata tracking:**
+- Complete audit trail of processing
+- Reproducible extraction (config stored)
+- Quality metrics for filtering
+- Artifact registry for lookup
+
+## Architecture
+
+```
+User uploads file
+       ↓
+S3: s3://rem/uploads/docs/report.pdf
+       ↓ (S3 ObjectCreated event)
+SQS: rem-file-processing queue
+       ↓ (KEDA monitors queue depth)
+K8s Deployment: file-processor (0-20 pods)
+       ↓ (SQSFileProcessor worker)
+ContentService.process_uri()
+       ↓ (Kreuzberg DocProvider)
+Extract: text + tables + images + metadata
+       ↓
+Generate structured outputs:
+  - report.pdf.parsed.md (markdown)
+  - report.pdf.meta.yaml (metadata)
+  - artifacts/images/*.png
+  - artifacts/tables/*.csv
+       ↓
+S3: s3://rem/parsed/docs/report-pdf/
+       ↓ (TODO: chunking + embedding)
+PostgreSQL + pgvector (as Resources)
+       ↓ (TODO: graph edges)
+Link to User, File entities
+```
+
+## Quick Start
+
+### 1. Complete Ingestion (MCP Tool Pattern)
+
+```python
+from rem.services.content import ContentService
+
+service = ContentService()
+
+# Full pipeline: read → store → parse → chunk → embed
+result = await service.ingest_file(
+    file_uri="/path/to/contract.pdf",  # or s3://, https://
+    user_id="user-123",
+    category="legal",
+    tags=["contract", "q1-2025"],
+    is_local_server=True  # Security: only local servers can read local files
+)
+
+print(f"File ID: {result['file_id']}")
+print(f"Storage: {result['storage_uri']}")
+print(f"Resources created: {result['resources_created']}")
+print(f"Status: {result['processing_status']}")
+print(f"Parsing metadata: {result['parsing_metadata']}")
+```
+
+### 2. Read-Only Extraction (CLI Pattern)
+
+```python
+service = ContentService()
+
+# Extract content only (no storage, no database)
+result = service.process_uri("./README.md")
+
+print(result["content"])  # Extracted text
+print(result["metadata"])  # File metadata
+print(result["provider"])  # "markdown"
+```
+
+### 3. Process Existing Files (Worker Pattern)
+
+```python
+service = ContentService()
+
+# Process file already in internal storage
+result = await service.process_and_save(
+    uri="file:///Users/.../.rem/fs/acme-corp/files/abc123/doc.pdf",
+    user_id="user-123"
+)
+
+print(f"Chunks created: {result['chunk_count']}")
+```
+
+### CLI Usage
+
+```bash
+# Read-only extraction (no storage)
+rem process uri ./README.md
+rem process uri s3://bucket/doc.md -o json
+rem process uri https://example.com/paper.pdf -s output.json
+```
+
+### MCP Tool Usage
+
+```python
+# Via MCP protocol (uses ingest_file internally)
+result = await parse_and_ingest_file(
+    file_uri="s3://bucket/report.pdf",
+    user_id="user-123",
+    is_local_server=False  # Remote server = no local file access
+)
+```
+
+## Supported File Formats
+
+### Currently Supported
+
+#### Markdown (`.md`, `.markdown`)
+- **Provider**: `TextProvider`
+- **Capabilities**: UTF-8 text extraction, heading analysis, line/char counts
+- **Use case**: Documentation, notes, structured text
+
+#### PDF (`.pdf`)
+- **Provider**: `DocProvider` (powered by Kreuzberg)
+- **Capabilities**:
+  - Text extraction with OCR fallback (Tesseract)
+  - Intelligent table detection and reconstruction
+  - Multi-format support (native PDF, scanned, password-protected)
+  - Daemon-safe subprocess workaround for ASGI servers
+  - Configurable accuracy thresholds
+- **Use case**: Documents, reports, forms, scanned papers
+
+#### Audio (`.wav`, `.mp3`, `.m4a`, `.flac`, `.ogg`)
+- **Provider**: `AudioProvider` (powered by OpenAI Whisper)
+- **Capabilities**:
+  - Speech-to-text transcription
+  - Automatic silence-based chunking
+  - Markdown-formatted output with timestamps
+  - Cost estimation ($0.006/minute)
+- **Use case**: Meeting recordings, interviews, podcasts, voice memos
+
+#### Images (`.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`)
+- **Provider**: `ImageProvider` (vision LLM + CLIP embeddings)
+- **Capabilities**:
+  - **Tier-based vision analysis**: Gold tier users always get vision LLM descriptions
+  - **Sampling-based analysis**: Non-gold users get analysis based on sample rate (0.0-1.0)
+  - **Multi-provider support**: Anthropic Claude, Google Gemini, OpenAI GPT-4o
+  - **Metadata extraction**: Image dimensions, format detection
+  - **Markdown descriptions**: Vision LLM generates detailed markdown descriptions
+  - **CLIP embeddings**: Semantic image search with Jina AI (512/768-dim vectors)
+  - **Graceful degradation**: Falls back when API keys unavailable
+- **Use case**: Screenshots, diagrams, charts, photos, scanned images
+- **Configuration**:
+  - Vision LLM (expensive, tier/sample-gated):
+    - `CONTENT__IMAGE_VLLM_SAMPLE_RATE`: Sampling rate (0.0 = never, 1.0 = always)
+    - `CONTENT__IMAGE_VLLM_PROVIDER`: Provider (anthropic, gemini, openai)
+    - `CONTENT__IMAGE_VLLM_MODEL`: Model name (optional, uses provider default)
+  - CLIP Embeddings (cheap, always-on when key provided):
+    - `CONTENT__CLIP_PROVIDER`: Provider (jina for API, self-hosted for future, default: jina)
+    - `CONTENT__CLIP_MODEL`: Model (jina-clip-v1 or jina-clip-v2, default: v2)
+    - `CONTENT__JINA_API_KEY`: Jina AI API key (get free key at https://jina.ai/embeddings/)
+- **Storage**: Saves to `ImageResource` table (separate from regular `Resource`)
+- **Pricing**:
+  - Vision LLM: ~$0.01-0.05 per image (varies by provider)
+  - CLIP: ~$0.02 per million tokens (~4K tokens per 512x512 image = $0.00008/image)
+  - Free tier: 10M CLIP tokens (~2,500 images)
+
+### Planned Support (via Kreuzberg)
+
+Kreuzberg supports 50+ file formats. Easy to add via provider pattern:
+
+#### Documents & Productivity
+- **Word**: `.docx`, `.doc` - Microsoft Word documents
+- **Excel**: `.xlsx`, `.xls`, `.ods` - Spreadsheets with table extraction
+- **PowerPoint**: `.pptx`, `.ppt` - Presentations
+- **Rich Text**: `.rtf` - Formatted text documents
+- **EPUB**: Digital books
+
+#### Structured Data
+- **Web**: `.html`, `.xml` - Web pages and markup
+- **Data**: `.json`, `.yaml`, `.toml` - Configuration and data files
+
+#### Academic & Technical
+- **LaTeX**: `.tex`, `.bib` - Academic papers and bibliographies
+- **Jupyter**: `.ipynb` - Notebooks with code and outputs
+- **Markup**: `.rst`, `.org` - reStructuredText, Org Mode
+- **Markdown variants**: Enhanced markdown processing
+
+#### Communication
+- **Email**: `.eml`, `.msg` - Email messages with attachments
+
+#### Archives
+- **Compressed**: `.zip`, `.tar`, `.gz`, `.7z` - Extract and process contents
+
+## Kreuzberg Document Intelligence
+
+### What is Kreuzberg?
+
+Kreuzberg is a polyglot document processing system with a Rust core that extracts text, metadata, and structured information from documents. It provides:
+
+- **50+ file format support** - PDFs, Office docs, images, HTML, XML, archives, email, and more
+- **OCR with table extraction** - Multiple backends (Tesseract, EasyOCR, PaddleOCR)
+- **Intelligent table detection** - Reconstructs table structure with configurable thresholds
+- **Batch processing** - Concurrent document handling with automatic resource management
+- **Memory efficiency** - Streaming parsers handle multi-GB files with constant memory
+- **Language detection** - Automatic detection with confidence thresholds
+- **Metadata extraction** - Authors, titles, dates, page counts, EXIF, format-specific props
+
+### ExtractionConfig Options
+
+Kreuzberg's `ExtractionConfig` provides fine-grained control over extraction:
+
+```python
+from kreuzberg import ExtractionConfig, GMFTConfig
+
+config = ExtractionConfig(
+    # Table extraction
+    extract_tables=True,                    # Enable table detection
+    extract_tables_from_ocr=False,          # Extract tables from OCR images
+    gmft_config=GMFTConfig(
+        detector_base_threshold=0.85,       # Detection confidence threshold
+        enable_multi_header=True,           # Support multi-row headers
+        remove_null_rows=True,              # Clean up empty rows
+    ),
+
+    # Image extraction
+    extract_images=True,                    # Extract embedded images
+    deduplicate_images=True,                # Remove duplicate images
+    ocr_extracted_images=False,             # Run OCR on extracted images
+    image_ocr_min_dimensions=(50, 50),      # Min image size for OCR
+    image_ocr_max_dimensions=(10000, 10000),# Max image size for OCR
+
+    # OCR configuration
+    force_ocr=False,                        # Force OCR even for native text
+    ocr_backend='tesseract',                # 'tesseract', 'paddle', 'easyocr'
+
+    # Content processing
+    chunk_content=False,                    # Enable semantic chunking
+    max_chars=2000,                         # Max chars per chunk
+    max_overlap=100,                        # Overlap between chunks
+
+    # Entity extraction
+    extract_entities=False,                 # Extract named entities
+    extract_keywords=False,                 # Extract keywords
+    keyword_count=10,                       # Number of keywords to extract
+
+    # Language detection
+    auto_detect_language=False,             # Auto-detect document language
+    language_detection_model='auto',        # 'lite', 'full', 'auto'
+
+    # Document classification
+    auto_detect_document_type=False,        # Classify document type
+    document_type_confidence_threshold=0.5, # Min confidence threshold
+
+    # Quality processing
+    enable_quality_processing=True,         # Enable quality assessment
+
+    # PDF-specific
+    pdf_password='',                        # Password for encrypted PDFs
+    target_dpi=150,                         # Target DPI for rendering
+    auto_adjust_dpi=True,                   # Auto-adjust DPI for large pages
+
+    # HTML/JSON-specific
+    html_to_markdown_config=None,           # HTML→Markdown options
+    json_config=None,                       # JSON extraction options
+
+    # Performance
+    use_cache=True,                         # Enable extraction cache
+)
+```
+
+### Extraction Result Structure
+
+Kreuzberg returns a rich result object with comprehensive metadata:
+
+```python
+from kreuzberg import extract_file_sync
+from pathlib import Path
+
+result = extract_file_sync(Path("document.pdf"), config=config)
+
+# Core content
+result.content                    # str: Full extracted text
+result.mime_type                  # str: Detected MIME type
+
+# Tables (list of dicts)
+for table in result.tables:
+    table['page_number']          # int: Source page
+    table['text']                 # str: Table as markdown/text
+    table['df']                   # pandas.DataFrame: Structured data
+    table['cropped_image']        # PIL.Image: Cropped table image
+
+# Images (list of dicts)
+for img in result.images:
+    img['page']                   # int: Source page
+    img['image']                  # PIL.Image: Image data
+    img['width'], img['height']   # int: Dimensions
+    img['format']                 # str: Image format
+
+# Document metadata
+result.metadata = {
+    'page_count': int,            # Number of pages
+    'author': str,                # Document author
+    'title': str,                 # Document title
+    'creation_date': str,         # ISO8601 timestamp
+    'modification_date': str,     # ISO8601 timestamp
+    'summary': str,               # Auto-generated summary
+    'quality_score': float,       # 0.0-1.0 quality metric
+}
+
+# Language detection
+result.detected_languages = [
+    {'language': 'en', 'confidence': 0.98},
+    {'language': 'es', 'confidence': 0.02},
+]
+
+# Document classification
+result.document_type              # str: 'report', 'invoice', etc.
+result.document_type_confidence   # float: 0.0-1.0
+
+# Entity extraction (if enabled)
+result.entities = {
+    'PERSON': ['John Doe', 'Jane Smith'],
+    'ORG': ['Acme Corp', 'TechCo'],
+    'GPE': ['New York', 'London'],
+    'DATE': ['2025-01-15', '2024-12-31'],
+}
+
+# Keywords (if enabled)
+result.keywords = ['machine learning', 'artificial intelligence', ...]
+
+# Chunks (if chunk_content=True)
+result.chunks = [
+    {'text': '...', 'start': 0, 'end': 2000},
+    {'text': '...', 'start': 1900, 'end': 3900},
+]
+
+# OCR results (if OCR was applied)
+result.image_ocr_results = [...]
+
+# Layout information
+result.layout                     # Document layout structure
+
+# Utility methods
+result.to_dict()                  # Convert to dict for serialization
+result.to_markdown()              # Convert to markdown format
+result.export_tables_to_csv(dir) # Export all tables to CSV files
+result.export_tables_to_tsv(dir) # Export all tables to TSV files
+result.get_table_summaries()      # Get summaries of all tables
+```
+
+### Daemon Process Workaround
+
+When running in ASGI servers (Hypercorn, Uvicorn), Kreuzberg's ProcessPoolExecutor may fail due to daemon restrictions. Our `DocProvider` implements a subprocess workaround:
+
+```python
+def _is_daemon_process(self) -> bool:
+    """Check if running in a daemon process."""
+    try:
+        return multiprocessing.current_process().daemon
+    except Exception:
+        return False
+
+def _parse_in_subprocess(self, file_path: Path) -> dict:
+    """Run kreuzberg in a separate subprocess to bypass daemon restrictions."""
+    # Executes parsing in isolated subprocess
+    # Serializes config and result as JSON
+    # 5 minute timeout for large documents
+```
+
+This pattern ensures reliable parsing in production deployments.
+
+## Provider Configuration
+
+### Default Provider Settings
+
+Providers are configured via `ContentSettings` in `settings.py`:
+
+```python
+from rem.settings import settings
+
+# View default supported types
+print(settings.content.supported_text_types)
+# [".txt", ".md", ".json", ".yaml", ".py", ".js", ...]
+
+print(settings.content.supported_doc_types)
+# [".pdf", ".docx", ".pptx", ".xlsx", ".png", ".jpg", ...]
+
+print(settings.content.supported_audio_types)
+# [".wav", ".mp3", ".m4a", ".flac", ".ogg"]
+```
+
+### Environment Variable Overrides
+
+You can override the default extension lists via environment variables:
+
+```bash
+# Override document types to only support PDFs
+export CONTENT__SUPPORTED_DOC_TYPES=".pdf"
+
+# Override text types to only support markdown and Python
+export CONTENT__SUPPORTED_TEXT_TYPES=".md,.py"
+
+# Override audio types to only support WAV
+export CONTENT__SUPPORTED_AUDIO_TYPES=".wav"
+
+# Disable audio transcription in development
+export CONTENT__SUPPORTED_AUDIO_TYPES=""
+```
+
+### Benefits of Settings-Based Configuration
+
+1. **No Long Lists** - Extension lists defined in one place (settings.py)
+2. **Environment Override** - Easy configuration via env vars
+3. **Clean Code** - ContentService.__init__() is concise
+4. **Testable** - Settings can be mocked/overridden in tests
+5. **DRY** - Single source of truth for supported file types
+
+## Kind-Based Document Routing
+
+**Special Processing for YAML/JSON Files**
+
+REM implements a **kind-based routing system** that intercepts YAML/JSON files during ingestion and routes them to specialized processors based on their `kind` field. This allows different document types to be stored in appropriate database tables instead of the default `resources` table.
+
+### Architecture
+
+```
+YAML/JSON file ingestion
+    ↓
+ContentService.process_and_save()
+    ↓
+Check for 'kind' field in document
+    ├─ kind=agent or kind=evaluator
+    │   ↓
+    │   SchemaProvider validates schema structure
+    │   ↓
+    │   Save to 'schemas' table (NOT resources)
+    │   ↓
+    │   Log: "🔧 Custom provider flow initiated: kind=agent"
+    │
+    ├─ kind=engram
+    │   ↓
+    │   EngramProcessor creates structured memory
+    │   ↓
+    │   Save to 'resources' table (parent engram)
+    │   + 'moments' table (temporal events)
+    │   ↓
+    │   Log: "🔧 Custom provider flow initiated: kind=engram"
+    │
+    └─ No kind field
+        ↓
+        Standard text processing
+        ↓
+        Save to 'resources' table (default)
+```
+
+### Supported Kinds
+
+#### 1. **kind=agent** (Agent Schemas)
+
+Agent schemas define AI agents with structured outputs and tool access.
+
+**Required fields:**
+- `kind: agent`
+- `name: <kebab-case-name>` (e.g., "cv-parser", "query-agent")
+- `type: object`
+- `properties: {...}` (output schema)
+- `json_schema_extra.version: "1.0.0"`
+
+**Storage:** `schemas` table with `category='agent'`
+
+**Example:**
+```yaml
+---
+type: object
+description: |
+  You are a CV parser that extracts structured candidate information.
+
+properties:
+  candidate_name:
+    type: string
+    description: Full name
+  skills:
+    type: array
+    items:
+      type: string
+
+required:
+  - candidate_name
+
+json_schema_extra:
+  kind: agent
+  name: cv-parser
+  version: "1.0.0"
+  tags: [recruitment, hr]
+```
+
+#### 2. **kind=evaluator** (Evaluator Schemas)
+
+Evaluator schemas define LLM-as-a-Judge evaluators for agent output assessment.
+
+**Required fields:**
+- `kind: evaluator`
+- `name: <kebab-case-name>` (e.g., "rem-lookup-correctness")
+- `type: object`
+- `properties: {...}` (evaluation criteria)
+- `json_schema_extra.version: "1.0.0"`
+
+**Storage:** `schemas` table with `category='evaluator'`
+
+**Example:**
+```yaml
+---
+type: object
+description: |
+  Evaluate REM LOOKUP query correctness.
+
+properties:
+  correctness:
+    type: number
+    minimum: 0.0
+    maximum: 1.0
+  reasoning:
+    type: string
+
+required:
+  - correctness
+
+json_schema_extra:
+  kind: evaluator
+  name: rem-lookup-correctness
+  version: "1.0.0"
+```
+
+#### 3. **kind=engram** (Memory Documents)
+
+Engrams are bio-inspired memory documents that combine resources and temporal moments.
+
+**Required fields:**
+- `kind: engram`
+- `name: <unique-identifier>`
+- `content: <text content>`
+- `moments: [...]` (optional temporal events)
+
+**Storage:**
+- Parent engram → `resources` table with `category='engram'`
+- Child moments → `moments` table with graph edges to parent
+
+**Example:**
+```yaml
+---
+kind: engram
+name: team-standup-2025-01-15
+category: meeting
+summary: Daily standup discussion
+content: |
+  Team discussed sprint progress and blockers.
+
+moments:
+  - start_time: 2025-01-15T09:00:00Z
+    end_time: 2025-01-15T09:15:00Z
+    summary: Sprint update
+    speakers: [sarah, mike]
+    topics: [sprint-planning, blockers]
+
+graph_edges:
+  - dst: sarah-chen
+    rel_type: participated_in
+    weight: 1.0
+```
+
+### Implementation Details
+
+#### ContentService._process_schema()
+
+Handles `kind=agent` and `kind=evaluator`:
+
+```python
+async def _process_schema(result: dict, uri: str, user_id: str) -> dict:
+    """
+    Save agent/evaluator schema to schemas table.
+
+    Args:
+        result: Extraction result from SchemaProvider with:
+            - metadata.kind: "agent" or "evaluator"
+            - metadata.name: Schema name
+            - schema_data: Full JSON Schema dict
+        uri: File URI
+        user_id: Tenant ID
+
+    Returns:
+        dict with schema_name, kind, version, status
+    """
+    # Create Schema entity
+    schema_entity = Schema(
+        tenant_id=user_id,
+        name=metadata['name'],
+        spec=schema_data,
+        category=metadata['kind'],  # "agent" or "evaluator"
+        provider_configs=metadata.get('provider_configs', []),
+        embedding_fields=metadata.get('embedding_fields', []),
+    )
+
+    # Save to schemas table (NOT resources)
+    await postgres.batch_upsert(
+        records=[schema_entity],
+        model=Schema,
+        table_name="schemas",
+        entity_key_field="name",
+    )
+
+    logger.info(f"✅ Schema saved: {name} (kind={kind})")
+```
+
+#### ContentService._process_engram()
+
+Handles `kind=engram`:
+
+```python
+async def _process_engram(data: dict, uri: str, user_id: str) -> dict:
+    """
+    Process engram into resources + moments.
+
+    Args:
+        data: Parsed engram with kind=engram
+        uri: File URI
+        user_id: User ID
+
+    Returns:
+        dict with resource_id, moment_ids, chunks_created
+    """
+    # Delegate to EngramProcessor
+    processor = EngramProcessor(postgres)
+    result = await processor.process_engram(
+        data=data,
+        user_id=user_id,
+    )
+
+    logger.info(f"✅ Engram processed: {result['resource_id']} "
+                f"with {len(result['moment_ids'])} moments")
+
+    return result
+```
+
+### Why Kind-Based Routing?
+
+**Separation of Concerns:**
+- Agents/evaluators belong in `schemas` table (metadata, no embeddings)
+- Engrams belong in `resources` + `moments` tables (dual indexing)
+- Regular files belong in `resources` table (standard chunking)
+
+**Intercepting Default Flow:**
+- Without routing, YAML/JSON files would be chunked and embedded as generic resources
+- `kind` field allows processors to intercept before default processing
+- Enables specialized handling while maintaining simple ingestion API
+
+**Extensibility:**
+- Easy to add new kinds (e.g., `kind=workflow`, `kind=config`)
+- Each kind maps to a specialized processor
+- Processors can save to any tables, not just resources
+
+### Logging
+
+Custom provider flows are logged with emoji markers for visibility:
+
+```
+🔧 Custom provider flow initiated: kind=agent for cv-parser.yaml
+Saving schema to schemas table: kind=agent, name=cv-parser, version=1.0.0
+✅ Schema saved: cv-parser (kind=agent)
+```
+
+```
+🔧 Custom provider flow initiated: kind=engram for team-standup.yaml
+Processing engram: team-standup-2025-01-15
+✅ Engram processed: res_abc123 with 3 moments
+```
+
+## Provider Plugin System
+
+You can register custom providers to either:
+1. **Add support for new file types** (e.g., `.epub`, `.rtf`)
+2. **Override default providers** for specific formats (e.g., use PyMuPDF instead of Kreuzberg for PDFs)
+
+### Example: Custom Provider for New File Type
+
+```python
+from rem.services.content import ContentService
+from rem.services.content.providers import ContentProvider
+
+class EpubProvider(ContentProvider):
+    @property
+    def name(self) -> str:
+        return "epub"
+
+    def extract(self, content: bytes, metadata: dict) -> dict:
+        # Use ebooklib or other EPUB parser
+        text = extract_text_from_epub(content)
+        return {
+            "text": text,
+            "metadata": {
+                "chapters": 12,
+                "author": "...",
+            }
+        }
+
+# Register for new file type
+service = ContentService()
+service.register_provider([".epub"], EpubProvider())
+```
+
+### Example: Override Default Provider
+
+**Note**: This is a hypothetical example. The default `DocProvider` already handles PDFs well via Kreuzberg.
+
+```python
+# Hypothetical: Override DocProvider with custom PDF parser
+class CustomPDFProvider(ContentProvider):
+    @property
+    def name(self) -> str:
+        return "custom_pdf"
+
+    def extract(self, content: bytes, metadata: dict) -> dict:
+        # Use PyMuPDF, pdfplumber, or other library instead of Kreuzberg
+        text = extract_text_with_pymupdf(content)
+        return {
+            "text": text,
+            "metadata": {"parser": "pymupdf"}
+        }
+
+# Override .pdf extension to use custom provider instead of DocProvider
+service = ContentService()
+service.register_provider([".pdf"], CustomPDFProvider())
+```
+
+### Provider Interface
+
+All providers must implement:
+
+```python
+class ContentProvider(ABC):
+    @property
+    @abstractmethod
+    def name(self) -> str:
+        """Provider name (e.g., 'markdown', 'pdf')."""
+        pass
+
+    @abstractmethod
+    def extract(self, content: bytes, metadata: dict) -> dict:
+        """
+        Extract content from file bytes.
+
+        Args:
+            content: Raw file bytes
+            metadata: File metadata (size, mime_type, etc.)
+
+        Returns:
+            Dict with 'text' and optional 'metadata' fields
+        """
+        pass
+```
+
+## Event-Driven Processing
+
+### SQS Worker
+
+Background worker that consumes S3 events from SQS queue.
+
+**Features:**
+- Long polling (20s) for efficient SQS usage
+- Graceful shutdown (SIGTERM/SIGINT)
+- Batch processing (up to 10 messages)
+- DLQ support (3 retries)
+- IRSA authentication (no credentials in code)
+
+**Entry point:**
+```bash
+python -m rem.workers.sqs_file_processor
+```
+
+### K8s Deployment
+
+Production deployment with KEDA autoscaling at `manifests/application/file-processor/`
+
+**Scaling Example:**
+- 0 messages → 0 pods (scale to zero)
+- 25 messages → 3 pods
+- 100 messages → 10 pods
+- 250 messages → 20 pods (max)
+
+## Configuration
+
+### Environment Variables
+
+```bash
+# .env or K8s ConfigMap
+SQS__QUEUE_URL=https://sqs.us-east-1.amazonaws.com/ACCOUNT_ID/rem-file-processing
+SQS__REGION=us-east-1
+SQS__MAX_MESSAGES=10
+SQS__WAIT_TIME_SECONDS=20
+SQS__VISIBILITY_TIMEOUT=300
+
+S3__BUCKET_NAME=rem
+S3__REGION=us-east-1
+```
+
+### IRSA Permissions
+
+Worker pods need two IAM roles:
+
+1. **KEDA operator role** - Read SQS metrics
+   ```
+   sqs:GetQueueAttributes
+   sqs:ListQueues
+   ```
+
+2. **File processor role** - Consume queue + read S3
+   ```
+   sqs:ReceiveMessage
+   sqs:DeleteMessage
+   sqs:ChangeMessageVisibility
+   s3:GetObject
+   ```
+
+## Local Development
+
+### Test ContentService
+
+```bash
+cd rem
+uv pip install -e .
+
+# Test with local file
+echo "# Test Document" > test.md
+python -c "from rem.services.content import ContentService; \
+           print(ContentService().process_uri('test.md'))"
+```
+
+### Test CLI
+
+```bash
+# Test with local file
+rem process uri test.md
+
+# Test with S3 (requires AWS credentials or IRSA)
+aws s3 cp test.md s3://rem/uploads/
+rem process uri s3://rem/uploads/test.md
+```
+
+### Test Worker
+
+```bash
+# Set environment variables
+export SQS__QUEUE_URL=https://sqs.us-east-1.amazonaws.com/ACCOUNT_ID/rem-file-processing
+export SQS__REGION=us-east-1
+export S3__BUCKET_NAME=rem
+export S3__REGION=us-east-1
+
+# Run worker
+python -m rem.workers.sqs_file_processor
+
+# Upload test file (in another terminal)
+aws s3 cp test.md s3://rem/uploads/
+
+# Watch worker logs process the file
+```
+
+## Production Deployment
+
+### 1. Build and push Docker image
+
+```bash
+cd rem
+docker build -f Dockerfile.worker -t your-registry/rem:latest .
+docker push your-registry/rem:latest
+```
+
+### 2. Deploy infrastructure (Pulumi)
+
+```bash
+cd ../manifests/infra/file-queue
+pulumi up
+# Note queue URL and IAM policy ARNs
+```
+
+### 3. Deploy KEDA platform
+
+```bash
+cd ../../platform/keda
+kubectl apply -f application.yaml
+```
+
+### 4. Configure IRSA roles
+
+```bash
+# Annotate KEDA operator ServiceAccount
+kubectl annotate sa keda-operator -n keda \
+  eks.amazonaws.com/role-arn=arn:aws:iam::ACCOUNT_ID:role/keda-operator-role
+```
+
+### 5. Deploy file processor
+
+```bash
+cd ../../application/file-processor
+
+# Update ACCOUNT_ID in deployment.yaml and keda-scaledobject.yaml
+sed -i 's/ACCOUNT_ID/123456789012/g' *.yaml
+
+kubectl apply -f .
+```
+
+### 6. Verify scaling
+
+```bash
+# Should start at 0 pods
+kubectl get pods -l app=file-processor
+
+# Upload test files
+for i in {1..25}; do
+  echo "test $i" | aws s3 cp - s3://rem/uploads/test-$i.md
+done
+
+# Watch scale up to 3 pods
+kubectl get pods -l app=file-processor -w
+
+# Watch HPA
+kubectl get hpa file-processor-scaler -w
+```
+
+## Cost Optimization
+
+- **Scale to zero**: No cost when idle
+- **Spot instances**: 70-90% savings (configured in Deployment affinity)
+- **Long polling**: Reduces SQS API calls
+- **Batch processing**: Up to 10 messages per receive
+- **KEDA efficiency**: Only scales when needed
+
+**Monthly cost estimate** (us-east-1, assuming 10k files/day):
+- SQS: ~$1 (requests + data transfer)
+- S3 storage: ~$5 (100 GB)
+- Compute: ~$10 (spot instances, avg 2 pods)
+- **Total: ~$16/month**
+
+## Future Enhancements
+
+### Short Term
+- [ ] PDF support (PyMuPDF/pdfplumber provider)
+- [ ] PostgreSQL storage integration
+- [ ] Embedding generation (OpenAI/local models)
+- [ ] Graph edge creation
+
+### Medium Term
+- [ ] HTML/web page extraction
+- [ ] DOCX/Office formats
+- [ ] Image OCR (Tesseract/cloud OCR)
+- [ ] Chunking strategies for large documents
+
+### Long Term
+- [ ] Video transcription
+- [ ] Audio processing
+- [ ] Custom ML model inference
+- [ ] Multi-language support
+
+## Monitoring
+
+### CloudWatch Metrics
+
+- **Queue depth**: `ApproximateNumberOfMessagesVisible`
+- **Processing rate**: Messages per second
+- **DLQ depth**: Failed messages
+- **Pod count**: Kubernetes metrics
+
+### Logs
+
+```bash
+# Worker logs
+kubectl logs -l app=file-processor -f
+
+# KEDA scaling events
+kubectl logs -n keda -l app.kubernetes.io/name=keda-operator --tail=100
+```
+
+### Alerts
+
+- Queue depth > 100 for 5+ minutes (backlog)
+- DLQ depth > 0 (processing failures)
+- Pod crash loops (worker errors)
+
+## Troubleshooting
+
+### Pods not scaling
+
+```bash
+# Check ScaledObject
+kubectl describe scaledobject file-processor-scaler
+
+# Check KEDA logs
+kubectl logs -n keda -l app.kubernetes.io/name=keda-operator --tail=50
+```
+
+### Access Denied errors
+
+```bash
+# Verify IRSA annotation
+kubectl get sa file-processor -o yaml | grep role-arn
+
+# Check pod environment
+kubectl get pod -l app=file-processor -o yaml | grep -A5 env:
+```
+
+### Messages not being processed
+
+```bash
+# Check queue has messages
+aws sqs get-queue-attributes \
+  --queue-url QUEUE_URL \
+  --attribute-names ApproximateNumberOfMessages
+
+# Check worker logs
+kubectl logs -l app=file-processor --tail=100
+```
+
+## See Also
+
+- Architecture: `/CLAUDE.md`
+- FS Service: `rem/src/rem/services/fs/` - Unified S3/local file operations
+- Settings: `rem/settings.py` - S3Settings, SQSSettings configuration
+- Infrastructure: `manifests/infra/file-queue/README.md`
+- KEDA: `manifests/platform/keda/README.md`
+- Deployment: `manifests/application/file-processor/README.md`