neotoma 0.3.0 → 0.3.2

package/docs/subsystems/ingestion/ingestion.md

@@ -1,772 +0,0 @@
-# Neotoma [Storing](#storing) Pipeline — [Source](#source) Processing and Truth [Extraction](#extraction)
-## Sources-First Architecture
-Neotoma uses a **sources-first architecture** that decouples raw content storage from [interpretation](#interpretation):
-```
-Raw Content → Sources → [Interpretations](#interpretation) → [Observations](#observation) → [Entity Snapshots](#entity-snapshot)
-```
-**Key Principles:**
-1. **Content-Addressed Storage**: Raw content [stored](#storing) with SHA-256 hash for deduplication
-2. **Versioned [Interpretation](#interpretation)**: Each [interpretation](#interpretation) creates a new [interpretation](#interpretation) record
-3. **Immutable [Observations](#observation)**: Reinterpretation creates NEW [observations](#observation); never modifies existing
-4. **User Isolation**: All tables user-scoped with RLS
-5. **Correction via Priority**: User corrections create priority-1000 [observations](#observation) that override AI
-**Benefits:**
-- **[Provenance](#provenance)**: Every [observation](#observation) traces to [source](#source) and [interpretation](#interpretation)
-- **Auditability**: [Interpretation](#interpretation) config logged; can understand how data was [extracted](#extraction)
-- **Flexibility**: Can reinterpret with new models without losing history
-- **Cost Control**: [Interpretation](#interpretation) quotas prevent runaway AI costs
-See [`docs/subsystems/sources.md`](../sources.md) for complete sources-first architecture details.
-## Scope
-This document covers:
-- File upload and normalization
-- **[Source](#source) storage and deduplication**
-- **[Interpretations](#interpretation)**
-- OCR and text [extraction](#extraction) (file uploads only)
-- [Entity schema](#entity-schema) detection
-- Field [extraction](#extraction) (rule-based, deterministic)
-- [Entity](#entity) resolution (user-scoped)
-- [Event](#event) generation
-- Graph insertion
-- **Correction mechanism**
-- **[Entity](#entity) merge**
-**[Storing](#storing) Paths:**
-1. **File Upload via `ingest()`**: Users explicitly upload PDFs, images, or text files through the UI or `ingest` MCP action. Files are [stored](#storing) in `sources`, undergo [interpretation](#interpretation), and [observations](#observation) are created.
-2. **Structured [Source](#source) via `ingest()`**: Agents provide pre-structured data with explicit [entity types](#entity-type). Bypasses AI [interpretation](#interpretation); creates [observations](#observation) directly with priority=100.
-3. **Corrections via `correct()`**: Users or agents correct [extracted](#extraction) values. Creates priority-1000 [observations](#observation) that override AI [extraction](#extraction).
-All `ingest()` and `correct()` requests MUST include an `idempotency_key`. Reuse with a different payload returns a validation error.
-All paths feed into the same truth model (Sources → [Interpretations](#interpretation) → [Observations](#observation) → [Entity Snapshots](#entity-snapshot)), ensuring unified memory regardless of [storing](#storing) source.
-This document does NOT cover:
-- UI implementation (see `docs/ui/`)
-- Database schema (see `docs/subsystems/schema.md`)
-- Search indexing (see `docs/subsystems/search/search.md`)
-- MCP action specifications (see `docs/specs/MCP_SPEC.md`)
-- Sources-first architecture details (see `docs/subsystems/sources.md`)
-## 1. [Storing](#storing) Pipeline Overview
-### 1.1 Sources-First Pipeline
-```mermaid
-%%{init: {'theme':'neutral'}}%%
-flowchart TD
-    Start[ingest MCP Call] --> Hash[Compute SHA-256 Hash]
-    Hash --> Dedup{Duplicate?}
-    Dedup -->|Yes| ReturnExisting[Return Existing source_id]
-    Dedup -->|No| StoreSource[Create Source Record]
-    StoreSource --> Upload[Upload to Storage]
-    Upload -->|Success| UploadOK[storage_status: uploaded]
-    Upload -->|Failure| Queue[Add to Upload Queue]
-    Queue --> UploadPending[storage_status: pending]
-    UploadOK --> CheckInterpret{interpret=true?}
-    UploadPending --> CheckInterpret
-    CheckInterpret -->|No| Done[Return source_id]
-    CheckInterpret -->|Yes| CheckQuota{Quota OK?}
-    CheckQuota -->|No| QuotaError[Error: QUOTA_EXCEEDED]
-    CheckQuota -->|Yes| CreateRun[Create interpretation]
-    CreateRun --> Interpret[Run Interpretation]
-    Interpret --> ExtractText[Extract Raw Text]
-    ExtractText --> DetectSchema[Detect Schema Type]
-    DetectSchema --> ExtractFields[Extract Fields]
-    ExtractFields --> ValidateSchema[Validate Against Schema]
-    ValidateSchema --> SplitFields{Field Valid?}
-    SplitFields -->|Yes| CreateObs[Create Observation]
-    SplitFields -->|No| StoreFragment[Store in raw_fragments]
-    CreateObs --> ResolveEntity[Resolve Entity User-Scoped]
-    ResolveEntity --> TriggerReducer[Trigger Reducer]
-    TriggerReducer --> ComputeSnapshot[Compute Entity Snapshot]
-    StoreFragment --> IncrementCount[Increment unknown_field_count]
-    IncrementCount --> CreateObs
-    ComputeSnapshot --> Done
-    style Start fill:#e1f5ff
-    style Done fill:#e6ffe6
-    style StoreSource fill:#fff4e6
-    style CreateRun fill:#ffe6f0
-    style ValidateSchema fill:#f0f0ff
-```
-### 1.2 Key Differences from Legacy Pipeline
-| Aspect | Legacy | Sources-First |
-|--------|--------|-------------------------|
-| Storage | Direct to sources | Sources → Interpretations → Observations |
-| Deduplication | Via external_hash | Content-addressed via SHA-256 per user |
-| Interpretation | Implicit | Explicit runs with config logging |
-| Reinterpretation | Not supported | Creates new run, new observations |
-| Corrections | Not supported | Priority-1000 observations |
-| User Isolation | Limited | Full RLS on all tables |
-| Unknown Fields | extraction_metadata | raw_fragments with provenance |
-### 1.3 Legacy Pipeline (Pre-Sources-First)
-```mermaid
-%%{init: {'theme':'neutral'}}%%
-flowchart TD
-    Start[File Upload] --> Validate[Validate File]
-    Validate -->|Valid| Store[Store File]
-    Validate -->|Invalid| RejectFile[Reject: Invalid Format]
-    Store --> Normalize[Normalize Format]
-    Normalize --> ExtractText[Extract Raw Text]
-    ExtractText -->|PDF/Image| OCR[OCR if Needed]
-    ExtractText -->|Text| RawText[Raw Text]
-    OCR --> RawText
-    RawText --> DetectSchema[Detect Schema Type]
-    DetectSchema --> ExtractFields[Extract Fields]
-    ExtractFields --> CategorizeFields[Categorize Known vs Unknown]
-    CategorizeFields --> StoreRawFragments[Store Raw Fragments]
-    CategorizeFields --> ResolveEntities[Resolve Entities]
-    ResolveEntities --> CreateObservations[Create Observations]
-    CreateObservations --> TriggerReducer[Trigger Reducer]
-    TriggerReducer --> ComputeSnapshots[Compute Entity Snapshots]
-    ComputeSnapshots --> GenerateEvents[Generate Events]
-    GenerateEvents --> InsertGraph[Insert into Graph]
-    InsertGraph --> EmitEvents[Emit Observability Events]
-    EmitEvents --> Done[Return Record]
-    RejectFile --> Done
-    style Start fill:#e1f5ff
-    style Done fill:#e6ffe6
-    style DetectSchema fill:#fff4e6
-    style InsertGraph fill:#ffe6e6
-```
-## 2. Step 1: File Upload and Validation
-### 2.1 Supported File Types
-**MVP:**
-- PDF (`.pdf`)
-- Images (`.jpg`, `.jpeg`, `.png`)
-- Text (`.txt`)
-- CSV/Spreadsheet (`.csv`, `.xlsx`) — row-level observation creation
-**Future:**
-- Office docs (`.docx`)
-- Email (`.eml`, `.msg`)
-- HTML/Web content
-**Special Case: Chat Transcripts**
-Chat transcripts (e.g., logs exported from LLM apps like ChatGPT) require **non-deterministic interpretation** that violates the Truth Layer's determinism constraints. Per `docs/specs/GENERAL_REQUIREMENTS.md`:
-- MVP provides a **separate CLI tool** (outside the ingestion pipeline) that can:
-  - Non-deterministically convert raw chat exports into well-structured JSON files (one observation per JSON object with explicit entity types and fields)
-  - Feed the resulting JSON files into the standard deterministic ingestion path
-- Neotoma's Truth Layer ingestion pipeline never performs non-deterministic interpretation of chat content
-- See `docs/specs/MVP_FEATURE_UNITS.md` for chat-to-JSON CLI feature unit details
-### 2.2 Validation Rules
-```typescript
-function validateFile(file: File): ValidationResult {
-  // Check file size (max 50MB)
-  if (file.size > 50 * 1024 * 1024) {
-    return { valid: false, error: "FILE_TOO_LARGE" };
-  }
-  // Check MIME type
-  const allowedTypes = [
-    "application/pdf",
-    "image/jpeg",
-    "image/png",
-    "text/plain",
-  ];
-  if (!allowedTypes.includes(file.type)) {
-    return { valid: false, error: "UNSUPPORTED_FILE_TYPE" };
-  }
-  return { valid: true };
-}
-```
-## 3. Step 2: File Storage and Normalization
-### 3.1 File Storage
-**Pattern:** Generate deterministic filename, store to S3 or local filesystem.
-```typescript
-function storeFile(file: File, userId: string): string {
-  const fileHash = hashFile(file); // SHA-256
-  const filename = `${userId}/${fileHash}.${file.extension}`;
-  // Upload to storage
-  await storage.upload(filename, file.buffer);
-  return `https://storage.example.com/${filename}`;
-}
-```
-**Determinism:** Same file content → same hash → same filename (deduplication).
-### 3.2 Format Normalization
-**Goal:** Convert all inputs to a common format for text extraction.
-```typescript
-function normalizeFile(file: File): Buffer {
-  if (file.type === "application/pdf") {
-    return file.buffer; // Already PDF
-  }
-  if (file.type.startsWith("image/")) {
-    // Convert image to PDF page (preserves quality for OCR)
-    return convertImageToPDF(file.buffer);
-  }
-  if (file.type === "text/plain") {
-    return file.buffer; // Already text
-  }
-  throw new Error("UNSUPPORTED_FILE_TYPE");
-}
-```
-## 4. Step 3: Text Extraction
-### 4.1 PDF Text Extraction
-```typescript
-import pdfParse from "pdf-parse";
-async function extractTextFromPDF(buffer: Buffer): Promise<string> {
-  const data = await pdfParse(buffer);
-  return data.text;
-}
-```
-**Determinism:** Same PDF → same extracted text (pdf-parse is deterministic).
-### 4.2 OCR for Images and Scanned PDFs
-**When to use OCR:**
-- Image files (`.jpg`, `.png`)
-- PDFs with no extractable text
-**OCR Provider (MVP):** Tesseract.js (local, deterministic)
-```typescript
-import Tesseract from "tesseract.js";
-async function performOCR(imageBuffer: Buffer): Promise<string> {
-  const {
-    data: { text },
-  } = await Tesseract.recognize(imageBuffer, "eng", {
-    logger: () => {}, // Suppress logs
-  });
-  return text;
-}
-```
-**Determinism:** Tesseract is deterministic for same image + same version.
-**Critical MVP Constraints (per `docs/specs/GENERAL_REQUIREMENTS.md`):**
-- **No randomness:** Given the same input image/PDF, OCR output text MUST be identical across runs
-- **No model drift:** OCR models/versions are pinned; updates require explicit testing and version bump
-- **Explicit failure handling:** OCR failures or low-confidence regions MUST be surfaced explicitly in metadata without fabricating or inferring missing content
-**Implementation Requirements:**
-- Pin Tesseract.js version in `package.json`
-- Include OCR confidence scores in extraction metadata
-- For low-confidence regions (confidence < 70%), mark as `ocr_low_confidence: true` in metadata rather than dropping text
-## 5. Step 4: Schema Detection and Field Extraction
-### 5.1 Extraction Approach: AI Interpretation with Idempotence
-**Current Architecture (v0.2.0+):** Neotoma uses **AI interpretation for unstructured files** (PDFs, images) with full auditability and system-level idempotence. Rule-based extraction available for structured data from agents.
-
-**Key Points:**
-- **AI interpretation** (via `interpretation.ts` service): Non-deterministic but auditable; interpretation config (model, temperature, prompt_hash) logged
-- **System-level idempotence**: Canonicalization + hashing ensures same source + same config → same final state (no duplicates)
-- **Rule-based extraction**: Available for structured data; deterministic regex/parsing patterns
-- See `docs/architecture/determinism.md` Section 1.4 for idempotence vs. replay determinism distinction
-### 5.2 Schema Detection Rules (Multi-Pattern Matching)
-**Pattern:** Apply **multi-pattern matching** — a document must match **2 or more patterns** for a given type to be classified as that type.
-**Fallback:** If no type matches 2+ patterns, classify as `document` (generic fallback).
-**Rationale:** Multi-pattern matching increases detection precision and reduces false positives.
-```typescript
-// Multi-pattern detection for each application type
-const SCHEMA_DETECTION_PATTERNS = {
-  invoice: [
-    /invoice\s*#?\s*:?\s*([A-Z0-9-]+)/i,
-    /bill\s*to:/i,
-    /amount\s*due:/i,
-    /invoice\s*date/i,
-    /payment\s*terms/i,
-  ],
-  receipt: [
-    /receipt/i,
-    /thank\s*you\s*for\s*your\s*purchase/i,
-    /items?\s*purchased/i,
-    /total\s*amount/i,
-    /payment\s*method/i,
-  ],
-  transaction: [
-    /transaction/i,
-    /debit|credit/i,
-    /account\s*number/i,
-    /balance/i,
-    /posted\s*date/i,
-  ],
-  statement: [
-    /statement/i,
-    /account\s*summary/i,
-    /period\s*from|statement\s*period/i,
-    /beginning\s*balance/i,
-    /ending\s*balance/i,
-  ],
-  contract: [
-    /agreement|contract/i,
-    /parties|between/i,
-    /effective\s*date/i,
-    /terms\s*and\s*conditions/i,
-    /signature/i,
-  ],
-  travel_document: [
-    /itinerary|booking|reservation/i,
-    /departure|arrival/i,
-    /flight|train|hotel/i,
-    /confirmation\s*number/i,
-    /passenger|guest/i,
-  ],
-  identity_document: [
-    /passport|driver.*license|national.*id/i,
-    /document\s*number/i,
-    /date\s*of\s*issue|issued/i,
-    /expiry|expires/i,
-    /nationality|country\s*of\s*issue/i,
-  ],
-  message: [/from:/i, /to:/i, /subject:/i, /sent:/i, /reply|forward/i],
-};
-function detectSchemaType(rawText: string): string {
-  const matchCounts: Record<string, number> = {};
-  // Count pattern matches for each type
-  for (const [type, patterns] of Object.entries(SCHEMA_DETECTION_PATTERNS)) {
-    matchCounts[type] = patterns.filter((pattern) =>
-      pattern.test(rawText)
-    ).length;
-  }
-  // Find types with 2+ matches
-  const candidates = Object.entries(matchCounts)
-    .filter(([_, count]) => count >= 2)
-    .sort(([_, countA], [__, countB]) => countB - countA);
-  // Return type with most matches, or fallback to 'document'
-  if (candidates.length > 0) {
-    return candidates[0][0];
-  }
-  return "document"; // Generic fallback
-}
-```
-**Example: Invoice Detection**
-```typescript
-// Raw text from invoice PDF
-const rawText = `
-INVOICE #INV-2024-001
-Bill To: Customer Corp
-Amount Due: $1,500.00
-Invoice Date: 2024-01-15
-Payment Terms: Net 30
-`;
-// Pattern matches:
-// ✅ /invoice\s*#?\s*:?\s*([A-Z0-9-]+)/i → Matches "INVOICE #INV-2024-001"
-// ✅ /bill\s*to:/i → Matches "Bill To:"
-// ✅ /amount\s*due:/i → Matches "Amount Due:"
-// ✅ /invoice\s*date/i → Matches "Invoice Date:"
-// ✅ /payment\s*terms/i → Matches "Payment Terms:"
-// Result: 5 matches → type = 'invoice' ✅
-```
-**Example: Generic Document Fallback**
-```typescript
-// Raw text from generic document
-const rawText = `
-This is a research paper about machine learning techniques.
-The paper discusses various approaches to neural networks.
-`;
-// No type matches 2+ patterns
-// Result: type = 'document' (fallback) ✅
-```
-**Note:** Schema detection patterns shown here are for rule-based fallback. In production, AI interpretation handles schema detection for unstructured files (with audit trail).
-**Tier 1 ICP Alignment:** Schema detection supports AI-Native Operators (research, contracts, travel), Knowledge Workers (legal docs, research papers, client communications), and Founders (company docs, product docs, investor materials).
-**Complete Patterns:** See [`docs/subsystems/record_types.md`](../record_types.md) section 7 for all detection patterns.
-### 5.3 Schema Registry
-**Location:** `src/config/record_types.ts`
-**Two-Tier System:**
-- **Application types** (this list): Used in code, database (`observations.entity_type`), MCP actions
-- **Schema families** (`Financial`, `Productivity`, etc.): Used for documentation only
-**Tier 1 ICP-aligned application types:**
-```typescript
-export const RECORD_TYPES = {
-  // Finance
-  INVOICE: 'invoice',
-  RECEIPT: 'receipt',
-  TRANSACTION: 'transaction',
-  STATEMENT: 'statement',
-  ACCOUNT: 'account',
-  // Productivity
-  NOTE: 'note',
-  DOCUMENT: 'document',  // Generic fallback
-  MESSAGE: 'message',
-  TASK: 'task',
-  PROJECT: 'project',
-  EVENT: 'event',
-  // Knowledge
-  "contact",
-  "dataset",
-  // Legal/Compliance
-  "contract",
-  // Travel
-  "travel_document",
-  // Identity
-  "identity_document",
-];
-// Fallback for unrecognized types
-const FALLBACK_TYPE = "document";
-```
-**Schema-to-Fields Mapping:**
-```typescript
-export const SCHEMA_FIELDS = {
-  invoice: ["invoice_number", "amount_due", "due_date", "vendor", "status"],
-  receipt: [
-    "receipt_number",
-    "amount_total",
-    "date",
-    "merchant_name",
-    "currency",
-  ],
-  contract: [
-    "contract_number",
-    "parties",
-    "effective_date",
-    "expiration_date",
-    "status",
-  ],
-  document: ["title", "summary", "source", "tags", "link"],
-  note: ["title", "content", "tags", "source", "summary"],
-  message: ["thread_id", "sender", "recipient", "subject", "body", "sent_at"],
-  travel_document: [
-    "booking_reference",
-    "departure_datetime",
-    "arrival_datetime",
-    "airline",
-  ],
-  identity_document: [
-    "full_name",
-    "document_number",
-    "date_expiry",
-    "nationality",
-  ],
-};
-```
-## 6. Step 5: Field Extraction
-### 6.1 Rule-Based Extraction
-**Pattern:** Use regex or parsing logic to extract fields deterministically.
-```typescript
-function extractFieldsForInvoice(rawText: string): any {
-  return {
-    schema_version: "1.0",
-    invoice_number: extractInvoiceNumber(rawText),
-    amount_due: extractAmount(rawText),
-    currency: extractCurrency(rawText) || "USD",
-    due_date: extractDate(rawText, "due|payable"),
-    vendor: extractVendorName(rawText),
-    status: extractPaymentStatus(rawText),
-  };
-}
-function extractFieldsForContract(rawText: string): any {
-  return {
-    schema_version: "1.0",
-    contract_number: extractContractNumber(rawText),
-    parties: extractParties(rawText),
-    effective_date: extractDate(rawText, "effective|commence"),
-    expiration_date: extractDate(rawText, "expir|terminat"),
-    status: extractContractStatus(rawText),
-  };
-}
-function extractFieldsForDocument(rawText: string): any {
-  return {
-    schema_version: "1.0",
-    title: extractTitle(rawText),
-    summary: extractFirstSentence(rawText),
-    source: "upload",
-  };
-}
-function extractInvoiceNumber(text: string): string | null {
-  const match = text.match(/invoice\s*#?\s*:?\s*([A-Z0-9-]+)/i);
-  return match ? match[1] : null;
-}
-function extractAmount(text: string): number | null {
-  const match = text.match(/total|amount\s*:?\s*\$?([0-9,]+\.?\d{0,2})/i);
-  return match ? parseFloat(match[1].replace(",", "")) : null;
-}
-```
-**Note:** Rule-based extractors shown here are for structured data from agents. For unstructured files (PDFs, images), the `interpretation.ts` service uses AI interpretation with full auditability (config logged) and system-level idempotence (canonicalization + hashing).
-### 6.2 Schema-Specific Extractors
-Each schema type has a dedicated extractor function:
-```typescript
-const EXTRACTORS = {
-  invoice: extractFieldsForInvoice,
-  receipt: extractFieldsForReceipt,
-  contract: extractFieldsForContract,
-  travel_document: extractFieldsForTravelDocument,
-  identity_document: extractFieldsForIdentityDocument,
-  message: extractFieldsForMessage,
-  note: extractFieldsForNote,
-  document: extractFieldsForDocument,
-};
-function extractFields(rawText: string, schemaType: string): any {
-  const extractor = EXTRACTORS[schemaType];
-  if (!extractor) {
-    // Fallback to generic document extraction
-    return extractFieldsForDocument(rawText);
-  }
-  return extractor(rawText);
-}
-```
-## 7. Step 6: Entity Resolution and Observation Creation
-**Four-Layer Truth Model Context:** This section implements the middle layers of Neotoma's four-layer truth model (Document → Entity → Observation → Entity Snapshot). After field extraction from documents, we:
-1. **Resolve Entities** — Identify canonical entities (people, companies, locations)
-2. **Create Observations** — Store granular, source-specific facts about entities
-3. **Trigger Reducers** — Compute entity snapshots from observations using merge policies from schema registry
-4. **Link to Graph** — Connect documents, entities, observations, and snapshots
-See [`docs/architecture/architectural_decisions.md`](../../architecture/architectural_decisions.md) for the complete four-layer model rationale and [`docs/subsystems/observation_architecture.md`](../observation_architecture.md) for detailed observation patterns.
-### 7.1 Entity Extraction
-**Entities:** People, companies, locations mentioned in extracted fields.
-```typescript
-function extractEntities(properties: any, schemaType: string): Entity[] {
-  const entities: Entity[] = [];
-  // Use application types (invoice, receipt) not schema families (FinancialRecord)
-  if (schemaType === "invoice" || schemaType === "receipt") {
-    if (properties.vendor_name) {
-      entities.push({
-        entity_type: "company",
-        raw_value: properties.vendor_name,
-      });
-    }
-  }
-  if (schemaType === "identity_document") {
-    if (properties.full_name) {
-      entities.push({
-        entity_type: "person",
-        raw_value: properties.full_name,
-      });
-    }
-  }
-  return entities;
-}
-```
-### 7.2 Entity ID Generation (Deterministic)
-```typescript
-import { createHash } from "crypto";
-function generateEntityId(entityType: string, canonicalName: string): string {
-  const normalized = normalizeEntityValue(entityType, canonicalName);
-  const hash = createHash("sha256")
-    .update(`${entityType}:${normalized}`)
-    .digest("hex");
-  return `ent_${hash.substring(0, 24)}`;
-}
-function normalizeEntityValue(entityType: string, raw: string): string {
-  let normalized = raw.trim().toLowerCase();
-  if (entityType === "company") {
-    normalized = normalized.replace(
-      /\s+(inc|llc|ltd|corp|corporation)\.?$/i,
-      ""
-    );
-  }
-  return normalized;
-}
-```
-**Determinism:** Same name → same ID (globally).
-### 7.3 User-Scoped Entity Resolution
-Entity resolution is **user-scoped** to ensure user isolation:
-```typescript
-async function resolveEntity(
-  entityType: string,
-  extractedFields: Record<string, unknown>,
-  userId: string
-): Promise<string> {
-  // 1. Exact match on external_id (user-scoped)
-  if (extractedFields.external_id) {
-    const existing = await findEntityByExternalId(
-      entityType, 
-      extractedFields.external_id, 
-      userId  // Always filter by user
-    );
-    if (existing) return existing.id;
-  }
-  // 2. Heuristic match on key fields (user-scoped)
-  const matchKey = getMatchKey(entityType, extractedFields);
-  if (matchKey) {
-    const existing = await findEntityByMatchKey(
-      entityType, 
-      matchKey, 
-      userId  // Always filter by user
-    );
-    if (existing) return existing.id;
-  }
-  // 3. Create new entity (stamp user_id)
-  return await createEntity(entityType, extractedFields, userId);
-}
-async function findEntityByExternalId(
-  entityType: string,
-  externalId: string,
-  userId: string
-): Promise<{ id: string } | null> {
-  const data = await db.queryOne("entities", {
-    entity_type: entityType,
-    external_id: externalId,
-    user_id: userId,
-    merged_to_entity_id: null,  // Exclude merged entities
-  }, ["id"]);
-  return data;
-}
-```
-**Important:** Entity resolution may create duplicates (heuristic matching). Use `merge_entities()` to repair duplicates.
-See [`docs/subsystems/entity_merge.md`](../entity_merge.md) for merge mechanism.
-## 7.5 Observation Creation
-**Overview:** After entity resolution, create observations — granular, source-specific facts extracted from documents. Observations are the intermediate layer between documents and entity snapshots in the four-layer truth model.
-**Process:**
-1. **Load Schema from Registry:** Fetch active schema version and field definitions from schema registry
-2. **Categorize Fields:** Separate known fields (match schema) from unknown fields
-3. **Store Raw Fragments:** Unknown fields stored in `raw_fragments` table with typed envelopes
-4. **Create Observations:** For each entity identified:
-   - Create observation with entity_id, entity_type, fields
-   - Set specificity_score based on field confidence
-   - Set source_priority based on document source
-   - Reference schema_version for deterministic replay
-   - Link to source_id
-5. **Trigger Reducer:** Call reducer engine to compute entity snapshot using merge policies from schema registry
-6. **Store Snapshot:** Reducer produces entity snapshot with provenance tracking
-**Schema Registry Integration:** Observations reference the schema version from schema registry, enabling deterministic replay if schemas evolve. Reducers use merge policies configured in schema registry to resolve conflicts between multiple observations.
-**Example:**
-```typescript
-async function createObservations(
-  sourceId: string,
-  entities: Entity[],
-  extractedFields: any,
-  schemaType: string,
-  schemaVersion: string
-): Promise<void> {
-  // Categorize known vs unknown fields
-  const schema = await schemaRegistry.loadActiveSchema(schemaType);
-  const { knownFields, unknownFields } = categorizeFields(
-    extractedFields,
-    schema
-  );
-  // Store raw fragments for unknown fields
-  for (const [key, value] of Object.entries(unknownFields)) {
-    await rawFragmentRepo.create({
-      source_id: sourceId,
-      fragment_key: key,
-      fragment_value: value,
-      fragment_envelope: { type: typeof value, confidence: "medium" },
-    });
-  }
-  // Create observations for each entity
-  for (const entity of entities) {
-    const observation = await observationRepo.create({
-      entity_id: entity.id,
-      entity_type: entity.type,
-      schema_version: schemaVersion,
-      source_id: sourceId,
-      observed_at: new Date(),
-      specificity_score: calculateSpecificity(entity, knownFields),
-      source_priority: calculatePriority(sourceId),
-      fields: extractEntityFields(knownFields, entity),
-    });
-    // Trigger reducer to compute snapshot
-    await reducerEngine.computeSnapshot(entity.id);
-  }
-}
-```
-**Related Documents:**
-- [`docs/architecture/architectural_decisions.md`](../../architecture/architectural_decisions.md) — Four-layer truth model
-- [`docs/subsystems/observation_architecture.md`](../observation_architecture.md) — Complete observation architecture
-- [`docs/subsystems/reducer.md`](../reducer.md) — Reducer merge strategies
-- [`docs/subsystems/schema_registry.md`](../schema_registry.md) — Schema registry patterns and merge policies
-## 8. Step 7: Event Generation
-### 8.1 Event Extraction
-**Events:** Timeline events derived from date fields.
-```typescript
-function generateEvents(
-  sourceId: string,
-  fields: any,
-  schemaType: string
-): Event[] {
-  const events: Event[] = [];
-  const dateFields = getDateFields(schemaType);
-  for (const fieldName of dateFields.sort()) {
-    const dateValue = fields[fieldName];
-    if (!dateValue) continue;
-    const eventType = mapFieldToEventType(fieldName, schemaType);
-    const eventId = generateEventId(sourceId, fieldName, dateValue);
-    events.push({
-      id: eventId,
-      event_type: eventType,
-      event_timestamp: dateValue,
-      source_id: sourceId,
-      source_field: fieldName,
-    });
-  }
-  return events;
-}
-function mapFieldToEventType(fieldName: string, schemaType: string): string {
-  // Use application types (invoice, travel_document) not schema families
-  if (schemaType === "invoice") {
-    if (fieldName === "date_issued") return "InvoiceIssued";
-    if (fieldName === "date_due") return "InvoiceDue";
-  }
-  if (schemaType === "travel_document") {
-    if (fieldName === "departure_datetime") return "FlightDeparture";
-    if (fieldName === "arrival_datetime") return "FlightArrival";
-  }
-  return "GenericEvent";
-}
-```
-## 9. Step 8: Graph Insertion
-### 9.1 Transactional Insert
-```typescript
-async function insertSourceAndGraph(
-  source: Source,
-  entities: Entity[],
-  events: Event[]
-): Promise<string> {
-  return await db.transaction(async (tx) => {
-    // Insert source
-    const sourceId = await tx.insert("sources", source);
-    // Insert entities (if not exist)
-    for (const entity of entities) {
-      await tx.upsert("entities", entity);
-      await tx.insert("source_entity_edges", {
-        source_id: sourceId,
-        entity_id: entity.id,
-      });
-    }
-    // Insert events
-    for (const event of events) {
-      await tx.insert("timeline_events", event);
-      await tx.insert("source_event_edges", {
-        source_id: sourceId,
-        event_id: event.id,
-      });
-    }
-    return sourceId;
-  });
-}
-```
-**Determinism:** Same inputs → same graph structure.
-## 10. Error Handling
-### 10.1 Error Types
-| Error Code              | Meaning                 | Retry? |
-| ----------------------- | ----------------------- | ------ |
-| `FILE_TOO_LARGE`        | File exceeds size limit | No     |
-| `UNSUPPORTED_FILE_TYPE` | File type not supported | No     |
-| `OCR_FAILED`            | OCR processing failed   | Yes    |
-| `EXTRACTION_FAILED`     | Field extraction failed | No     |
-| `DB_INSERT_FAILED`      | Database write failed   | Yes    |
-### 10.2 Retry Logic
-```typescript
-async function ingestFileWithRetry(file: File): Promise<Record> {
-  const maxRetries = 3;
-  let attempt = 0;
-  while (attempt < maxRetries) {
-    try {
-      return await ingestFile(file);
-    } catch (error) {
-      if (isTransientError(error) && attempt < maxRetries - 1) {
-        attempt++;
-        await sleep(1000 * attempt); // Exponential backoff
-      } else {
-        throw error;
-      }
-    }
-  }
-}
-```
-## Agent Instructions
-### When to Load This Document
-Load `docs/subsystems/ingestion/ingestion.md` when:
-- Implementing file upload or ingestion logic
-- Adding new schema types or field extractors
-- Modifying entity resolution or event generation
-- Debugging ingestion errors
-- Working with OCR or text extraction
-### Required Co-Loaded Documents
-- `docs/NEOTOMA_MANIFEST.md` (determinism, immutability)
-- `docs/architecture/determinism.md` (deterministic extraction)
-- `docs/subsystems/schema.md` (schema structure)
-- `docs/subsystems/ingestion/state_machines.md` (ingestion states)
-### Constraints Agents Must Enforce
-1. **AI interpretation for unstructured files** with audit trail (interpretation config logged)
-2. **System-level idempotence** enforced (canonicalization + hashing)
-3. **Entity IDs MUST be hash-based**
-4. **Event IDs MUST be hash-based**
-5. **All ingestion MUST be transactional**
-6. **Same file → same source_id** (via content hash; deduplication)
-7. **All errors MUST use ErrorEnvelope**
-### Forbidden Patterns
-- Bypassing interpretation service for unstructured files
-- Nondeterministic entity IDs
-- Non-transactional graph inserts
-- Skipping error handling
-- Not logging interpretation config
-### Validation Checklist
-- [ ] AI interpretation uses interpretation service (with config logging)
-- [ ] Idempotence enforced (canonicalization + hashing)
-- [ ] Entity IDs are hash-based
-- [ ] Event IDs are hash-based
-- [ ] Graph insertion is transactional
-- [ ] Error handling covers all failure modes
-- [ ] Tests cover all schema types
-- [ ] Interpretation config logged for audit trail