neotoma 0.3.0 → 0.3.2

package/docs/architecture/schema_expansion.md

@@ -1,260 +0,0 @@
-# Automatic Schema Expansion Architecture
-## Overview
-**Problem:** Users upload files with fields that don't match existing schemas. Currently, these fields are preserved in `extraction_metadata.unknown_fields`, but users must manually create schemas.
-**Solution:** Automatic schema expansion detects patterns in `extraction_metadata.unknown_fields` across multiple records and automatically creates schemas when patterns are strong enough, with user opt-in control and retroactive review.
-**Key Principle:** "Explicit control" means the user explicitly enables the capability once, not approval every time. Similar to how `cloudStorageEnabled` works.
-## 1. User Preference Model
-### Settings
-Users can opt-in to automatic schema expansion via settings:
-```typescript
-interface Settings {
-  autoExpandSchemas: boolean;  // Default: false
-  autoSchemaConfidenceThreshold: 'high' | 'medium' | 'low';  // Default: 'high'
-  autoSchemaMinOccurrences: number;  // Default: 5
-  autoReextractOnSchemaCreate: boolean;  // Default: false
-}
-```
-**Settings UI:**
-- Checkbox: "Automatically expand schemas"
-- Dropdown: Confidence threshold (high/medium/low)
-- Number input: Minimum occurrences (field must appear in N+ records)
-- Checkbox: "Re-extract records when schema created"
-**Rationale:**
-- Explicit user opt-in (respects NEOTOMA_MANIFEST.md Section 5.2)
-- Configurable thresholds (user controls sensitivity)
-- Reversible (user can disable anytime)
-## 2. Pattern Detection Architecture
-### Data Source
-Pattern detection analyzes `extraction_metadata.unknown_fields` across records:
-```sql
-SELECT type, extraction_metadata->'unknown_fields' as unknown_fields
-FROM records
-WHERE created_at > NOW() - INTERVAL '30 days'
-  AND extraction_metadata->'unknown_fields' IS NOT NULL
-ORDER BY created_at DESC
-LIMIT 1000;
-```
-### Pattern Analysis
-**Algorithm:**
-1. Group records by `type` (existing schema)
-2. Extract field names from `unknown_fields` across records
-3. Calculate frequency: how many records contain each unknown field
-4. Calculate consistency: field appears in same context across records
-5. Generate suggestions when patterns are strong
-**Example:**
-- 10 records with `type = 'document'` all have `unknown_fields.purchase_order`
-- 9 records have `unknown_fields.client_organization`
-- 8 records have `unknown_fields.timeline_weeks`
-- → Suggest creating `project_proposal` schema or expanding `document` schema
-### Schema Suggestion Structure
-```typescript
-interface SchemaSuggestion {
-  type: 'new_type' | 'expand_existing';
-  type_name?: string;  // For new types
-  existing_type?: string;  // For expansions
-  suggested_fields: Array<{
-    field_name: string;
-    field_type: 'string' | 'number' | 'date' | 'array' | 'object';
-    frequency: number;  // How often this field appears
-    sample_values: unknown[];  // Examples for user review
-    confidence: 'high' | 'medium' | 'low';
-  }>;
-  affected_records: number;  // How many records would benefit
-  pattern_evidence: {
-    records_analyzed: number;
-    pattern_consistency: number;  // 0-1 score
-    extraction_quality: number;  // 0-1 score
-  };
-}
-```
-## 3. Automatic Schema Creation
-### Deterministic Fingerprinting
-**Requirement:** Same patterns MUST produce same schema (deterministic).
-**Implementation:**
-- Generate fingerprint from pattern: hash of field names + types + frequencies
-- Check if schema with fingerprint already exists (idempotent)
-- If exists, use existing schema; if not, create new
-```typescript
-function generateSchemaFingerprint(suggestion: SchemaSuggestion): string {
-  const pattern = {
-    fields: suggestion.suggested_fields.map(f => ({
-      name: f.field_name,
-      type: f.field_type,
-      frequency: f.frequency
-    })).sort((a, b) => a.name.localeCompare(b.name)),
-    evidence: suggestion.pattern_evidence
-  };
-  return sha256(JSON.stringify(pattern));
-}
-```
-### Auto-Creation Criteria
-Schema is auto-created if:
-1. User preference `autoExpandSchemas` is enabled
-2. Pattern consistency score > 0.8
-3. Affected records >= `autoSchemaMinOccurrences`
-4. All suggested fields have confidence = 'high' (if threshold is 'high')
-### Schema Storage
-**Custom Schemas Table:**
-```sql
-CREATE TABLE custom_schemas (
-  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
-  type TEXT NOT NULL UNIQUE,
-  schema_version TEXT NOT NULL DEFAULT '1.0',
-  definition JSONB NOT NULL,
-  detection_patterns TEXT[],
-  extraction_rules JSONB,
-  fingerprint TEXT UNIQUE,  -- Deterministic hash of patterns
-  auto_created BOOLEAN DEFAULT false,
-  created_by UUID REFERENCES auth.users(id),
-  created_at TIMESTAMPTZ DEFAULT NOW(),
-  updated_at TIMESTAMPTZ DEFAULT NOW()
-);
-```
-**Schema Definition Structure:**
-```typescript
-interface SchemaDefinition {
-  type: string;
-  schema_version: string;
-  fields: Array<{
-    name: string;
-    type: 'string' | 'number' | 'date' | 'array' | 'object';
-    required: boolean;
-    extraction_pattern?: string;  // Regex pattern
-    description?: string;
-  }>;
-  detection_patterns: string[];  // Regex patterns for type detection
-}
-```
-## 4. Background Processing
-### Integration Point
-Automatic schema expansion runs in background (non-blocking) after record creation:
-```typescript
-async function processUploadWithAutoSchema(
-  rawText: string,
-  fileName: string,
-  userId: string
-): Promise<NeotomaRecord> {
-  // Normal extraction
-  const extraction = await extractAndValidate(rawText, detectedType);
-  const record = await createRecord({ ...extraction, user_id: userId });
-  
-  // Background auto-expansion (non-blocking)
-  if (await getSetting(userId, 'autoExpandSchemas')) {
-    analyzeAndAutoExpandSchemas(userId, record).catch(err => 
-      logError('Auto schema expansion failed', err)
-    );
-  }
-  
-  return record;
-}
-```
-### Processing Flow
-1. Query recent records with `unknown_fields` (last 30 days)
-2. Analyze patterns (group by type, calculate frequencies)
-3. Generate suggestions if patterns are strong
-4. Auto-create schemas if criteria met
-5. Optionally re-extract affected records with new schema
-**Error Handling:**
-- Failures in background processing don't block record creation
-- Errors logged for debugging
-- User can review failed expansions in UI
-## 5. Schema Review and Management
-### Schema Management UI
-**Features:**
-- List canonical schemas (read-only, 16 MVP types)
-- List custom schemas (manual + auto-created)
-- Show record count per schema
-- Actions per schema:
-  - View details
-  - Lock (auto-created → manual, prevent deletion)
-  - Delete (if unused)
-**Badges:**
-- "Auto-created" vs "Manual"
-- "Locked" (previously auto-created, now locked)
-**Filters:**
-- All schemas
-- Canonical only
-- Custom only
-- Auto-created only
-- Manual only
-### Retroactive Control
-**User Actions:**
-1. **Disable auto-expansion:** Turn off `autoExpandSchemas` setting
-2. **Delete auto-created schema:** Remove schema if unused
-3. **Lock schema:** Convert auto-created to manual (prevent deletion)
-4. **Review suggestions:** View pending suggestions before auto-creation (optional UI)
-**Rationale:**
-- User maintains control retroactively
-- Can disable feature anytime
-- Can clean up unwanted auto-created schemas
-- Can lock important auto-created schemas
-## 6. Deterministic Behavior
-### Requirements
-**MUST:**
-- Same patterns → same schema (fingerprinting ensures idempotent creation)
-- Deterministic extraction with new schemas (regex patterns only)
-- Same input → same output (maintains manifest requirement)
-**MUST NOT:**
-- Create different schemas for same patterns
-- Use non-deterministic logic in pattern detection
-- Use LLM for schema creation (respects MVP constraint)
-### Fingerprinting Example
-```typescript
-// Pattern detected:
-// - purchase_order (string, appears in 8/10 records)
-// - client_organization (string, appears in 9/10 records)
-// - timeline_weeks (number, appears in 7/10 records)
-// Fingerprint: sha256("purchase_order:string:0.8,client_organization:string:0.9,timeline_weeks:number:0.7")
-// Result: Same pattern always generates same fingerprint
-// → If schema with this fingerprint exists, use it
-// → If not, create new schema with this fingerprint
-```
-## 7. Future Implementation Notes
-### MCP Actions Required
-1. **`suggest_schema_expansion`**: Return schema suggestions for review
-2. **`create_schema`**: Create custom schema (manual or auto)
-3. **`list_schemas`**: List all schemas (canonical + custom)
-4. **`delete_schema`**: Delete custom schema (if unused)
-5. **`lock_schema`**: Convert auto-created schema to manual
-### UI Components Required
-1. **Settings UI:** Schema management section with auto-expansion toggle
-2. **Schemas View:** List, filter, and manage all schemas
-3. **Schema Details:** View schema definition, extraction rules, record count
-4. **Schema Suggestions:** Review pending suggestions (optional)
-### Integration Points
-1. **Ingestion pipeline:** Trigger background analysis after record creation
-2. **Extraction service:** Check custom schemas before canonical schemas
-3. **Settings service:** Store and retrieve user preferences
-4. **Schema service:** Pattern analysis, fingerprinting, schema creation
-### Database Migrations
-1. Create `custom_schemas` table
-2. Add indexes for fast schema lookup
-3. Add indexes for pattern analysis queries
-## 8. Constraints and Compliance
-### Manifest Compliance
-**Respects:**
-- ✅ Explicit user control (user opts in via settings)
-- ✅ No synthetic data (schemas derived from actual extraction patterns)
-- ✅ Deterministic (same patterns → same schema via fingerprinting)
-- ✅ Reversible (can disable, delete, or revert schemas)
-- ✅ No LLM (pattern detection uses frequency analysis, not inference)
-**Post-MVP Rationale:**
-- MVP requires only 16 canonical schemas + `document` fallback
-- Automatic expansion is enhancement, not core requirement
-- Reduces MVP complexity and risk
-## 9. Related Documentation
-- **Layered storage model:** `docs/architecture/schema_handling.md`
-- **Extraction metadata:** `docs/subsystems/schema.md` Section 3.11
-- **Schema definitions:** `docs/subsystems/record_types.md` Section 4
-- **Manifest constraints:** `docs/NEOTOMA_MANIFEST.md` Sections 5.2, 5.6, 12.2
-## Agent Instructions
-### When to Load This Document
-Load `docs/architecture/schema_expansion.md` when:
-- Planning automatic schema expansion implementation (post-MVP)
-- Understanding pattern detection requirements
-- Designing schema review and management UI
-- Implementing MCP actions for schema management
-### Implementation Status
-**MVP:** Documented but not implemented. Focus on core extraction and layered storage.
-**Post-MVP:** Implement automatic schema expansion following this architecture.
-**END OF DOCUMENT**

package/docs/architecture/schema_handling.md

@@ -1,209 +0,0 @@
-# Schema Handling Architecture
-## 1. Three-Layer Storage Model
-Neotoma uses a **three-layer storage model** to reconcile two requirements:
-1. **Preserve all extracted data** (users can upload any [source](#source) without data loss)
-2. **Maintain deterministic [entity schemas](#entity-schema)** (structured, queryable truth layer)
-### Layer 1: `raw_text` (Immutable Original)
-**Purpose:** Store the original extracted text exactly as extracted.
-**Characteristics:**
-- Immutable after [storing](#storing) (per NEOTOMA_MANIFEST.md Section 5.4)
-- Full [source](#source) preserved
-- Used for re-[extraction](#extraction) if [entity schemas](#entity-schema) evolve
-**Storage:** Stored with [source](#source) in the `sources` table
-### Layer 2: `properties` ([Entity Schema](#entity-schema)-Compliant, Deterministic)
-**Purpose:** Store only fields that match the assigned [entity schema](#entity-schema) definition.
-**Characteristics:**
-- [Entity schema](#entity-schema)-compliant fields only
-- Deterministic (same input → same output)
-- Queryable via JSONB indexes
-- Must include `schema_version: "1.0"`
-- Used for [entity](#entity) [extraction](#extraction), [event](#event) generation, structured queries
-**Storage:** Stored in [observations](#observation) `fields` JSONB column
-**Example:**
-```json
-{
-  "schema_version": "1.0",
-  "invoice_number": "INV-001",
-  "amount": 1500.00,
-  "currency": "USD",
-  "date_issued": "2024-01-15T00:00:00Z",
-  "vendor_name": "Acme Corp"
-}
-```
-### Layer 3: `extraction_metadata` (Preservation Layer)
-**Purpose:** Store unknown fields, validation warnings, and [extraction](#extraction) quality metrics.
-**Characteristics:**
-- Unknown fields ([extracted](#extraction) but not in [entity schema](#entity-schema) definition)
-- Validation warnings (missing required fields, unknown fields)
-- [Extraction](#extraction) quality metrics (fields_extracted_count, fields_filtered_count)
-- Used for debugging, quality tracking, future [entity schema](#entity-schema) expansion
-**Storage:** Stored in [observations](#observation) or [raw_fragments](#raw-fragments) table
-**Example:**
-```json
-{
-  "unknown_fields": {
-    "purchase_order": "PO-789",
-    "internal_cost_center": "CC-456"
-  },
-  "warnings": [
-    {
-      "type": "unknown_field",
-      "field": "purchase_order",
-      "message": "Field 'purchase_order' not defined for type 'invoice' - preserved in extraction_metadata"
-    }
-  ],
-  "extraction_quality": {
-    "fields_extracted_count": 7,
-    "fields_filtered_count": 2,
-    "matched_patterns": ["invoice_number_pattern", "amount_due_pattern"]
-  }
-}
-```
-### Why Three Layers?
-**Problem:** Users want to upload any [source](#source) without losing data, but we need deterministic [entity schemas](#entity-schema) for queries and AI reasoning.
-**Solution:**
-- `raw_text`: Never loses [source](#source)
-- `properties`: Maintains [entity schema](#entity-schema) compliance for deterministic queries
-- `extraction_metadata`: Preserves everything else for future use
-**Benefits:**
-1. **Zero data loss:** All [extracted](#extraction) data preserved somewhere
-2. **[Entity schema](#entity-schema) compliance:** `properties` contains only valid [entity schema](#entity-schema) fields
-3. **Determinism:** Same input → same `properties` structure
-4. **Queryability:** Structured queries use `properties` (indexed)
-5. **Explainability:** Warnings explain what was filtered and why
-6. **Future evolution:** Unknown fields available for [entity schema](#entity-schema) expansion
-## 2. Field Validation Pattern
-### [Extraction](#extraction) and Validation Flow
-```typescript
-function extractAndValidate(
-  rawText: string,
-  detectedEntityType: string
-): {
-  properties: Record<string, unknown>;
-  extraction_metadata: ExtractionMetadata;
-} {
-  // 1. [Extract](#extraction) ALL fields (including unknown ones)
-  const allExtracted = extractAllFields(rawText, detectedEntityType);
-  
-  // 2. Get [entity schema](#entity-schema) definition for detected [entity type](#entity-type)
-  const entitySchema = getEntitySchema(detectedEntityType);
-  
-  // 3. Partition into structured vs unknown
-  const { structured, unknown } = partitionFields(allExtracted, entitySchema);
-  
-  // 4. Validate required fields
-  const warnings = validateRequired(structured, entitySchema);
-  
-  // 5. Build structured properties ([entity schema](#entity-schema)-compliant only)
-  const properties = {
-    schema_version: "1.0",
-    ...structured
-  };
-  
-  // 6. Build [extraction](#extraction) metadata (preservation layer)
-  const extraction_metadata: ExtractionMetadata = {
-    ...(Object.keys(unknown).length > 0 && { unknown_fields: unknown }),
-    ...(warnings.length > 0 && { warnings }),
-    extraction_quality: {
-      fields_extracted_count: Object.keys(allExtracted).length,
-      fields_filtered_count: Object.keys(unknown).length,
-      matched_patterns: getMatchedPatterns(rawText, detectedEntityType)
-    }
-  };
-  
-  return { properties, extraction_metadata };
-}
-```
-### Partition Logic
-**Function:** Separate [extracted](#extraction) fields into [entity schema](#entity-schema)-compliant vs unknown
-**Rules:**
-- Fields defined in [entity schema](#entity-schema) definition → `structured` → `properties`
-- Fields not defined in [entity schema](#entity-schema) definition → `unknown` → `extraction_metadata.unknown_fields`
-- Missing required fields → warning in `extraction_metadata.warnings`
-- Unknown fields → warning in `extraction_metadata.warnings`
-**Example:**
-```typescript
-// [Extracted](#extraction): { invoice_number, amount, purchase_order, internal_cost_center }
-// [Entity schema](#entity-schema) definition for 'invoice': { invoice_number, amount, currency, date_issued }
-// Partition result:
-// structured: { invoice_number, amount } → goes to properties
-// unknown: { purchase_order, internal_cost_center } → goes to extraction_metadata.unknown_fields
-```
-### Validation Rules
-**MUST:**
-- Always create [observation](#observation) (never reject entire [source](#source))
-- Filter unknown fields to `extraction_metadata.unknown_fields`
-- Log warnings for filtered fields and missing required fields
-- Include `schema_version: "1.0"` in all `properties`
-- Preserve all [extracted](#extraction) data (in `properties` or `extraction_metadata`)
-**MUST NOT:**
-- Reject [observations](#observation) due to unknown fields
-- Reject [observations](#observation) due to missing optional fields
-- Store unknown fields in `properties`
-- Modify or guess field values
-## 3. [Extraction](#extraction) Metadata Usage
-### Query Patterns
-**Structured queries (use `properties`):**
-```sql
--- Find invoices from Acme Corp
-SELECT * FROM observations
-WHERE entity_type = 'invoice'
-  AND fields @> '{"vendor_name": "Acme Corp"}'::jsonb;
-```
-**Quality monitoring (use `extraction_metadata`):**
-```sql
--- Find [observations](#observation) with [extraction](#extraction) issues
-SELECT * FROM raw_fragments
-WHERE fragment_type = 'unknown_field';
-```
-**[Entity schema](#entity-schema) evolution data (use `extraction_metadata`):**
-```sql
--- Find [observations](#observation) with unknown fields (potential [entity schema](#entity-schema) expansion candidates)
-SELECT * FROM raw_fragments
-WHERE fragment_type = 'unknown_field';
-```
-### [Entity](#entity) and [Event](#event) [Extraction](#extraction)
-**Important:** [Entity](#entity) [extraction](#extraction) and [event](#event) generation MUST use fields from `properties` only, NOT from `extraction_metadata.unknown_fields`.
-**Rationale:**
-- [Entities](#entity) and [events](#event) must be deterministic and [entity schema](#entity-schema)-compliant
-- Unknown fields are not yet part of the [entity schema](#entity-schema) definition
-- Future [entity schema](#entity-schema) expansion can migrate `unknown_fields` to `properties` and then [extract](#extraction) [entities](#entity)/[events](#event)
-**Example:**
-```typescript
-// [Entity](#entity) [extraction](#extraction) from invoice
-const vendorName = properties.vendor_name; // ✅ Use this
-// NOT: extraction_metadata.unknown_fields.vendor_name // ❌ Don't use this
-```
-## 4. Migration and Evolution
-### When [Entity Schemas](#entity-schema) Evolve
-**Additive Evolution Only:**
-- New fields added to [entity schema](#entity-schema) definition
-- Existing [observations](#observation) retain old `schema_version`
-- New [observations](#observation) use new `schema_version`
-- Application handles both versions gracefully
-### Future: Automatic [Entity Schema](#entity-schema) Expansion
-Unknown fields in `extraction_metadata.unknown_fields` can be analyzed to suggest [entity schema](#entity-schema) expansions:
-- Pattern detection across multiple [observations](#observation)
-- User approval for [entity schema](#entity-schema) changes
-- Deterministic [entity schema](#entity-schema) fingerprinting
-- See `docs/architecture/schema_expansion.md` (post-MVP)
-## 5. Related Documentation
-- **[Entity schema](#entity-schema) structure:** `docs/subsystems/schema.md` Section 2.9
-- **[Entity schema](#entity-schema) definitions:** `docs/subsystems/schema_registry.md`
-- **Validation rules:** `docs/subsystems/schema.md` Section 1.2
-- **Manifest constraints:** `docs/NEOTOMA_MANIFEST.md` Sections 5.4, 12.2, 14.4
-- **Canonical vocabulary:** `docs/vocabulary/canonical_terms.md`
-## Agent Instructions
-### When to Load This Document
-Load `docs/architecture/schema_handling.md` when:
-- Implementing field extraction and validation logic
-- Understanding how unknown fields are preserved
-- Designing queries that need to access unknown fields
-- Planning schema evolution strategies
-### Constraints Agents Must Enforce
-1. **Always preserve data:** Unknown fields MUST go to `extraction_metadata` or `raw_fragments`, never discarded
-2. **Never reject [observations](#observation):** [Observations](#observation) MUST be created even with missing required fields (warn only)
-3. **[Entity schema](#entity-schema) compliance:** Only [entity schema](#entity-schema)-defined fields in `properties`
-4. **Determinism:** Same input → same `properties` structure always
-5. **[Entity](#entity)/[Event](#event) [extraction](#extraction):** Use `properties` only, not `extraction_metadata`
-**END OF DOCUMENT**