neotoma 0.3.0 → 0.3.2

package/docs/architecture/idempotence_pattern.md

@@ -1,139 +0,0 @@
-# Idempotence Pattern for LLM Interpretation
-
-**Reference:** [`docs/architecture/determinism.md`](./determinism.md) — Section 1.4 on idempotence vs determinism
-
-## Core Principle
-
-**Idempotence is enforced by system design, not by making the LLM deterministic.**
-
-This is a **category error** to expect determinism from an LLM. The LLM is a stochastic proposal engine; determinism is enforced downstream through canonicalization, hashing, and deduplication.
-
-**Key Insight**: The LLM proposes candidates (stochastic). The system enforces acceptance criteria (deterministic). Idempotence emerges post-generation, not during it.
-
-## Design Pattern
-
-The idempotence pattern for LLM interpretation follows these steps:
-
-1. **Constrain Outputs**
-   - Strict schemas (JSON Schema/Zod)
-   - Enums over free text
-   - Required fields only
-
-2. **Deterministic Decoding (Optional Aid)**
-   - temperature = 0
-   - top_p = 1
-   - Fixed prompts
-   - Reduces variance; does not guarantee identity
-
-3. **Validate → Reject → Retry**
-   - Validate against schema
-   - Reject invalid outputs
-   - Retry with identical inputs until valid
-
-4. **Canonicalize**
-   - Sort keys and arrays
-   - Normalize whitespace and line endings
-   - Round numbers; normalize dates
-   - Strip comments/non-semantic text
-
-5. **Hash and Cache**
-   - Hash canonical output (e.g., SHA-256)
-   - Use hash for replay, caching, and verification
-
-6. **Commit via Protocol**
-   - Check for existing observation with same hash
-   - Create observation only if not exists
-   - Enforce idempotence at database level
-
-## Fixed-Point Guarantee
-
-Accept output only when canonical hash stabilizes:
-
-- Run interpretation → canonicalize → hash
-- If hash differs from previous attempt → retry
-- Accept only when `hash(n) == hash(n-1)` (convergence)
-
-Idempotence achieved post-generation.
-
-## Example: Neotoma-Style Data Ingestion
-
-**LLM Behavior (Stochastic)**:
-
-- Two runs might produce:
-  - Run 1: `{"date":"2026-01-10","amount":1200,"currency":"EUR"}`
-  - Run 2: `{"currency":"EUR","amount":1200.00,"date":"2026-01-10"}`
-
-**System Enforcement (Deterministic)**:
-
-- JSON Schema validation
-- Key ordering (sorted keys)
-- Numeric normalization (1200.00 → 1200)
-- ISO date enforcement
-- Result: `{"amount":1200,"currency":"EUR","date":"2026-01-10"}`
-
-**Outcome**: Same input → same canonical output → same hash. Idempotence achieved without touching the model.
-
-## Integration with Neotoma Architecture
-
-### Interpretation Pipeline
-
-The idempotence pattern integrates with Neotoma's interpretation pipeline:
-
-1. **LLM Extraction** (`src/services/llm_extraction.ts`)
-   - Stochastic model output
-   - Temperature = 0.1 (can be reduced to 0 for variance reduction)
-
-2. **Schema Validation** (`src/services/interpretation.ts`)
-   - Validate against `schema_registry`
-   - Route unknown fields to `raw_fragments`
-   - Only schema-valid fields proceed
-
-3. **Canonicalization** (to be implemented)
-   - Normalize strings (trim, lowercase where appropriate)
-   - Sort arrays by deterministic key
-   - Normalize dates (ISO 8601, timezone handling)
-   - Round numbers (consistent precision)
-
-4. **Hash-Based Identity** (to be implemented)
-   - Generate observation ID from: `source_id + interpretation_id + entity_id + canonical_fields_hash`
-   - Check for existing observation before creating
-
-5. **Deduplication** (to be implemented)
-   - If observation with same hash exists → no duplicate created
-   - Final state is same regardless of LLM variance
-
-### Database Schema
-
-Observations table already supports:
-
-- `source_id` (links to source)
-- `interpretation_id` (links to interpretation run)
-- `entity_id` (links to entity)
-- `fields` (JSONB with validated fields)
-
-**To add:**
-
-- `canonical_hash` column (computed from canonicalized fields)
-- Unique constraint on `(source_id, interpretation_id, entity_id, canonical_hash)`
-
-## Benefits
-
-- **Safe for infrastructure**: Idempotent operations can be retried safely
-- **Cacheable**: Same inputs → same outputs → cacheable results
-- **Replayable**: Can replay interpretation logs instead of re-running model
-- **Model-agnostic**: Works with any LLM provider
-- **Verifiable**: Hash-based identity enables verification
-
-## Trade-offs
-
-- **Engineering overhead**: Requires canonicalization, hashing, deduplication logic
-- **Reduced creative latitude**: Schema constraints limit model flexibility
-- **Schema quality is critical**: Poor schemas cause false convergence or missed valid outputs
-- **Latency**: Retry loops and fixed-point checks add latency
-
-## Related Documents
-
-- [`docs/architecture/determinism.md`](./determinism.md) — Determinism doctrine and idempotence distinction
-- [`docs/architecture/sources_first_ingestion_final.md`](./sources_first_ingestion_final.md) — Sources-first ingestion architecture
-- [`docs/subsystems/ingestion/ingestion.md`](../subsystems/ingestion/ingestion.md) — Ingestion pipeline
-- [`docs/foundation/philosophy.md`](../foundation/philosophy.md) — Core philosophy (section 5.1 on idempotence)

package/docs/architecture/mcp_actions_assessment.md

@@ -1,182 +0,0 @@
-# MCP Actions Architecture Assessment
-
-**Date:** 2026-01-01  
-**Status:** Assessment Complete
-
-## Executive Summary
-
-Assessment of all current MCP actions against documented architecture (`docs/vocabulary/canonical_terms.md`, `docs/specs/MCP_SPEC.md`, `docs/architecture/source_material_model.md`). Identifies missing actions, terminology misalignments, and implementation gaps.
-
-## Current Actions Inventory
-
-**Total Actions Implemented:** 15
-
-**Note:** `upload_file` was intentionally removed from the MCP spec. File uploads are handled via the unified `store` action. Agents can read files from the local filesystem, base64-encode the content, and use `store` with `file_content` and `mime_type` parameters.
-
-### ✅ Core Ingestion (1/1)
-- `store` - Unified ingestion for structured and unstructured source (replaces `ingest` terminology in implementation)
-
-### ✅ File Operations (1/1)
-- `get_file_url` - Get signed URL for file access
-
-**Note:** `upload_file` is intentionally not implemented as an MCP action. File uploads are handled via the unified `store` action. Agents can read files from the local filesystem, base64-encode the content, and use `store` with `file_content` and `mime_type` parameters. The REST endpoint `/upload_file` exists for UI/frontend use but is not part of the MCP protocol.
-
-### ✅ Entity Operations (6/6)
-- `get_entity_snapshot` - Get entity snapshot with provenance
-- `retrieve_entities` - Query entities with filters
-- `get_entity_by_identifier` - Find entity by identifier
-- `get_related_entities` - Get entities via relationships (n-hop)
-- `get_graph_neighborhood` - Get complete graph context around node
-- `merge_entities` - Merge duplicate entities
-
-### ✅ Observation & Relationship Operations (5/5)
-- `list_observations` - Query observations for entity
-- `get_field_provenance` - Trace field to source
-- `create_relationship` - Create typed relationship between entities
-- `list_relationships` - Query entity relationships
-- `list_timeline_events` - Query timeline events with filters
-
-### ✅ Correction & Reinterpretation (2/2)
-- `correct` - Create high-priority correction observation
-- `reinterpret` - Re-run AI interpretation on existing source
-
-### ❌ Integration Operations (0/2)
-- ❌ `list_provider_catalog` - **MISSING** (spec requires for MVP)
-- ❌ `sync_provider_imports` - **MISSING** (spec requires for MVP)
-
-## Architecture Alignment Issues
-
-### 1. Missing MVP Actions
-
-**Critical:** Two MVP-required actions are missing:
-
-#### `list_provider_catalog`
-- **Status:** Missing
-- **Spec Requirement:** MVP required (`docs/specs/MCP_SPEC.md` section 2.3)
-- **Purpose:** List available external providers
-- **Impact:** Agents cannot discover available provider integrations
-- **Recommendation:** Implement provider catalog listing
-
-#### `sync_provider_imports`
-- **Status:** Missing
-- **Spec Requirement:** MVP required (`docs/specs/MCP_SPEC.md` section 2.3)
-- **Purpose:** Trigger provider import sync
-- **Impact:** Agents cannot trigger provider data synchronization
-- **Recommendation:** Implement provider sync action
-
-### 2. Legacy Terminology Usage
-
-**Violates:** `docs/vocabulary/canonical_terms.md` canonical vocabulary rules
-
-#### `get_graph_neighborhood`
-- **Issue:** Uses "record" terminology instead of "source"
-- **Location:** `src/server.ts:1150, 1152, 1273-1287`
-- **Details:**
-  - Parameter `node_type: "record"` should be `"source_material"` (or removed if redundant)
-  - Parameter `include_records` is legacy; `include_source_material` exists but both are present
-  - Implementation queries `records` table instead of `sources` table
-  - Response includes `related_records` instead of `related_source_material`
-- **Impact:** Violates canonical vocabulary, confuses "record" (legacy) vs "source" (canonical)
-- **Recommendation:** Remove `node_type: "record"` option, remove `include_records` parameter, update implementation to use `sources` table
-
-#### `list_timeline_events`
-- **Issue:** Legacy parameter `source_record_id` alongside canonical `source_id`
-- **Location:** `src/server.ts:223-225, 860, 884-885, 909-910`
-- **Details:**
-  - Accepts both `source_record_id` (legacy) and `source_id` (canonical)
-  - Implementation prefers `source_id` but maintains backward compatibility
-- **Impact:** Maintains backward compatibility but violates vocabulary purity
-- **Recommendation:** Remove `source_record_id` parameter in next breaking change release
-
-#### `get_field_provenance`
-- **Issue:** Uses `records` table and `source_record_id` field
-- **Location:** `src/server.ts:688-712`
-- **Details:**
-  - Queries `records` table instead of `sources` table
-  - Uses `observation.source_record_id` instead of `observation.source_material_id`
-  - Response includes `source_record_id` and `record_type` instead of `source_material_id` and source metadata
-- **Impact:** Returns legacy terminology in response, breaks provenance chain to source
-- **Recommendation:** Update to query `sources` table, use `source_material_id` field, return canonical terminology
-
-### 3. Database Schema Misalignment
-
-**Issue:** Actions reference legacy `records` table instead of `sources` table
-
-**Affected Actions:**
-- `get_field_provenance` - Queries `records` table (line 690)
-- `get_graph_neighborhood` - Queries `records` table (lines 1249, 1275)
-
-**Impact:** 
-- Breaks architecture migration to source model
-- Observations may reference `source_record_id` instead of `source_material_id`
-- Provenance chain incomplete
-
-**Recommendation:** 
-- Update all actions to use `sources` table
-- Ensure observations table uses `source_material_id` field
-- Update all queries to reference `sources` instead of `records`
-
-### 4. Field Name Inconsistencies
-
-**Issue:** Mixed use of `source_record_id` vs `source_material_id`
-
-**Locations:**
-- `get_field_provenance`: Uses `observation.source_record_id` (line 692)
-- `get_graph_neighborhood`: Uses `obs.source_record_id` (line 1245)
-- `list_timeline_events`: Accepts both `source_record_id` and `source_id` parameters
-
-**Impact:** Inconsistent field naming breaks canonical vocabulary compliance
-
-**Recommendation:** 
-- Standardize on `source_material_id` throughout
-- Update database schema if needed
-- Migrate existing data references
-
-## Architecture Compliance Summary
-
-| Category | Status | Issues |
-|----------|--------|--------|
-| **Action Completeness** | ⚠️ Partial | Missing 2 MVP actions |
-| **Terminology** | ❌ Non-compliant | Legacy "record" terminology in 3 actions |
-| **Database Schema** | ❌ Misaligned | References `records` table instead of `sources` |
-| **Field Naming** | ❌ Inconsistent | Mixed `source_record_id` / `source_material_id` |
-| **Vocabulary Compliance** | ❌ Non-compliant | Violates canonical terms in multiple places |
-
-## Recommendations
-
-### Priority 1: Critical (MVP Blockers)
-1. **Implement `list_provider_catalog` action** - Required for MVP per spec
-2. **Implement `sync_provider_imports` action** - Required for MVP per spec
-
-### Priority 2: Architecture Alignment
-4. **Update `get_field_provenance`** - Use `sources` table, `source_material_id` field, canonical terminology
-5. **Update `get_graph_neighborhood`** - Remove `node_type: "record"`, remove `include_records`, use `sources` table
-6. **Update `list_timeline_events`** - Remove `source_record_id` parameter (breaking change)
-
-### Priority 3: Database Migration
-7. **Verify observations table schema** - Ensure `source_material_id` field exists
-8. **Migrate data references** - Update all `source_record_id` references to `source_material_id`
-9. **Update all queries** - Replace `records` table references with `sources` table
-
-### Priority 4: Documentation
-10. **Update action descriptions** - Ensure all descriptions use canonical vocabulary
-11. **Remove legacy parameter documentation** - Clean up deprecated parameter references
-
-## Compliance Checklist
-
-- [ ] All MVP-required actions implemented
-- [ ] All actions use canonical vocabulary (`source`, not `record`)
-- [ ] All actions query `sources` table (not `records`)
-- [ ] All actions use `source_material_id` field (not `source_record_id`)
-- [ ] All action descriptions conform to `docs/vocabulary/canonical_terms.md`
-- [ ] No legacy terminology in action schemas or responses
-- [ ] All database queries align with source architecture
-
-## References
-
-- **Canonical Vocabulary:** `docs/vocabulary/canonical_terms.md`
-- **MCP Specification:** `docs/specs/MCP_SPEC.md`
-- **Source Architecture:** `docs/architecture/source_material_model.md`
-- **Implementation:** `src/server.ts`
-
-

package/docs/architecture/ner_vs_schema_first_extraction.md

@@ -1,280 +0,0 @@
-# NER vs Schema-First Extraction in Neotoma
-
-## Scope
-
-This document covers:
-- Why Neotoma does NOT use traditional Named Entity Recognition (NER)
-- Schema-first extraction architecture
-- Architectural constraints against probabilistic NER
-- How entity resolution works instead
-- Potential future NER integration patterns
-
-This document does NOT cover:
-- NER implementation details (not used)
-- Complete extraction pipeline (see `docs/subsystems/ingestion/ingestion.md`)
-- Entity resolution implementation (see `docs/foundation/entity_resolution.md`)
-- Observation architecture (see `docs/subsystems/observation_architecture.md`)
-
-## 1. Current State: NER is NOT Used in MVP
-
-Neotoma deliberately **does not use** traditional NER (Named Entity Recognition) in its MVP. Instead, it uses:
-
-### 1.1 Schema-First Extraction (Not NER)
-
-**What it is**: Rule-based, deterministic field extraction guided by entity schemas
-
-**How it works**:
-- Files are interpreted using AI models (GPT-4o-mini)
-- Entity schemas define extraction rules that specify which fields to extract
-- Extraction is deterministic within the system (same source + same interpretation config → same observations)
-
-**Key difference from NER**: Rather than using ML models to detect "person", "company", "location" entities in unstructured text, Neotoma uses schema-driven extraction rules.
-
-### 1.2 Entity Resolution (Deterministic, Hash-Based)
-
-From `src/services/entity_resolution.ts`:
-
-```typescript
-function generateEntityId(entityType: string, canonicalName: string): string {
-  const normalized = normalizeEntityValue(entityType, canonicalName);
-  const hash = sha256(`${entityType}:${normalized}`);
-  return `ent_${hash.substring(0, 24)}`;
-}
-```
-
-**Key architectural principle**: Same entity name → same entity ID globally, using deterministic normalization and hashing, NOT probabilistic NER models.
-
-## 2. Three-Layer Truth Model
-
-Neotoma uses this architecture:
-
-```
-Source → Interpretation (AI) → Observation → Entity Snapshot
-```
-
-**Layers:**
-- **Source**: Raw content (PDFs, images, structured JSON)
-- **Interpretation**: AI-based extraction with config logging (model, temperature, prompt hash)
-- **Observation**: Granular facts extracted from sources
-- **Entity Snapshot**: Deterministic reducer output representing current truth
-
-### 2.1 Where AI is Used (vs. Where NER Would Be)
-
-**AI interpretation is used for**:
-- Extracting structured fields from unstructured documents (invoices, receipts, contracts)
-- Converting raw text into schema-compliant observations
-- But NOT for probabilistic entity detection
-
-**NER would typically be used for**:
-- Detecting named entities ("Apple Inc.", "Steve Jobs", "Cupertino") in running text
-- Classifying entities by type (PERSON, ORG, LOCATION)
-- Probabilistic, non-deterministic entity boundaries
-
-## 3. Architectural Constraints Against Traditional NER
-
-From `docs/foundation/philosophy.md`:
-
-### 3.1 Determinism Above All
-
-**Constraint**: "A given operation MUST always produce the same final state"
-
-**Why this blocks NER**:
-- NER models are inherently probabilistic and non-deterministic
-- Same text → different entity boundaries/classifications across runs
-- Neotoma achieves system-level determinism through canonicalization + hashing + deduplication
-
-### 3.2 Schema-First Processing
-
-**Constraint**: "All extraction, linking, and event creation derive from: schema_type → extraction rules → extracted_fields → entities + events"
-
-**Why this blocks NER**:
-- NER is typically schema-agnostic and bottom-up (detect entities first, classify later)
-- Neotoma is schema-first and top-down (schema defines what to extract)
-- Entity types are determined by schema, not by ML classification
-
-### 3.3 Provenance and Auditability
-
-**Requirement**: Every observation links to both `source_id` (what raw content) and `interpretation_id` (how it was extracted)
-
-**How this shapes extraction**:
-- Interpretation config is logged (model, temperature, prompt_hash)
-- This provides auditability without claiming replay determinism
-- NER would need similar config logging but couldn't provide deterministic replay
-
-## 4. What Replaces NER
-
-### 4.1 Schema-First AI Interpretation
-
-```
-Entity Schema → Extraction Rules → AI Interpretation → Structured Fields
-```
-
-**Example**: Invoice schema defines fields like `vendor_name`, `customer_name`, `amount`. AI interprets the invoice to extract these specific fields, not to discover arbitrary entities.
-
-### 4.2 Rule-Based Entity Normalization
-
-From `src/services/entity_resolution.ts`:
-
-```typescript
-function normalizeEntityValue(entityType: string, raw: string): string {
-  let normalized = raw.trim().toLowerCase();
-  
-  // Remove common suffixes for companies
-  if (entityType === "company" || entityType === "organization") {
-    normalized = normalized
-      .replace(/\s+(inc|llc|ltd|corp|corporation|co|company|limited)\.?$/i, "")
-      .trim();
-  }
-  
-  // Remove extra whitespace
-  normalized = normalized.replace(/\s+/g, " ").trim();
-  
-  return normalized;
-}
-```
-
-**Deterministic canonicalization**: Same entity name → same normalized form → same entity ID
-
-### 4.3 Hash-Based Entity Resolution
-
-```typescript
-const entityId = generateEntityId(entityType, canonicalName);
-// Same entityType + canonicalName → same entityId globally
-```
-
-**Benefits**:
-- Deterministic entity IDs
-- Global deduplication (same entity across multiple documents)
-- No probabilistic matching required
-
-### 4.4 Reducer-Based Snapshot Computation
-
-```
-Observations → Reducer (merge policies) → Entity Snapshot
-```
-
-**Deterministic merging**: Multiple observations about same entity → single deterministic snapshot
-
-## 5. Architectural Comparison
-
-| Aspect | Traditional NER | Neotoma Schema-First |
-|--------|----------------|---------------------|
-| **Entity Detection** | Bottom-up (find entities in text) | Top-down (schema defines entities) |
-| **Determinism** | Probabilistic, non-deterministic | System-level deterministic (canonicalization + hashing) |
-| **Entity Types** | ML classification (PERSON, ORG, LOC) | Schema-defined (invoice, company, person) |
-| **Entity IDs** | Generated or probabilistic matching | Hash-based, deterministic |
-| **Provenance** | Model-specific, opaque | Source + Interpretation + Observation chain |
-| **Auditability** | Model outputs, hard to explain | Full config logging, explainable |
-| **Reprocessing** | Non-deterministic re-extraction | Reinterpretation creates NEW observations |
-
-## 6. If NER Were Added (Post-MVP)
-
-Traditional NER could potentially be used for:
-
-### 6.1 Entity Discovery
-
-**Use case**: Finding entities in unstructured text that aren't captured by schema-driven extraction
-
-**Example**: User uploads free-form notes mentioning companies, people, locations that don't fit a specific schema
-
-**Integration pattern**:
-- NER runs as separate interpretation pass
-- Creates observations with `source_priority: 0` (lowest, AI interpretation)
-- Logs NER model/config in `interpretation_id`
-- Does NOT claim deterministic replay
-
-### 6.2 Relationship Discovery
-
-**Use case**: Detecting relationships between entities mentioned in documents
-
-**Example**: "John Smith works at Acme Corp" → relationship between person and company
-
-**Integration pattern**:
-- NER detects entities + relations
-- Creates relationship observations
-- Links to source + interpretation for provenance
-
-### 6.3 Enhanced Search
-
-**Use case**: Finding entities by type across all documents
-
-**Example**: "Find all companies mentioned in my documents"
-
-**Integration pattern**:
-- NER provides entity type tags
-- Indexed for search
-- Maintains provenance chain
-
-### 6.4 Required Adaptations
-
-Any NER integration would need to:
-
-1. **Respect the three-layer model**: Source → Interpretation → Observation → Snapshot
-2. **Log NER model/config** for provenance (`interpretation_id` with model, temperature, prompt_hash)
-3. **Not claim deterministic replay** (AI outputs vary)
-4. **Maintain system-level idempotence** through canonicalization + hashing + deduplication
-5. **Use low source priority** (`source_priority: 0`) so user corrections override
-6. **Support reinterpretation** (new NER model → new observations, old preserved)
-
-## 7. Verifiable Domain Principle
-
-**Why Neotoma avoids probabilistic NER**:
-
-Neotoma creates a **verifiable domain** for personal data where same operations produce same final states. This enables:
-- Reliable, consistent results LLMs can depend on
-- No hallucinations or duplicates
-- Reproducible, explainable outputs
-- Tamper-evident records (hash-based IDs)
-
-Traditional probabilistic NER would violate this principle:
-- Non-deterministic entity detection
-- Varying entity boundaries across runs
-- Probabilistic entity matching
-- Unexplainable ML classifications
-
-**Solution**: System-level idempotence through:
-- Canonicalization (normalize, sort, round)
-- Hashing (create deterministic identity)
-- Deduplication (prevent duplicates)
-
-## Agent Instructions
-
-### When to Load This Document
-
-Load this document when:
-- Understanding why Neotoma doesn't use NER
-- Comparing NER vs schema-first extraction
-- Evaluating NER integration for post-MVP
-- Explaining entity resolution to stakeholders
-
-### Required Co-Loaded Documents
-
-- `docs/foundation/philosophy.md` — Determinism and schema-first principles
-- `docs/foundation/entity_resolution.md` — Entity ID generation and normalization
-- `docs/subsystems/observation_architecture.md` — Three-layer truth model
-- `docs/subsystems/ingestion/ingestion.md` — Extraction pipeline
-
-### Constraints Agents Must Enforce
-
-1. **Do NOT implement traditional NER in MVP** (violates determinism constraints)
-2. **Use schema-first extraction** (entity types from schemas, not ML)
-3. **Generate deterministic entity IDs** (hash-based, not probabilistic)
-4. **Log all interpretation config** (model, temperature, prompt_hash)
-5. **Maintain system-level idempotence** (canonicalization + hashing)
-
-### Forbidden Patterns
-
-- ❌ Using probabilistic NER models for entity detection in MVP
-- ❌ Generating non-deterministic entity IDs
-- ❌ Bottom-up entity discovery without schema guidance
-- ❌ Claiming replay determinism for AI interpretation
-- ❌ Skipping interpretation config logging
-
-### Validation Checklist
-
-- [ ] Schema-first extraction (not NER-based)
-- [ ] Deterministic entity ID generation (hash-based)
-- [ ] Interpretation config logged (model, temperature, prompt_hash)
-- [ ] System-level idempotence maintained
-- [ ] Provenance chain complete (Source → Interpretation → Observation → Snapshot)
-- [ ] No probabilistic entity matching in MVP