neotoma 0.3.0 → 0.3.2

package/docs/specs/INTERNAL_MCP_RELEASE.md

@@ -1,300 +0,0 @@
-# Neotoma Internal MCP Release — Specification
-## Scope
-### Included Capabilities
-**1. Core Backend Services (Domain Layer)**
-- ✅ **FU-100:** File Analysis Service
-  - PDF text extraction (pdf-parse)
-  - OCR for images (Tesseract)
-  - Schema type detection (rule-based, deterministic)
-  - Field extraction per schema type (rule-based regex, no LLM)
-  - Content hash for deduplication
-- ✅ **FU-101:** Entity Resolution Service
-  - Entity extraction from fields
-  - Normalization rules (lowercase, trim, suffix removal)
-  - Deterministic entity ID generation (hash-based)
-  - Deduplication logic
-  - Entity-record edge creation
-- ✅ **FU-102:** Event Generation Service
-  - Date field detection per schema type
-  - Event type mapping (date_issued → InvoiceIssued)
-  - Deterministic event ID generation (hash-based)
-  - Event-record and event-entity edge creation
-  - Timeline ordering (chronological)
-- ✅ **FU-103:** Graph Builder Service
-  - Transactional insert (record + entities + events + edges)
-  - Orphan node prevention
-  - Cycle detection
-  - Edge type validation
-- ✅ **FU-105:** Search Service
-  - Structured filters (type, properties, date range)
-  - Full-text search (GIN indexes)
-  - Deterministic ranking with tiebreakers
-  - Pagination
-  - **Constraint:** Structured search only (no semantic/vector search in this release)
-- ⏳ **FU-104:** Embedding Service (Optional)
-  - OpenAI embedding generation (ada-002, 1536-dim)
-  - Async embedding queue
-  - Vector storage in PostgreSQL (pgvector)
-  - **Note:** Embeddings not required for MCP validation, but may be useful for future semantic search
-**2. MCP Actions (Application Layer)**
-- ✅ **FU-200:** MCP Server Core
-  - MCP server initialization
-  - Tool registration
-  - Request validation (Zod schemas)
-  - Error envelope handling
-  - WebSocket bridge
-- ✅ **FU-201:** MCP Action — `store_record`
-  - Create new record with properties
-  - Input validation
-  - Emit `RecordCreated` event via `EventRepository.appendEvent()`
-  - Apply reducer (`reduceRecordCreated`) to compute state
-  - Refresh materialized view after event emission
-  - Response with created record
-- ✅ **FU-202:** MCP Action — `retrieve_records`
-  - Query records with filters (type, properties, search)
-  - Retrieve state via `StateRepository.getState()` (repository abstraction)
-  - Search service integration
-  - Deterministic result ordering
-  - Pagination
-- ✅ **FU-203:** MCP Action — `update_record`
-  - Update record metadata/properties
-  - Emit `RecordUpdated` event via `EventRepository.appendEvent()`
-  - Apply reducer (`reduceRecordUpdated`) to compute updated state
-  - Refresh materialized view after event emission
-  - Property merging
-  - Response with updated record
-- ✅ **FU-204:** MCP Action — `delete_record`
-  - Delete record and associated files
-  - Emit `RecordDeleted` event via `EventRepository.appendEvent()`
-  - Apply reducer (`reduceRecordDeleted`) to mark record as deleted
-  - Refresh materialized view after event emission
-  - Cascade delete (edges, files)
-  - Response with confirmation
-- ✅ **FU-205:** MCP Action — `upload_file`
-  - Upload file from local path
-  - Upload to storage (cloud storage or S3)
-  - Trigger file analysis pipeline
-  - Create record with extracted fields (uses `store_record` internally, inherits event-sourcing)
-  - Use `StateRepository` and `EventRepository` when creating records
-  - Response with record + file URL
-- ✅ **FU-206:** MCP Action — `get_file_url`
-  - Get signed URL for file access
-  - Expiration handling
-- ⏳ **FU-208:** MCP Provider Integrations (Optional)
-  - `list_provider_catalog` — List available providers
-  - `sync_provider_imports` — Trigger provider import sync
-  - **Note:** Provider integrations not required for core validation, but may be useful for Gmail attachment testing
-**3. Infrastructure**
-- ✅ **FU-000:** Database Schema v1.0
-  - `records`, `record_relationships` tables
-  - Indexes (GIN, B-tree)
-  - Migration scripts
-  - **Note:** RLS policies not required (single-user release)
-- ✅ **FU-002:** Configuration Management
-  - Environment variable loading
-  - Canonical type mappings
-  - Configuration validation
-## Explicitly Excluded
-**1. UI Components (All Phase 3 UI Feature Units)**
-- ❌ FU-300: Design System Implementation
-- ❌ FU-301: Records List View
-- ❌ FU-302: Record Detail View
-- ❌ FU-303: Timeline View
-- ❌ FU-304: File Upload UI
-- ❌ FU-305: Dashboard View
-- ❌ FU-306: Settings UI
-- ❌ FU-307: Chat/AI Panel
-**2. Multi-User Infrastructure**
-- ❌ FU-700: Authentication UI (OAuth)
-- ❌ FU-701: Row-Level Security (RLS)
-- ❌ FU-702: Billing and Subscription Management
-- ❌ FU-703: Local Storage / Offline Mode (optional)
-**3. Onboarding Flow**
-- ❌ FU-400-403: Onboarding Feature Units (welcome modal, upload flow, etc.)
-**4. Post-MVP Features**
-- ❌ FU-207: Plaid Integration (post-MVP)
-- ❌ Semantic/vector search (structured search only)
-- ❌ LLM-based extraction (rule-based only)
-## MCP Actions Available
-This release provides **6 core MCP actions** (plus 2 optional provider actions):
-| Action | Purpose | Status |
-|--------|---------|--------|
-| `store_record` | Create new record | ✅ Required |
-| `retrieve_records` | Query records with filters | ✅ Required |
-| `update_record` | Update record metadata | ✅ Required |
-| `delete_record` | Delete record | ✅ Required |
-| `upload_file` | Upload file and create record | ✅ Required |
-| `get_file_url` | Get signed URL for file | ✅ Required |
-| `list_provider_catalog` | List available providers | ⏳ Optional |
-| `sync_provider_imports` | Trigger provider sync | ⏳ Optional |
-## Validation Workflows
-**1. File Upload → Extraction → Query**
-```
-Cursor/ChatGPT → upload_file("invoice.pdf")
-  → File analyzed, fields extracted
-  → Entities resolved, events generated
-  → Graph inserted
-  → retrieve_records(type="invoice")
-  → Verify extracted fields, entities, events
-```
-**2. Manual Record Creation → Query**
-```
-Cursor/ChatGPT → store_record({
-  type: "invoice",
-  properties: { invoice_number: "INV-001", amount: 1000 }
-})
-  → retrieve_records(type="invoice")
-  → Verify record stored correctly
-```
-**3. Entity Resolution Validation**
-```
-Cursor/ChatGPT → upload_file("invoice1.pdf") // Contains "Acme Corp"
-  → upload_file("invoice2.pdf") // Contains "ACME CORP"
-  → retrieve_records(type="invoice")
-  → Verify both records link to same entity ID
-```
-**4. Timeline Event Validation**
-```
-Cursor/ChatGPT → upload_file("invoice.pdf") // Contains date_issued: "2024-01-15"
-  → retrieve_records(type="invoice")
-  → Verify event generated with correct timestamp
-  → Query events chronologically
-```
-**5. Search and Filtering**
-```
-Cursor/ChatGPT → retrieve_records({
-  type: "invoice",
-  properties: { vendor: "Acme Corp" },
-  search: ["invoice"],
-  limit: 10
-})
-  → Verify deterministic ordering
-  → Verify filters applied correctly
-```
-## Technical Requirements
-### Database
-- Single-user database (no RLS policies required)
-- All tables from FU-000 schema
-- Indexes for search performance
-### Storage
-- File storage (cloud storage or S3)
-- Signed URL generation for file access
-### MCP Server
-- MCP protocol compliance
-- WebSocket bridge for Cursor/ChatGPT integration
-- Error envelope handling
-- Request validation (Zod schemas)
-### Determinism Requirements
-- Same file → same record_id, entities, events (100% deterministic)
-- Same query + same DB state → same result order
-- Same entity name → same entity_id (globally)
-## Testing Strategy
-**1. Unit Tests**
-- Schema detection rules (determinism)
-- Field extraction regex (determinism)
-- Entity ID generation (determinism)
-- Event ID generation (determinism)
-- Search ranking (determinism)
-**2. Integration Tests**
-- Full ingestion pipeline (upload → extraction → graph)
-- MCP action validation
-- Graph integrity (no orphans, no cycles)
-- Transaction rollback on error
-**3. E2E Tests (MCP Client)**
-- Cursor integration (upload_file → retrieve_records)
-- ChatGPT integration (store_record → retrieve_records)
-- Entity resolution across multiple uploads
-- Timeline event generation and ordering
-**4. Property-Based Tests**
-- Determinism proofs (same input → same output, 100 runs)
-- Graph integrity invariants (no orphans, no cycles)
-## Acceptance Criteria
-**Functional**
-- ✅ Upload PDF → extract fields → create record (95% success rate)
-- ✅ Upload image → OCR → extract fields → create record (90% success rate)
-- ✅ Same file uploaded twice → same record_id (deduplication)
-- ✅ Entity resolution: "Acme Corp" and "ACME CORP" → same entity_id
-- ✅ Event generation: date fields → timeline events
-- ✅ Graph integrity: 0 orphan nodes, 0 cycles
-- ✅ Search: deterministic ordering (same query → same order)
-**Performance**
-- ✅ Upload latency: <5s P95 for 10MB PDF
-- ✅ Search latency: <500ms P95 for structured queries
-- ✅ Graph insertion: <1s P95 (transactional)
-**Determinism**
-- ✅ Same file → same extraction (100% deterministic, 100 runs)
-- ✅ Same query → same order (100% deterministic, 100 runs)
-- ✅ Same entity name → same entity_id (100% deterministic)
-## Dependencies
-**Required Feature Units (Execution Order)**
-1. **Phase 0: Foundation**
-   - FU-000: Database Schema ✅
-   - FU-002: Configuration ✅
-2. **Phase 1: Core Services**
-   - FU-100: File Analysis Service 🔨
-   - FU-101: Entity Resolution Service 🔨
-   - FU-102: Event Generation Service 🔨
-   - FU-103: Graph Builder Service 🔨
-   - FU-105: Search Service 🔨
-   - FU-104: Embedding Service (Optional) ⏳
-3. **Phase 2: MCP Layer**
-   - FU-200: MCP Server Core ✅
-   - FU-201: store_record ✅
-   - FU-202: retrieve_records ✅
-   - FU-203: update_record ✅
-   - FU-204: delete_record ✅
-   - FU-205: upload_file ✅
-   - FU-206: get_file_url ✅
-   - FU-208: Provider Integrations (Optional) ⏳
-**Status Legend:**
-- ✅ Complete
-- 🔨 Partial / Needs Work
-- ⏳ Not Started
-- ❌ Excluded
-## Differences from Full MVP
-| Feature | Internal MCP Release | Full MVP |
-|---------|---------------------|----------|
-| **UI** | ❌ None | ✅ 7 views (Dashboard, Records, Timeline, Entities, Upload, Settings, Chat) |
-| **Multi-User** | ❌ Single-user only | ✅ Auth + RLS (2–20 people) |
-| **Billing** | ❌ None | ✅ Stripe integration |
-| **Onboarding** | ❌ None | ✅ Welcome modal, upload flow |
-| **Provider Integrations** | ⏳ Optional | ✅ Gmail (required) |
-| **Local Storage** | ❌ None | ✅ Offline mode |
-| **Semantic Search** | ❌ Structured only | ❌ Structured only (post-MVP) |
-| **LLM Extraction** | ❌ Rule-based only | ❌ Rule-based only |
-## Success Criteria
-**Internal Release is Complete When:**
-1. ✅ All 6 core MCP actions functional
-2. ✅ File upload → extraction → graph insertion working end-to-end
-3. ✅ Entity resolution validated (canonical IDs)
-4. ✅ Event generation validated (timeline events)
-5. ✅ Graph integrity validated (0 orphans, 0 cycles)
-6. ✅ Search deterministic (same query → same order)
-7. ✅ Cursor integration tested (upload_file, retrieve_records)
-8. ✅ ChatGPT integration tested (store_record, retrieve_records)
-9. ✅ All critical path tests passing (100% coverage)
-10. ✅ Determinism validated (100 runs, same input → same output)
-## Next Steps After Internal Release
-Once internal MCP release is validated:
-1. **Add UI Layer** (Phase 3 Feature Units)
-   - Design system
-   - Records list, detail, timeline views
-   - Upload UI
-   - Dashboard, settings, chat panel
-2. **Add Multi-User Support** (Phase 7 Feature Units)
-   - Authentication (OAuth)
-   - Row-Level Security (RLS)
-   - Billing (Stripe)
-3. **Add Onboarding Flow** (Phase 4 Feature Units)
-   - Welcome modal
-   - Upload flow guidance
-4. **Add Provider Integrations** (Phase 5 Feature Units)
-   - Gmail attachment import
-## Documentation References
-- [`docs/NEOTOMA_MANIFEST.md`](../NEOTOMA_MANIFEST.md) — Foundational principles
-- [`docs/specs/MCP_SPEC.md`](./MCP_SPEC.md) — MCP action specifications
-- [`docs/specs/MVP_FEATURE_UNITS.md`](./MVP_FEATURE_UNITS.md) — Complete Feature Unit inventory
-- [`docs/architecture/architecture.md`](../architecture/architecture.md) — System architecture
-- [`docs/subsystems/ingestion/ingestion.md`](../subsystems/ingestion/ingestion.md) — Ingestion pipeline
-**Status:** 📋 Specification Complete
-**Last Updated:** 2024-12-02

package/docs/specs/MANIFEST_ALIGNMENT_SUMMARY.md

@@ -1,162 +0,0 @@
-# Manifest Alignment Summary
-*(MVP Plan Updates to Match NEOTOMA_MANIFEST.md)*
-## Changes Made
-### 1. MVP_FEATURE_UNITS.md
-**FU-100: File Analysis Service**
-- Status: Changed from "✅ Complete" to "🔨 Needs Update"
-- Added manifest constraints section
-- Added implementation notes requiring LLM removal
-- Updated deliverables to specify rule-based extraction only
-- Added PDFDocument fallback requirement
-- Updated tests to include determinism property tests (100 runs)
-**FU-105: Search Service**
-- Removed semantic search from deliverables
-- Added manifest constraints section
-- Added implementation notes requiring semantic search removal
-- Updated deliverables to structured + full-text only
-- Added deterministic ranking requirement with tiebreakers
-**FU-600: Advanced Search UI**
-- Removed search mode toggle (keyword/semantic/both)
-- Added manifest constraints section
-- Added implementation notes to remove semantic search UI
-- Updated deliverables to structured filters only
-**Summary Table:**
-- Updated FU-100 status to "🔨 Needs Update"
-### 2. MVP_EXECUTION_PLAN.md
-**Phase 1 Updates:**
-- Updated status: "File Analysis: 🔨 Needs Update (uses OpenAI LLM; MUST be replaced)"
-- Added FU-100 as first P0 task in execution sequence
-- Added detailed FU-100 execution steps with LLM removal instructions
-**Critical Path Updates:**
-- Added FU-100 to P0 critical path (was missing)
-- Updated estimates to include FU-100 (8-10h agent time, 2-3h human time)
-**Cleanup Phase:**
-- Added LLM removal as first cleanup task
-- Added explicit steps to remove OpenAI calls
-- Updated cleanup verification to include LLM removal
-**Execution Strategy:**
-- Added "Critical: Implementation Must Overwrite to Match Manifest" section
-- Listed specific violations requiring overwrites
-- Added explicit instructions that agents MUST overwrite non-compliant code
-**Estimates Updated:**
-- Minimal MVP: 12-20h manual (was 10-17h), 51-67h agent (was 43-57h)
-- Full MVP: 21-33.5h manual (was 19-30.5h), 82-109h agent (was 74-99h)
-- Calendar: 8-10 days (was 7-9 days) for minimal, 13-16 days (was 12-15 days) for full
-**References:**
-- Added link to IMPLEMENTATION_OVERWRITE_GUIDE.md
-**Constraints:**
-- Added constraint #6: MUST overwrite non-compliant code
-**Forbidden Patterns:**
-- Added: Preserving non-compliant code
-- Added: Adding feature flags to conditionally enable manifest violations
-### 3. MVP_OVERVIEW.md
-**Post-MVP Section:**
-- Added note: "Current implementation uses LLM but MUST be removed for MVP per manifest"
-**Extraction Section:**
-- Updated to emphasize "regex-based, no LLM"
-- Updated to list Tier 1 ICP-aligned schema types
-### 4. FUNCTIONAL_REQUIREMENTS.md
-**Schema Requirements:**
-- Replaced legacy types (FinancialRecord, IdentityDocument, etc.)
-- Added Tier 1 ICP-aligned types organized by category
-- Added rationale section linking schema types to Tier 1 ICP workflows
-### 5. schema.md (docs/subsystems/)
-**JSONB Examples:**
-- Added Contract example (section 3.5)
-- Added Message (Email/Chat) example (section 3.6)
-- Added Document example (section 3.7)
-- Added Note example (section 3.8)
-- Updated fallback example to use "document" (section 3.9)
-- Updated section numbering (3.10 for querying)
-- Added Tier 1 ICP use cases for each schema type
-### 6. ingestion.md (docs/subsystems/ingestion/)
-**Schema Detection:**
-- Updated `detectSchemaType()` example to use Tier 1 ICP types
-- Changed return values from `FinancialRecord` → `invoice`, `receipt`, etc.
-- Updated fallback from `PDFDocument` → `document`
-- Added Tier 1 ICP alignment note
-**Schema Registry:**
-- Updated to show `CANONICAL_TYPES` array
-- Updated to show `SCHEMA_FIELDS` mapping
-- Added `FALLBACK_TYPE = 'document'`
-**Field Extraction:**
-- Updated extractors to use Tier 1 ICP type names
-- Added `extractFieldsForContract()`, `extractFieldsForDocument()`, etc.
-- Added implementation note: "No LLM calls permitted in MVP"
-### 7. NEW: IMPLEMENTATION_OVERWRITE_GUIDE.md
-**Created comprehensive overwrite guide with:**
-- Specific violations requiring overwrites
-- Current code that MUST be removed/changed
-- Required replacement code
-- Feature Unit execution checklists
-- Verification tests
-- Cleanup phase grep commands
-- Agent instructions emphasizing overwrite requirement
-## Manifest Compliance Summary
-### Violations Identified
-| Violation | Location | Manifest Line | Fix Required | FU |
-|-----------|----------|---------------|--------------|-----|
-| LLM Extraction | `src/services/file_analysis.ts` | 125, 393, 789 | Remove OpenAI calls, add regex | FU-100 |
-| Custom Types | `src/config/record_types.ts` | 497, 505 | Return `'document'` fallback only | FU-100 |
-| Semantic Search | `src/actions.ts` (if present) | 394, 590 | Disable vector similarity | FU-105 |
-| Semantic UI Toggle | Search components (if present) | 394 | Remove mode toggle | FU-600 |
-### Compliance Actions
-**MUST Overwrite:**
-- `analyzeFileForRecord()` — Replace LLM with regex
-- `normalizeRecordType()` — Replace custom types with `'document'` fallback
-- `retrieve_records()` — Remove semantic search
-- Search UI — Remove semantic mode toggle
-**MUST Remove:**
-- All `openai.chat.completions.create()` calls
-- `sanitizeCustomType()` function
-- Vector similarity queries
-- Semantic search mode UI
-**MUST Add:**
-- Rule-based schema detection (regex patterns)
-- Rule-based field extractors per schema type
-- Determinism tests (100 runs → same output)
-## Execution Priority
-**Before Any Other Feature Units:**
-1. Execute FU-100 to remove LLM and implement rule-based extraction
-2. Verify determinism tests pass (100 runs → same output)
-3. Verify no OpenAI calls in codebase
-**During Execution:**
-- FU-100 blocks FU-101 (entity resolution depends on extraction)
-- FU-100 blocks FU-102 (event generation depends on extraction)
-- FU-105 must remove semantic search
-- FU-600 must remove semantic UI
-**Before Launch:**
-- Verify all overwrites complete
-- Run cleanup verification (grep commands in IMPLEMENTATION_OVERWRITE_GUIDE.md)
-- Confirm 100% manifest compliance
-## Agent Instructions
-### Loading Order for Feature Unit Execution
-1. Load `docs/NEOTOMA_MANIFEST.md` (architectural truth)
-2. Load `docs/specs/IMPLEMENTATION_OVERWRITE_GUIDE.md` (required changes)
-3. Load `docs/specs/MVP_EXECUTION_PLAN.md` (execution sequence)
-4. Load `docs/specs/MVP_FEATURE_UNITS.md` (feature inventory)
-5. Load relevant subsystem docs (e.g., `docs/subsystems/ingestion/ingestion.md` for FU-100)
-### When Executing FU-100, FU-105, FU-600
-**DO:**
-- **OVERWRITE** non-compliant code completely
-- Replace with manifest-compliant implementations
-- Write determinism tests
-- Verify no external API calls during extraction
-**DO NOT:**
-- Preserve LLM code "for compatibility"
-- Add feature flags to conditionally enable violations
-- Keep custom type logic "for flexibility"
-- Implement hybrid approaches (rule-based + LLM)
-**The manifest is absolute. Implementation adapts, not the manifest.**
-## References
-**Primary Compliance Documents:**
-- [`docs/NEOTOMA_MANIFEST.md`](../NEOTOMA_MANIFEST.md) — Architectural constraints
-- [`docs/specs/IMPLEMENTATION_OVERWRITE_GUIDE.md`](./IMPLEMENTATION_OVERWRITE_GUIDE.md) — Required code changes
-**Updated Planning Documents:**
-- [`docs/specs/MVP_FEATURE_UNITS.md`](./MVP_FEATURE_UNITS.md) — Feature inventory with compliance notes
-- [`docs/specs/MVP_EXECUTION_PLAN.md`](./MVP_EXECUTION_PLAN.md) — Execution sequence with overwrite instructions
-**Updated Schema Documents:**
-- [`docs/specs/FUNCTIONAL_REQUIREMENTS.md`](./FUNCTIONAL_REQUIREMENTS.md) — Tier 1 ICP schema types
-- [`docs/subsystems/schema.md`](../subsystems/schema.md) — Schema examples with Tier 1 ICP use cases
-- [`docs/subsystems/ingestion/ingestion.md`](../subsystems/ingestion/ingestion.md) — Rule-based detection patterns
-**Status:** All MVP planning documents now align with NEOTOMA_MANIFEST.md. Implementation overwrites will occur during Feature Unit execution.