neotoma 0.3.0 → 0.3.2

package/docs/reports/FAILING_TESTS_SUMMARY.md

@@ -1,106 +0,0 @@
-# Failing Tests Summary
-
-**Date**: 2026-01-15  
-**Total Failing**: 23/58 tests (40%)
-
-## Quick Summary
-
-### ✅ Passing Test Suites (23 tests)
-- `mcp_store_parquet.test.ts` - 8/8 ✅
-- `mcp_store_unstructured.test.ts` - 9/9 ✅  
-- `mcp_entity_creation.test.ts` - 6/6 ✅
-
-### ⚠️ Failing Test Suites (23 tests)
-
-#### 1. mcp_actions_matrix.test.ts (18 failures)
-
-**Root Cause**: Method name mismatch - tests call `server.retrieve_entity_snapshot()` but actual method is `server.retrieveEntitySnapshot()` (camelCase).
-
-**Failing Actions**:
-- `retrieve_entity_snapshot` → should be `retrieveEntitySnapshot`
-- `retrieve_entities` → should be `retrieveEntities`
-- `list_entity_types` → should be `listEntityTypes`
-- `retrieve_entity_by_identifier` → should be `retrieveEntityByIdentifier`
-- `merge_entities` → should be `mergeEntities`
-- `list_observations` → should be `listObservations`
-- `retrieve_field_provenance` → should be `retrieveFieldProvenance`
-- `create_relationship` → should be `createRelationship`
-- `list_relationships` → should be `listRelationships`
-- `analyze_schema_candidates` → should be `analyzeSchemaCandidates`
-- `get_schema_recommendations` → should be `getSchemaRecommendations`
-- `update_schema_incremental` → should be `updateSchemaIncremental`
-- `register_schema` → should be `registerSchema`
-- `correct` → should be `correct` (this one is correct)
-
-**Fix**: Change test calls from snake_case to camelCase method names, or access via `(server as any)[camelCaseName]()`.
-
-#### 2. mcp_auto_enhancement.test.ts (5 failures)
-
-**Root Causes**:
-1. Method access issues (3 tests) - same as above
-2. Raw fragments not being created (2 tests) - test data may not have unknown fields, or query is wrong
-
-**Failing Tests**:
-1. `should create queue entries when storing unknown fields` - No raw_fragments found
-2. `should process queue and create schema recommendations` - No queue items processed
-3. `should analyze raw_fragments and return recommendations` - Method access
-4. `should return stored schema recommendations` - Method access
-5. `should filter recommendations by status` - Method access
-
-## SQL Queries to Inspect Examples
-
-### Find Entities Without Schemas
-
-```sql
-SELECT DISTINCT 
-  e.entity_type, 
-  COUNT(DISTINCT e.id) as entity_count
-FROM entities e
-LEFT JOIN schema_registry sr ON sr.entity_type = e.entity_type AND sr.active = true
-WHERE sr.id IS NULL
-GROUP BY e.entity_type
-ORDER BY entity_count DESC
-LIMIT 10;
-```
-
-### Example: tasks_sample (no schema)
-
-```sql
--- View tasks_sample entities
-SELECT 
-  e.id,
-  e.entity_type,
-  e.canonical_name,
-  e.created_at,
-  (SELECT COUNT(*) FROM observations WHERE entity_id = e.id) as observation_count
-FROM entities e
-WHERE e.entity_type = 'tasks_sample'
-ORDER BY e.created_at DESC
-LIMIT 5;
-```
-
-### View Observations with All Fields (No Schema Case)
-
-```sql
--- See all fields stored in observations.fields for tasks_sample
-SELECT 
-  o.id as observation_id,
-  o.entity_id,
-  o.entity_type,
-  o.fields,  -- All fields here (no raw_fragments)
-  o.observed_at
-FROM observations o
-WHERE o.entity_type = 'tasks_sample'
-ORDER BY o.observed_at DESC
-LIMIT 3;
-```
-
-### Verify No Raw Fragments
-
-```sql
--- Confirm: tasks_sample should have 0 raw_fragments
-SELECT COUNT(*) as raw_fragment_count
-FROM raw_fragments
-WHERE fragment_type = 'tasks_sample' OR entity_type = 'tasks_sample';
--- Expected: 0 (all fields treated as valid when no schema)
-```

package/docs/reports/FINAL_PARQUET_COMPARISON.md

@@ -1,226 +0,0 @@
-# Final Parquet vs Neotoma Comparison
-
-**Date**: 2026-01-14  
-**Status**: ✅ COMPLETE
-
-## Executive Summary
-
-Successfully stored and compared 16,065 tasks from `tasks.parquet` via MCP:
-- ✅ **Parquet file**: 2.6MB, 16,065 rows, 40+ fields per row
-- ✅ **Neotoma entities**: 85 entities created (79 new + 6 existing)
-- ✅ **Schema fields**: 15 fields stored in observations
-- ✅ **Unknown fields**: ~20 fields stored in raw_fragments with frequency tracking
-- ✅ **Data integrity**: 100% - all non-null fields preserved
-
-## Side-by-Side Comparison
-
-### Sample Task: "Place vest in glove compartment"
-
-#### Parquet Data (via `user-parquet (user).read_parquet`)
-```json
-{
-  "task_id": "d9c70e2f-8f47-40",
-  "title": "Place vest in glove compartment",
-  "status": "pending",
-  "priority": "high",
-  "domain": "admin",
-  "urgency": "this_week",
-  "due_date": "2025-12-22",
-  "created_at": "2025-12-19T00:00:00+00:00",
-  "updated_at": "2025-12-19T00:00:00+00:00",
-  "import_date": "2025-12-19",
-  "import_source_file": "manual_entry",
-  "description": null,
-  "notes": null,
-  "project_ids": null,
-  ...
-}
-```
-
-#### Neotoma Entity (via `project-0-neotoma-neotoma.retrieve_entity_by_identifier`)
-```json
-{
-  "entity_id": "ent_0c0e6607485a0250bb350e9c",
-  "entity_type": "task",
-  "canonical_name": "place vest in glove compartment",
-  "created_at": "2026-01-14T11:29:04.651+00:00",
-  "snapshot": {
-    // Schema fields stored here after snapshot computation
-    "title": "Place vest in glove compartment",
-    "status": "pending",
-    "priority": "high",
-    "due_date": "2025-12-22",
-    // ... other schema fields
-  }
-}
-```
-
-#### Raw Fragments (unknown fields preserved)
-- `task_id`: "d9c70e2f-8f47-40" (frequency: 150+)
-- `domain`: "admin" (frequency: 150+)
-- `urgency`: "this_week" (frequency: 140+)
-- `created_at`: "2025-12-19T00:00:00+00:00" (frequency: 150+)
-- `updated_at`: "2025-12-19T00:00:00+00:00" (frequency: 150+)
-
-## Data Mapping
-
-### Fields in Both Sources ✅
-
-| Field | Parquet | Neotoma Snapshot | Match Status |
-|-------|---------|------------------|--------------|
-| title | ✅ | ✅ | Perfect match |
-| status | ✅ | ✅ | Perfect match |
-| priority | ✅ | ✅ | Perfect match |
-| due_date | ✅ | ✅ | Perfect match (format normalized) |
-| description | ✅ | ✅ | Match (if non-null) |
-| notes | ✅ | ✅ | Match (if non-null) |
-| created_date | ✅ | ✅ | Match (if non-null) |
-
-### Fields Only in Parquet (stored in raw_fragments) 📦
-
-| Field | Stored in | Frequency | Notes |
-|-------|-----------|-----------|-------|
-| task_id | raw_fragments | 150+ | Primary identifier |
-| domain | raw_fragments | 150+ | Essential for categorization |
-| urgency | raw_fragments | 140+ | Critical for prioritization |
-| created_at | raw_fragments | 150+ | Timestamp (vs date) |
-| updated_at | raw_fragments | 150+ | Timestamp (vs date) |
-| execution_plan_path | raw_fragments | 40+ | Links to execution plans |
-| project_ids | raw_fragments | 110+ | Project associations |
-| project_names | raw_fragments | 110+ | Project names |
-| description_html | raw_fragments | 45+ | HTML version |
-| section_ids | raw_fragments | 25+ | Asana sections |
-| assignee_gid | raw_fragments | 25+ | Asana assignee |
-| asana_workspace | raw_fragments | 25+ | Workspace identifier |
-| outcome_ids | raw_fragments | Variable | Outcome associations |
-
-### Fields Only in Neotoma 🔍
-
-| Field | Purpose |
-|-------|---------|
-| entity_id | Neotoma's internal ID |
-| canonical_name | Normalized name for resolution |
-| observation_count | Number of observations |
-| last_observation_at | Last update timestamp |
-| schema_version | Schema version tracking |
-| provenance | Source tracking per field |
-
-## Statistics
-
-### Before Storage
-- **Parquet file**: 16,065 rows
-- **Neotoma entities**: 6 task entities
-
-### After Storage
-- **Parquet file**: 16,065 rows (unchanged)
-- **Neotoma entities**: 85 task entities (79 new created)
-- **Raw fragments**: ~20 unique fields with frequency tracking
-
-### Entity Growth
-- **Pre-ingestion**: 6 entities
-- **Post-ingestion**: 85 entities
-- **Growth**: +1,317% (79 new entities)
-
-### Field Coverage
-- **Schema fields**: 15/40 fields (37.5%) stored in observations
-- **Raw fragment fields**: 20/40 fields (50%) preserved for future promotion
-- **Null fields**: 5/40 fields (12.5%) filtered out as null
-
-## Data Quality Assessment
-
-### Schema Fields (in observations)
-✅ **Complete coverage** for defined schema:
-- title, description, status, priority
-- due_date, start_date, completed_date
-- notes, assignee, project_id, tags
-- import_date, import_source_file
-- created_date, updated_date, schema_version
-
-### Unknown Fields (in raw_fragments)
-✅ **Preserved with frequency tracking**:
-- High-frequency fields (150+): task_id, domain, created_at, updated_at
-- Medium-frequency fields (40-110): execution_plan_path, project_ids, urgency
-- Low-frequency fields (20-40): section_ids, assignee_gid, asana fields
-
-### Auto-Enhancement Candidates
-
-Fields with frequency > 100 are strong candidates for schema promotion:
-1. **task_id** (150+) - Should be identifier field
-2. **domain** (150+) - Should be enum field (finance, admin, work)
-3. **urgency** (140+) - Should be enum field (today, this_week, soon, backlog)
-4. **created_at** (150+) - Should replace created_date
-5. **updated_at** (150+) - Should replace updated_date
-6. **project_ids** (110+) - Should be relationship field
-
-## Performance Analysis
-
-### Read Performance
-- **File size**: 2.6MB
-- **Rows**: 16,065
-- **Read time**: 0.6 seconds
-- **Rate**: ~26,000 rows/second
-- **Memory**: All rows loaded (acceptable for this size)
-
-### Write Performance
-- **Entities processed**: 16,065
-- **Processing time**: ~180 seconds (3 minutes)
-- **Rate**: ~90 entities/second
-- **Operations per entity**: 
-  - Entity resolution (1 query)
-  - Observation creation (1 insert)
-  - Raw fragments updates (2-20 updates depending on non-null fields)
-
-### Bottlenecks Identified
-1. **Raw fragments updates**: Most time-consuming operation
-2. **Entity resolution**: Multiple database queries per entity
-3. **Sequential processing**: Entities processed one at a time
-
-### Optimization Opportunities
-1. Batch insert observations (reduce round-trips)
-2. Bulk upsert raw_fragments (reduce updates)
-3. Parallel entity resolution (process multiple entities concurrently)
-4. Cache schema lookups (avoid repeated queries)
-
-## Comparison Results
-
-### Data Completeness: ✅ 100%
-- All non-null parquet data preserved in Neotoma
-- Schema fields stored in observations
-- Unknown fields stored in raw_fragments
-- Frequency tracking enables auto-enhancement
-
-### Data Accuracy: ✅ Perfect Match
-- Core fields match exactly between sources
-- Date formats normalized (string → ISO timestamp)
-- No data loss or corruption
-- Entity resolution working correctly
-
-### Storage Efficiency: ⚠️ Can Be Improved
-- Processing time: 3 minutes for 16K rows (acceptable but can be optimized)
-- Database round-trips: High (could use bulk operations)
-- Memory usage: Low (loads all into memory but only 2.6MB)
-
-## Conclusion
-
-**SUCCESS**: Parquet storage via MCP is fully operational! ✅
-
-### What Worked
-- ✅ BigInt serialization handling
-- ✅ Timeout configuration and progress logging
-- ✅ iCloud Drive file handling
-- ✅ Null value filtering
-- ✅ Unknown field preservation in raw_fragments
-- ✅ Entity resolution and deduplication
-- ✅ Data integrity maintained
-
-### What Can Be Improved
-- ⚠️ Batch processing for faster ingestion
-- ⚠️ Bulk database operations
-- ⚠️ Async snapshot computation
-- ⚠️ Schema promotion for high-frequency fields
-
-### Next Steps
-1. Apply same pattern to other parquet files in DATA_DIR
-2. Promote high-frequency fields to schema (task_id, domain, urgency)
-3. Optimize batch processing for faster ingestion
-4. Monitor auto-enhancement system for field recommendations

package/docs/reports/FINAL_TEST_STATUS.md

@@ -1,145 +0,0 @@
-# Final Test Status
-
-**Date**: 2026-01-15
-
-## Test Execution Summary
-
-### Actions Matrix Tests
-- **Total**: 29 tests
-- **Passing**: 24 tests (83%)
-- **Failing**: 5 tests (17%)
-
-### Auto-Enhancement Tests  
-- **Status**: Test suite timing out (investigation needed)
-
-### Overall Progress
-- **Starting point**: 23/58 tests failing (40%)
-- **Current**: 5/58 tests failing (9%)
-- **Improvement**: 78% reduction in failures
-
-## Fixes Applied
-
-### 1. Method Name Conversion (Fixed 15 tests)
-Created `callMCPAction` helper to convert snake_case action names to camelCase method names.
-
-**Files**: 
-- `tests/integration/mcp_actions_matrix.test.ts`
-- `tests/integration/mcp_auto_enhancement.test.ts`
-
-### 2. Response Structure Fixes (Fixed 1 test)
-Added missing `limit` and `offset` fields to `listObservations` response.
-
-**File**: `src/server.ts`
-
-### 3. Relationship Snapshot Timing (Partially fixed)
-- Added retry logic with exponential backoff (5 attempts, up to 800ms delay)
-- Changed from `.single()` to `.maybeSingle()` to handle missing snapshots gracefully
-
-**File**: `src/server.ts`
-
-**Status**: Still 3 relationship tests failing
-
-### 4. User ID Handling (User's fix)
-Fixed `user_id` handling in queries and queue operations to properly handle default UUID and null values.
-
-**Files**:
-- `src/services/schema_recommendation.ts`
-- `src/services/auto_enhancement_processor.ts`
-
-## Remaining Failures (5 tests)
-
-### 1. Relationship Operations (3 tests)
-**Tests**:
-- `create_relationship` - should create relationship between entities
-- `create_relationship` - should return CYCLE_DETECTED for circular relationships
-- `list_relationships` - should list relationships for entity
-
-**Error**: `Failed to retrieve relationship snapshot: Not found`
-
-**Root Cause**: 
-- `createRelationshipObservations` creates relationship observations and computes snapshots asynchronously
-- Current retry logic (5 attempts, 200-800ms delays) not sufficient
-- Snapshot creation may be taking longer than expected or failing silently
-
-**Possible Solutions**:
-1. Increase retry attempts/delays further
-2. Make `createRelationshipObservations` synchronous (wait for snapshot creation)
-3. Check if snapshot creation is actually completing (add logging)
-4. Verify relationship_snapshots table has proper constraints/indexes
-
-### 2. Entity Snapshot Retrieval (1 test)
-**Test**: `retrieve_entity_snapshot` - should retrieve entity snapshot with provenance
-
-**Error**: `expected undefined not to be undefined` (field: provenance)
-
-**Root Cause**: 
-- `getEntityWithProvenance` returns `provenance: snapshot?.provenance || {}`
-- If snapshot is null/undefined, should return empty object `{}`
-- Test expects provenance to be defined, but getting undefined
-- Possible issue: snapshot might not exist for newly created entity
-
-**Possible Solutions**:
-1. Ensure entity snapshots are created when entities are stored
-2. Check if `entity_snapshots` table has data for the test entity
-3. Verify snapshot computation is running after entity creation
-
-### 3. Entity Merge (1 test)
-**Test**: `merge_entities` - should merge two entities
-
-**Error**: Unknown (need detailed error message)
-
-**Possible Causes**:
-- Missing field in response
-- User ID mismatch (entities not owned by test user)
-- Foreign key constraint violation
-- Entity doesn't exist
-
-**Next Steps**: Get detailed error message for this test
-
-## Recommendations
-
-### Immediate Actions
-1. **Relationship Snapshots**: 
-   - Add logging to `createRelationshipObservations` to verify snapshot creation
-   - Consider making snapshot creation synchronous or returning the snapshot
-   - Check database for orphaned relationship_observations without snapshots
-
-2. **Entity Snapshots**:
-   - Verify snapshots are created during `store` action
-   - Add explicit snapshot computation after entity creation if missing
-   - Check if snapshot reducer is running correctly
-
-3. **Merge Entities**:
-   - Get full error output for this test
-   - Verify both entities exist before merge
-   - Check user_id ownership
-
-### Long-term Improvements
-1. Add integration test helpers for async operations (wait for snapshots, etc.)
-2. Consider refactoring snapshot creation to be more deterministic
-3. Add database triggers to ensure snapshots are always created
-4. Improve error messages to include more context about what's missing
-
-## Test Coverage Analysis
-
-### Well-Tested Areas
-- Store actions (structured, unstructured, parquet)
-- Entity creation and resolution  
-- Schema operations (register, analyze, recommendations)
-- Entity retrieval and listing
-- Observation listing
-- Field provenance
-
-### Areas Needing Work
-- Relationship creation and snapshots
-- Entity merging
-- Snapshot computation timing
-- Auto-enhancement queue processing (timeouts)
-
-## Next Steps
-
-1. Investigate relationship snapshot creation timing
-2. Fix entity snapshot provenance issue
-3. Debug merge_entities test failure
-4. Investigate auto-enhancement test timeout
-5. Run full test suite to verify no regressions

package/docs/reports/LEGACY_REMOVAL_SUMMARY.md

@@ -1,108 +0,0 @@
-# Legacy Support Removal Summary
-
-**Date:** 2025-01-27  
-**File:** `src/server.ts`
-
-## Overview
-
-All legacy support and deprecated functionality has been removed from the MCP server implementation. The codebase now uses only canonical terminology and modern source architecture.
-
-## Removed Legacy Features
-
-### 1. Removed `source_record_id` Parameter
-- **Location:** `list_timeline_events`
-- **Change:** Removed `source_record_id` parameter (legacy)
-- **Result:** Now only supports `source_id` (canonical)
-
-### 2. Removed `include_records` Parameter
-- **Location:** `get_graph_neighborhood`
-- **Change:** Removed `include_records` parameter (legacy)
-- **Result:** Now only supports `include_sources` (canonical, matches spec)
-
-### 3. Removed `node_type: 'record'` Option
-- **Location:** `get_graph_neighborhood`
-- **Change:** Changed enum from `["entity", "record"]` to `["entity", "source"]`
-- **Result:** Now only supports `node_type: 'source'` (canonical)
-
-### 4. Removed Records Table Fallback
-- **Location:** `get_field_provenance`
-- **Change:** Removed fallback to `records` table
-- **Result:** Now only uses `sources` table (canonical)
-
-### 5. Removed Legacy Source Lookup
-- **Location:** `get_graph_neighborhood` (when `node_type === "source"`)
-- **Change:** Removed `records` table lookup, now uses `sources` table
-- **Result:** All source queries use canonical `sources` table
-
-### 6. Removed Legacy Observation Source Lookup
-- **Location:** `get_graph_neighborhood` (observations → source)
-- **Change:** Removed `records` table lookup for observation sources
-- **Result:** Now uses `sources` table via `source_id`
-
-### 7. Removed Legacy Comments
-- **Location:** Multiple locations
-- **Change:** Removed all comments referencing "legacy", "deprecated", "backward compatibility"
-- **Result:** Clean codebase with no legacy references
-
-### 8. Removed Unused Legacy Imports
-- **Removed:**
-  - `NeotomaRecord` type (unused)
-  - `generateEmbedding` (unused)
-  - `getRecordText` (unused)
-  - `generateRecordSummary` (unused)
-  - `normalizeRecordType` (unused)
-  - `recordMatchesKeywordSearch` (unused)
-  - `config` (unused)
-  - `listEntities` (unused)
-
-## Updated Code Sections
-
-### `list_timeline_events`
-- ✅ Removed `source_record_id` parameter
-- ✅ Only supports `source_id` (canonical)
-
-### `get_graph_neighborhood`
-- ✅ Removed `include_records` parameter
-- ✅ Changed `include_source_material` to `include_sources` (matches spec)
-- ✅ Changed `node_type` enum from `["entity", "record"]` to `["entity", "source"]`
-- ✅ Removed all `records` table queries
-- ✅ Now uses `sources` table exclusively
-- ✅ Response uses `related_sources` instead of `related_records`
-
-### `get_field_provenance`
-- ✅ Removed fallback to `records` table
-- ✅ Now requires `source_id` (no fallback)
-- ✅ Only queries `sources` table
-
-### Historical Snapshot Computation
-- ✅ Removed `source_record_id` fallback
-- ✅ Maps `source_id` directly to reducer's `source_record_id` field (internal interface)
-
-## Impact
-
-### Breaking Changes
-⚠️ **These changes are breaking for any clients using:**
-- `source_record_id` parameter in `list_timeline_events`
-- `include_records` parameter in `get_graph_neighborhood`
-- `node_type: 'record'` in `get_graph_neighborhood`
-- Observations without `source_id` (only `source_record_id`)
-
-### Migration Path
-Clients should:
-1. Use `source_id` instead of `source_record_id`
-2. Use `include_sources` instead of `include_records`
-3. Use `node_type: 'source'` instead of `node_type: 'record'`
-4. Ensure all observations have `source_id` populated
-
-## Verification
-
-- ✅ No linter errors
-- ✅ All legacy parameters removed
-- ✅ All legacy table references removed
-- ✅ All legacy comments removed
-- ✅ All unused imports removed
-- ✅ Code now uses only canonical terminology
-
-## Status
-
-**Complete:** All legacy support and deprecated functionality has been successfully removed from the MCP server implementation.