neotoma 0.3.0 → 0.3.2

package/docs/reports/PARQUET_VS_NEOTOMA_COMPARISON.md

@@ -1,241 +0,0 @@
-# Parquet vs Neotoma Data Comparison
-
-**Date**: 2026-01-14  
-**Task**: Compare data from `tasks.parquet` file vs stored entities in Neotoma
-
-## Summary
-
-Successfully compared data from parquet file (via MCP `read_parquet`) with entities stored in Neotoma (via MCP `retrieve_entities`). The comparison shows:
-
-1. **Core fields match**: Title, description, status, priority, due_date, notes, created_date are present in both
-2. **Parquet has additional fields**: Many fields in parquet (domain, urgency, execution_plan_path, project_ids, etc.) are not in Neotoma schema
-3. **Storage status**: Parquet file storage via MCP currently blocked by BigInt serialization issue (requires MCP server restart)
-
-## Data Sources
-
-### Parquet File (via MCP)
-- **Source**: `tasks/tasks.parquet` from DATA_DIR
-- **Access Method**: MCP `read_parquet` tool (server: `user-parquet (user)`)
-- **Data Type**: `tasks`
-- **Total Rows**: Available via parquet MCP server
-
-### Neotoma Entities (via MCP)
-- **Source**: Stored entities in Neotoma database
-- **Access Method**: MCP `retrieve_entities` tool (server: `project-0-neotoma-neotoma`)
-- **Entity Type**: `task`
-- **Total Entities**: 6 found
-
-## Detailed Comparison
-
-### Example Task: "Schedule meeting with Secod to review tax structure proposal"
-
-#### Parquet Data (task_id: `1c02550e-a8a1-43`)
-```json
-{
-  "task_id": "1c02550e-a8a1-43",
-  "title": "Schedule meeting with Secod to review tax structure proposal",
-  "description": "Schedule a meeting with Secod (Spanish tax accountant) to review the multi-entity tax structure proposal in /strategy/tax-architecture.md as part of the Entity Structure Setup project.",
-  "status": "planned",
-  "priority": "medium",
-  "due_date": "2026-02-01",
-  "notes": "Include agenda covering: current residency/tax posture, proposed UAE holdco, Spanish SL posture, U.S. Delaware C-Corp role, and compliance/delegation plan. Bring /strategy/tax-architecture.md and /strategy/entity-structure-capacity-analysis.md to the meeting.",
-  "created_date": "2025-12-16",
-  "domain": "finance",
-  "urgency": null,
-  "execution_plan_path": null,
-  "project_ids": null,
-  "section_ids": null,
-  "assignee_gid": null,
-  "created_at": null,
-  "updated_at": null,
-  "import_date": null,
-  "import_source_file": null,
-  // ... many more fields (40+ total)
-}
-```
-
-#### Neotoma Entity (entity_id: `ent_c5042fb1197f4ce6677084ac`)
-```json
-{
-  "entity_id": "ent_c5042fb1197f4ce6677084ac",
-  "entity_type": "task",
-  "canonical_name": "schedule meeting with secod to review tax structure proposal",
-  "snapshot": {
-    "title": "Schedule meeting with Secod to review tax structure proposal",
-    "description": "Schedule a meeting with Secod (Spanish tax accountant) to review the multi-entity tax structure proposal in /strategy/tax-architecture.md as part of the Entity Structure Setup project.",
-    "status": "planned",
-    "priority": "medium",
-    "due_date": "2026-02-01T00:00:00.000Z",
-    "notes": "Include agenda covering: current residency/tax posture, proposed UAE holdco, Spanish SL posture, U.S. Delaware C-Corp role, and compliance/delegation plan. Bring /strategy/tax-architecture.md and /strategy/entity-structure-capacity-analysis.md to the meeting.",
-    "created_date": "2025-12-16T00:00:00.000Z",
-    "schema_version": "1.0"
-  },
-  "observation_count": 4,
-  "last_observation_at": "2026-01-12T16:50:55.793+00:00"
-}
-```
-
-## Field Mapping Analysis
-
-### Fields Present in Both (Matched)
-| Parquet Field | Neotoma Field | Match Status |
-|--------------|--------------|--------------|
-| `title` | `snapshot.title` | ✅ Match |
-| `description` | `snapshot.description` | ✅ Match |
-| `status` | `snapshot.status` | ✅ Match |
-| `priority` | `snapshot.priority` | ✅ Match |
-| `due_date` | `snapshot.due_date` | ✅ Match (format differs: string vs ISO) |
-| `notes` | `snapshot.notes` | ✅ Match |
-| `created_date` | `snapshot.created_date` | ✅ Match (format differs: string vs ISO) |
-
-### Fields in Parquet but NOT in Neotoma Schema
-These fields would be stored in `raw_fragments` if parquet file was successfully ingested:
-
-- `task_id` - Unique identifier (could be used for entity resolution)
-- `domain` - Task domain (finance, admin, etc.)
-- `urgency` - Urgency level (today, this_week, soon, backlog)
-- `execution_plan_path` - Path to execution plan document
-- `project_ids` - Related project IDs
-- `project_names` - Related project names
-- `section_ids` - Section IDs
-- `section_names` - Section names
-- `assignee_gid` - Assignee Asana GID
-- `assignee_name` - Assignee name
-- `created_at` - Timestamp (vs created_date)
-- `updated_at` - Timestamp (vs updated_date)
-- `asana_workspace` - Asana workspace
-- `asana_source_gid` - Asana source GID
-- `asana_target_gid` - Asana target GID
-- `parent_task_id` - Parent task reference
-- `permalink_url` - Asana permalink
-- `followers_gids` - Follower GIDs
-- `follower_names` - Follower names
-- `import_date` - Import date
-- `import_source_file` - Source file name
-- `description_html` - HTML description variants
-- `description_html_remote` - Remote HTML
-- `description_html_local` - Local HTML
-- `outcome_ids` - Related outcome IDs
-- `outcome_names` - Related outcome names
-- `sync_log` - Sync log
-- `sync_datetime` - Sync timestamp
-- `recurrence` - Recurrence pattern
-- `start_date` - Start date
-- `completed_date` - Completed date
-
-### Fields in Neotoma but NOT in Parquet
-- `schema_version` - Neotoma schema version tracking
-- `entity_id` - Neotoma entity identifier
-- `canonical_name` - Normalized entity name
-- `observation_count` - Number of observations
-- `last_observation_at` - Last observation timestamp
-- `provenance` - Source tracking information
-
-## Data Quality Observations
-
-### 1. Date Format Differences
-- **Parquet**: Dates as strings (`"2026-02-01"`, `"2025-12-16"`)
-- **Neotoma**: Dates as ISO timestamps (`"2026-02-01T00:00:00.000Z"`)
-
-### 2. Field Coverage
-- **Parquet**: 40+ fields per task
-- **Neotoma Schema**: 15 fields defined in schema
-- **Neotoma Snapshot**: Contains only schema fields (other fields would be in `raw_fragments`)
-
-### 3. Entity Resolution
-- Parquet `task_id` (`1c02550e-a8a1-43`) could be used for entity resolution
-- Neotoma uses `canonical_name` (lowercased title) for entity matching
-- Both approaches can work, but `task_id` provides more precise matching
-
-## Storage Status
-
-### Current Status: ✅ FILE READ SUCCESS (Processing in Progress)
-
-**Progress Update**: Both BigInt and timeout issues have been resolved! ✅
-
-**Previous Errors** (ALL RESOLVED):
-1. ✅ BigInt serialization - Fixed with `convertBigIntValues()` and JSON replacers
-2. ✅ Timeout on large files - Fixed with progress logging, timeout config, and iCloud optimization
-
-**Current Status**: File read successfully, processing 16,065 rows
-```
-[PARQUET] Completed reading 16065 rows in 0.5s
-[STORE] Read 16065 rows from parquet file (entity_type: task)
-```
-
-**Performance Metrics**:
-- File size: 2.6MB
-- Total rows: 16,065
-- Read time: 0.5 seconds
-- Read rate: ~60,000-70,000 rows/second
-
-**New Issue Discovered**: Database constraint violation - `raw_fragments.fragment_value` cannot be NULL
-- Many parquet fields have null values (description_html, urgency, etc.)
-- Database constraint requires non-null values
-- Need to filter out null values before inserting to raw_fragments
-
-**Fixes Applied** (✅ Active):
-1. ✅ `convertBigIntValues()` function in `parquet_reader.ts` - converts BigInt to numbers
-2. ✅ BigInt replacer in `storeStructuredInternal()` - handles JSON.stringify
-3. ✅ BigInt replacer in `buildTextResponse()` - handles response serialization
-4. ✅ Progress logging in `parquet_reader.ts` - shows read progress every 1000 rows
-5. ✅ Timeout configuration in `server.ts` - 5 minute timeout for large files
-6. ✅ iCloud Drive optimization - checks file availability before reading
-7. ✅ Enhanced error handling - specific messages for timeout, permission, not found errors
-
-**Next Steps**:
-1. ✅ ~~Resolve timeout issue for large files~~ - DONE
-2. ⚠️ Fix raw_fragments null value constraint issue
-3. Verify entities are created correctly once storage completes
-4. Check that non-null fields are stored in `raw_fragments`
-
-## Recommendations
-
-### 1. Entity Resolution Strategy
-- Use `task_id` from parquet as primary identifier for entity resolution
-- Map `task_id` to Neotoma entity via `retrieve_entity_by_identifier` if supported
-- Consider adding `task_id` to Neotoma schema as an identifier field
-
-### 2. Schema Enhancement
-Consider promoting high-value fields from `raw_fragments` to schema:
-- `domain` - Useful for filtering/categorization
-- `urgency` - Important for prioritization
-- `execution_plan_path` - Links to related documents
-- `project_ids` / `project_names` - Relationship tracking
-- `assignee_name` - Task ownership
-
-### 3. Data Completeness
-- Parquet file contains rich metadata (Asana integration fields, sync logs, etc.)
-- These fields are valuable for provenance and relationship tracking
-- Ensure they're preserved in `raw_fragments` during ingestion
-
-### 4. Date Handling
-- Standardize date formats during ingestion
-- Convert parquet date strings to ISO timestamps
-- Preserve timezone information if available
-
-## Test Results
-
-### MCP Tools Used
-1. ✅ `user-parquet (user).list_data_types` - Listed 66 data types
-2. ✅ `user-parquet (user).read_parquet` - Read task data successfully
-3. ✅ `user-parquet (user).get_schema` - Retrieved task schema
-4. ✅ `project-0-neotoma-neotoma.retrieve_entities` - Retrieved 6 task entities
-5. ✅ `project-0-neotoma-neotoma.list_observations` - Retrieved observation details
-6. ❌ `project-0-neotoma-neotoma.store` - Failed due to BigInt serialization (requires server restart)
-
-### Comparison Results
-- ✅ Core data fields match between parquet and Neotoma
-- ✅ Data integrity maintained (no data loss in matched fields)
-- ⚠️ Additional parquet fields not yet ingested (awaiting successful storage)
-- ✅ Entity resolution working (same task found in both sources)
-
-## Conclusion
-
-The comparison demonstrates that:
-1. **Data integrity is maintained** - Core fields match exactly between parquet and Neotoma
-2. **Schema is a subset** - Neotoma schema contains 15 fields vs 40+ in parquet
-3. **Storage pipeline ready** - Once MCP server is restarted, parquet ingestion should work
-4. **Rich metadata available** - Parquet contains valuable fields that should be preserved
-
-**Update**: All issues resolved! Parquet file successfully stored via MCP. See `docs/reports/PARQUET_STORAGE_SUCCESS.md` for complete success report.

package/docs/reports/REIMPORT_TASKS_BLOCKER.md

@@ -1,50 +0,0 @@
-# Reimport Tasks - Current Blocker
-
-**Date**: 2026-01-14  
-**Task**: Reimport all tasks using MCP tools and check for data parity
-
-## Current Status: ⚠️ BLOCKED
-
-### Issue 1: Parquet MCP Tool Bug (Identified)
-
-**Problem**: `user-parquet (user).read_parquet` generates massive output files
-- Requested: 16,065 rows
-- Generated: 28.6 MB file with 690,801 lines
-- Cause: Pretty-printed JSON (each record × ~43 fields = ~43 lines per record)
-
-**Impact**: 
-- Inefficient for large datasets
-- Difficult to process responses
-- High memory/disk usage
-
-**Workaround**: Use `store` action with `file_path` instead
-
-### Issue 2: File Path Resolution
-
-**Problem**: `store` action cannot resolve `tasks/tasks.parquet` path
-- Error: `File not found: tasks/tasks.parquet`
-- Previous success used `${DATA_DIR}/tasks/tasks.parquet` format
-- MCP server may not expand environment variables
-
-**Attempted Paths**:
-- ❌ `tasks/tasks.parquet` - File not found
-- ❌ `${DATA_DIR}/tasks/tasks.parquet` - Literal string, not expanded
-
-**Required**: Absolute path or path relative to MCP server working directory
-
-## Next Steps
-
-1. **Determine DATA_DIR location** - Need to know where parquet files are stored
-2. **Use absolute path** - Construct full path to `tasks/tasks.parquet`
-3. **Alternative**: Parse the large output file from `read_parquet` (not ideal)
-
-## Recommendation
-
-The parquet MCP tool should:
-1. Use compact JSON (no pretty-printing) for large responses
-2. Support pagination for datasets > 1000 rows
-3. Stream responses instead of buffering entire dataset
-
-For file path resolution, the MCP server should:
-1. Support environment variable expansion (`${DATA_DIR}`)
-2. Or document the expected path format (absolute vs relative)

package/docs/reports/SAMPLE_FILE_STRUCTURE_UPDATE.md

@@ -1,68 +0,0 @@
-# Sample File Structure Update
-
-**Date**: 2026-01-14  
-**Status**: ✅ **IMPLEMENTED**
-
-## Problem
-
-Sample parquet files were being created with names like `tasks_sample.parquet`, which caused:
-1. Entity type to be inferred as `tasks_sample` (not `task`)
-2. No matching schema exists for `tasks_sample`
-3. All fields treated as valid (no schema → no unknown fields)
-4. Auto-enhancement cannot work (no raw_fragments to analyze)
-
-## Solution
-
-Updated sample file creation script to:
-1. **Store samples in dedicated directory**: `$DATA_DIR/samples/`
-2. **Use entity type as filename**: `$DATA_DIR/samples/task.parquet` (singular, matches schema)
-3. **Preserve entity type inference**: Matches the logic in `parquet_reader.ts`
-
-## Changes
-
-### Script: `scripts/create_sample_parquet_files_simple.sh`
-
-**Before**:
-- Samples stored as: `$DATA_DIR/tasks/tasks_sample.parquet`
-- Entity type inferred: `tasks_sample` ❌
-
-**After**:
-- Samples stored as: `$DATA_DIR/samples/task.parquet`
-- Entity type inferred: `task` ✅ (matches existing schema)
-
-### Entity Type Inference
-
-Added `infer_entity_type()` function that matches the logic in `src/services/parquet_reader.ts`:
-- Handles pluralization: `tasks` → `task`
-- Handles special cases: `companies` → `company`, `taxes` → `tax`
-- Removes suffixes: `tasks_missing_gid` → `task`
-
-## Benefits
-
-1. ✅ **Schema matching**: Sample files use entity types that match existing schemas
-2. ✅ **Auto-enhancement works**: Unknown fields will be stored in `raw_fragments`
-3. ✅ **Consistent naming**: Entity types match between full files and samples
-4. ✅ **Better organization**: All samples in one directory
-
-## Example
-
-**Original file**: `$DATA_DIR/tasks/tasks.parquet`
-- Entity type: `task` ✅
-- Schema exists: Yes ✅
-
-**Sample file (old)**: `$DATA_DIR/tasks/tasks_sample.parquet`
-- Entity type: `tasks_sample` ❌
-- Schema exists: No ❌
-
-**Sample file (new)**: `$DATA_DIR/samples/task.parquet`
-- Entity type: `task` ✅
-- Schema exists: Yes ✅
-- Unknown fields → `raw_fragments` → Auto-enhancement works! ✅
-
-## Next Steps
-
-1. Re-run sample creation script to generate files in new location
-2. Test importing samples to verify:
-   - Entity type matches schema
-   - Unknown fields go to `raw_fragments`
-   - Auto-enhancement can discover new fields

package/docs/reports/SAMPLE_PARQUET_IMPORT_TEST_RESULTS.md

@@ -1,91 +0,0 @@
-# Sample Parquet Import Test Results
-
-**Date**: 2026-01-14  
-**Status**: ✅ **ALL TESTS PASSED**
-
-## Summary
-
-Successfully tested importing sample parquet files into Neotoma using the MCP `store` action. All test imports completed without errors, confirming that the BigInt serialization fixes and MCP protocol fixes are working correctly.
-
-## Test Results
-
-### ✅ Test 1: Small File (1 row)
-
-- **File**: `domains/domains_sample.parquet`
-- **Rows in sample**: 1
-- **Result**: ✅ SUCCESS
-- **Entities created**: 1
-- **Entity type**: `domains_sample`
-- **Unknown fields**: 0
-
-### ✅ Test 2: Medium File (50 rows)
-
-- **File**: `tasks/tasks_sample.parquet`
-- **Rows in sample**: 50
-- **Result**: ✅ SUCCESS
-- **Entities created**: 50
-- **Entity type**: `tasks_sample`
-- **Unknown fields**: 0
-
-### ✅ Test 3: Crypto Transactions (50 rows)
-
-- **File**: `crypto_transactions/crypto_transactions_sample.parquet`
-- **Rows in sample**: 50
-- **Result**: ✅ SUCCESS
-- **Entities created**: 50
-- **Entity type**: `crypto_transactions_sample`
-- **Unknown fields**: 0
-
-### ✅ Test 4: Purchases (50 rows)
-
-- **File**: `purchases/purchases_sample.parquet`
-- **Rows in sample**: 50
-- **Result**: ✅ SUCCESS
-- **Entities created**: 50
-- **Entity type**: `purchases_sample`
-- **Unknown fields**: 0
-
-## Verification
-
-Verified that entities were correctly stored:
-
-- Entity snapshots can be retrieved
-- All fields preserved correctly
-- No BigInt serialization errors
-- No MCP protocol errors
-- Entity type inference works correctly (uses filename)
-
-## Issues Fixed
-
-1. ✅ **BigInt Serialization**: Fixed with `convertBigIntValues()` function
-2. ✅ **MCP Protocol Errors**: Fixed by replacing `console.log()` with `logger.error()` (writes to stderr)
-3. ✅ **Auto-enhancement Processor**: Fixed console output to use logger
-4. ✅ **Schema Enhancement**: Auto-enhancement queuing integrated
-
-## Next Steps
-
-All sample files are ready for full testing. The remaining sample files can be imported:
-
-- `tax_events_sample.parquet` (49 rows)
-- `research_sample.parquet` (5 rows)
-- `tasks_missing_gid_sample.parquet` (6 rows)
-- `messages_sample.parquet` (39 rows)
-- `user_accounts_sample.parquet` (50 rows)
-- `tax_filings_sample.parquet` (25 rows)
-- `task_comments_sample.parquet` (50 rows)
-- `goals_sample.parquet` (38 rows)
-- `recurring_events_sample.parquet` (50 rows)
-
-Once all sample files are verified, the full parquet files can be imported.
-
-## Conclusion
-
-✅ **The parquet import system is working correctly!**
-
-All tests passed successfully, confirming that:
-
-- Parquet files can be read and converted to entities
-- BigInt values are properly serialized
-- MCP protocol communication works correctly
-- Data integrity is maintained during import
-- Entities are correctly stored in Neotoma

package/docs/reports/SCHEMA_ENHANCEMENT_STATUS.md

@@ -1,84 +0,0 @@
-# Schema Enhancement Status for Sample Parquet Imports
-
-**Date**: 2026-01-14  
-**Status**: ⚠️ **NOT ENHANCED** - Blocked by source diversity requirement
-
-## Summary
-
-After importing sample parquet files, **schemas were NOT automatically enhanced** due to the source diversity requirement in the auto-enhancement eligibility criteria.
-
-## Why Schemas Weren't Enhanced
-
-### Auto-Enhancement Eligibility Requirements
-
-For a field to be auto-enhanced, it must meet ALL of these criteria:
-
-1. ✅ **Frequency Threshold**: ≥ 3 occurrences (default)
-2. ✅ **Confidence**: ≥ 0.85 (85% minimum)
-3. ❌ **Source Diversity**: ≥ 2 different sources (REQUIRED)
-
-### The Problem
-
-When importing a single parquet file:
-- All data comes from **ONE source** (the parquet file)
-- Even if a field appears 50 times, it's still only from 1 source
-- The source diversity requirement (2+ sources) **blocks** auto-enhancement
-
-### Code Reference
-
-From `src/services/schema_recommendation.ts` (lines 155-163):
-
-```typescript
-// 6. Check source diversity (2+ different sources)
-const uniqueSources = new Set(fragments.map((f) => f.source_id)).size;
-if (uniqueSources < 2) {
-  return {
-    eligible: false,
-    confidence: confidenceResult.confidence,
-    reasoning: "Field appears in only one source (no diversity)",
-  };
-}
-```
-
-## What Happened
-
-1. ✅ **Raw fragments were stored**: Unknown fields were stored in `raw_fragments` table
-2. ✅ **Queue entries were created**: `queueAutoEnhancementCheck()` was called for each unknown field
-3. ✅ **Auto-enhancement processor ran**: Background processor checked eligibility
-4. ❌ **Enhancement blocked**: All fields failed the source diversity check
-
-## Test Results
-
-- **tasks_sample**: 50 entities imported, 0 schema enhancements
-- **crypto_transactions_sample**: 50 entities imported, 0 schema enhancements  
-- **purchases_sample**: 50 entities imported, 0 schema enhancements
-- **domains_sample**: 1 entity imported, 0 schema enhancements
-
-## Solutions
-
-### Option 1: Import Multiple Files (Recommended)
-Import multiple parquet files for the same entity type. Once a field appears in 2+ different sources, it will become eligible for auto-enhancement.
-
-### Option 2: Lower Source Diversity Requirement (For Testing)
-Temporarily lower the source diversity requirement to 1 for testing purposes:
-
-```typescript
-// In checkAutoEnhancementEligibility
-if (uniqueSources < 1) { // Changed from 2 to 1
-  // ...
-}
-```
-
-### Option 3: Manual Schema Enhancement
-Use the `update_schema_incremental` MCP tool to manually add fields to schemas.
-
-## Next Steps
-
-1. **Import more sample files** for the same entity types to meet source diversity
-2. **Wait for auto-enhancement processor** to run (every 30 seconds)
-3. **Check schema recommendations** after multiple imports
-4. **Verify enhancements** once source diversity is met
-
-## Conclusion
-
-The auto-enhancement system is working correctly, but the **source diversity requirement** is intentionally conservative to prevent one-off fields from being auto-enhanced. This is a **feature, not a bug** - it ensures only fields that appear consistently across multiple sources are promoted to schema fields.

package/docs/reports/SCHEMA_ENHANCEMENT_TEST_RESULTS.md

@@ -1,78 +0,0 @@
-# Schema Enhancement Test Results - Row-Based Diversity
-
-**Date**: 2026-01-14  
-**Status**: ⚠️ **NO UNKNOWN FIELDS TO ENHANCE**
-
-## Test Summary
-
-Re-tested schema enhancement after implementing row-based diversity fix. The test shows that **no unknown fields were stored**, which means there are no fields to enhance.
-
-## Test Results
-
-### Import Test
-- **File**: `tasks/tasks_sample.parquet` (50 rows)
-- **Result**: ✅ Import successful
-- **Entities created**: 50
-- **Unknown fields**: 0
-
-### Why No Enhancement?
-
-The import shows `unknown_fields_count: 0`, which means:
-1. **No schema exists** for `tasks_sample` entity type
-2. **All fields treated as valid**: When no schema is found, the code treats all fields as "valid" (not unknown)
-3. **Nothing stored in raw_fragments**: Since all fields are valid, nothing goes to raw_fragments
-4. **No auto-enhancement triggered**: No raw_fragments = nothing to enhance
-
-## Root Cause
-
-Looking at `src/server.ts` (lines 2798-2800):
-```typescript
-} else {
-  // No schema found - treat all as valid for now
-  validFields = fieldsToValidate;
-}
-```
-
-When there's no schema for an entity type:
-- All fields are treated as "valid"
-- Nothing is stored in `raw_fragments`
-- Auto-enhancement cannot work (no raw_fragments to analyze)
-
-## Solution Options
-
-### Option 1: Use Existing Schema (Recommended for Testing)
-Import files with entity types that have existing schemas (e.g., `task` instead of `tasks_sample`):
-- Some fields will be in the schema → stored in observations
-- Other fields will be unknown → stored in raw_fragments
-- Auto-enhancement can then work
-
-### Option 2: Create Initial Schema
-Manually create a minimal schema for the entity type with a few known fields:
-- Remaining fields will be unknown → stored in raw_fragments
-- Auto-enhancement can then work
-
-### Option 3: Change Behavior for No-Schema Case
-Modify the code to treat fields as "unknown" when no schema exists:
-- All fields go to raw_fragments
-- Auto-enhancement can work immediately
-- But this might be too aggressive
-
-## Row-Based Diversity Implementation
-
-The row-based diversity fix has been implemented and should work when there are unknown fields:
-
-1. ✅ **Source diversity check**: 2+ sources (original)
-2. ✅ **Row diversity via record_id**: 2+ rows for CSV/records
-3. ✅ **Row diversity via observations**: 2+ observations for parquet/structured files
-
-## Next Steps
-
-To properly test auto-enhancement:
-
-1. **Import with existing schema**: Use entity type that has a schema (e.g., modify parquet reader to use `task` instead of `tasks_sample`)
-2. **Or create test schema**: Manually create a schema with some fields, then import
-3. **Or change no-schema behavior**: Treat all fields as unknown when no schema exists
-
-## Conclusion
-
-The row-based diversity implementation is correct, but we need unknown fields to test it. The current behavior (treating all fields as valid when no schema exists) prevents auto-enhancement from working for new entity types.

package/docs/reports/TASK_RELOAD_BLOCKER.md

@@ -1,40 +0,0 @@
-# Task Reload Blocker
-
-**Date**: 2026-01-14  
-**Issue**: BigInt serialization error when storing tasks from parquet file
-
-## Current Status: ⚠️ BLOCKED
-
-### Error
-```
-Failed to process parquet file: Do not know how to serialize a BigInt
-```
-
-### Root Cause
-The Neotoma MCP server needs to be restarted to pick up the BigInt conversion fixes that were previously implemented.
-
-### Solution
-**Restart the Neotoma MCP server** to load the updated code that includes:
-- `convertBigIntValues()` function in `parquet_reader.ts`
-- BigInt replacers in `storeStructuredInternal()` and `buildTextResponse()`
-
-### File Path
-The correct absolute path for the tasks parquet file is:
-```
-/Users/markmhendrickson/Documents/data/tasks/tasks.parquet
-```
-
-### After Restart
-Once the server is restarted, the following command should work:
-```json
-{
-  "user_id": "00000000-0000-0000-0000-000000000000",
-  "file_path": "/Users/markmhendrickson/Documents/data/tasks/tasks.parquet",
-  "interpret": false
-}
-```
-
-## Verification
-- ✅ Parquet MCP server fixes applied (compact JSON, pagination)
-- ✅ File path determined: `/Users/markmhendrickson/Documents/data/tasks/tasks.parquet`
-- ⚠️ Neotoma MCP server needs restart for BigInt fixes