@prmichaelsen/remember-mcp 2.0.5 → 2.2.1

package/agent/tasks/task-26-migrate-parm-schema-to-remember-schema.md

@@ -0,0 +1,331 @@
+# Task 26: Migrate PARM Schema to Remember Schema
+
+**Milestone**: M8 - Testing & Quality  
+**Estimated Time**: 4 hours  
+**Dependencies**: None  
+**Status**: Not Started  
+**Priority**: Medium
+
+---
+
+## Objective
+
+Migrate existing PARM (Personal Agent Relationship Manager) memories from the old schema to the new remember-mcp schema. This involves mapping old fields to new fields, handling missing fields, and ensuring data integrity.
+
+## Problem Statement
+
+Existing memories in Weaviate were created with the PARM schema which has different field names and structure than the current remember-mcp schema:
+
+### Old PARM Schema (Observed)
+```yaml
+author: string (UUID)
+content: string (long text)
+contentType: string (e.g., "note")
+createdAt: datetime
+description: string
+filePath: string (firestore:// URL)
+project: string (e.g., "parm-migration")
+status: string (e.g., "migrated")
+tags: string[] (e.g., ["love", "relationships", "poetry"])
+title: string
+updatedAt: datetime
+```
+
+### New Remember-MCP Schema (Current)
+```yaml
+# Core Identity
+user_id: string
+doc_type: string ("memory" or "relationship")
+
+# Content
+content: string
+title: string
+summary: string
+type: string (ContentType - 45 types)
+
+# Scoring
+weight: number (0-1)
+trust: number (0-1)
+confidence: number (0-1)
+
+# Location
+location_gps_lat: number
+location_gps_lng: number
+location_address: string
+location_city: string
+location_country: string
+
+# Locale
+locale_language: string
+locale_timezone: string
+
+# Context
+context_conversation_id: string
+context_summary: string
+context_timestamp: date
+
+# Relationships
+relationships: string[] (relationship IDs)
+
+# Access Tracking
+access_count: number
+last_accessed_at: date
+
+# Metadata
+tags: string[]
+references: string[] (URLs)
+created_at: date
+updated_at: date
+version: number
+
+# Template
+template_id: string
+```
+
+## Field Mapping
+
+### Direct Mappings
+| PARM Field | Remember Field | Notes |
+|------------|----------------|-------|
+| `content` | `content` | Direct copy |
+| `title` | `title` | Direct copy |
+| `tags` | `tags` | Direct copy |
+| `createdAt` | `created_at` | Rename |
+| `updatedAt` | `updated_at` | Rename |
+| `contentType` | `type` | Map to valid ContentType |
+
+### Derived Mappings
+| PARM Field | Remember Field | Transformation |
+|------------|----------------|----------------|
+| `author` | `user_id` | Extract user ID from author UUID |
+| `description` | `summary` | Use as summary |
+| `title` | `summary` | If no description, use title |
+| N/A | `doc_type` | Set to "memory" |
+| N/A | `version` | Set to 1 |
+| N/A | `weight` | Default to 0.5 |
+| N/A | `trust` | Default to 0.5 |
+| N/A | `confidence` | Default to 1.0 |
+
+### Fields to Drop
+- `filePath` - Firestore reference, not needed in Weaviate
+- `project` - Migration metadata, not needed
+- `status` - Migration metadata, not needed
+
+### New Fields (Defaults)
+- `relationships: []` - Empty array
+- `references: []` - Empty array
+- `access_count: 0` - Default 0
+- `last_accessed_at: <created_at>` - Set to creation time
+- `access_frequency: 0` - Default 0
+- `base_weight: 0.5` - Same as weight
+- `computed_weight: 0.5` - Same as weight
+- `location: { gps: null, address: null, source: 'unavailable', confidence: 0, is_approximate: true }`
+- `context: { timestamp: <created_at>, source: { type: 'migration', platform: 'parm' }, summary: 'Migrated from PARM' }`
+- `template_id: null` - No template
+- `structured_content: null` - No structured content
+
+## ContentType Mapping
+
+Map old `contentType` values to new `type` values:
+
+| Old contentType | New type | Reasoning |
+|-----------------|----------|-----------|
+| `note` | `note` | Direct match |
+| `person` | `person` | Direct match |
+| `event` | `event` | Direct match |
+| `bookmark` | `bookmark` | Direct match |
+| `recipe` | `recipe` | Direct match |
+| `meeting` | `meeting` | Direct match |
+| `project` | `project` | Direct match |
+| (unknown) | `note` | Default fallback |
+
+**Special Case**: If `tags` contains "poetry", consider setting `type: "note"` but keep the poetry tag.
+
+## Steps
+
+### 1. Analyze Existing Data
+- [ ] Query Weaviate for all memories in user's collection
+- [ ] Count total memories to migrate
+- [ ] Identify unique `contentType` values
+- [ ] Identify unique `author` values (should be one user)
+- [ ] Check for any unexpected fields
+
+### 2. Create Migration Script
+- [ ] Create `scripts/migrate-parm-to-remember.ts`
+- [ ] Implement field mapping logic
+- [ ] Implement contentType to type conversion
+- [ ] Add validation for required fields
+- [ ] Add dry-run mode (preview without writing)
+
+### 3. Implement Migration Logic
+```typescript
+interface PARMMemory {
+  author: string;
+  content: string;
+  contentType: string;
+  createdAt: string;
+  description?: string;
+  filePath?: string;
+  project?: string;
+  status?: string;
+  tags: string[];
+  title: string;
+  updatedAt: string;
+}
+
+interface RememberMemory {
+  // All remember-mcp fields
+}
+
+function migratePARMToRemember(
+  parmMemory: PARMMemory,
+  userId: string
+): RememberMemory {
+  return {
+    // Core identity
+    user_id: userId,
+    doc_type: 'memory',
+    
+    // Content
+    content: parmMemory.content,
+    title: parmMemory.title,
+    summary: parmMemory.description || parmMemory.title,
+    type: mapContentType(parmMemory.contentType),
+    
+    // Scoring (defaults)
+    weight: 0.5,
+    trust: 0.5,
+    confidence: 1.0,
+    
+    // Location (null)
+    location: {
+      gps: null,
+      address: null,
+      source: 'unavailable',
+      confidence: 0,
+      is_approximate: true,
+    },
+    
+    // Context
+    context: {
+      timestamp: parmMemory.createdAt,
+      source: { type: 'migration', platform: 'parm' },
+      summary: 'Migrated from PARM',
+    },
+    
+    // Relationships
+    relationships: [],
+    
+    // Access tracking
+    access_count: 0,
+    last_accessed_at: parmMemory.createdAt,
+    access_frequency: 0,
+    
+    // Metadata
+    tags: parmMemory.tags,
+    references: [],
+    created_at: parmMemory.createdAt,
+    updated_at: parmMemory.updatedAt,
+    version: 1,
+    
+    // Template
+    template_id: null,
+    structured_content: null,
+    
+    // Computed weight
+    base_weight: 0.5,
+    computed_weight: 0.5,
+  };
+}
+
+function mapContentType(oldType: string): string {
+  const mapping: Record<string, string> = {
+    'note': 'note',
+    'person': 'person',
+    'event': 'event',
+    'bookmark': 'bookmark',
+    'recipe': 'recipe',
+    'meeting': 'meeting',
+    'project': 'project',
+  };
+  
+  return mapping[oldType] || 'note';
+}
+```
+
+### 4. Add Safety Checks
+- [ ] Verify user_id matches expected user
+- [ ] Validate all required fields are present
+- [ ] Check for duplicate IDs
+- [ ] Verify contentType mapping is valid
+- [ ] Ensure no data loss (all old fields accounted for)
+
+### 5. Create Backup Strategy
+- [ ] Export all existing memories to JSON file
+- [ ] Store backup with timestamp
+- [ ] Document rollback procedure
+
+### 6. Test Migration
+- [ ] Test with 1 memory (dry-run)
+- [ ] Test with 10 memories (dry-run)
+- [ ] Verify migrated data structure
+- [ ] Test search on migrated memories
+- [ ] Verify tags are preserved
+
+### 7. Execute Migration
+- [ ] Run full migration (dry-run first)
+- [ ] Review migration report
+- [ ] Execute actual migration
+- [ ] Verify all memories migrated
+- [ ] Test search functionality
+- [ ] Verify no data loss
+
+### 8. Post-Migration Validation
+- [ ] Count memories before and after
+- [ ] Spot-check 10 random memories
+- [ ] Search for poems by poetry tag
+- [ ] Verify all contentTypes mapped correctly
+- [ ] Check that old fields are gone
+- [ ] Verify new fields have correct defaults
+
+## Verification
+
+- [ ] All PARM memories successfully migrated
+- [ ] No data loss (content, title, tags preserved)
+- [ ] All memories have valid remember-mcp schema
+- [ ] Search works correctly on migrated memories
+- [ ] Poems findable via poetry tag
+- [ ] No old PARM fields remain
+- [ ] Backup created and verified
+- [ ] Migration report generated
+
+## Files to Create
+
+- `scripts/migrate-parm-to-remember.ts` - Migration script
+- `scripts/backup-memories.ts` - Backup utility
+- `scripts/validate-migration.ts` - Validation script
+- `agent/design/parm-migration-report.md` - Migration report template
+
+## Expected Outcome
+
+- All existing PARM memories converted to remember-mcp schema
+- No data loss
+- Improved searchability with proper schema
+- Poems findable via tags
+- Clean, consistent data structure
+- Documented migration process for future reference
+
+## Rollback Plan
+
+If migration fails:
+1. Stop migration immediately
+2. Restore from backup JSON file
+3. Re-import using Weaviate batch import
+4. Verify restoration
+5. Debug migration script
+6. Retry with fixes
+
+---
+
+**Next Task**: Task 27 - Add Poetry Content Type  
+**Blockers**: Need user confirmation before executing migration

package/dist/constants/content-types.d.ts

@@ -30,7 +30,7 @@ export declare const CONTENT_TYPE_CATEGORIES: {
     readonly communication: readonly ["email", "conversation", "meeting", "person"];
     readonly content: readonly ["article", "webpage", "social", "presentation", "spreadsheet", "pdf"];
     readonly media: readonly ["image", "video", "audio", "transcript"];
-    readonly creative: readonly ["screenplay", "recipe", "idea", "quote"];
+    readonly creative: readonly ["song", "screenplay", "recipe", "idea", "quote"];
     readonly personal: readonly ["journal", "memory", "event"];
     readonly organizational: readonly ["bookmark", "form", "location"];
     readonly business: readonly ["invoice", "contract"];

package/dist/server-factory.js

@@ -839,6 +839,7 @@ var CONTENT_TYPES = [
   "spreadsheet",
   "pdf",
   // Creative
+  "song",
   "screenplay",
   "recipe",
   "idea",
@@ -1024,6 +1025,13 @@ var CONTENT_TYPE_METADATA = {
     common_fields: ["pages", "file_size"]
   },
   // Creative
+  song: {
+    name: "song",
+    category: "creative",
+    description: "Music tracks and songs",
+    examples: ["Songs", "Music tracks", "Albums", "Playlists"],
+    common_fields: ["artist", "album", "genre", "duration", "release_date", "url"]
+  },
   screenplay: {
     name: "screenplay",
     category: "creative",
@@ -1425,6 +1433,8 @@ var searchMemoryTool = {
   name: "remember_search_memory",
   description: `Search memories AND relationships using hybrid semantic and keyword search.
   
+  **BEST FOR**: Precise searches with specific keywords or exact phrases. Good for finding specific items when you know what you're looking for.
+  
   By default, searches BOTH memories and relationships to provide comprehensive results.
   Relationships contain valuable context in their observations.
   
@@ -1439,6 +1449,15 @@ var searchMemoryTool = {
   - "Find memories about camping trips" \u2192 returns memories + relationships about camping
   - "Search for recipes I saved" \u2192 returns recipe memories + related relationships
   - "Show me notes from last week" \u2192 returns notes + any relationships created that week
+  
+  **AGENT GUIDANCE**:
+  - If search results are too narrow or miss relevant content, try remember_query_memory instead - it uses pure semantic search which is better for broader, concept-based queries. You can inform the user: "I didn't find what you're looking for with keyword search. Let me try a broader semantic search using the query tool."
+  - **CRITICAL**: If no results are returned, DO NOT make up or fabricate memories. Only report what was actually found. Tell the user honestly that no matching memories were found and suggest they:
+    * Create a new memory with the information they're looking for
+    * Try the other search tool (remember_query_memory for broader semantic search)
+    * Remove or relax filters if they applied any
+    * Increase the limit parameter to see more results
+    * Try different search terms or keywords
   `,
   inputSchema: {
     type: "object",
@@ -1979,6 +1998,8 @@ var queryMemoryTool = {
   name: "remember_query_memory",
   description: `Query memories using natural language for RAG (Retrieval-Augmented Generation).
   
+  **BEST FOR**: Broad, concept-based searches and natural language questions. Uses pure semantic search to find memories by meaning, not just keywords. Great for exploratory queries and when you're not sure of exact terms.
+  
   This tool is optimized for LLM context retrieval. It returns relevant memories
   with their full content and context, formatted for easy consumption by LLMs.
   
@@ -1987,12 +2008,23 @@ var queryMemoryTool = {
   - Provide context for conversations
   - Retrieve information for decision-making
   - Build responses using past knowledge
+  - Find memories by concept rather than exact keywords
   
   Examples:
   - "What do I know about camping?"
   - "Tell me about the recipes I've saved"
   - "What meetings did I have last week?"
   - "What are my project goals?"
+  
+  **AGENT GUIDANCE**:
+  - If query results are too broad or include irrelevant content, try remember_search_memory instead - it uses hybrid search with keyword matching which is better for precise, specific searches. You can inform the user: "The results were too broad. Let me try a more precise keyword search using the search tool."
+  - **CRITICAL**: If no results are returned, DO NOT make up or fabricate memories. Only report what was actually found. Tell the user honestly that no matching memories were found and suggest they:
+    * Create a new memory with the information they're looking for
+    * Try the other search tool (remember_search_memory for more precise keyword-based search)
+    * Remove or relax filters if they applied any
+    * Increase the limit parameter to see more results
+    * Lower the min_relevance threshold to include less relevant matches
+    * Try rephrasing the query with different terms
   `,
   inputSchema: {
     type: "object",

package/dist/server.js

@@ -881,6 +881,7 @@ var CONTENT_TYPES = [
   "spreadsheet",
   "pdf",
   // Creative
+  "song",
   "screenplay",
   "recipe",
   "idea",
@@ -1066,6 +1067,13 @@ var CONTENT_TYPE_METADATA = {
     common_fields: ["pages", "file_size"]
   },
   // Creative
+  song: {
+    name: "song",
+    category: "creative",
+    description: "Music tracks and songs",
+    examples: ["Songs", "Music tracks", "Albums", "Playlists"],
+    common_fields: ["artist", "album", "genre", "duration", "release_date", "url"]
+  },
   screenplay: {
     name: "screenplay",
     category: "creative",
@@ -1467,6 +1475,8 @@ var searchMemoryTool = {
   name: "remember_search_memory",
   description: `Search memories AND relationships using hybrid semantic and keyword search.
   
+  **BEST FOR**: Precise searches with specific keywords or exact phrases. Good for finding specific items when you know what you're looking for.
+  
   By default, searches BOTH memories and relationships to provide comprehensive results.
   Relationships contain valuable context in their observations.
   
@@ -1481,6 +1491,15 @@ var searchMemoryTool = {
   - "Find memories about camping trips" \u2192 returns memories + relationships about camping
   - "Search for recipes I saved" \u2192 returns recipe memories + related relationships
   - "Show me notes from last week" \u2192 returns notes + any relationships created that week
+  
+  **AGENT GUIDANCE**:
+  - If search results are too narrow or miss relevant content, try remember_query_memory instead - it uses pure semantic search which is better for broader, concept-based queries. You can inform the user: "I didn't find what you're looking for with keyword search. Let me try a broader semantic search using the query tool."
+  - **CRITICAL**: If no results are returned, DO NOT make up or fabricate memories. Only report what was actually found. Tell the user honestly that no matching memories were found and suggest they:
+    * Create a new memory with the information they're looking for
+    * Try the other search tool (remember_query_memory for broader semantic search)
+    * Remove or relax filters if they applied any
+    * Increase the limit parameter to see more results
+    * Try different search terms or keywords
   `,
   inputSchema: {
     type: "object",
@@ -2021,6 +2040,8 @@ var queryMemoryTool = {
   name: "remember_query_memory",
   description: `Query memories using natural language for RAG (Retrieval-Augmented Generation).
   
+  **BEST FOR**: Broad, concept-based searches and natural language questions. Uses pure semantic search to find memories by meaning, not just keywords. Great for exploratory queries and when you're not sure of exact terms.
+  
   This tool is optimized for LLM context retrieval. It returns relevant memories
   with their full content and context, formatted for easy consumption by LLMs.
   
@@ -2029,12 +2050,23 @@ var queryMemoryTool = {
   - Provide context for conversations
   - Retrieve information for decision-making
   - Build responses using past knowledge
+  - Find memories by concept rather than exact keywords
   
   Examples:
   - "What do I know about camping?"
   - "Tell me about the recipes I've saved"
   - "What meetings did I have last week?"
   - "What are my project goals?"
+  
+  **AGENT GUIDANCE**:
+  - If query results are too broad or include irrelevant content, try remember_search_memory instead - it uses hybrid search with keyword matching which is better for precise, specific searches. You can inform the user: "The results were too broad. Let me try a more precise keyword search using the search tool."
+  - **CRITICAL**: If no results are returned, DO NOT make up or fabricate memories. Only report what was actually found. Tell the user honestly that no matching memories were found and suggest they:
+    * Create a new memory with the information they're looking for
+    * Try the other search tool (remember_search_memory for more precise keyword-based search)
+    * Remove or relax filters if they applied any
+    * Increase the limit parameter to see more results
+    * Lower the min_relevance threshold to include less relevant matches
+    * Try rephrasing the query with different terms
   `,
   inputSchema: {
     type: "object",

package/dist/types/memory.d.ts

@@ -5,7 +5,7 @@
  * Content types for memories
  * Based on agent/design/content-types-expansion.md
  */
-export type ContentType = 'code' | 'note' | 'documentation' | 'reference' | 'todo' | 'checklist' | 'project' | 'goal' | 'habit' | 'email' | 'conversation' | 'meeting' | 'person' | 'article' | 'webpage' | 'social' | 'image' | 'video' | 'audio' | 'transcript' | 'presentation' | 'spreadsheet' | 'pdf' | 'screenplay' | 'recipe' | 'idea' | 'quote' | 'journal' | 'memory' | 'event' | 'bookmark' | 'form' | 'location' | 'invoice' | 'contract' | 'system' | 'action' | 'audit' | 'history';
+export type ContentType = 'code' | 'note' | 'documentation' | 'reference' | 'todo' | 'checklist' | 'project' | 'goal' | 'habit' | 'email' | 'conversation' | 'meeting' | 'person' | 'article' | 'webpage' | 'social' | 'image' | 'video' | 'audio' | 'song' | 'transcript' | 'presentation' | 'spreadsheet' | 'pdf' | 'screenplay' | 'recipe' | 'idea' | 'quote' | 'journal' | 'memory' | 'event' | 'bookmark' | 'form' | 'location' | 'invoice' | 'contract' | 'system' | 'action' | 'audit' | 'history';
 /**
  * GPS coordinates
  */

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@prmichaelsen/remember-mcp",
-  "version": "2.0.5",
+  "version": "2.2.1",
   "description": "Multi-tenant memory system MCP server with vector search and relationships",
   "main": "dist/server.js",
   "type": "module",

package/src/constants/content-types.ts

@@ -37,6 +37,7 @@ export const CONTENT_TYPES: readonly ContentType[] = [
   'spreadsheet',
   'pdf',
   // Creative
+  'song',
   'screenplay',
   'recipe',
   'idea',
@@ -241,6 +242,13 @@ export const CONTENT_TYPE_METADATA: Record<ContentType, ContentTypeMetadata> = {
   },
 
   // Creative
+  song: {
+    name: 'song',
+    category: 'creative',
+    description: 'Music tracks and songs',
+    examples: ['Songs', 'Music tracks', 'Albums', 'Playlists'],
+    common_fields: ['artist', 'album', 'genre', 'duration', 'release_date', 'url'],
+  },
   screenplay: {
     name: 'screenplay',
     category: 'creative',
@@ -371,7 +379,7 @@ export const CONTENT_TYPE_CATEGORIES = {
   communication: ['email', 'conversation', 'meeting', 'person'],
   content: ['article', 'webpage', 'social', 'presentation', 'spreadsheet', 'pdf'],
   media: ['image', 'video', 'audio', 'transcript'],
-  creative: ['screenplay', 'recipe', 'idea', 'quote'],
+  creative: ['song', 'screenplay', 'recipe', 'idea', 'quote'],
   personal: ['journal', 'memory', 'event'],
   organizational: ['bookmark', 'form', 'location'],
   business: ['invoice', 'contract'],

package/src/tools/query-memory.ts

@@ -16,6 +16,8 @@ export const queryMemoryTool = {
   name: 'remember_query_memory',
   description: `Query memories using natural language for RAG (Retrieval-Augmented Generation).
   
+  **BEST FOR**: Broad, concept-based searches and natural language questions. Uses pure semantic search to find memories by meaning, not just keywords. Great for exploratory queries and when you're not sure of exact terms.
+  
   This tool is optimized for LLM context retrieval. It returns relevant memories
   with their full content and context, formatted for easy consumption by LLMs.
   
@@ -24,12 +26,23 @@ export const queryMemoryTool = {
   - Provide context for conversations
   - Retrieve information for decision-making
   - Build responses using past knowledge
+  - Find memories by concept rather than exact keywords
   
   Examples:
   - "What do I know about camping?"
   - "Tell me about the recipes I've saved"
   - "What meetings did I have last week?"
   - "What are my project goals?"
+  
+  **AGENT GUIDANCE**:
+  - If query results are too broad or include irrelevant content, try remember_search_memory instead - it uses hybrid search with keyword matching which is better for precise, specific searches. You can inform the user: "The results were too broad. Let me try a more precise keyword search using the search tool."
+  - **CRITICAL**: If no results are returned, DO NOT make up or fabricate memories. Only report what was actually found. Tell the user honestly that no matching memories were found and suggest they:
+    * Create a new memory with the information they're looking for
+    * Try the other search tool (remember_search_memory for more precise keyword-based search)
+    * Remove or relax filters if they applied any
+    * Increase the limit parameter to see more results
+    * Lower the min_relevance threshold to include less relevant matches
+    * Try rephrasing the query with different terms
   `,
   inputSchema: {
     type: 'object',

package/src/tools/search-memory.ts

@@ -16,6 +16,8 @@ export const searchMemoryTool = {
   name: 'remember_search_memory',
   description: `Search memories AND relationships using hybrid semantic and keyword search.
   
+  **BEST FOR**: Precise searches with specific keywords or exact phrases. Good for finding specific items when you know what you're looking for.
+  
   By default, searches BOTH memories and relationships to provide comprehensive results.
   Relationships contain valuable context in their observations.
   
@@ -30,6 +32,15 @@ export const searchMemoryTool = {
   - "Find memories about camping trips" → returns memories + relationships about camping
   - "Search for recipes I saved" → returns recipe memories + related relationships
   - "Show me notes from last week" → returns notes + any relationships created that week
+  
+  **AGENT GUIDANCE**:
+  - If search results are too narrow or miss relevant content, try remember_query_memory instead - it uses pure semantic search which is better for broader, concept-based queries. You can inform the user: "I didn't find what you're looking for with keyword search. Let me try a broader semantic search using the query tool."
+  - **CRITICAL**: If no results are returned, DO NOT make up or fabricate memories. Only report what was actually found. Tell the user honestly that no matching memories were found and suggest they:
+    * Create a new memory with the information they're looking for
+    * Try the other search tool (remember_query_memory for broader semantic search)
+    * Remove or relax filters if they applied any
+    * Increase the limit parameter to see more results
+    * Try different search terms or keywords
   `,
   inputSchema: {
     type: 'object',

package/src/types/memory.ts

@@ -30,6 +30,7 @@ export type ContentType =
   | 'image'
   | 'video'
   | 'audio'
+  | 'song'
   | 'transcript'
   | 'presentation'
   | 'spreadsheet'