@prmichaelsen/remember-mcp 2.8.0 → 3.12.0

package/agent/design/soft-delete-system.md

@@ -0,0 +1,291 @@
+# Soft Delete System
+
+**Concept**: Implement soft delete with confirmation flow for memory deletion
+**Created**: 2026-02-25
+**Status**: Design Specification
+
+---
+
+## Overview
+
+Implement a soft delete system that prevents accidental data loss by requiring confirmation before deletion and marking memories as deleted rather than permanently removing them. This enables recovery and maintains data integrity while providing a safer deletion workflow.
+
+---
+
+## Problem Statement
+
+Current deletion implementation has critical issues:
+- **No confirmation flow**: Memories are instantly deleted without user confirmation
+- **No recovery**: Deleted memories cannot be recovered
+- **No visibility**: Cannot search or view deleted memories
+- **Data loss risk**: Accidental deletions result in permanent data loss
+
+---
+
+## Solution
+
+Implement a comprehensive soft delete system with:
+
+1. **Confirmation Flow**: Reuse existing token-based confirmation system
+2. **Soft Delete Schema**: Add `deleted_at`, `deleted_by`, `deletion_reason` fields
+3. **Default Filtering**: Exclude deleted memories from all searches by default
+4. **Explicit Override**: `deleted_filter` parameter to include/only show deleted memories
+5. **Relationship Handling**: Mark relationships as "orphaned" when memories deleted
+6. **Breaking Change**: Immediate migration to new behavior (no feature flags)
+
+---
+
+## Implementation
+
+### Schema Changes
+
+Add three new fields to Memory schema (both `Memory_{user_id}` and `Memory_public`):
+
+```typescript
+{
+  name: 'deleted_at',
+  dataType: 'date' as any,
+  description: 'Timestamp when memory was soft-deleted (null = not deleted)'
+},
+{
+  name: 'deleted_by',
+  dataType: 'text' as any,
+  description: 'User ID who deleted the memory'
+},
+{
+  name: 'deletion_reason',
+  dataType: 'text' as any,
+  description: 'Optional reason for deletion'
+}
+```
+
+**Note**: `deleted_at` is nullable. `null` or missing = not deleted.
+
+### Tool Modifications
+
+#### `remember_delete_memory` (Modified)
+
+**Old Behavior**: Immediately deletes memory from Weaviate
+
+**New Behavior**: Creates confirmation token and returns it
+
+```typescript
+// Input
+{
+  memory_id: string;
+  reason?: string;  // Optional deletion reason
+}
+
+// Output
+{
+  success: true;
+  token: string;
+  expires_at: string;
+  preview: {
+    memory_id: string;
+    content: string;
+    relationships_count: number;
+    will_orphan: string[];  // IDs of relationships that will be orphaned
+  }
+}
+```
+
+#### `remember_confirm` (Enhanced)
+
+Already handles confirmation. Will be enhanced to support `delete_memory` action:
+
+```typescript
+// When confirming deletion
+{
+  action: 'delete_memory',
+  payload: {
+    memory_id: string;
+    reason?: string;
+  }
+}
+
+// Execution:
+// 1. Update memory: deleted_at = now(), deleted_by = user_id, deletion_reason = reason
+// 2. Mark relationships as orphaned (implementation TBD)
+// 3. Return success
+```
+
+#### `remember_deny` (No Changes)
+
+Already handles denial of any pending action.
+
+### Search Tool Modifications
+
+All search tools get new `deleted_filter` parameter:
+
+```typescript
+deleted_filter?: 'exclude' | 'include' | 'only'
+// Default: 'exclude'
+```
+
+**Affected Tools**:
+- `remember_search_memory`
+- `remember_query_memory`
+- `remember_find_similar`
+- `remember_search_relationship`
+
+**Filter Implementation** (Weaviate query level):
+
+```typescript
+// deleted_filter: 'exclude' (default)
+.where(Filters.or(
+  collection.filter.byProperty('deleted_at').isNull(true),
+  // ... other filters
+))
+
+// deleted_filter: 'include'
+// No filter applied
+
+// deleted_filter: 'only'
+.where(
+  collection.filter.byProperty('deleted_at').isNull(false)
+)
+```
+
+### Relationship Handling
+
+**When memory is soft-deleted**:
+- Relationships remain in database
+- Relationships are marked as "orphaned" (implementation TBD - may use a flag or computed property)
+- Searching relationships excludes deleted memories by default
+
+**Creating relationships**:
+- Cannot create relationship with deleted memory
+- Error: "Cannot create relationship: memory {id} is deleted"
+
+**Updating deleted memories**:
+- Error: "Cannot update deleted memory"
+- User must restore first (future enhancement)
+
+---
+
+## Benefits
+
+1. **Safety**: Confirmation flow prevents accidental deletions
+2. **Recovery**: Soft delete enables future restoration feature
+3. **Audit Trail**: Track who deleted what and when
+4. **Flexibility**: `deleted_filter` parameter allows searching deleted memories when needed
+5. **Data Integrity**: Relationships preserved (orphaned but not lost)
+
+---
+
+## Trade-offs
+
+1. **Storage**: Deleted memories consume storage (acceptable trade-off)
+2. **Complexity**: Additional filtering logic in all search tools
+3. **Breaking Change**: Immediate behavior change (no backward compatibility)
+4. **No Restoration**: Phase 1 doesn't include restoration tool (future enhancement)
+5. **Shared Spaces**: Deleted published memories remain in spaces (future: `remember_retract`)
+
+---
+
+## Migration Strategy
+
+### Existing Data
+
+**Approach**: Treat missing `deleted_at` field as "not deleted"
+- No migration script needed
+- Existing memories implicitly have `deleted_at: null`
+- Weaviate `isNull(true)` filter handles this correctly
+
+### API Changes
+
+**Breaking Change**: Immediate migration
+- `remember_delete_memory` behavior changes immediately
+- All search tools add `deleted_filter` parameter (default: 'exclude')
+- No feature flags or gradual rollout
+- Version bump: v2.8.0 → v3.0.0 (major version)
+
+---
+
+## Future Enhancements
+
+### Phase 2: Recovery (Future)
+- `remember_restore_memory` tool
+- Confirmation flow for restoration
+- Automatic relationship restoration
+
+### Phase 3: Permanent Deletion (Not Planned)
+- No permanent deletion feature
+- Soft delete is sufficient
+- Storage cost is acceptable
+
+### Phase 4: Shared Space Integration (Future)
+- `remember_retract` tool to unpublish memories
+- Separate from deletion
+- Requires confirmation
+
+### Phase 5: Moderation (Future)
+- Space moderators can hide memories
+- Uses `moderation_flags` instead of `deleted_at`
+- Separate from user deletion
+
+---
+
+## Security Considerations
+
+### Access Control
+
+**Deleted memories**:
+- Only owner can search their deleted memories (`deleted_filter: 'include'` or `'only'`)
+- Other users cannot see deleted memories (filtered out)
+- Shared space memories remain visible until retracted (future enhancement)
+
+**Confirmation tokens**:
+- Reuse existing token service
+- 5-minute expiry
+- One-time use
+
+---
+
+## Testing Strategy
+
+1. **Unit Tests**:
+   - Schema field validation
+   - Filter logic (exclude/include/only)
+   - Confirmation flow
+   - Relationship orphaning
+
+2. **Integration Tests**:
+   - End-to-end deletion workflow
+   - Search with deleted_filter variations
+   - Relationship creation with deleted memories (should fail)
+   - Update deleted memory (should fail)
+
+3. **Edge Cases**:
+   - Delete already deleted memory
+   - Confirm expired token
+   - Search with invalid deleted_filter value
+
+---
+
+## Documentation Updates
+
+1. **README.md**: Update deletion workflow examples
+2. **CHANGELOG.md**: Document breaking change (v3.0.0)
+3. **Tool Descriptions**: Update all affected tools
+4. **Migration Guide**: Document behavior changes
+
+---
+
+## Open Questions
+
+**From Clarification Document**:
+
+1. **Agent Permission** (Item 3.2, Question 3):
+   > "Should agents be able to search deleted memories without explicit user permission, or should this require a special flag?"
+   > Response: "Clarification needed"
+
+   **Recommendation**: Allow agents to search deleted memories if user explicitly requests it in natural language (e.g., "search my deleted memories"). The `deleted_filter` parameter provides the mechanism. No additional permission system needed.
+
+---
+
+**Status**: Design Specification Complete
+**Recommendation**: Proceed with milestone and task creation
+**Priority**: Highest (per user feedback)
+**Estimated Effort**: 2-3 weeks

package/agent/design/trust-escalation-prevention.md

@@ -1,9 +1,16 @@
 # Trust Escalation Prevention
 
-**Concept**: Automatic trust reduction for repeated unauthorized access attempts  
-**Created**: 2026-02-11  
+**Concept**: Automatic trust reduction for repeated unauthorized access attempts
+**Created**: 2026-02-11
+**Updated**: 2026-02-27
 **Status**: Design Specification
 
+> **NOTE (2026-02-27)**: Trust escalation applies to ghost/persona conversations (confirmed in
+> clarification-3). When a user repeatedly asks a ghost about topics above their trust level,
+> the escalation system triggers: -0.1 trust per failed attempt, block after 3 attempts. The
+> ghost's system prompt warns progressively before backend escalation kicks in.
+> See `local.ghost-persona-system.md` for ghost-specific escalation behavior.
+
 ---
 
 ## Overview

package/agent/design/trust-system-implementation.md

@@ -1,8 +1,17 @@
 # Trust System Implementation - Prompt-Based Enforcement
 
-**Concept**: Trust enforcement through LLM prompting and validation  
-**Created**: 2026-02-11  
-**Status**: Design Specification
+**Concept**: Trust enforcement through LLM prompting and validation
+**Created**: 2026-02-11
+**Updated**: 2026-02-27
+**Status**: Superseded by Ghost/Persona Design
+
+> **NOTE (2026-02-27)**: This document describes **prompt-based enforcement** as the primary
+> trust mechanism. Per the ghost/persona design (`local.ghost-persona-system.md`), the primary
+> enforcement is now **query-level filtering** at the Weaviate layer — memories above the
+> accessor's trust threshold are never returned. Prompt-based enforcement remains as one of
+> three configurable modes (query, prompt, hybrid), but query is the default. The prompt-based
+> approach described here applies only when enforcement_mode is set to 'prompt' or 'hybrid'.
+> See `local.ghost-persona-system.md` for the current architecture.
 
 ---
 

package/agent/milestones/milestone-13-soft-delete-system.md

@@ -0,0 +1,306 @@
+# Milestone 13: Soft Delete System
+
+**Goal**: Implement safe deletion with confirmation flow and recovery capabilities
+**Duration**: 2-3 weeks
+**Dependencies**: M10 (Shared Spaces - for confirmation token service)
+**Status**: Not Started
+**Priority**: Highest
+
+---
+
+## Overview
+
+Implement a comprehensive soft delete system that prevents accidental data loss by requiring confirmation before deletion and marking memories as deleted rather than permanently removing them. This enables future recovery features and maintains data integrity while providing a safer deletion workflow.
+
+**Key Innovation**: Reuse existing confirmation token service from `remember_publish` for a consistent user experience across all destructive operations.
+
+---
+
+## Deliverables
+
+### 1. Schema Updates (3 new fields)
+- Add `deleted_at` field (date, nullable) - Timestamp of deletion
+- Add `deleted_by` field (text) - User ID who deleted the memory
+- Add `deletion_reason` field (text) - Optional reason for deletion
+
+### 2. Confirmation Flow
+- Modify `remember_delete_memory` to create confirmation token
+- Enhance `remember_confirm` to handle `delete_memory` action
+- Include deletion preview (content, relationship count, orphaned relationships)
+
+### 3. Search Tool Updates (4 tools)
+- Add `deleted_filter` parameter to `remember_search_memory`
+- Add `deleted_filter` parameter to `remember_query_memory`
+- Add `deleted_filter` parameter to `remember_find_similar`
+- Add `deleted_filter` parameter to `remember_search_relationship`
+
+### 4. Relationship Handling
+- Mark relationships as "orphaned" when memory deleted
+- Prevent creating relationships with deleted memories
+- Exclude deleted memories from relationship searches by default
+
+### 5. Error Handling
+- Return error when updating deleted memory
+- Return error when creating relationship with deleted memory
+- Clear error messages with actionable guidance
+
+### 6. Documentation Updates
+- Update README with new deletion workflow
+- Document breaking changes in CHANGELOG (v3.0.0)
+- Update tool descriptions
+- Add migration guide
+
+### 7. Testing
+- Unit tests for schema fields
+- Unit tests for confirmation flow
+- Unit tests for deleted_filter parameter
+- Unit tests for relationship orphaning
+- Integration tests for full deletion workflow
+
+---
+
+## Success Criteria
+
+- [ ] Schema has 3 new fields: `deleted_at`, `deleted_by`, `deletion_reason`
+- [ ] `remember_delete_memory` creates confirmation token (not immediate delete)
+- [ ] `remember_confirm` handles `delete_memory` action
+- [ ] Deletion preview shows content and relationship impact
+- [ ] All 4 search tools have `deleted_filter` parameter
+- [ ] Default behavior excludes deleted memories from all searches
+- [ ] `deleted_filter: "include"` shows all memories (deleted + active)
+- [ ] `deleted_filter: "only"` shows only deleted memories
+- [ ] Relationships marked as orphaned when memory deleted
+- [ ] Cannot create relationship with deleted memory (returns error)
+- [ ] Cannot update deleted memory (returns error)
+- [ ] All existing tests still passing
+- [ ] New soft delete tests passing
+- [ ] TypeScript compiles without errors
+- [ ] Build successful
+- [ ] Documentation updated with examples
+- [ ] CHANGELOG documents breaking changes
+- [ ] Version bumped to v3.0.0 (major version)
+
+---
+
+## Key Files to Create/Modify
+
+```
+src/
+├── weaviate/
+│   ├── schema.ts                    # Add 3 new fields to Memory schema
+│   └── space-schema.ts              # Add 3 new fields to Memory_public
+├── tools/
+│   ├── delete-memory.ts             # Modify to create confirmation token
+│   ├── confirm.ts                   # Enhance to handle delete_memory action
+│   ├── search-memory.ts             # Add deleted_filter parameter
+│   ├── query-memory.ts              # Add deleted_filter parameter
+│   ├── find-similar.ts              # Add deleted_filter parameter
+│   ├── search-relationship.ts       # Add deleted_filter parameter
+│   ├── create-relationship.ts       # Prevent relationships with deleted memories
+│   └── update-memory.ts             # Prevent updating deleted memories
+├── utils/
+│   └── weaviate-filters.ts          # Add deleted_filter helper functions
+└── types/
+    └── memory.ts                    # Add DeletedFilter type
+
+tests/
+└── unit/
+    ├── schema.test.ts               # Test new fields
+    ├── space-schema.test.ts         # Test new fields in public collection
+    ├── delete-memory.test.ts        # Test confirmation flow
+    ├── confirm.test.ts              # Test delete_memory action
+    ├── search-memory.test.ts        # Test deleted_filter
+    ├── query-memory.test.ts         # Test deleted_filter
+    ├── find-similar.test.ts         # Test deleted_filter
+    └── search-relationship.test.ts  # Test deleted_filter
+
+agent/
+└── design/
+    └── soft-delete-system.md        # ✅ Already created
+```
+
+---
+
+## Implementation Tasks
+
+See individual task documents:
+- Task 70: Add Soft Delete Schema Fields
+- Task 71: Implement Delete Confirmation Flow
+- Task 72: Add deleted_filter to Search Tools
+- Task 73: Update Relationship Handling for Deleted Memories
+- Task 74: Add Unit Tests for Soft Delete
+- Task 75: Update Documentation and CHANGELOG
+
+---
+
+## Architecture Notes
+
+### Soft Delete vs Hard Delete
+
+**Soft Delete** (Implemented):
+- Memory remains in Weaviate
+- `deleted_at` field set to current timestamp
+- Filtered out by default
+- Can be searched with `deleted_filter: "include"` or `"only"`
+- Enables future restoration feature
+
+**Hard Delete** (Not Implemented):
+- Memory permanently removed from Weaviate
+- Cannot be recovered
+- Not planned for this milestone
+
+### Confirmation Token Reuse
+
+**Existing Service**: `ConfirmationTokenService` (from M10)
+- Already handles token generation, validation, expiry
+- Stores tokens in Firestore: `users/{user_id}/requests/{request_id}`
+- 5-minute expiry
+- One-time use
+
+**New Action**: `delete_memory`
+```typescript
+{
+  action: 'delete_memory',
+  payload: {
+    memory_id: string;
+    reason?: string;
+  }
+}
+```
+
+### Filter Implementation
+
+**Weaviate Query Level**:
+```typescript
+// deleted_filter: 'exclude' (default)
+collection.query.hybrid(query)
+  .where(Filters.or(
+    collection.filter.byProperty('deleted_at').isNull(true),
+    // ... other filters
+  ))
+
+// deleted_filter: 'include'
+// No deleted_at filter applied
+
+// deleted_filter: 'only'
+collection.query.hybrid(query)
+  .where(
+    collection.filter.byProperty('deleted_at').isNull(false)
+  )
+```
+
+### Relationship Orphaning
+
+**Implementation Options**:
+1. **Computed Property** (Recommended): Check if any memory in relationship is deleted
+2. **Flag Field**: Add `is_orphaned` boolean to relationship
+3. **Status Field**: Add `status: "active" | "orphaned"` to relationship
+
+**Recommendation**: Use computed property (Option 1) - no schema changes needed, always accurate.
+
+### Breaking Changes
+
+**Version**: v2.8.0 → v3.0.0 (major version bump)
+
+**Changes**:
+1. `remember_delete_memory` behavior changes (immediate → confirmation)
+2. All search tools add `deleted_filter` parameter
+3. Default search behavior excludes deleted memories
+
+**Migration Path**:
+- No code changes required for users
+- Behavior change is immediate
+- Existing memories implicitly have `deleted_at: null`
+
+---
+
+## Testing Strategy
+
+1. **Unit Tests**: Schema fields, filter logic, confirmation flow
+2. **Integration Tests**: End-to-end deletion workflow
+3. **Edge Cases**: Delete deleted memory, expired tokens, invalid filters
+4. **Performance Tests**: Search with large numbers of deleted memories
+
+---
+
+## Future Phases
+
+**Phase 2: Recovery** (Future - Not in M13):
+- `remember_restore_memory` tool
+- Confirmation flow for restoration
+- Automatic relationship restoration
+- Restore to original state
+
+**Phase 3: Shared Space Integration** (Future):
+- `remember_retract` tool to unpublish memories
+- Separate from deletion
+- Requires confirmation
+- Removes from shared spaces
+
+**Phase 4: Moderation** (Future):
+- Space moderators can hide memories
+- Uses `moderation_flags` instead of `deleted_at`
+- Separate from user deletion
+- Per-space moderation
+
+---
+
+## Breaking Changes
+
+**API Changes**:
+- `remember_delete_memory` now returns confirmation token (not immediate delete)
+- All search tools add `deleted_filter` parameter (default: 'exclude')
+- Default search behavior changes (excludes deleted memories)
+
+**Data Changes**:
+- None (existing memories implicitly have `deleted_at: null`)
+
+**Behavior Changes**:
+- Deletion requires confirmation (two-step process)
+- Deleted memories remain in database (soft delete)
+- Cannot update or create relationships with deleted memories
+
+---
+
+## Security Considerations
+
+**Access Control**:
+- Only owner can delete their memories
+- Only owner can search their deleted memories
+- Confirmation tokens are user-specific
+- Deleted memories not visible to other users
+
+**Data Retention**:
+- Deleted memories remain in database indefinitely
+- No automatic purge policy
+- Storage cost is acceptable trade-off for safety
+
+---
+
+## Documentation Requirements
+
+1. **README.md**: 
+   - Update deletion workflow section
+   - Add examples of `deleted_filter` usage
+   - Document confirmation flow
+
+2. **CHANGELOG.md**:
+   - Document breaking changes (v3.0.0)
+   - List all modified tools
+   - Provide migration guidance
+
+3. **Tool Descriptions**:
+   - Update `remember_delete_memory` description
+   - Add `deleted_filter` parameter to 4 search tools
+   - Update error messages
+
+4. **Migration Guide**:
+   - Explain behavior changes
+   - Provide code examples
+   - Document version bump rationale
+
+---
+
+**Next Milestone**: M5 - Template System (deferred until M13 complete)
+**Blockers**: None (builds on M10 confirmation token service)
+**Priority**: Highest (per user feedback)