memorylink 1.0.0

package/V1.4_BUILD_PLAN_IMPLEMENTATION.md

@@ -0,0 +1,698 @@
+# MemoryLink v1.4 - Build Plan & Implementation Guide
+
+**Date**: 2025-12-19  
+**Version Target**: 1.4.0  
+**Status**: Ready for Implementation  
+**Based On**: Cursor-facing v1.4 Build Plan
+
+---
+
+## 📋 EXECUTIVE SUMMARY
+
+**v1.4 = Agent-Ready Repo Memory + Safety Layer (Buddy-Check)**
+
+MemoryLink v1.4 introduces **Buddy-Check**, a preflight/postflight safety system that validates memory records before and after AI agent operations. This version focuses on:
+
+1. **Schema Evolution**: Enhanced MemoryRecord with trust levels, sensitivity, status, and provenance
+2. **Buddy-Check Engine**: Preflight/postflight validation with GREEN/YELLOW/RED status
+3. **VS Code Integration**: Full extension with status indicators and blocking
+4. **Developer/Enterprise Modes**: Flexible enforcement levels
+5. **Migration Path**: Seamless upgrade from v1.3 to v1.4
+
+---
+
+## 🎯 WHAT v1.4 IS
+
+### Core Concept
+- **Storage**: `.memorylink/` JSONL files (canonical, Git-native) - **UNCHANGED**
+- **Guardrails**: **Buddy-Check** (preflight + postflight, GREEN/YELLOW/RED) - **NEW**
+- **Agents**: Cursor/Claude/etc. must pass checks and use filtered, trusted memory - **NEW**
+
+### Target Users
+- **Individual Developers**: Developer mode (default) - light checks, low friction
+- **Small Teams**: Enterprise/Strict mode (opt-in) - full guardrails, CI enforcement
+
+---
+
+## 📦 IMPLEMENTATION TASKS
+
+### Phase 1: Schema Changes & Types
+
+#### Task 1.1: Update Type Definitions
+**File**: `src/types.ts`
+
+**Changes**:
+```typescript
+// New types
+type MemoryStatus = 'DRAFT' | 'VERIFIED';
+type Sensitivity = 'public' | 'internal' | 'secret';
+type TrustLevel = 'high' | 'medium' | 'low';
+type SourceType = 'human' | 'code' | 'agent' | 'external';
+type RecordType = 'NODE' | 'EDGE' | 'NOTE' | 'EVENT';
+
+// Updated MemoryRecord interface
+interface MemoryRecord {
+  id: string;
+  schemaVersion: string;        // "1.4.0"
+  recordVersion: number;        // start at 1, increment on update
+
+  type: RecordType;
+  projectPath: string;
+  branch?: string;
+  location?: {
+    lineStart: number;
+    lineEnd: number;
+    commit: string;             // commitOid
+  };
+  content: string;
+
+  sensitivity: Sensitivity;
+  trustLevel: TrustLevel;
+  status: MemoryStatus;         // 'DRAFT' or 'VERIFIED'
+
+  sourceType: SourceType;
+  provenance: {
+    createdAt: string;          // ISO 8601 UTC
+    createdBy: string;          // user or agent id
+    model?: string;
+    tool?: string;              // cursor, vscode, cli, etc.
+    commitOid?: string;
+    branch?: string;
+    prNumber?: string;
+    kind: 'user' | 'app' | 'signed-token';
+    tokenIssuer?: string;
+  };
+
+  expiresAt?: string;           // optional TTL hint
+  incident?: boolean;           // optional incident flag
+}
+```
+
+**Action Items**:
+- [ ] Add new type definitions
+- [ ] Update MemoryEntry interface to extend MemoryRecord
+- [ ] Update LearningEntry and ChangeEntry with new fields
+- [ ] Maintain backward compatibility with v1.3 records
+
+**Estimated Time**: 2-3 hours
+
+---
+
+#### Task 1.2: Update JSONL Parsing
+**File**: `src/utils.ts`
+
+**Changes**:
+- Update `readJsonlEntries()` to handle new schema fields
+- Add default values for missing v1.3 fields
+- Validate schema version on read
+
+**Action Items**:
+- [ ] Update JSONL reader to handle v1.4 schema
+- [ ] Add schema version detection
+- [ ] Add default value injection for v1.3 records
+- [ ] Add validation for required fields
+
+**Estimated Time**: 2-3 hours
+
+---
+
+### Phase 2: Migration System
+
+#### Task 2.1: Migration Command
+**File**: `src/commands/migrate.ts` (update existing)
+
+**New Command**:
+```bash
+memorylink migrate --to 1.4
+```
+
+**Behavior**:
+- Read all existing v1.3 records
+- Transform to v1.4 schema:
+  - `schemaVersion = "1.4.0"`
+  - `recordVersion = 1`
+  - `sensitivity = 'internal'` (default)
+  - `trustLevel = 'medium'` (default)
+  - `status = 'VERIFIED'` for human-origin, `'DRAFT'` if unknown
+  - `sourceType = 'agent'` if tool-logged, else `'human'`
+  - `provenance` from existing timestamps
+- Write migrated records back
+- Create backup of original files
+
+**Action Items**:
+- [ ] Add `--to` flag to migrate command
+- [ ] Implement v1.3 → v1.4 transformation
+- [ ] Add backup creation
+- [ ] Add validation after migration
+- [ ] Add rollback capability
+
+**Estimated Time**: 4-5 hours
+
+---
+
+### Phase 3: Buddy-Check Engine
+
+#### Task 3.1: Buddy-Check Types
+**File**: `src/types.ts` (add new types)
+
+**Types**:
+```typescript
+type BuddyStatus = 'GREEN' | 'YELLOW' | 'RED';
+
+interface BuddyIssue {
+  id?: string;             // record id if applicable
+  type: string;            // SCHEMA, BROKEN_REF, SECRET, PII, STALE, CONFLICT, POLICY, etc.
+  severity: 'info' | 'warning' | 'error';
+  message: string;
+  projectPath?: string;
+}
+
+interface BuddyReport {
+  status: BuddyStatus;
+  summary: string;
+  issues: BuddyIssue[];
+  override?: {
+    approvedBy: string;
+    reason: string;
+    timestamp: string;
+  };
+}
+```
+
+**Action Items**:
+- [ ] Add BuddyStatus type
+- [ ] Add BuddyIssue interface
+- [ ] Add BuddyReport interface
+- [ ] Add override tracking interface
+
+**Estimated Time**: 1 hour
+
+---
+
+#### Task 3.2: Preflight Check Implementation
+**File**: `src/commands/doctor.ts` (major update)
+
+**New Command**:
+```bash
+memorylink doctor --preflight
+memorylink doctor --preflight --json > .memorylink/preflight.json
+```
+
+**Checks to Implement**:
+1. **Schema Validation**
+   - Required fields present
+   - Valid enum values
+   - Valid data types
+
+2. **Referential Integrity**
+   - Files exist
+   - Line numbers in range
+   - Commit OIDs exist in Git
+
+3. **Security Checks**
+   - Secret detection (API keys, passwords, tokens)
+   - PII detection (emails, IDs)
+   - Sensitivity classification
+
+4. **Staleness Detection**
+   - File changed since commit
+   - Mark as STALE
+   - Downgrade trust level
+
+5. **Policy Validation**
+   - "Prefer fresh code over memory"
+   - "No secrets in memory"
+   - Custom policy rules
+
+**Status Rules**:
+- **GREEN**: No errors; issues empty or informational
+- **YELLOW**: Warnings (stale/low-trust/conflicts) but no hard security errors
+- **RED**: Any SECRET/PII error, severe schema/ref issues on critical paths
+
+**Action Items**:
+- [ ] Implement schema validation check
+- [ ] Implement referential integrity check
+- [ ] Implement secret detection (enhance existing)
+- [ ] Implement PII detection
+- [ ] Implement staleness detection
+- [ ] Implement policy validation
+- [ ] Implement status calculation logic
+- [ ] Add JSON output option
+- [ ] Add file output option
+
+**Estimated Time**: 8-10 hours
+
+---
+
+#### Task 3.3: Postflight Check Implementation
+**File**: `src/commands/doctor.ts` (add to existing)
+
+**New Command**:
+```bash
+memorylink doctor --postflight --changed
+memorylink doctor --postflight --changed --json > .memorylink/postflight.json
+memorylink doctor --postflight --changed --fix   # optional auto-fix
+```
+
+**Behavior**:
+- Scope: Only changed files + new/updated MemoryRecords
+- Run same checks as preflight
+- Additional checks:
+  - Code + memory changes consistency
+  - Mark new agent records as `status='DRAFT'`
+- If `--fix`:
+  - Drop irreparable broken memory records
+  - Update `location.commit`/`provenance.commitOid` when file moved
+  - Never edit code automatically
+
+**Status Rules**:
+- **RED**: Do not allow commit/merge in strict mode
+- **YELLOW**: Allow with warning; show in IDE
+
+**Action Items**:
+- [ ] Implement changed file detection
+- [ ] Implement new/updated record detection
+- [ ] Implement consistency checks
+- [ ] Implement auto-fix logic (memory only)
+- [ ] Add JSON output option
+- [ ] Add file output option
+
+**Estimated Time**: 6-8 hours
+
+---
+
+#### Task 3.4: Human Override Logging
+**File**: `src/commands/doctor.ts` (add override handling)
+
+**Behavior**:
+- When status is RED in strict mode and human overrides:
+  - Require `approvedBy` and `reason`
+  - Append to report:
+    ```typescript
+    override?: {
+      approvedBy: string;
+      reason: string;
+      timestamp: string;
+    };
+    ```
+  - Store in `.memorylink/preflight.json` / `.memorylink/postflight.json`
+  - Store in `.memorylink/overrides.log` (JSONL)
+
+**Action Items**:
+- [ ] Add override prompt/interactive input
+- [ ] Add override to report structure
+- [ ] Implement override logging to JSON files
+- [ ] Implement override logging to JSONL file
+- [ ] Add override history viewing
+
+**Estimated Time**: 3-4 hours
+
+---
+
+#### Task 3.5: Observability Log
+**File**: `src/utils/observability.ts` (new file)
+
+**New Log File**: `.memorylink/observability.log` (JSONL)
+
+**Log Entry Format**:
+```json
+{
+  "timestamp": "2025-12-19T14:30:12Z",
+  "runType": "preflight",
+  "status": "YELLOW",
+  "agentId": "cursor-claude-3.5",
+  "branch": "feature/x",
+  "recordsUsed": 47,
+  "highTrust": 40,
+  "mediumTrust": 5,
+  "lowTrust": 2,
+  "staleCount": 3,
+  "issues": ["STALE", "LOW_TRUST"],
+  "durationMs": 420
+}
+```
+
+**Action Items**:
+- [ ] Create observability utility module
+- [ ] Implement log entry creation
+- [ ] Implement log writing (JSONL append)
+- [ ] Integrate into preflight/postflight
+- [ ] Add log viewing command (optional)
+
+**Estimated Time**: 2-3 hours
+
+---
+
+### Phase 4: Configuration & Modes
+
+#### Task 4.1: Mode Configuration
+**File**: `src/types.ts` (update MemoryLinkConfig)
+
+**Config Structure**:
+```typescript
+interface MemoryLinkConfig {
+  // ... existing fields
+  mode?: 'developer' | 'enterprise';
+  strict?: boolean;
+}
+```
+
+**Default**: `mode: 'developer'`, `strict: false`
+
+**Action Items**:
+- [ ] Add mode field to config interface
+- [ ] Add strict field to config interface
+- [ ] Update config reading/writing
+- [ ] Add mode switching command (optional)
+
+**Estimated Time**: 1-2 hours
+
+---
+
+#### Task 4.2: Mode Behavior Implementation
+**Files**: `src/commands/doctor.ts`, `src/utils/doctor.ts`
+
+**Developer Mode** (default):
+- `doctor` available, but not automatically blocking
+- VS Code shows warnings
+- Full schema fields exist but can be auto-defaulted
+
+**Enterprise Mode**:
+- Preflight/postflight automatically called around AI actions
+- RED blocks agent runs and commits (unless override logged)
+- CI/GitHub Action recommended
+
+**Action Items**:
+- [ ] Implement mode detection in doctor command
+- [ ] Implement blocking logic for enterprise mode
+- [ ] Add mode-specific behavior
+- [ ] Update error messages based on mode
+
+**Estimated Time**: 3-4 hours
+
+---
+
+### Phase 5: VS Code Extension Integration
+
+#### Task 5.1: Status Bar Indicator
+**File**: `extensions/vscode/src/extension.ts`
+
+**Features**:
+- Small indicator (● GREEN / ● YELLOW / ● RED)
+- Shows last preflight result
+- Click to open report
+
+**Action Items**:
+- [ ] Add status bar item
+- [ ] Implement status polling/updates
+- [ ] Add click handler to open report
+- [ ] Add color coding
+
+**Estimated Time**: 2-3 hours
+
+---
+
+#### Task 5.2: Sidebar Enhancements
+**File**: `extensions/vscode/src/sidebar.ts`
+
+**Features**:
+- List recent MemoryRecords
+- Show trustLevel, sensitivity, status (DRAFT/VERIFIED)
+- Buddy-Check issues list (from latest report)
+- Group issues by file
+
+**Action Items**:
+- [ ] Update sidebar to show new fields
+- [ ] Add trust level indicators
+- [ ] Add sensitivity indicators
+- [ ] Add status indicators (DRAFT/VERIFIED)
+- [ ] Add issues list view
+- [ ] Group issues by file
+
+**Estimated Time**: 4-5 hours
+
+---
+
+#### Task 5.3: AI Action Hooks
+**File**: `extensions/vscode/src/extension.ts`
+
+**Workflow**:
+1. **Before AI Action**:
+   - Call `memorylink doctor --preflight --json`
+   - If status = RED and strict mode ON → show errors, block run (with override option)
+
+2. **After AI Action**:
+   - Apply to temp workspace or diff
+   - Run `memorylink doctor --postflight --changed --json`
+   - If RED → show blocking UI, offer:
+     - Drop bad memory (`--fix`)
+     - Let user adjust and rerun
+   - If GREEN/YELLOW → allow commit of code + memory
+
+**Action Items**:
+- [ ] Hook into Cursor AI action triggers
+- [ ] Implement preflight before AI action
+- [ ] Implement blocking UI for RED status
+- [ ] Implement postflight after AI action
+- [ ] Implement fix/drop memory options
+- [ ] Implement override flow
+
+**Estimated Time**: 8-10 hours
+
+---
+
+#### Task 5.4: VS Code Commands
+**File**: `extensions/vscode/src/extension.ts`
+
+**Commands to Add**:
+- "MemoryLink: Run Preflight"
+- "MemoryLink: Run Postflight for Changed Files"
+- "MemoryLink: Open Buddy-Check Report"
+- "MemoryLink: Promote DRAFT → VERIFIED"
+
+**Action Items**:
+- [ ] Register new commands
+- [ ] Implement preflight command
+- [ ] Implement postflight command
+- [ ] Implement report viewer
+- [ ] Implement promote command
+
+**Estimated Time**: 3-4 hours
+
+---
+
+### Phase 6: CI/GitHub Action Integration
+
+#### Task 6.1: GitHub Action Example
+**File**: `.github/workflows/memorylink-preflight.yml` (new)
+
+**Workflow**:
+```yaml
+name: MemoryLink Preflight
+on: [pull_request]
+
+jobs:
+  preflight:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-node@v3
+        with:
+          node-version: '18'
+      - run: npm install -g memorylink
+      - name: MemoryLink Preflight
+        run: |
+          memorylink doctor --preflight --for "${{ github.actor }}" --pr "${{ github.event.number }}" --branch "${{ github.ref }}" --json > .memorylink/preflight.json
+          STATUS=$(jq -r '.status' .memorylink/preflight.json)
+          if [ "$STATUS" = "RED" ]; then
+            echo "Buddy-Check RED: blocking merge."
+            exit 1
+          fi
+```
+
+**Action Items**:
+- [ ] Create GitHub Action workflow file
+- [ ] Add preflight step
+- [ ] Add status checking
+- [ ] Add blocking logic
+- [ ] Add documentation
+
+**Estimated Time**: 2-3 hours
+
+---
+
+#### Task 6.2: CI Integration Helpers
+**File**: `docs/CI_INTEGRATION.md` (new)
+
+**Content**:
+- GitHub Actions example
+- GitLab CI example
+- Jenkins example
+- General CI integration guide
+
+**Action Items**:
+- [ ] Create CI integration documentation
+- [ ] Add GitHub Actions example
+- [ ] Add GitLab CI example
+- [ ] Add Jenkins example
+- [ ] Add troubleshooting guide
+
+**Estimated Time**: 2-3 hours
+
+---
+
+### Phase 7: JSON/TOON Adapter (Boundary Only)
+
+#### Task 7.1: Formatter Interface
+**File**: `src/utils/formatters.ts` (new)
+
+**Interface**:
+```typescript
+type MemoryFormat = 'json' | 'toon';
+
+interface MemoryFormatter {
+  serialize(records: MemoryRecord[]): string;
+  parse(payload: string): MemoryRecord[];
+}
+```
+
+**Action Items**:
+- [ ] Create formatter interface
+- [ ] Implement JsonFormatter (existing logic)
+- [ ] Create ToonFormatter stub (for future)
+- [ ] Update get-context to support --format flag
+
+**Estimated Time**: 2-3 hours
+
+---
+
+## 🚫 WHAT NOT TO BUILD IN v1.4
+
+**Intentionally Out of Scope** (v1.5/v2.0):
+
+- ❌ Real-time checkpoints (`--checkpoint`)
+- ❌ Agent locking
+- ❌ Smart filtering
+- ❌ Project auto-scoping for huge monorepos
+- ❌ Encryption
+- ❌ RBAC (Role-Based Access Control)
+- ❌ Audit log export
+- ❌ Binary/streaming formats
+- ❌ Full dashboard (beyond simple read-only UI if time permits)
+
+---
+
+## 📅 IMPLEMENTATION TIMELINE
+
+### Week 1: Foundation
+- **Days 1-2**: Schema changes & types (Tasks 1.1, 1.2)
+- **Days 3-4**: Migration system (Task 2.1)
+- **Day 5**: Buddy-Check types & observability (Tasks 3.1, 3.5)
+
+### Week 2: Core Engine
+- **Days 1-3**: Preflight implementation (Task 3.2)
+- **Days 4-5**: Postflight implementation (Task 3.3)
+
+### Week 3: Integration
+- **Days 1-2**: Override logging & modes (Tasks 3.4, 4.1, 4.2)
+- **Days 3-5**: VS Code extension (Tasks 5.1, 5.2, 5.3, 5.4)
+
+### Week 4: Polish & CI
+- **Days 1-2**: CI integration (Tasks 6.1, 6.2)
+- **Days 3-4**: Formatter interface (Task 7.1)
+- **Day 5**: Testing, documentation, release prep
+
+**Total Estimated Time**: 4 weeks (20 working days)
+
+---
+
+## 🧪 TESTING REQUIREMENTS
+
+### Unit Tests
+- [ ] Schema validation tests
+- [ ] Migration tests
+- [ ] Preflight check tests
+- [ ] Postflight check tests
+- [ ] Mode behavior tests
+
+### Integration Tests
+- [ ] VS Code extension integration
+- [ ] CI workflow tests
+- [ ] End-to-end workflow tests
+
+### Test Scenarios
+- [ ] Developer mode (non-blocking)
+- [ ] Enterprise mode (blocking)
+- [ ] Override flow
+- [ ] Migration from v1.3
+- [ ] Secret detection
+- [ ] PII detection
+- [ ] Staleness detection
+
+---
+
+## 📝 DOCUMENTATION REQUIREMENTS
+
+### User Documentation
+- [ ] v1.4 migration guide
+- [ ] Buddy-Check user guide
+- [ ] Developer vs Enterprise mode guide
+- [ ] VS Code extension guide
+- [ ] CI integration guide
+
+### Developer Documentation
+- [ ] Schema v1.4 specification
+- [ ] Buddy-Check API documentation
+- [ ] Extension development guide
+
+---
+
+## ✅ ACCEPTANCE CRITERIA
+
+### Must Have (v1.4 Release)
+- [ ] Schema v1.4 implemented
+- [ ] Migration from v1.3 works
+- [ ] Preflight check works (GREEN/YELLOW/RED)
+- [ ] Postflight check works
+- [ ] VS Code status bar shows status
+- [ ] VS Code sidebar shows issues
+- [ ] Developer mode works (non-blocking)
+- [ ] Enterprise mode works (blocking)
+- [ ] Override logging works
+- [ ] Observability log works
+
+### Nice to Have (If Time Permits)
+- [ ] GitHub Action example
+- [ ] TOON formatter stub
+- [ ] Simple read-only dashboard
+
+---
+
+## 🎯 SUCCESS METRICS
+
+### Technical Metrics
+- All tests passing
+- Migration successful for v1.3 projects
+- Preflight/postflight accuracy >95%
+- Performance: <2s for preflight on 1000 records
+
+### User Metrics
+- Zero breaking changes for v1.3 users
+- Smooth migration experience
+- Clear status indicators
+- Helpful error messages
+
+---
+
+## 📚 REFERENCES
+
+- Master AI Verification Document: `MASTER_AI_VERIFICATION_DOCUMENT.md`
+- Current State Summary: `SESSION_SUMMARY_CURRENT_STATE.md`
+- v1.3 Completion: `V1.3_COMPETITIVE_LAUNCH_COMPLETE.md`
+
+---
+
+**Document Created**: 2025-12-19  
+**Status**: Ready for Implementation  
+**Next Step**: Begin Phase 1 (Schema Changes)
+