@soulcraft/brainy 4.10.4 → 4.11.1

package/CHANGELOG.md

@@ -2,6 +2,131 @@
 
 All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
 
+### [4.11.1](https://github.com/soulcraftlabs/brainy/compare/v4.11.0...v4.11.1) (2025-10-30)
+
+- fix: prevent orphaned relationships in restore() and add VFS progress tracking (549d773)
+
+
+## [4.11.1](https://github.com/soulcraftlabs/brainy/compare/v4.11.0...v4.11.1) (2025-10-30)
+
+### 🐛 Bug Fixes
+
+* **fix(api)**: DataAPI.restore() now filters orphaned relationships (P0 Critical)
+  - **Issue**: restore() created relationships to entities that failed to restore, causing "Entity not found" errors
+  - **Root Cause**: Relationships were not filtered based on successfully restored entities
+  - **Fix**: Now builds Set of successful entity IDs and filters relationships accordingly
+  - **New Tracking**: Added `relationshipsSkipped` to return type for visibility
+  - **Impact**: Prevents complete data corruption when some entities fail to restore
+
+* **fix(import)**: VFS creation now reports progress during import (P1 High)
+  - **Issue**: 3-5 minute VFS creation showed no progress (stuck at 0%), causing users to think import froze
+  - **Root Cause**: VFSStructureGenerator.generate() had no progress callback parameter
+  - **Fix**: Added onProgress callback to VFSStructureOptions interface
+  - **Progress Stages**: Reports 'directories', 'entities', 'metadata' with detailed messages
+  - **Frequency**: Reports every 10 entity files to avoid excessive updates
+  - **Integration**: Wired through ImportCoordinator to main progress callback
+
+### 📝 Files Modified
+
+* `src/api/DataAPI.ts` (lines 173-350) - Added orphaned relationship filtering
+* `src/importers/VFSStructureGenerator.ts` (lines 18-53, 110-347) - Added progress callback
+* `src/import/ImportCoordinator.ts` (lines 438-459) - Wired progress callback
+
+## [4.11.0](https://github.com/soulcraftlabs/brainy/compare/v4.10.4...v4.11.0) (2025-10-30)
+
+### 🚨 CRITICAL BUG FIX
+
+**DataAPI.restore() Complete Data Loss Bug Fixed**
+
+Previous versions (v4.10.4 and earlier) had a critical bug where `DataAPI.restore()` did NOT persist data to storage, causing complete data loss after instance restart or cache clear. **If you used backup/restore in v4.10.4 or earlier, your restored data was NOT saved.**
+
+### 🔧 What Was Fixed
+
+* **fix(api)**: DataAPI.restore() now properly persists data to all storage adapters
+  - **Root Cause**: restore() called `storage.saveNoun()` directly, bypassing all indexes and proper persistence
+  - **Fix**: Now uses `brain.addMany()` and `brain.relateMany()` (proper persistence path)
+  - **Result**: Data now survives instance restart and is fully indexed/searchable
+
+### ✨ Improvements
+
+* **feat(api)**: Enhanced restore() with progress reporting and error tracking
+  - **New Return Type**: Returns `{ entitiesRestored, relationshipsRestored, errors }` instead of `void`
+  - **Progress Callback**: Optional `onProgress(completed, total)` parameter for UI updates
+  - **Error Details**: Returns array of failed entities/relations with error messages
+  - **Verification**: Automatically verifies first entity is retrievable after restore
+
+* **feat(api)**: Cross-storage restore support
+  - Backup from any storage adapter, restore to any other
+  - Example: Backup from GCS → Restore to Filesystem
+  - Automatically uses target storage's optimal batch configuration
+
+* **perf(api)**: Storage-aware batching for restore operations
+  - Leverages v4.10.4's storage-aware batching (10-100x faster on cloud storage)
+  - Automatic backpressure management prevents circuit breaker activation
+  - Separate read/write circuit breakers (backup can run during restore throttling)
+
+### 📊 What's Now Guaranteed
+
+| Feature | v4.10.4 | v4.11.0 |
+|---------|---------|---------|
+| Data Persists to Storage | ❌ No | ✅ Yes |
+| Data Survives Restart | ❌ No | ✅ Yes |
+| HNSW Index Updated | ❌ No | ✅ Yes |
+| Metadata Index Updated | ❌ No | ✅ Yes |
+| Searchable After Restore | ❌ No | ✅ Yes |
+| Progress Reporting | ❌ No | ✅ Yes |
+| Error Tracking | ❌ Silent | ✅ Detailed |
+| Cross-Storage Support | ❌ No | ✅ Yes |
+
+### 🔄 Migration Guide
+
+**No code changes required!** The fix is backward compatible:
+
+```typescript
+// Old code (still works)
+await brain.data().restore({ backup, overwrite: true })
+
+// New code (with progress tracking)
+const result = await brain.data().restore({
+  backup,
+  overwrite: true,
+  onProgress: (done, total) => {
+    console.log(`Restoring... ${done}/${total}`)
+  }
+})
+
+console.log(`✅ Restored ${result.entitiesRestored} entities`)
+if (result.errors.length > 0) {
+  console.warn(`⚠️ ${result.errors.length} failures`)
+}
+```
+
+### ⚠️ Breaking Changes (Minor API Change)
+
+* **DataAPI.restore()** return type changed from `Promise<void>` to `Promise<{ entitiesRestored, relationshipsRestored, errors }>`
+  - Impact: Minimal - most code doesn't use the return value
+  - Fix: Remove explicit `Promise<void>` type annotations if present
+
+### 📝 Files Modified
+
+* `src/api/DataAPI.ts` - Complete rewrite of restore() method (lines 161-338)
+
+### [4.10.4](https://github.com/soulcraftlabs/brainy/compare/v4.10.3...v4.10.4) (2025-10-30)
+
+* fix: prevent circuit breaker activation and data loss during bulk imports
+  - Storage-aware batching system prevents rate limiting on cloud storage (GCS, S3, R2, Azure)
+  - Separate read/write circuit breakers prevent read lockouts during write throttling
+  - ImportCoordinator uses addMany()/relateMany() for 10-100x performance improvement
+  - Fixes silent data loss and 30+ second lockouts on 1000+ row imports
+
+### [4.10.3](https://github.com/soulcraftlabs/brainy/compare/v4.10.2...v4.10.3) (2025-10-29)
+
+* fix: add atomic writes to ALL file operations to prevent concurrent write corruption
+
+### [4.10.2](https://github.com/soulcraftlabs/brainy/compare/v4.10.1...v4.10.2) (2025-10-29)
+
+* fix: VFS not initialized during Excel import, causing 0 files accessible
+
 ### [4.10.1](https://github.com/soulcraftlabs/brainy/compare/v4.10.0...v4.10.1) (2025-10-29)
 
 - fix: add mutex locks to FileSystemStorage for HNSW concurrency (CRITICAL) (ff86e88)

package/dist/api/DataAPI.d.ts

@@ -81,13 +81,32 @@ export declare class DataAPI {
     }>;
     /**
      * Restore data from a backup
+     *
+     * v4.11.1: CRITICAL FIX - Now uses brain.addMany() and brain.relateMany()
+     * Previous implementation only wrote to storage cache without updating indexes,
+     * causing complete data loss on restart. This fix ensures:
+     * - All 5 indexes updated (HNSW, metadata, adjacency, sparse, type-aware)
+     * - Proper persistence to disk/cloud storage
+     * - Storage-aware batching for optimal performance
+     * - Atomic writes to prevent corruption
+     * - Data survives instance restart
      */
     restore(params: {
         backup: BackupData;
         merge?: boolean;
         overwrite?: boolean;
         validate?: boolean;
-    }): Promise<void>;
+        onProgress?: (completed: number, total: number) => void;
+    }): Promise<{
+        entitiesRestored: number;
+        relationshipsRestored: number;
+        relationshipsSkipped: number;
+        errors: Array<{
+            type: 'entity' | 'relation';
+            id: string;
+            error: string;
+        }>;
+    }>;
     /**
      * Clear data
      */

package/dist/api/DataAPI.js

@@ -75,89 +75,171 @@ export class DataAPI {
     }
     /**
      * Restore data from a backup
+     *
+     * v4.11.1: CRITICAL FIX - Now uses brain.addMany() and brain.relateMany()
+     * Previous implementation only wrote to storage cache without updating indexes,
+     * causing complete data loss on restart. This fix ensures:
+     * - All 5 indexes updated (HNSW, metadata, adjacency, sparse, type-aware)
+     * - Proper persistence to disk/cloud storage
+     * - Storage-aware batching for optimal performance
+     * - Atomic writes to prevent corruption
+     * - Data survives instance restart
      */
     async restore(params) {
-        const { backup, merge = false, overwrite = false, validate = true } = params;
+        const { backup, merge = false, overwrite = false, validate = true, onProgress } = params;
+        const result = {
+            entitiesRestored: 0,
+            relationshipsRestored: 0,
+            relationshipsSkipped: 0,
+            errors: []
+        };
         // Validate backup format
         if (validate) {
             if (!backup.version || !backup.entities || !backup.relations) {
-                throw new Error('Invalid backup format');
+                throw new Error('Invalid backup format: missing version, entities, or relations');
             }
         }
+        // Validate brain instance is available (required for v4.11.1+ restore)
+        if (!this.brain) {
+            throw new Error('Restore requires brain instance. DataAPI must be initialized with brain reference. ' +
+                'Use: await brain.data() instead of constructing DataAPI directly.');
+        }
         // Clear existing data if not merging
         if (!merge && overwrite) {
             await this.clear({ entities: true, relations: true });
         }
-        // Restore entities
-        for (const entity of backup.entities) {
+        // ============================================
+        // Phase 1: Restore entities using addMany()
+        // v4.11.1: Uses proper persistence path through brain.addMany()
+        // ============================================
+        // Prepare entity parameters for addMany()
+        const entityParams = backup.entities
+            .filter(entity => {
+            // Skip existing entities when merging without overwrite
+            if (merge && !overwrite) {
+                // Note: We'll rely on addMany's internal duplicate handling
+                // rather than checking each entity individually (performance)
+                return true;
+            }
+            return true;
+        })
+            .map(entity => {
+            // Extract data field from metadata (backup format compatibility)
+            // Backup stores the original data in metadata.data
+            const data = entity.metadata?.data || entity.id;
+            return {
+                id: entity.id,
+                data, // Required field for brainy.add()
+                type: entity.type,
+                metadata: entity.metadata || {},
+                vector: entity.vector, // Preserve original vectors from backup
+                service: entity.service,
+                // Preserve confidence and weight if available
+                confidence: entity.metadata?.confidence,
+                weight: entity.metadata?.weight
+            };
+        });
+        // Restore entities in batches using storage-aware batching (v4.11.0)
+        // v4.11.1: Track successful entity IDs to prevent orphaned relationships
+        const successfulEntityIds = new Set();
+        if (entityParams.length > 0) {
             try {
-                // v4.0.0: Prepare noun and metadata separately
-                const noun = {
-                    id: entity.id,
-                    vector: entity.vector || new Array(384).fill(0), // Default vector if missing
-                    connections: new Map(),
-                    level: 0
-                };
-                const metadata = {
-                    ...entity.metadata,
-                    noun: entity.type,
-                    service: entity.service,
-                    createdAt: Date.now()
-                };
-                // Check if entity exists when merging
-                if (merge) {
-                    const existing = await this.storage.getNoun(entity.id);
-                    if (existing && !overwrite) {
-                        continue; // Skip existing entities unless overwriting
+                const addResult = await this.brain.addMany({
+                    items: entityParams,
+                    continueOnError: true,
+                    onProgress: (done, total) => {
+                        onProgress?.(done, backup.entities.length + backup.relations.length);
                     }
-                }
-                await this.storage.saveNoun(noun);
-                await this.storage.saveNounMetadata(entity.id, metadata);
+                });
+                result.entitiesRestored = addResult.successful.length;
+                // Build Set of successfully restored entity IDs (v4.11.1 Bug Fix)
+                addResult.successful.forEach((entityId) => {
+                    successfulEntityIds.add(entityId);
+                });
+                // Track errors
+                addResult.failed.forEach((failure) => {
+                    result.errors.push({
+                        type: 'entity',
+                        id: failure.item?.id || 'unknown',
+                        error: failure.error || 'Unknown error'
+                    });
+                });
             }
             catch (error) {
-                console.error(`Failed to restore entity ${entity.id}:`, error);
+                throw new Error(`Failed to restore entities: ${error.message}`);
             }
         }
-        // Restore relations
-        for (const relation of backup.relations) {
-            try {
-                // Get source and target entities to compute relation vector
-                const sourceNoun = await this.storage.getNoun(relation.from);
-                const targetNoun = await this.storage.getNoun(relation.to);
-                if (!sourceNoun || !targetNoun) {
-                    console.warn(`Skipping relation ${relation.id}: missing entities`);
-                    continue;
-                }
-                // Compute relation vector as average of source and target
-                const relationVector = sourceNoun.vector.map((v, i) => (v + targetNoun.vector[i]) / 2);
-                // v4.0.0: Prepare verb and metadata separately
-                const verb = {
-                    id: relation.id,
-                    vector: relationVector,
-                    connections: new Map(),
-                    verb: relation.type,
-                    sourceId: relation.from,
-                    targetId: relation.to
-                };
-                const verbMetadata = {
-                    weight: relation.weight,
-                    ...relation.metadata,
-                    createdAt: Date.now()
-                };
-                // Check if relation exists when merging
-                if (merge) {
-                    const existing = await this.storage.getVerb(relation.id);
-                    if (existing && !overwrite) {
-                        continue;
-                    }
+        // ============================================
+        // Phase 2: Restore relationships using relateMany()
+        // v4.11.1: CRITICAL FIX - Filter orphaned relationships
+        // ============================================
+        // Prepare relationship parameters - filter out orphaned references
+        const relationParams = backup.relations
+            .filter(relation => {
+            // Skip existing relations when merging without overwrite
+            if (merge && !overwrite) {
+                // Note: We'll rely on relateMany's internal duplicate handling
+                return true;
+            }
+            // v4.11.1 CRITICAL BUG FIX: Skip relationships where source or target entity failed to restore
+            // This prevents creating orphaned references that cause "Entity not found" errors
+            const sourceExists = successfulEntityIds.has(relation.from);
+            const targetExists = successfulEntityIds.has(relation.to);
+            if (!sourceExists || !targetExists) {
+                // Track skipped relationship
+                result.relationshipsSkipped++;
+                // Optionally log for debugging (can be removed in production)
+                if (process.env.NODE_ENV !== 'production') {
+                    console.debug(`Skipping orphaned relationship: ${relation.from} -> ${relation.to} ` +
+                        `(${!sourceExists ? 'missing source' : 'missing target'})`);
                 }
-                await this.storage.saveVerb(verb);
-                await this.storage.saveVerbMetadata(relation.id, verbMetadata);
+                return false;
+            }
+            return true;
+        })
+            .map(relation => ({
+            from: relation.from,
+            to: relation.to,
+            type: relation.type,
+            metadata: relation.metadata || {},
+            weight: relation.weight || 1.0
+            // Note: relation.id is ignored - brain.relate() generates new IDs
+            // This is intentional to avoid ID conflicts
+        }));
+        // Restore relationships in batches using storage-aware batching (v4.11.0)
+        if (relationParams.length > 0) {
+            try {
+                const relateResult = await this.brain.relateMany({
+                    items: relationParams,
+                    continueOnError: true
+                });
+                result.relationshipsRestored = relateResult.successful.length;
+                // Track errors
+                relateResult.failed.forEach((failure) => {
+                    result.errors.push({
+                        type: 'relation',
+                        id: failure.item?.from + '->' + failure.item?.to || 'unknown',
+                        error: failure.error || 'Unknown error'
+                    });
+                });
             }
             catch (error) {
-                console.error(`Failed to restore relation ${relation.id}:`, error);
+                throw new Error(`Failed to restore relationships: ${error.message}`);
+            }
+        }
+        // ============================================
+        // Phase 3: Verify restoration succeeded
+        // ============================================
+        // Sample verification: Check that first entity is actually retrievable
+        if (backup.entities.length > 0 && result.entitiesRestored > 0) {
+            const firstEntityId = backup.entities[0].id;
+            const verified = await this.brain.get(firstEntityId);
+            if (!verified) {
+                console.warn(`⚠️  Restore completed but verification failed - entity ${firstEntityId} not retrievable. ` +
+                    `This may indicate a persistence issue with the storage adapter.`);
             }
         }
+        return result;
     }
     /**
      * Clear data

package/dist/import/ImportCoordinator.js

@@ -137,7 +137,16 @@ export class ImportCoordinator {
             sourceFilename: normalizedSource.filename || `import.${detection.format}`,
             createRelationshipFile: true,
             createMetadataFile: true,
-            trackingContext // v4.10.0: Pass tracking metadata to VFS
+            trackingContext, // v4.10.0: Pass tracking metadata to VFS
+            // v4.11.1: Pass progress callback for VFS creation updates
+            onProgress: (vfsProgress) => {
+                options.onProgress?.({
+                    stage: 'storing-vfs',
+                    message: vfsProgress.message,
+                    processed: vfsProgress.processed,
+                    total: vfsProgress.total
+                });
+            }
         });
         // Report graph storage stage
         options.onProgress?.({

package/dist/importers/VFSStructureGenerator.d.ts

@@ -30,6 +30,13 @@ export interface VFSStructureOptions {
     createMetadataFile?: boolean;
     /** Import tracking context (v4.10.0) */
     trackingContext?: TrackingContext;
+    /** Progress callback (v4.11.1) - Reports VFS creation progress */
+    onProgress?: (progress: {
+        stage: 'directories' | 'entities' | 'metadata';
+        message: string;
+        processed: number;
+        total: number;
+    }) => void;
 }
 export interface VFSStructureResult {
     /** Root path created */

package/dist/importers/VFSStructureGenerator.js

@@ -48,6 +48,26 @@ export class VFSStructureGenerator {
         };
         // Ensure VFS is initialized
         await this.init();
+        // v4.11.1: Calculate total operations for progress tracking
+        const groups = this.groupEntities(importResult, options);
+        const totalEntities = Array.from(groups.values()).reduce((sum, entities) => sum + entities.length, 0);
+        const totalOperations = 1 + // root directory
+            (options.preserveSource ? 1 : 0) + // source file
+            groups.size + // group directories
+            totalEntities + // entity files
+            (options.createRelationshipFile !== false ? 1 : 0) + // relationships file
+            (options.createMetadataFile !== false ? 1 : 0); // metadata file
+        let completedOperations = 0;
+        // Helper to report progress
+        const reportProgress = (stage, message) => {
+            completedOperations++;
+            options.onProgress?.({
+                stage,
+                message,
+                processed: completedOperations,
+                total: totalOperations
+            });
+        };
         // Extract tracking metadata if provided
         const trackingMetadata = options.trackingContext ? {
             importIds: [options.trackingContext.importId],
@@ -65,6 +85,7 @@ export class VFSStructureGenerator {
             });
             result.directories.push(options.rootPath);
             result.operations++;
+            reportProgress('directories', `Created root directory: ${options.rootPath}`);
         }
         catch (error) {
             // Directory might already exist, that's fine
@@ -72,6 +93,7 @@ export class VFSStructureGenerator {
                 throw error;
             }
             result.directories.push(options.rootPath);
+            reportProgress('directories', `Root directory exists: ${options.rootPath}`);
         }
         // Preserve source file if requested
         if (options.preserveSource && options.sourceBuffer && options.sourceFilename) {
@@ -84,9 +106,10 @@ export class VFSStructureGenerator {
                 type: 'source'
             });
             result.operations++;
+            reportProgress('metadata', `Preserved source file: ${options.sourceFilename}`);
         }
-        // Group entities
-        const groups = this.groupEntities(importResult, options);
+        // Note: groups already calculated above for progress tracking
+        // const groups = this.groupEntities(importResult, options)
         // Create directories and files for each group
         for (const [groupName, entities] of groups.entries()) {
             const groupPath = `${options.rootPath}/${groupName}`;
@@ -98,6 +121,7 @@ export class VFSStructureGenerator {
                 });
                 result.directories.push(groupPath);
                 result.operations++;
+                reportProgress('directories', `Created directory: ${groupName} (${entities.length} entities)`);
             }
             catch (error) {
                 // Directory might already exist
@@ -105,9 +129,12 @@ export class VFSStructureGenerator {
                     throw error;
                 }
                 result.directories.push(groupPath);
+                reportProgress('directories', `Directory exists: ${groupName}`);
             }
             // Create entity files
+            let entityCount = 0;
             for (const extracted of entities) {
+                entityCount++;
                 const sanitizedName = this.sanitizeFilename(extracted.entity.name);
                 const entityPath = `${groupPath}/${sanitizedName}.json`;
                 // Create entity JSON
@@ -140,6 +167,10 @@ export class VFSStructureGenerator {
                     type: 'entity'
                 });
                 result.operations++;
+                // v4.11.1: Report progress every 10 entities (or on last entity)
+                if (entityCount % 10 === 0 || entityCount === entities.length) {
+                    reportProgress('entities', `Created ${entityCount}/${entities.length} ${groupName} files`);
+                }
             }
         }
         // Create relationships file
@@ -167,6 +198,7 @@ export class VFSStructureGenerator {
                 type: 'relationships'
             });
             result.operations++;
+            reportProgress('metadata', `Created relationships file (${allRelationships.length} relationships)`);
         }
         // Create metadata file
         if (options.createMetadataFile !== false) {
@@ -206,6 +238,16 @@ export class VFSStructureGenerator {
                 type: 'metadata'
             });
             result.operations++;
+            reportProgress('metadata', 'Created metadata file');
+        }
+        // v4.11.1: Final progress update
+        if (options.onProgress) {
+            options.onProgress({
+                stage: 'metadata',
+                message: `VFS structure created successfully (${result.files.length} files, ${result.directories.length} directories)`,
+                processed: totalOperations,
+                total: totalOperations
+            });
         }
         result.duration = Date.now() - startTime;
         return result;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@soulcraft/brainy",
-  "version": "4.10.4",
+  "version": "4.11.1",
   "description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. 31 nouns × 40 verbs for infinite expressiveness.",
   "main": "dist/index.js",
   "module": "dist/index.js",