@soulcraft/brainy 0.31.0 → 0.32.0

package/README.md

@@ -1379,6 +1379,106 @@ terabyte-scale data that can't fit entirely in memory, we provide several approa
 
 For detailed information on how to scale Brainy for large datasets, vector dimension standardization, threading implementation, storage testing, and other technical topics, see our comprehensive [Technical Guides](TECHNICAL_GUIDES.md).
 
+## Recent Changes and Performance Improvements
+
+### Enhanced Memory Management and Scalability
+
+Brainy has been significantly improved to handle larger datasets more efficiently:
+
+- **Pagination Support**: All data retrieval methods now support pagination to avoid loading entire datasets into memory at once. The deprecated `getAllNouns()` and `getAllVerbs()` methods have been replaced with `getNouns()` and `getVerbs()` methods that support pagination, filtering, and cursor-based navigation.
+
+- **Multi-level Caching**: A sophisticated three-level caching strategy has been implemented:
+  - **Level 1**: Hot cache (most accessed nodes) - RAM (automatically detecting and adjusting in each environment)
+  - **Level 2**: Warm cache (recent nodes) - OPFS, Filesystem or S3 depending on environment
+  - **Level 3**: Cold storage (all nodes) - OPFS, Filesystem or S3 depending on environment
+
+- **Adaptive Memory Usage**: The system automatically detects available memory and adjusts cache sizes accordingly:
+  - In Node.js: Uses 10% of free memory (minimum 1000 entries)
+  - In browsers: Scales based on device memory (500 entries per GB, minimum 1000)
+
+- **Intelligent Cache Eviction**: Implements a Least Recently Used (LRU) policy that evicts the oldest 20% of items when the cache reaches the configured threshold.
+
+- **Prefetching Strategy**: Implements batch prefetching to improve performance while avoiding overwhelming system resources.
+
+### S3-Compatible Storage Improvements
+
+- **Enhanced Cloud Storage**: Improved support for S3-compatible storage services including AWS S3, Cloudflare R2, and others.
+
+- **Optimized Data Access**: Batch operations and error handling for efficient cloud storage access.
+
+- **Change Log Management**: Efficient synchronization through change logs to track updates.
+
+### Data Compatibility
+
+Yes, you can use existing data indexed from an old version. Brainy includes robust data migration capabilities:
+
+- **Vector Regeneration**: If vectors are missing in imported data, they will be automatically created using the embedding function.
+
+- **HNSW Index Reconstruction**: The system can reconstruct the HNSW index from backup data, ensuring compatibility with previous versions.
+
+- **Sparse Data Import**: Support for importing sparse data (without vectors) through the `importSparseData()` method.
+
+### System Requirements
+
+#### Default Mode
+
+- **Memory**: 
+  - Minimum: 512MB RAM
+  - Recommended: 2GB+ RAM for medium datasets, 8GB+ for large datasets
+  
+- **CPU**: 
+  - Minimum: 2 cores
+  - Recommended: 4+ cores for better performance with parallel operations
+
+- **Storage**:
+  - Minimum: 1GB available storage
+  - Recommended: Storage space at least 3x the size of your dataset
+
+#### Read-Only Mode
+
+Read-only mode prevents all write operations (add, update, delete) and is optimized for search operations.
+
+- **Memory**: 
+  - Minimum: 256MB RAM
+  - Recommended: 1GB+ RAM
+  
+- **CPU**: 
+  - Minimum: 1 core
+  - Recommended: 2+ cores
+
+- **Storage**:
+  - Minimum: Storage space equal to the size of your dataset
+  - Recommended: 2x the size of your dataset for caching
+
+- **New Feature**: Lazy loading support in read-only mode for improved performance with large datasets.
+
+#### Write-Only Mode
+
+Write-only mode prevents all search operations and is optimized for initial data loading or when you want to optimize for write performance.
+
+- **Memory**: 
+  - Minimum: 512MB RAM
+  - Recommended: 2GB+ RAM
+  
+- **CPU**: 
+  - Minimum: 2 cores
+  - Recommended: 4+ cores for faster data ingestion
+
+- **Storage**:
+  - Minimum: Storage space at least 2x the size of your dataset
+  - Recommended: 4x the size of your dataset for optimal performance
+
+### Performance Tuning Parameters
+
+Brainy offers several configuration options for performance tuning:
+
+- **Hot Cache Size**: Control the maximum number of items to keep in memory.
+- **Eviction Threshold**: Set the threshold at which cache eviction begins (default: 0.8 or 80% of max size).
+- **Warm Cache TTL**: Set the time-to-live for items in the warm cache (default: 24 hours).
+- **Batch Size**: Control the number of items to process in a single batch for operations like prefetching (default: 10).
+
+These improvements make Brainy more efficient, scalable, and adaptable to different environments and usage patterns.
+
 ## Testing
 
 Brainy uses Vitest for testing. For detailed information about testing in Brainy, including test configuration, scripts, reporting tools, and best practices, see our [Testing Guide](TESTING.md).

package/dist/brainyData.d.ts

@@ -65,6 +65,13 @@ export interface BrainyDataConfig {
      * When true, all write operations will throw an error
      */
     readOnly?: boolean;
+    /**
+     * Enable lazy loading in read-only mode
+     * When true and in read-only mode, the index is not fully loaded during initialization
+     * Nodes are loaded on-demand during search operations
+     * This improves startup performance for large datasets
+     */
+    lazyLoadInReadOnlyMode?: boolean;
     /**
      * Set the database to write-only mode
      * When true, the index is not loaded into memory and search operations will throw an error
@@ -183,6 +190,7 @@ export declare class BrainyData<T = any> implements BrainyDataInterface<T> {
     private distanceFunction;
     private requestPersistentStorage;
     private readOnly;
+    private lazyLoadInReadOnlyMode;
     private writeOnly;
     private storageConfig;
     private useOptimizedIndex;

package/dist/coreTypes.d.ts

@@ -92,6 +92,7 @@ export interface HNSWConfig {
     efConstruction: number;
     efSearch: number;
     ml: number;
+    useDiskBasedIndex?: boolean;
 }
 /**
  * Storage interface for persistence

package/dist/storage/adapters/s3CompatibleStorage.d.ts

@@ -58,6 +58,8 @@ export declare class S3CompatibleStorage extends BaseStorage {
     private activeLocks;
     private changeLogPrefix;
     private operationExecutors;
+    private nounCacheManager;
+    private verbCacheManager;
     /**
      * Initialize the storage adapter
      * @param options Configuration options for the S3-compatible storage
@@ -72,6 +74,11 @@ export declare class S3CompatibleStorage extends BaseStorage {
         sessionToken?: string;
         serviceType?: string;
         operationConfig?: OperationConfig;
+        cacheConfig?: {
+            hotCacheMaxSize?: number;
+            hotCacheEvictionThreshold?: number;
+            warmCacheTTL?: number;
+        };
     });
     /**
      * Initialize the storage adapter
@@ -97,10 +104,27 @@ export declare class S3CompatibleStorage extends BaseStorage {
      * Get all nouns from storage (internal implementation)
      */
     protected getAllNouns_internal(): Promise<HNSWNoun[]>;
+    private nodeCache;
     /**
      * Get all nodes from storage
+     * @deprecated This method is deprecated and will be removed in a future version.
+     * It can cause memory issues with large datasets. Use getNodesWithPagination() instead.
      */
     protected getAllNodes(): Promise<HNSWNode[]>;
+    /**
+     * Get nodes with pagination
+     * @param options Pagination options
+     * @returns Promise that resolves to a paginated result of nodes
+     */
+    protected getNodesWithPagination(options?: {
+        limit?: number;
+        cursor?: string;
+        useCache?: boolean;
+    }): Promise<{
+        nodes: HNSWNode[];
+        hasMore: boolean;
+        nextCursor?: string;
+    }>;
     /**
      * Get nouns by noun type (internal implementation)
      * @param nounType The noun type to filter by
@@ -139,12 +163,63 @@ export declare class S3CompatibleStorage extends BaseStorage {
     protected getEdge(id: string): Promise<Edge | null>;
     /**
      * Get all verbs from storage (internal implementation)
+     * @deprecated This method is deprecated and will be removed in a future version.
+     * It can cause memory issues with large datasets. Use getVerbsWithPagination() instead.
      */
     protected getAllVerbs_internal(): Promise<GraphVerb[]>;
     /**
      * Get all edges from storage
+     * @deprecated This method is deprecated and will be removed in a future version.
+     * It can cause memory issues with large datasets. Use getEdgesWithPagination() instead.
      */
     protected getAllEdges(): Promise<Edge[]>;
+    /**
+     * Get edges with pagination
+     * @param options Pagination options
+     * @returns Promise that resolves to a paginated result of edges
+     */
+    protected getEdgesWithPagination(options?: {
+        limit?: number;
+        cursor?: string;
+        useCache?: boolean;
+        filter?: {
+            sourceId?: string;
+            targetId?: string;
+            type?: string;
+        };
+    }): Promise<{
+        edges: Edge[];
+        hasMore: boolean;
+        nextCursor?: string;
+    }>;
+    /**
+     * Filter an edge based on filter criteria
+     * @param edge The edge to filter
+     * @param filter The filter criteria
+     * @returns True if the edge matches the filter, false otherwise
+     */
+    private filterEdge;
+    /**
+     * Get verbs with pagination
+     * @param options Pagination options
+     * @returns Promise that resolves to a paginated result of verbs
+     */
+    getVerbsWithPagination(options?: {
+        limit?: number;
+        cursor?: string;
+        filter?: {
+            verbType?: string | string[];
+            sourceId?: string | string[];
+            targetId?: string | string[];
+            service?: string | string[];
+            metadata?: Record<string, any>;
+        };
+    }): Promise<{
+        items: GraphVerb[];
+        totalCount?: number;
+        hasMore: boolean;
+        nextCursor?: string;
+    }>;
     /**
      * Get verbs by source (internal implementation)
      */

package/dist/storage/adapters/s3CompatibleStorage.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"s3CompatibleStorage.d.ts","sourceRoot":"","sources":["../../../src/storage/adapters/s3CompatibleStorage.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAEH,OAAO,EAAC,SAAS,EAAE,QAAQ,EAAE,cAAc,EAAC,MAAM,oBAAoB,CAAA;AACtE,OAAO,EAAC,WAAW,EAAgE,MAAM,mBAAmB,CAAA;AAC5G,OAAO,EAA4B,eAAe,EAAC,MAAM,+BAA+B,CAAA;AAIxF,KAAK,QAAQ,GAAG,QAAQ,CAAA;AACxB,KAAK,IAAI,GAAG,SAAS,CAAA;AAGrB,UAAU,cAAc;IACpB,SAAS,EAAE,MAAM,CAAA;IACjB,SAAS,EAAE,KAAK,GAAG,QAAQ,GAAG,QAAQ,CAAA;IACtC,UAAU,EAAE,MAAM,GAAG,MAAM,GAAG,UAAU,CAAA;IACxC,QAAQ,EAAE,MAAM,CAAA;IAChB,IAAI,CAAC,EAAE,GAAG,CAAA;IACV,UAAU,CAAC,EAAE,MAAM,CAAA;CACtB;AAGD,OAAO,EAAC,mBAAmB,IAAI,SAAS,EAAC,CAAA;AAMzC;;;;;;;;;;;;;;;;;;;;;GAqBG;AACH,qBAAa,mBAAoB,SAAQ,WAAW;IAChD,OAAO,CAAC,QAAQ,CAAwB;IACxC,OAAO,CAAC,UAAU,CAAQ;IAC1B,OAAO,CAAC,WAAW,CAAQ;IAC3B,OAAO,CAAC,MAAM,CAAQ;IACtB,OAAO,CAAC,QAAQ,CAAC,CAAQ;IACzB,OAAO,CAAC,SAAS,CAAC,CAAQ;IAC1B,OAAO,CAAC,WAAW,CAAQ;IAC3B,OAAO,CAAC,eAAe,CAAQ;IAC/B,OAAO,CAAC,YAAY,CAAC,CAAQ;IAG7B,OAAO,CAAC,UAAU,CAAQ;IAC1B,OAAO,CAAC,UAAU,CAAQ;IAC1B,OAAO,CAAC,cAAc,CAAQ;IAC9B,OAAO,CAAC,WAAW,CAAQ;IAG3B,SAAS,CAAC,eAAe,EAAE,cAAc,GAAG,IAAI,CAAO;IAGvD,OAAO,CAAC,UAAU,CAAmB;IACrC,OAAO,CAAC,WAAW,CAAyB;IAG5C,OAAO,CAAC,eAAe,CAAwB;IAG/C,OAAO,CAAC,kBAAkB,CAA2B;IAErD;;;OAGG;gBACS,OAAO,EAAE;QACjB,UAAU,EAAE,MAAM,CAAA;QAClB,MAAM,CAAC,EAAE,MAAM,CAAA;QACf,QAAQ,CAAC,EAAE,MAAM,CAAA;QACjB,SAAS,CAAC,EAAE,MAAM,CAAA;QAClB,WAAW,EAAE,MAAM,CAAA;QACnB,eAAe,EAAE,MAAM,CAAA;QACvB,YAAY,CAAC,EAAE,MAAM,CAAA;QACrB,WAAW,CAAC,EAAE,MAAM,CAAA;QACpB,eAAe,CAAC,EAAE,eAAe,CAAA;KACpC;IAqBD;;OAEG;IACU,IAAI,IAAI,OAAO,CAAC,IAAI,CAAC;IAqDlC;;OAEG;cACa,iBAAiB,CAAC,IAAI,EAAE,QAAQ,GAAG,OAAO,CAAC,IAAI,CAAC;IAIhE;;OAEG;cACa,QAAQ,CAAC,IAAI,EAAE,QAAQ,GAAG,OAAO,CAAC,IAAI,CAAC;IAuEvD;;OAEG;cACa,gBAAgB,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,GAAG,IAAI,CAAC;IAItE;;OAEG;cACa,OAAO,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,GAAG,IAAI,CAAC;IAiE7D;;OAEG;cACa,oBAAoB,IAAI,OAAO,CAAC,QAAQ,EAAE,CAAC;IAI3D;;OAEG;cACa,WAAW,IAAI,OAAO,CAAC,QAAQ,EAAE,CAAC;IAyHlD;;;;OAIG;cACa,2BAA2B,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,EAAE,CAAC;IAIlF;;;;OAIG;cACa,kBAAkB,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,EAAE,CAAC;IAuBzE;;OAEG;cACa,mBAAmB,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAI9D;;OAEG;cACa,UAAU,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IA4BrD;;OAEG;cACa,iBAAiB,CAAC,IAAI,EAAE,SAAS,GAAG,OAAO,CAAC,IAAI,CAAC;IAIjE;;OAEG;cACa,QAAQ,CAAC,IAAI,EAAE,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC;IA6CnD;;OAEG;cACa,gBAAgB,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,GAAG,IAAI,CAAC;IAIvE;;OAEG;cACa,OAAO,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,GAAG,IAAI,CAAC;IA2FzD;;OAEG;cACa,oBAAoB,IAAI,OAAO,CAAC,SAAS,EAAE,CAAC;IAI5D;;OAEG;cACa,WAAW,IAAI,OAAO,CAAC,IAAI,EAAE,CAAC;IA6F9C;;OAEG;cACa,yBAAyB,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC;IAIjF;;OAEG;cACa,gBAAgB,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,EAAE,CAAC;IAKnE;;OAEG;cACa,yBAAyB,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC;IAIjF;;OAEG;cACa,gBAAgB,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,EAAE,CAAC;IAKnE;;OAEG;cACa,uBAAuB,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC;IAI3E;;OAEG;cACa,cAAc,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,EAAE,CAAC;IAK7D;;OAEG;cACa,mBAAmB,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAI9D;;OAEG;cACa,UAAU,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IA4BrD;;OAEG;IACU,YAAY,CAAC,EAAE,EAAE,MAAM,EAAE,QAAQ,EAAE,GAAG,GAAG,OAAO,CAAC,IAAI,CAAC;IA6DnE;;OAEG;IACU,WAAW,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,GAAG,GAAG,IAAI,CAAC;IA6DzD;;OAEG;IACU,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;IAsDnC;;OAEG;IACU,gBAAgB,IAAI,OAAO,CAAC;QACrC,IAAI,EAAE,MAAM,CAAA;QACZ,IAAI,EAAE,MAAM,CAAA;QACZ,KAAK,EAAE,MAAM,GAAG,IAAI,CAAA;QACpB,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAA;KAChC,CAAC;IAwJF,SAAS,CAAC,4BAA4B,EAAE,MAAM,CAAC,OAAO,GAAG,IAAI,CAAO;IAEpE,SAAS,CAAC,kBAAkB,UAAQ;IAEpC,SAAS,CAAC,uBAAuB,SAAI;IAErC,SAAS,CAAC,QAAQ,CAAC,qBAAqB,QAAO;IAE/C,SAAS,CAAC,QAAQ,CAAC,kBAAkB,SAAQ;IAE7C;;;;OAIG;IACH,OAAO,CAAC,uBAAuB;IAO/B;;;OAGG;IACH,OAAO,CAAC,uBAAuB;IAI/B;;;OAGG;IACH,OAAO,CAAC,sBAAsB;IAI9B;;OAEG;IACH,SAAS,CAAC,mBAAmB,IAAI,IAAI;IAwBrC;;OAEG;cACa,eAAe,IAAI,OAAO,CAAC,IAAI,CAAC;IAoGhD;;;;;OAKG;IACH,OAAO,CAAC,eAAe;IA4BvB;;;OAGG;cACa,kBAAkB,CAAC,UAAU,EAAE,cAAc,GAAG,OAAO,CAAC,IAAI,CAAC;IAqB7E;;;OAGG;cACa,iBAAiB,IAAI,OAAO,CAAC,cAAc,GAAG,IAAI,CAAC;IAuDnE;;;;OAIG;YACW,uBAAuB;IAyCrC;;;OAGG;YACW,iBAAiB;IAmC/B;;;;;OAKG;IACU,eAAe,CAAC,cAAc,EAAE,MAAM,EAAE,UAAU,GAAE,MAAa,GAAG,OAAO,CAAC,cAAc,EAAE,CAAC;IA4D1G;;;OAGG;IACU,oBAAoB,CAAC,kBAAkB,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IA6D5E;;;;;OAKG;YACW,WAAW;IAmEzB;;;;;OAKG;YACW,WAAW;IAmDzB;;;OAGG;YACW,mBAAmB;CAkEpC"}
+{"version":3,"file":"s3CompatibleStorage.d.ts","sourceRoot":"","sources":["../../../src/storage/adapters/s3CompatibleStorage.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAEH,OAAO,EAAE,SAAS,EAAE,QAAQ,EAAE,cAAc,EAAE,MAAM,oBAAoB,CAAA;AACxE,OAAO,EACL,WAAW,EAMZ,MAAM,mBAAmB,CAAA;AAC1B,OAAO,EAEL,eAAe,EAChB,MAAM,+BAA+B,CAAA;AAKtC,KAAK,QAAQ,GAAG,QAAQ,CAAA;AACxB,KAAK,IAAI,GAAG,SAAS,CAAA;AAGrB,UAAU,cAAc;IACtB,SAAS,EAAE,MAAM,CAAA;IACjB,SAAS,EAAE,KAAK,GAAG,QAAQ,GAAG,QAAQ,CAAA;IACtC,UAAU,EAAE,MAAM,GAAG,MAAM,GAAG,UAAU,CAAA;IACxC,QAAQ,EAAE,MAAM,CAAA;IAChB,IAAI,CAAC,EAAE,GAAG,CAAA;IACV,UAAU,CAAC,EAAE,MAAM,CAAA;CACpB;AAGD,OAAO,EAAE,mBAAmB,IAAI,SAAS,EAAE,CAAA;AAM3C;;;;;;;;;;;;;;;;;;;;;GAqBG;AACH,qBAAa,mBAAoB,SAAQ,WAAW;IAClD,OAAO,CAAC,QAAQ,CAAwB;IACxC,OAAO,CAAC,UAAU,CAAQ;IAC1B,OAAO,CAAC,WAAW,CAAQ;IAC3B,OAAO,CAAC,MAAM,CAAQ;IACtB,OAAO,CAAC,QAAQ,CAAC,CAAQ;IACzB,OAAO,CAAC,SAAS,CAAC,CAAQ;IAC1B,OAAO,CAAC,WAAW,CAAQ;IAC3B,OAAO,CAAC,eAAe,CAAQ;IAC/B,OAAO,CAAC,YAAY,CAAC,CAAQ;IAG7B,OAAO,CAAC,UAAU,CAAQ;IAC1B,OAAO,CAAC,UAAU,CAAQ;IAC1B,OAAO,CAAC,cAAc,CAAQ;IAC9B,OAAO,CAAC,WAAW,CAAQ;IAG3B,SAAS,CAAC,eAAe,EAAE,cAAc,GAAG,IAAI,CAAO;IAGvD,OAAO,CAAC,UAAU,CAAmB;IACrC,OAAO,CAAC,WAAW,CAAyB;IAG5C,OAAO,CAAC,eAAe,CAAwB;IAG/C,OAAO,CAAC,kBAAkB,CAA2B;IAGrD,OAAO,CAAC,gBAAgB,CAAwB;IAChD,OAAO,CAAC,gBAAgB,CAAoB;IAE5C;;;OAGG;gBACS,OAAO,EAAE;QACnB,UAAU,EAAE,MAAM,CAAA;QAClB,MAAM,CAAC,EAAE,MAAM,CAAA;QACf,QAAQ,CAAC,EAAE,MAAM,CAAA;QACjB,SAAS,CAAC,EAAE,MAAM,CAAA;QAClB,WAAW,EAAE,MAAM,CAAA;QACnB,eAAe,EAAE,MAAM,CAAA;QACvB,YAAY,CAAC,EAAE,MAAM,CAAA;QACrB,WAAW,CAAC,EAAE,MAAM,CAAA;QACpB,eAAe,CAAC,EAAE,eAAe,CAAA;QACjC,WAAW,CAAC,EAAE;YACZ,eAAe,CAAC,EAAE,MAAM,CAAA;YACxB,yBAAyB,CAAC,EAAE,MAAM,CAAA;YAClC,YAAY,CAAC,EAAE,MAAM,CAAA;SACtB,CAAA;KACF;IA2BD;;OAEG;IACU,IAAI,IAAI,OAAO,CAAC,IAAI,CAAC;IA4IlC;;OAEG;cACa,iBAAiB,CAAC,IAAI,EAAE,QAAQ,GAAG,OAAO,CAAC,IAAI,CAAC;IAIhE;;OAEG;cACa,QAAQ,CAAC,IAAI,EAAE,QAAQ,GAAG,OAAO,CAAC,IAAI,CAAC;IA8EvD;;OAEG;cACa,gBAAgB,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,GAAG,IAAI,CAAC;IAItE;;OAEG;cACa,OAAO,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,GAAG,IAAI,CAAC;IAwE7D;;OAEG;cACa,oBAAoB,IAAI,OAAO,CAAC,QAAQ,EAAE,CAAC;IAK3D,OAAO,CAAC,SAAS,CAA8B;IAE/C;;;;OAIG;cACa,WAAW,IAAI,OAAO,CAAC,QAAQ,EAAE,CAAC;IAwBlD;;;;OAIG;cACa,sBAAsB,CAAC,OAAO,GAAE;QAC9C,KAAK,CAAC,EAAE,MAAM,CAAA;QACd,MAAM,CAAC,EAAE,MAAM,CAAA;QACf,QAAQ,CAAC,EAAE,OAAO,CAAA;KACd,GAAG,OAAO,CAAC;QACf,KAAK,EAAE,QAAQ,EAAE,CAAA;QACjB,OAAO,EAAE,OAAO,CAAA;QAChB,UAAU,CAAC,EAAE,MAAM,CAAA;KACpB,CAAC;IAwGF;;;;OAIG;cACa,2BAA2B,CACzC,QAAQ,EAAE,MAAM,GACf,OAAO,CAAC,QAAQ,EAAE,CAAC;IAItB;;;;OAIG;cACa,kBAAkB,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,EAAE,CAAC;IA2CzE;;OAEG;cACa,mBAAmB,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAI9D;;OAEG;cACa,UAAU,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IA4BrD;;OAEG;cACa,iBAAiB,CAAC,IAAI,EAAE,SAAS,GAAG,OAAO,CAAC,IAAI,CAAC;IAIjE;;OAEG;cACa,QAAQ,CAAC,IAAI,EAAE,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC;IA6CnD;;OAEG;cACa,gBAAgB,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,GAAG,IAAI,CAAC;IAIvE;;OAEG;cACa,OAAO,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,GAAG,IAAI,CAAC;IAkGzD;;;;OAIG;cACa,oBAAoB,IAAI,OAAO,CAAC,SAAS,EAAE,CAAC;IAK5D;;;;OAIG;cACa,WAAW,IAAI,OAAO,CAAC,IAAI,EAAE,CAAC;IAwB9C;;;;OAIG;cACa,sBAAsB,CAAC,OAAO,GAAE;QAC9C,KAAK,CAAC,EAAE,MAAM,CAAA;QACd,MAAM,CAAC,EAAE,MAAM,CAAA;QACf,QAAQ,CAAC,EAAE,OAAO,CAAA;QAClB,MAAM,CAAC,EAAE;YACP,QAAQ,CAAC,EAAE,MAAM,CAAA;YACjB,QAAQ,CAAC,EAAE,MAAM,CAAA;YACjB,IAAI,CAAC,EAAE,MAAM,CAAA;SACd,CAAA;KACG,GAAG,OAAO,CAAC;QACf,KAAK,EAAE,IAAI,EAAE,CAAA;QACb,OAAO,EAAE,OAAO,CAAA;QAChB,UAAU,CAAC,EAAE,MAAM,CAAA;KACpB,CAAC;IAiHF;;;;;OAKG;IACH,OAAO,CAAC,UAAU;IA4BlB;;;;OAIG;IACU,sBAAsB,CAAC,OAAO,GAAE;QAC3C,KAAK,CAAC,EAAE,MAAM,CAAA;QACd,MAAM,CAAC,EAAE,MAAM,CAAA;QACf,MAAM,CAAC,EAAE;YACP,QAAQ,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,CAAA;YAC5B,QAAQ,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,CAAA;YAC5B,QAAQ,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,CAAA;YAC5B,OAAO,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,CAAA;YAC3B,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAA;SAC/B,CAAA;KACG,GAAG,OAAO,CAAC;QACf,KAAK,EAAE,SAAS,EAAE,CAAA;QAClB,UAAU,CAAC,EAAE,MAAM,CAAA;QACnB,OAAO,EAAE,OAAO,CAAA;QAChB,UAAU,CAAC,EAAE,MAAM,CAAA;KACpB,CAAC;IAiDF;;OAEG;cACa,yBAAyB,CACvC,QAAQ,EAAE,MAAM,GACf,OAAO,CAAC,SAAS,EAAE,CAAC;IAIvB;;OAEG;cACa,gBAAgB,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,EAAE,CAAC;IAKnE;;OAEG;cACa,yBAAyB,CACvC,QAAQ,EAAE,MAAM,GACf,OAAO,CAAC,SAAS,EAAE,CAAC;IAIvB;;OAEG;cACa,gBAAgB,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,EAAE,CAAC;IAKnE;;OAEG;cACa,uBAAuB,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC;IAI3E;;OAEG;cACa,cAAc,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,EAAE,CAAC;IAK7D;;OAEG;cACa,mBAAmB,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAI9D;;OAEG;cACa,UAAU,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IA4BrD;;OAEG;IACU,YAAY,CAAC,EAAE,EAAE,MAAM,EAAE,QAAQ,EAAE,GAAG,GAAG,OAAO,CAAC,IAAI,CAAC;IAoEnE;;OAEG;IACU,WAAW,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,GAAG,GAAG,IAAI,CAAC;IA+DzD;;OAEG;IACU,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;IA0DnC;;OAEG;IACU,gBAAgB,IAAI,OAAO,CAAC;QACvC,IAAI,EAAE,MAAM,CAAA;QACZ,IAAI,EAAE,MAAM,CAAA;QACZ,KAAK,EAAE,MAAM,GAAG,IAAI,CAAA;QACpB,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAA;KAC9B,CAAC;IA+KF,SAAS,CAAC,4BAA4B,EAAE,MAAM,CAAC,OAAO,GAAG,IAAI,CAAO;IAEpE,SAAS,CAAC,kBAAkB,UAAQ;IAEpC,SAAS,CAAC,uBAAuB,SAAI;IAErC,SAAS,CAAC,QAAQ,CAAC,qBAAqB,QAAO;IAE/C,SAAS,CAAC,QAAQ,CAAC,kBAAkB,SAAQ;IAE7C;;;;OAIG;IACH,OAAO,CAAC,uBAAuB;IAO/B;;;OAGG;IACH,OAAO,CAAC,uBAAuB;IAI/B;;;OAGG;IACH,OAAO,CAAC,sBAAsB;IAI9B;;OAEG;IACH,SAAS,CAAC,mBAAmB,IAAI,IAAI;IAyBrC;;OAEG;cACa,eAAe,IAAI,OAAO,CAAC,IAAI,CAAC;IA4GhD;;;;;OAKG;IACH,OAAO,CAAC,eAAe;IAgDvB;;;OAGG;cACa,kBAAkB,CAChC,UAAU,EAAE,cAAc,GACzB,OAAO,CAAC,IAAI,CAAC;IAqBhB;;;OAGG;cACa,iBAAiB,IAAI,OAAO,CAAC,cAAc,GAAG,IAAI,CAAC;IAuDnE;;;;OAIG;YACW,uBAAuB;IA0CrC;;;OAGG;YACW,iBAAiB;IAmC/B;;;;;OAKG;IACU,eAAe,CAC1B,cAAc,EAAE,MAAM,EACtB,UAAU,GAAE,MAAa,GACxB,OAAO,CAAC,cAAc,EAAE,CAAC;IA8D5B;;;OAGG;IACU,oBAAoB,CAAC,kBAAkB,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAiE5E;;;;;OAKG;YACW,WAAW;IA0EzB;;;;;OAKG;YACW,WAAW;IA0DzB;;;OAGG;YACW,mBAAmB;CAmElC"}

package/dist/storage/baseStorage.d.ts

@@ -34,6 +34,8 @@ export declare abstract class BaseStorage extends BaseStorageAdapter {
     getNoun(id: string): Promise<HNSWNoun | null>;
     /**
      * Get all nouns from storage
+     * @deprecated This method is deprecated and will be removed in a future version.
+     * It can cause memory issues with large datasets. Use getNouns() with pagination instead.
      */
     getAllNouns(): Promise<HNSWNoun[]>;
     /**
@@ -56,6 +58,8 @@ export declare abstract class BaseStorage extends BaseStorageAdapter {
     getVerb(id: string): Promise<GraphVerb | null>;
     /**
      * Get all verbs from storage
+     * @deprecated This method is deprecated and will be removed in a future version.
+     * It can cause memory issues with large datasets. Use getVerbs() with pagination instead.
      */
     getAllVerbs(): Promise<GraphVerb[]>;
     /**

package/dist/storage/baseStorage.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"baseStorage.d.ts","sourceRoot":"","sources":["../../src/storage/baseStorage.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAEH,OAAO,EAAE,SAAS,EAAE,QAAQ,EAAE,cAAc,EAAE,MAAM,iBAAiB,CAAA;AACrE,OAAO,EAAE,kBAAkB,EAAE,MAAM,kCAAkC,CAAA;AAGrE,eAAO,MAAM,SAAS,UAAU,CAAA;AAChC,eAAO,MAAM,SAAS,UAAU,CAAA;AAChC,eAAO,MAAM,YAAY,aAAa,CAAA;AACtC,eAAO,MAAM,SAAS,UAAU,CAAA;AAChC,eAAO,MAAM,cAAc,eAAe,CAAA;AAE1C;;;GAGG;AACH,8BAAsB,WAAY,SAAQ,kBAAkB;IAC1D,SAAS,CAAC,aAAa,UAAQ;IAE/B;;;OAGG;aACa,IAAI,IAAI,OAAO,CAAC,IAAI,CAAC;IAErC;;OAEG;cACa,iBAAiB,IAAI,OAAO,CAAC,IAAI,CAAC;IAMlD;;OAEG;IACU,QAAQ,CAAC,IAAI,EAAE,QAAQ,GAAG,OAAO,CAAC,IAAI,CAAC;IAKpD;;OAEG;IACU,OAAO,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,GAAG,IAAI,CAAC;IAK1D;;OAEG;IACU,WAAW,IAAI,OAAO,CAAC,QAAQ,EAAE,CAAC;IAK/C;;;;OAIG;IACU,kBAAkB,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,EAAE,CAAC;IAKtE;;OAEG;IACU,UAAU,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAKlD;;OAEG;IACU,QAAQ,CAAC,IAAI,EAAE,SAAS,GAAG,OAAO,CAAC,IAAI,CAAC;IAKrD;;OAEG;IACU,OAAO,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,GAAG,IAAI,CAAC;IAK3D;;OAEG;IACU,WAAW,IAAI,OAAO,CAAC,SAAS,EAAE,CAAC;IAKhD;;OAEG;IACU,gBAAgB,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC;IAKrE;;OAEG;IACU,gBAAgB,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC;IAKrE;;OAEG;IACU,cAAc,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC;IAK/D;;;;OAIG;IACU,QAAQ,CAAC,OAAO,CAAC,EAAE;QAC9B,UAAU,CAAC,EAAE;YACX,MAAM,CAAC,EAAE,MAAM,CAAA;YACf,KAAK,CAAC,EAAE,MAAM,CAAA;YACd,MAAM,CAAC,EAAE,MAAM,CAAA;SAChB,CAAA;QACD,MAAM,CAAC,EAAE;YACP,QAAQ,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,CAAA;YAC5B,OAAO,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,CAAA;YAC3B,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAA;SAC/B,CAAA;KACF,GAAG,OAAO,CAAC;QACV,KAAK,EAAE,QAAQ,EAAE,CAAA;QACjB,UAAU,CAAC,EAAE,MAAM,CAAA;QACnB,OAAO,EAAE,OAAO,CAAA;QAChB,UAAU,CAAC,EAAE,MAAM,CAAA;KACpB,CAAC;IAiIF;;;;OAIG;IACU,QAAQ,CAAC,OAAO,CAAC,EAAE;QAC9B,UAAU,CAAC,EAAE;YACX,MAAM,CAAC,EAAE,MAAM,CAAA;YACf,KAAK,CAAC,EAAE,MAAM,CAAA;YACd,MAAM,CAAC,EAAE,MAAM,CAAA;SAChB,CAAA;QACD,MAAM,CAAC,EAAE;YACP,QAAQ,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,CAAA;YAC5B,QAAQ,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,CAAA;YAC5B,QAAQ,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,CAAA;YAC5B,OAAO,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,CAAA;YAC3B,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAA;SAC/B,CAAA;KACF,GAAG,OAAO,CAAC;QACV,KAAK,EAAE,SAAS,EAAE,CAAA;QAClB,UAAU,CAAC,EAAE,MAAM,CAAA;QACnB,OAAO,EAAE,OAAO,CAAA;QAChB,UAAU,CAAC,EAAE,MAAM,CAAA;KACpB,CAAC;IAsNF;;OAEG;IACU,UAAU,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAKlD;;;OAGG;aACa,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;IAEtC;;;OAGG;aACa,gBAAgB,IAAI,OAAO,CAAC;QAC1C,IAAI,EAAE,MAAM,CAAA;QACZ,IAAI,EAAE,MAAM,CAAA;QACZ,KAAK,EAAE,MAAM,GAAG,IAAI,CAAA;QACpB,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAA;KAC9B,CAAC;IAEF;;;OAGG;aACa,YAAY,CAAC,EAAE,EAAE,MAAM,EAAE,QAAQ,EAAE,GAAG,GAAG,OAAO,CAAC,IAAI,CAAC;IAEtE;;;OAGG;aACa,WAAW,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,GAAG,GAAG,IAAI,CAAC;IAE5D;;;OAGG;IACH,SAAS,CAAC,QAAQ,CAAC,iBAAiB,CAAC,IAAI,EAAE,QAAQ,GAAG,OAAO,CAAC,IAAI,CAAC;IAEnE;;;OAGG;IACH,SAAS,CAAC,QAAQ,CAAC,gBAAgB,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,GAAG,IAAI,CAAC;IAEzE;;;OAGG;IACH,SAAS,CAAC,QAAQ,CAAC,oBAAoB,IAAI,OAAO,CAAC,QAAQ,EAAE,CAAC;IAE9D;;;OAGG;IACH,SAAS,CAAC,QAAQ,CAAC,2BAA2B,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,EAAE,CAAC;IAErF;;;OAGG;IACH,SAAS,CAAC,QAAQ,CAAC,mBAAmB,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAEjE;;;OAGG;IACH,SAAS,CAAC,QAAQ,CAAC,iBAAiB,CAAC,IAAI,EAAE,SAAS,GAAG,OAAO,CAAC,IAAI,CAAC;IAEpE;;;OAGG;IACH,SAAS,CAAC,QAAQ,CAAC,gBAAgB,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,GAAG,IAAI,CAAC;IAE1E;;;OAGG;IACH,SAAS,CAAC,QAAQ,CAAC,oBAAoB,IAAI,OAAO,CAAC,SAAS,EAAE,CAAC;IAE/D;;;OAGG;IACH,SAAS,CAAC,QAAQ,CAAC,yBAAyB,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC;IAEpF;;;OAGG;IACH,SAAS,CAAC,QAAQ,CAAC,yBAAyB,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC;IAEpF;;;OAGG;IACH,SAAS,CAAC,QAAQ,CAAC,uBAAuB,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC;IAE9E;;;OAGG;IACH,SAAS,CAAC,QAAQ,CAAC,mBAAmB,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAEjE;;OAEG;IACH,SAAS,CAAC,WAAW,CAAC,CAAC,SAAS,MAAM,GAAG,MAAM,EAAE,CAAC,EAChD,GAAG,EAAE,GAAG,CAAC,CAAC,EAAE,CAAC,CAAC,EACd,gBAAgB,GAAE,CAAC,KAAK,EAAE,CAAC,KAAK,GAAc,GAC7C,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC;IAQtB;;;;OAIG;IACH,SAAS,CAAC,QAAQ,CAAC,kBAAkB,CAAC,UAAU,EAAE,cAAc,GAAG,OAAO,CAAC,IAAI,CAAC;IAEhF;;;;OAIG;IACH,SAAS,CAAC,QAAQ,CAAC,iBAAiB,IAAI,OAAO,CAAC,cAAc,GAAG,IAAI,CAAC;CACvE"}
+{"version":3,"file":"baseStorage.d.ts","sourceRoot":"","sources":["../../src/storage/baseStorage.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAEH,OAAO,EAAE,SAAS,EAAE,QAAQ,EAAE,cAAc,EAAE,MAAM,iBAAiB,CAAA;AACrE,OAAO,EAAE,kBAAkB,EAAE,MAAM,kCAAkC,CAAA;AAGrE,eAAO,MAAM,SAAS,UAAU,CAAA;AAChC,eAAO,MAAM,SAAS,UAAU,CAAA;AAChC,eAAO,MAAM,YAAY,aAAa,CAAA;AACtC,eAAO,MAAM,SAAS,UAAU,CAAA;AAChC,eAAO,MAAM,cAAc,eAAe,CAAA;AAE1C;;;GAGG;AACH,8BAAsB,WAAY,SAAQ,kBAAkB;IAC1D,SAAS,CAAC,aAAa,UAAQ;IAE/B;;;OAGG;aACa,IAAI,IAAI,OAAO,CAAC,IAAI,CAAC;IAErC;;OAEG;cACa,iBAAiB,IAAI,OAAO,CAAC,IAAI,CAAC;IAMlD;;OAEG;IACU,QAAQ,CAAC,IAAI,EAAE,QAAQ,GAAG,OAAO,CAAC,IAAI,CAAC;IAKpD;;OAEG;IACU,OAAO,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,GAAG,IAAI,CAAC;IAK1D;;;;OAIG;IACU,WAAW,IAAI,OAAO,CAAC,QAAQ,EAAE,CAAC;IAM/C;;;;OAIG;IACU,kBAAkB,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,EAAE,CAAC;IAKtE;;OAEG;IACU,UAAU,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAKlD;;OAEG;IACU,QAAQ,CAAC,IAAI,EAAE,SAAS,GAAG,OAAO,CAAC,IAAI,CAAC;IAKrD;;OAEG;IACU,OAAO,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,GAAG,IAAI,CAAC;IAK3D;;;;OAIG;IACU,WAAW,IAAI,OAAO,CAAC,SAAS,EAAE,CAAC;IAMhD;;OAEG;IACU,gBAAgB,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC;IAKrE;;OAEG;IACU,gBAAgB,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC;IAKrE;;OAEG;IACU,cAAc,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC;IAK/D;;;;OAIG;IACU,QAAQ,CAAC,OAAO,CAAC,EAAE;QAC9B,UAAU,CAAC,EAAE;YACX,MAAM,CAAC,EAAE,MAAM,CAAA;YACf,KAAK,CAAC,EAAE,MAAM,CAAA;YACd,MAAM,CAAC,EAAE,MAAM,CAAA;SAChB,CAAA;QACD,MAAM,CAAC,EAAE;YACP,QAAQ,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,CAAA;YAC5B,OAAO,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,CAAA;YAC3B,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAA;SAC/B,CAAA;KACF,GAAG,OAAO,CAAC;QACV,KAAK,EAAE,QAAQ,EAAE,CAAA;QACjB,UAAU,CAAC,EAAE,MAAM,CAAA;QACnB,OAAO,EAAE,OAAO,CAAA;QAChB,UAAU,CAAC,EAAE,MAAM,CAAA;KACpB,CAAC;IAwLF;;;;OAIG;IACU,QAAQ,CAAC,OAAO,CAAC,EAAE;QAC9B,UAAU,CAAC,EAAE;YACX,MAAM,CAAC,EAAE,MAAM,CAAA;YACf,KAAK,CAAC,EAAE,MAAM,CAAA;YACd,MAAM,CAAC,EAAE,MAAM,CAAA;SAChB,CAAA;QACD,MAAM,CAAC,EAAE;YACP,QAAQ,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,CAAA;YAC5B,QAAQ,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,CAAA;YAC5B,QAAQ,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,CAAA;YAC5B,OAAO,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,CAAA;YAC3B,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAA;SAC/B,CAAA;KACF,GAAG,OAAO,CAAC;QACV,KAAK,EAAE,SAAS,EAAE,CAAA;QAClB,UAAU,CAAC,EAAE,MAAM,CAAA;QACnB,OAAO,EAAE,OAAO,CAAA;QAChB,UAAU,CAAC,EAAE,MAAM,CAAA;KACpB,CAAC;IAoRF;;OAEG;IACU,UAAU,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAKlD;;;OAGG;aACa,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;IAEtC;;;OAGG;aACa,gBAAgB,IAAI,OAAO,CAAC;QAC1C,IAAI,EAAE,MAAM,CAAA;QACZ,IAAI,EAAE,MAAM,CAAA;QACZ,KAAK,EAAE,MAAM,GAAG,IAAI,CAAA;QACpB,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAA;KAC9B,CAAC;IAEF;;;OAGG;aACa,YAAY,CAAC,EAAE,EAAE,MAAM,EAAE,QAAQ,EAAE,GAAG,GAAG,OAAO,CAAC,IAAI,CAAC;IAEtE;;;OAGG;aACa,WAAW,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,GAAG,GAAG,IAAI,CAAC;IAE5D;;;OAGG;IACH,SAAS,CAAC,QAAQ,CAAC,iBAAiB,CAAC,IAAI,EAAE,QAAQ,GAAG,OAAO,CAAC,IAAI,CAAC;IAEnE;;;OAGG;IACH,SAAS,CAAC,QAAQ,CAAC,gBAAgB,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,GAAG,IAAI,CAAC;IAEzE;;;OAGG;IACH,SAAS,CAAC,QAAQ,CAAC,oBAAoB,IAAI,OAAO,CAAC,QAAQ,EAAE,CAAC;IAE9D;;;OAGG;IACH,SAAS,CAAC,QAAQ,CAAC,2BAA2B,CAC5C,QAAQ,EAAE,MAAM,GACf,OAAO,CAAC,QAAQ,EAAE,CAAC;IAEtB;;;OAGG;IACH,SAAS,CAAC,QAAQ,CAAC,mBAAmB,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAEjE;;;OAGG;IACH,SAAS,CAAC,QAAQ,CAAC,iBAAiB,CAAC,IAAI,EAAE,SAAS,GAAG,OAAO,CAAC,IAAI,CAAC;IAEpE;;;OAGG;IACH,SAAS,CAAC,QAAQ,CAAC,gBAAgB,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,GAAG,IAAI,CAAC;IAE1E;;;OAGG;IACH,SAAS,CAAC,QAAQ,CAAC,oBAAoB,IAAI,OAAO,CAAC,SAAS,EAAE,CAAC;IAE/D;;;OAGG;IACH,SAAS,CAAC,QAAQ,CAAC,yBAAyB,CAC1C,QAAQ,EAAE,MAAM,GACf,OAAO,CAAC,SAAS,EAAE,CAAC;IAEvB;;;OAGG;IACH,SAAS,CAAC,QAAQ,CAAC,yBAAyB,CAC1C,QAAQ,EAAE,MAAM,GACf,OAAO,CAAC,SAAS,EAAE,CAAC;IAEvB;;;OAGG;IACH,SAAS,CAAC,QAAQ,CAAC,uBAAuB,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC;IAE9E;;;OAGG;IACH,SAAS,CAAC,QAAQ,CAAC,mBAAmB,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAEjE;;OAEG;IACH,SAAS,CAAC,WAAW,CAAC,CAAC,SAAS,MAAM,GAAG,MAAM,EAAE,CAAC,EAChD,GAAG,EAAE,GAAG,CAAC,CAAC,EAAE,CAAC,CAAC,EACd,gBAAgB,GAAE,CAAC,KAAK,EAAE,CAAC,KAAK,GAAc,GAC7C,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC;IAQtB;;;;OAIG;IACH,SAAS,CAAC,QAAQ,CAAC,kBAAkB,CACnC,UAAU,EAAE,cAAc,GACzB,OAAO,CAAC,IAAI,CAAC;IAEhB;;;;OAIG;IACH,SAAS,CAAC,QAAQ,CAAC,iBAAiB,IAAI,OAAO,CAAC,cAAc,GAAG,IAAI,CAAC;CACvE"}

package/dist/storage/cacheManager.d.ts

@@ -0,0 +1,250 @@
+/**
+ * Multi-level Cache Manager
+ *
+ * Implements a three-level caching strategy:
+ * - Level 1: Hot cache (most accessed nodes) - RAM (automatically detecting and adjusting in each environment)
+ * - Level 2: Warm cache (recent nodes) - OPFS, Filesystem or S3 depending on environment
+ * - Level 3: Cold storage (all nodes) - OPFS, Filesystem or S3 depending on environment
+ */
+import { HNSWNoun, GraphVerb } from '../coreTypes.js';
+declare global {
+    interface Navigator {
+        deviceMemory?: number;
+    }
+    interface WorkerGlobalScope {
+        storage?: {
+            getDirectory?: () => Promise<any>;
+            [key: string]: any;
+        };
+    }
+}
+type HNSWNode = HNSWNoun;
+type Edge = GraphVerb;
+interface CacheStats {
+    hits: number;
+    misses: number;
+    evictions: number;
+    size: number;
+    maxSize: number;
+}
+/**
+ * Multi-level cache manager for efficient data access
+ */
+export declare class CacheManager<T extends HNSWNode | Edge> {
+    private hotCache;
+    private stats;
+    private environment;
+    private warmStorageType;
+    private coldStorageType;
+    private hotCacheMaxSize;
+    private hotCacheEvictionThreshold;
+    private warmCacheTTL;
+    private batchSize;
+    private autoTune;
+    private lastAutoTuneTime;
+    private autoTuneInterval;
+    private storageStatistics;
+    private warmStorage;
+    private coldStorage;
+    /**
+     * Initialize the cache manager
+     * @param options Configuration options
+     */
+    constructor(options?: {
+        hotCacheMaxSize?: number;
+        hotCacheEvictionThreshold?: number;
+        warmCacheTTL?: number;
+        batchSize?: number;
+        autoTune?: boolean;
+        warmStorage?: any;
+        coldStorage?: any;
+    });
+    /**
+     * Detect the current environment
+     */
+    private detectEnvironment;
+    /**
+     * Detect the optimal cache size based on available memory
+     */
+    private detectOptimalCacheSize;
+    /**
+     * Tune cache parameters based on statistics and environment
+     * This method is called periodically if auto-tuning is enabled
+     *
+     * The auto-tuning process:
+     * 1. Retrieves storage statistics if available
+     * 2. Tunes each parameter based on statistics and environment
+     * 3. Logs the tuned parameters if debug is enabled
+     *
+     * Auto-tuning helps optimize cache performance by adapting to:
+     * - The current environment (Node.js, browser, worker)
+     * - Available system resources (memory, CPU)
+     * - Usage patterns (read-heavy vs. write-heavy workloads)
+     * - Cache efficiency (hit/miss ratios)
+     */
+    private tuneParameters;
+    /**
+     * Tune hot cache size based on statistics and environment
+     *
+     * The hot cache size is tuned based on:
+     * 1. Available memory in the current environment
+     * 2. Total number of nodes and edges in the system
+     * 3. Cache hit/miss ratio
+     *
+     * Algorithm:
+     * - Start with a size based on available memory
+     * - If storage statistics are available, consider caching a percentage of total items
+     * - If hit ratio is low, increase the cache size to improve performance
+     * - Ensure a reasonable minimum size to maintain basic functionality
+     */
+    private tuneHotCacheSize;
+    /**
+     * Tune eviction threshold based on statistics
+     *
+     * The eviction threshold determines when items start being evicted from the hot cache.
+     * It is tuned based on:
+     * 1. Cache hit/miss ratio
+     * 2. Operation patterns (read-heavy vs. write-heavy workloads)
+     *
+     * Algorithm:
+     * - Start with a default threshold of 0.8 (80% of max size)
+     * - For high hit ratios, increase the threshold to keep more items in cache
+     * - For low hit ratios, decrease the threshold to evict items more aggressively
+     * - For read-heavy workloads, use a higher threshold
+     * - For write-heavy workloads, use a lower threshold
+     */
+    private tuneEvictionThreshold;
+    /**
+     * Tune warm cache TTL based on statistics
+     *
+     * The warm cache TTL determines how long items remain in the warm cache.
+     * It is tuned based on:
+     * 1. Update frequency from operation statistics
+     *
+     * Algorithm:
+     * - Start with a default TTL of 24 hours
+     * - For frequently updated data, use a shorter TTL
+     * - For rarely updated data, use a longer TTL
+     */
+    private tuneWarmCacheTTL;
+    /**
+     * Tune batch size based on statistics and environment
+     *
+     * The batch size determines how many items are processed in a single batch
+     * for operations like prefetching. It is tuned based on:
+     * 1. Current environment (Node.js, browser, worker)
+     * 2. Available memory
+     * 3. Operation patterns
+     * 4. Cache hit/miss ratio
+     *
+     * Algorithm:
+     * - Start with a default based on the environment
+     * - Adjust based on available memory in browsers
+     * - For bulk-heavy workloads, use a larger batch size
+     * - For high hit ratios, use smaller batches (items likely in cache)
+     * - For low hit ratios, use larger batches (need to fetch more items)
+     */
+    private tuneBatchSize;
+    /**
+     * Detect the appropriate warm storage type based on environment
+     */
+    private detectWarmStorageType;
+    /**
+     * Detect the appropriate cold storage type based on environment
+     */
+    private detectColdStorageType;
+    /**
+     * Initialize warm storage adapter
+     */
+    private initializeWarmStorage;
+    /**
+     * Initialize cold storage adapter
+     */
+    private initializeColdStorage;
+    /**
+     * Get an item from cache, trying each level in order
+     * @param id The item ID
+     * @returns The cached item or null if not found
+     */
+    get(id: string): Promise<T | null>;
+    /**
+     * Get an item from warm cache
+     * @param id The item ID
+     * @returns The cached item or null if not found
+     */
+    private getFromWarmCache;
+    /**
+     * Get an item from cold storage
+     * @param id The item ID
+     * @returns The item or null if not found
+     */
+    private getFromColdStorage;
+    /**
+     * Add an item to hot cache
+     * @param id The item ID
+     * @param item The item to cache
+     */
+    private addToHotCache;
+    /**
+     * Add an item to warm cache
+     * @param id The item ID
+     * @param item The item to cache
+     */
+    private addToWarmCache;
+    /**
+     * Evict items from hot cache based on LRU policy
+     */
+    private evictFromHotCache;
+    /**
+     * Set an item in all cache levels
+     * @param id The item ID
+     * @param item The item to cache
+     */
+    set(id: string, item: T): Promise<void>;
+    /**
+     * Delete an item from all cache levels
+     * @param id The item ID to delete
+     */
+    delete(id: string): Promise<void>;
+    /**
+     * Clear all cache levels
+     */
+    clear(): Promise<void>;
+    /**
+     * Get cache statistics
+     * @returns Cache statistics
+     */
+    getStats(): CacheStats;
+    /**
+     * Prefetch items based on ID patterns or relationships
+     * @param ids Array of IDs to prefetch
+     */
+    prefetch(ids: string[]): Promise<void>;
+    /**
+     * Check if it's time to tune parameters and do so if needed
+     * This is called before operations that might benefit from tuned parameters
+     *
+     * This method serves as a checkpoint for auto-tuning, ensuring that:
+     * 1. Parameters are tuned periodically based on the auto-tune interval
+     * 2. Tuning happens before critical operations that would benefit from optimized parameters
+     * 3. Tuning doesn't happen too frequently, which could impact performance
+     *
+     * By calling this method before get(), getMany(), and prefetch() operations,
+     * we ensure that the cache parameters are optimized for the current workload
+     * without adding unnecessary overhead to every operation.
+     */
+    private checkAndTuneParameters;
+    /**
+     * Get multiple items at once, optimizing for batch retrieval
+     * @param ids Array of IDs to get
+     * @returns Map of ID to item
+     */
+    getMany(ids: string[]): Promise<Map<string, T>>;
+    /**
+     * Set the storage adapters for warm and cold caches
+     * @param warmStorage Warm cache storage adapter
+     * @param coldStorage Cold storage adapter
+     */
+    setStorageAdapters(warmStorage: any, coldStorage: any): void;
+}
+export {};

package/dist/storage/cacheManager.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"cacheManager.d.ts","sourceRoot":"","sources":["../../src/storage/cacheManager.ts"],"names":[],"mappings":"AAAA;;;;;;;GAOG;AAEH,OAAO,EAAE,QAAQ,EAAE,SAAS,EAAE,MAAM,iBAAiB,CAAA;AAKrD,OAAO,CAAC,MAAM,CAAC;IACb,UAAU,SAAS;QACjB,YAAY,CAAC,EAAE,MAAM,CAAC;KACvB;IAED,UAAU,iBAAiB;QACzB,OAAO,CAAC,EAAE;YACR,YAAY,CAAC,EAAE,MAAM,OAAO,CAAC,GAAG,CAAC,CAAC;YAClC,CAAC,GAAG,EAAE,MAAM,GAAG,GAAG,CAAC;SACpB,CAAC;KACH;CACF;AAGD,KAAK,QAAQ,GAAG,QAAQ,CAAA;AACxB,KAAK,IAAI,GAAG,SAAS,CAAA;AAWrB,UAAU,UAAU;IAClB,IAAI,EAAE,MAAM,CAAA;IACZ,MAAM,EAAE,MAAM,CAAA;IACd,SAAS,EAAE,MAAM,CAAA;IACjB,IAAI,EAAE,MAAM,CAAA;IACZ,OAAO,EAAE,MAAM,CAAA;CAChB;AAiBD;;GAEG;AACH,qBAAa,YAAY,CAAC,CAAC,SAAS,QAAQ,GAAG,IAAI;IAEjD,OAAO,CAAC,QAAQ,CAAmC;IAGnD,OAAO,CAAC,KAAK,CAMZ;IAGD,OAAO,CAAC,WAAW,CAAa;IAChC,OAAO,CAAC,eAAe,CAAa;IACpC,OAAO,CAAC,eAAe,CAAa;IAGpC,OAAO,CAAC,eAAe,CAAQ;IAC/B,OAAO,CAAC,yBAAyB,CAAQ;IACzC,OAAO,CAAC,YAAY,CAAQ;IAC5B,OAAO,CAAC,SAAS,CAAQ;IAGzB,OAAO,CAAC,QAAQ,CAAS;IACzB,OAAO,CAAC,gBAAgB,CAAY;IACpC,OAAO,CAAC,gBAAgB,CAAwB;IAChD,OAAO,CAAC,iBAAiB,CAAY;IAGrC,OAAO,CAAC,WAAW,CAAK;IACxB,OAAO,CAAC,WAAW,CAAK;IAExB;;;OAGG;gBACS,OAAO,GAAE;QACnB,eAAe,CAAC,EAAE,MAAM,CAAA;QACxB,yBAAyB,CAAC,EAAE,MAAM,CAAA;QAClC,YAAY,CAAC,EAAE,MAAM,CAAA;QACrB,SAAS,CAAC,EAAE,MAAM,CAAA;QAClB,QAAQ,CAAC,EAAE,OAAO,CAAA;QAClB,WAAW,CAAC,EAAE,GAAG,CAAA;QACjB,WAAW,CAAC,EAAE,GAAG,CAAA;KACb;IAyCN;;OAEG;IACH,OAAO,CAAC,iBAAiB;IAWzB;;OAEG;IACH,OAAO,CAAC,sBAAsB;IA8C9B;;;;;;;;;;;;;;OAcG;YACW,cAAc;IA2C5B;;;;;;;;;;;;;OAaG;IACH,OAAO,CAAC,gBAAgB;IA0CxB;;;;;;;;;;;;;;OAcG;IACH,OAAO,CAAC,qBAAqB;IAiD7B;;;;;;;;;;;OAWG;IACH,OAAO,CAAC,gBAAgB;IA6BxB;;;;;;;;;;;;;;;;OAgBG;IACH,OAAO,CAAC,aAAa;IAwDrB;;OAEG;IACH,OAAO,CAAC,qBAAqB;IAmB7B;;OAEG;IACH,OAAO,CAAC,qBAAqB;IAmB7B;;OAEG;IACH,OAAO,CAAC,qBAAqB;IAM7B;;OAEG;IACH,OAAO,CAAC,qBAAqB;IAM7B;;;;OAIG;IACU,GAAG,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,CAAC,GAAG,IAAI,CAAC;IAuD/C;;;;OAIG;YACW,gBAAgB;IAW9B;;;;OAIG;YACW,kBAAkB;IAWhC;;;;OAIG;IACH,OAAO,CAAC,aAAa;IAkBrB;;;;OAIG;YACW,cAAc;IAa5B;;OAEG;IACH,OAAO,CAAC,iBAAiB;IAsBzB;;;;OAIG;IACU,GAAG,CAAC,EAAE,EAAE,MAAM,EAAE,IAAI,EAAE,CAAC,GAAG,OAAO,CAAC,IAAI,CAAC;IAiBpD;;;OAGG;IACU,MAAM,CAAC,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IA0B9C;;OAEG;IACU,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;IAgCnC;;;OAGG;IACI,QAAQ,IAAI,UAAU;IAI7B;;;OAGG;IACU,QAAQ,CAAC,GAAG,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC,IAAI,CAAC;IAkCnD;;;;;;;;;;;;OAYG;YACW,sBAAsB;IAWpC;;;;OAIG;IACU,OAAO,CAAC,GAAG,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC,GAAG,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC;IAqF5D;;;;OAIG;IACI,kBAAkB,CAAC,WAAW,EAAE,GAAG,EAAE,WAAW,EAAE,GAAG,GAAG,IAAI;CAIpE"}