@soulcraft/brainy 4.1.3 → 4.2.0

package/CHANGELOG.md

@@ -2,6 +2,11 @@
 
 All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
 
+### [4.1.4](https://github.com/soulcraftlabs/brainy/compare/v4.1.3...v4.1.4) (2025-10-21)
+
+- feat: add import API validation and v4.x migration guide (a1a0576)
+
+
 ### [4.1.3](https://github.com/soulcraftlabs/brainy/compare/v4.1.2...v4.1.3) (2025-10-21)
 
 - perf: make getRelations() pagination consistent and efficient (54d819c)
@@ -223,22 +228,110 @@ $ brainy import ./research-papers --extract-concepts --progress
 
 ### ⚠️ Breaking Changes
 
-**NONE** - v4.0.0 is 100% backward compatible!
+#### 💥 Import API Redesign
+
+The import API has been redesigned for clarity and better feature control. **Old v3.x option names are no longer recognized** and will throw errors.
+
+**What Changed:**
+
+| v3.x Option | v4.x Option | Action Required |
+|-------------|-------------|-----------------|
+| `extractRelationships` | `enableRelationshipInference` | **Rename option** |
+| `autoDetect` | *(removed)* | **Delete option** (always enabled) |
+| `createFileStructure` | `vfsPath` | **Replace** with VFS path |
+| `excelSheets` | *(removed)* | **Delete option** (all sheets processed) |
+| `pdfExtractTables` | *(removed)* | **Delete option** (always enabled) |
+| - | `enableNeuralExtraction` | **Add option** (new in v4.x) |
+| - | `enableConceptExtraction` | **Add option** (new in v4.x) |
+| - | `preserveSource` | **Add option** (new in v4.x) |
+
+**Why These Changes?**
+
+1. **Clearer option names**: `enableRelationshipInference` explicitly indicates AI-powered relationship inference
+2. **Separation of concerns**: Neural extraction, relationship inference, and VFS are now separate, explicit options
+3. **Better defaults**: Auto-detection and AI features are enabled by default
+4. **Reduced confusion**: Removed redundant options like `autoDetect` and format-specific options
+
+**Migration Examples:**
+
+<details>
+<summary>Example 1: Basic Excel Import</summary>
+
+```typescript
+// v3.x (OLD - Will throw error)
+await brain.import('./glossary.xlsx', {
+  extractRelationships: true,
+  createFileStructure: true
+})
+
+// v4.x (NEW - Use this)
+await brain.import('./glossary.xlsx', {
+  enableRelationshipInference: true,
+  vfsPath: '/imports/glossary'
+})
+```
+</details>
+
+<details>
+<summary>Example 2: Full-Featured Import</summary>
 
-All v4.0.0 features are:
+```typescript
+// v3.x (OLD - Will throw error)
+await brain.import('./data.xlsx', {
+  extractRelationships: true,
+  autoDetect: true,
+  createFileStructure: true
+})
+
+// v4.x (NEW - Use this)
+await brain.import('./data.xlsx', {
+  enableNeuralExtraction: true,      // Extract entity names
+  enableRelationshipInference: true, // Infer semantic relationships
+  enableConceptExtraction: true,     // Extract entity types
+  vfsPath: '/imports/data',          // VFS directory
+  preserveSource: true               // Save original file
+})
+```
+</details>
+
+**Error Messages:**
+
+If you use old v3.x options, you'll get a clear error message:
+
+```
+❌ Invalid import options detected (Brainy v4.x breaking changes)
+
+The following v3.x options are no longer supported:
+
+  ❌ extractRelationships
+     → Use: enableRelationshipInference
+     → Why: Option renamed for clarity in v4.x
+
+📖 Migration Guide: https://brainy.dev/docs/guides/migrating-to-v4
+```
+
+**Other v4.0.0 Features (Non-Breaking):**
+
+All other v4.0.0 features are:
 - ✅ Opt-in (lifecycle, compression, batch operations)
 - ✅ Additive (new CLI commands, new methods)
 - ✅ Non-breaking (existing code continues to work)
 
 ### 📝 Migration
 
-**No migration required!** All v4.0.0 features are optional enhancements.
+**Import API migration required** if you use `brain.import()` with the old v3.x option names.
 
-To use new features:
+#### Required Changes:
 1. Update to v4.0.0: `npm install @soulcraft/brainy@4.0.0`
-2. Enable lifecycle policies: `brainy storage lifecycle set`
-3. Use batch operations: `brainy storage batch-delete entities.txt`
-4. See `docs/MIGRATION-V3-TO-V4.md` for full feature documentation
+2. Update import calls to use new option names (see table above)
+3. Test your imports - you'll get clear error messages if you use old options
+
+#### Optional Enhancements:
+- Enable lifecycle policies: `brainy storage lifecycle set`
+- Use batch operations: `brainy storage batch-delete entities.txt`
+- See full migration guide: `docs/guides/migrating-to-v4.md`
+
+**Complete Migration Guide:** [docs/guides/migrating-to-v4.md](./docs/guides/migrating-to-v4.md)
 
 ### 🎓 What This Means
 

package/dist/brainy.d.ts

@@ -686,33 +686,91 @@ export declare class Brainy<T = any> implements BrainyInterface<T> {
         limit?: number;
     }): Promise<string[]>;
     /**
-     * Import files with auto-detection and dual storage (VFS + Knowledge Graph)
+     * Import files with intelligent extraction and dual storage (VFS + Knowledge Graph)
      *
      * Unified import system that:
      * - Auto-detects format (Excel, PDF, CSV, JSON, Markdown)
-     * - Extracts entities and relationships
+     * - Extracts entities with AI-powered name/type detection
+     * - Infers semantic relationships from context
      * - Stores in both VFS (organized files) and Knowledge Graph (connected entities)
      * - Links VFS files to graph entities
      *
-     * @example
-     * // Import from file path
-     * const result = await brain.import('/path/to/file.xlsx')
+     * @since 4.0.0
      *
-     * @example
-     * // Import from buffer
+     * @example Quick Start (All AI features enabled by default)
+     * ```typescript
+     * const result = await brain.import('./glossary.xlsx')
+     * // Auto-detects format, extracts entities, infers relationships
+     * ```
+     *
+     * @example Full-Featured Import (v4.x)
+     * ```typescript
+     * const result = await brain.import('./data.xlsx', {
+     *   // AI features
+     *   enableNeuralExtraction: true,      // Extract entity names/metadata
+     *   enableRelationshipInference: true, // Detect semantic relationships
+     *   enableConceptExtraction: true,     // Extract types/concepts
+     *
+     *   // VFS features
+     *   vfsPath: '/imports/my-data',       // Store in VFS directory
+     *   groupBy: 'type',                   // Organize by entity type
+     *   preserveSource: true,              // Keep original file
+     *
+     *   // Progress tracking
+     *   onProgress: (p) => console.log(p.message)
+     * })
+     * ```
+     *
+     * @example Performance Tuning (Large Files)
+     * ```typescript
+     * const result = await brain.import('./huge-file.csv', {
+     *   enableDeduplication: false,  // Skip dedup for speed
+     *   confidenceThreshold: 0.8,    // Higher threshold = fewer entities
+     *   onProgress: (p) => console.log(`${p.processed}/${p.total}`)
+     * })
+     * ```
+     *
+     * @example Import from Buffer or Object
+     * ```typescript
+     * // From buffer
      * const result = await brain.import(buffer, { format: 'pdf' })
      *
-     * @example
-     * // Import JSON object
+     * // From object
      * const result = await brain.import({ entities: [...] })
+     * ```
      *
-     * @example
-     * // Custom VFS path and grouping
-     * const result = await brain.import(buffer, {
-     *   vfsPath: '/my-imports/data',
-     *   groupBy: 'type',
-     *   onProgress: (progress) => console.log(progress.message)
-     * })
+     * @throws {Error} If invalid options are provided (v4.x breaking changes)
+     *
+     * @see {@link https://brainy.dev/docs/api/import API Documentation}
+     * @see {@link https://brainy.dev/docs/guides/migrating-to-v4 Migration Guide}
+     *
+     * @remarks
+     * **⚠️ Breaking Changes from v3.x:**
+     *
+     * The import API was redesigned in v4.0.0 for clarity and better feature control.
+     * Old v3.x option names are **no longer recognized** and will throw errors.
+     *
+     * **Option Changes:**
+     * - ❌ `extractRelationships` → ✅ `enableRelationshipInference`
+     * - ❌ `createFileStructure` → ✅ `vfsPath: '/your/path'`
+     * - ❌ `autoDetect` → ✅ *(removed - always enabled)*
+     * - ❌ `excelSheets` → ✅ *(removed - all sheets processed)*
+     * - ❌ `pdfExtractTables` → ✅ *(removed - always enabled)*
+     *
+     * **New Options:**
+     * - ✅ `enableNeuralExtraction` - Extract entity names via AI
+     * - ✅ `enableConceptExtraction` - Extract entity types via AI
+     * - ✅ `preserveSource` - Save original file in VFS
+     *
+     * **If you get an error:**
+     * The error message includes migration instructions and examples.
+     * See the complete migration guide for all details.
+     *
+     * **Why these changes?**
+     * - Clearer option names (explicitly describe what they do)
+     * - Separation of concerns (neural, relationships, VFS are separate)
+     * - Better defaults (AI features enabled by default)
+     * - Reduced confusion (removed redundant options)
      */
     import(source: Buffer | string | object, options?: {
         format?: 'excel' | 'pdf' | 'csv' | 'json' | 'markdown';

package/dist/brainy.js

@@ -1593,33 +1593,91 @@ export class Brainy {
         return options?.limit ? concepts.slice(0, options.limit) : concepts;
     }
     /**
-     * Import files with auto-detection and dual storage (VFS + Knowledge Graph)
+     * Import files with intelligent extraction and dual storage (VFS + Knowledge Graph)
      *
      * Unified import system that:
      * - Auto-detects format (Excel, PDF, CSV, JSON, Markdown)
-     * - Extracts entities and relationships
+     * - Extracts entities with AI-powered name/type detection
+     * - Infers semantic relationships from context
      * - Stores in both VFS (organized files) and Knowledge Graph (connected entities)
      * - Links VFS files to graph entities
      *
-     * @example
-     * // Import from file path
-     * const result = await brain.import('/path/to/file.xlsx')
+     * @since 4.0.0
      *
-     * @example
-     * // Import from buffer
+     * @example Quick Start (All AI features enabled by default)
+     * ```typescript
+     * const result = await brain.import('./glossary.xlsx')
+     * // Auto-detects format, extracts entities, infers relationships
+     * ```
+     *
+     * @example Full-Featured Import (v4.x)
+     * ```typescript
+     * const result = await brain.import('./data.xlsx', {
+     *   // AI features
+     *   enableNeuralExtraction: true,      // Extract entity names/metadata
+     *   enableRelationshipInference: true, // Detect semantic relationships
+     *   enableConceptExtraction: true,     // Extract types/concepts
+     *
+     *   // VFS features
+     *   vfsPath: '/imports/my-data',       // Store in VFS directory
+     *   groupBy: 'type',                   // Organize by entity type
+     *   preserveSource: true,              // Keep original file
+     *
+     *   // Progress tracking
+     *   onProgress: (p) => console.log(p.message)
+     * })
+     * ```
+     *
+     * @example Performance Tuning (Large Files)
+     * ```typescript
+     * const result = await brain.import('./huge-file.csv', {
+     *   enableDeduplication: false,  // Skip dedup for speed
+     *   confidenceThreshold: 0.8,    // Higher threshold = fewer entities
+     *   onProgress: (p) => console.log(`${p.processed}/${p.total}`)
+     * })
+     * ```
+     *
+     * @example Import from Buffer or Object
+     * ```typescript
+     * // From buffer
      * const result = await brain.import(buffer, { format: 'pdf' })
      *
-     * @example
-     * // Import JSON object
+     * // From object
      * const result = await brain.import({ entities: [...] })
+     * ```
      *
-     * @example
-     * // Custom VFS path and grouping
-     * const result = await brain.import(buffer, {
-     *   vfsPath: '/my-imports/data',
-     *   groupBy: 'type',
-     *   onProgress: (progress) => console.log(progress.message)
-     * })
+     * @throws {Error} If invalid options are provided (v4.x breaking changes)
+     *
+     * @see {@link https://brainy.dev/docs/api/import API Documentation}
+     * @see {@link https://brainy.dev/docs/guides/migrating-to-v4 Migration Guide}
+     *
+     * @remarks
+     * **⚠️ Breaking Changes from v3.x:**
+     *
+     * The import API was redesigned in v4.0.0 for clarity and better feature control.
+     * Old v3.x option names are **no longer recognized** and will throw errors.
+     *
+     * **Option Changes:**
+     * - ❌ `extractRelationships` → ✅ `enableRelationshipInference`
+     * - ❌ `createFileStructure` → ✅ `vfsPath: '/your/path'`
+     * - ❌ `autoDetect` → ✅ *(removed - always enabled)*
+     * - ❌ `excelSheets` → ✅ *(removed - all sheets processed)*
+     * - ❌ `pdfExtractTables` → ✅ *(removed - always enabled)*
+     *
+     * **New Options:**
+     * - ✅ `enableNeuralExtraction` - Extract entity names via AI
+     * - ✅ `enableConceptExtraction` - Extract entity types via AI
+     * - ✅ `preserveSource` - Save original file in VFS
+     *
+     * **If you get an error:**
+     * The error message includes migration instructions and examples.
+     * See the complete migration guide for all details.
+     *
+     * **Why these changes?**
+     * - Clearer option names (explicitly describe what they do)
+     * - Separation of concerns (neural, relationships, VFS are separate)
+     * - Better defaults (AI features enabled by default)
+     * - Reduced confusion (removed redundant options)
      */
     async import(source, options) {
         // Lazy load ImportCoordinator

package/dist/import/FormatDetector.d.ts

@@ -8,7 +8,7 @@
  *
  * NO MOCKS - Production-ready implementation
  */
-export type SupportedFormat = 'excel' | 'pdf' | 'csv' | 'json' | 'markdown';
+export type SupportedFormat = 'excel' | 'pdf' | 'csv' | 'json' | 'markdown' | 'yaml' | 'docx';
 export interface DetectionResult {
     format: SupportedFormat;
     confidence: number;
@@ -54,6 +54,11 @@ export declare class FormatDetector {
      * Check if content looks like CSV
      */
     private looksLikeCSV;
+    /**
+     * Check if content looks like YAML
+     * v4.2.0: Added YAML detection
+     */
+    private looksLikeYAML;
     /**
      * Check if content is text-based (not binary)
      */

package/dist/import/FormatDetector.js

@@ -38,7 +38,11 @@ export class FormatDetector {
             '.csv': 'csv',
             '.json': 'json',
             '.md': 'markdown',
-            '.markdown': 'markdown'
+            '.markdown': 'markdown',
+            '.yaml': 'yaml',
+            '.yml': 'yaml',
+            '.docx': 'docx',
+            '.doc': 'docx'
         };
         const format = extensionMap[ext];
         if (format) {
@@ -63,6 +67,14 @@ export class FormatDetector {
                 evidence: ['Content starts with { or [', 'Valid JSON structure']
             };
         }
+        // YAML detection (v4.2.0)
+        if (this.looksLikeYAML(trimmed)) {
+            return {
+                format: 'yaml',
+                confidence: 0.90,
+                evidence: ['Contains YAML key: value patterns', 'YAML-style indentation']
+            };
+        }
         // Markdown detection
         if (this.looksLikeMarkdown(trimmed)) {
             return {
@@ -233,6 +245,33 @@ export class FormatDetector {
         }
         return false;
     }
+    /**
+     * Check if content looks like YAML
+     * v4.2.0: Added YAML detection
+     */
+    looksLikeYAML(content) {
+        const lines = content.split('\n').filter(l => l.trim()).slice(0, 20);
+        if (lines.length < 2)
+            return false;
+        let yamlIndicators = 0;
+        for (const line of lines) {
+            const trimmed = line.trim();
+            // Check for YAML key: value pattern
+            if (/^[\w-]+:\s/.test(trimmed)) {
+                yamlIndicators++;
+            }
+            // Check for YAML list items (- item)
+            if (/^-\s+\w/.test(trimmed)) {
+                yamlIndicators++;
+            }
+            // Check for YAML document separator (---)
+            if (trimmed === '---' || trimmed === '...') {
+                yamlIndicators += 2;
+            }
+        }
+        // If >50% of lines have YAML indicators, it's likely YAML
+        return yamlIndicators / lines.length > 0.5;
+    }
     /**
      * Check if content is text-based (not binary)
      */

package/dist/import/ImportCoordinator.d.ts

@@ -15,13 +15,23 @@ import { ImportHistory } from './ImportHistory.js';
 import { NounType, VerbType } from '../types/graphTypes.js';
 export interface ImportSource {
     /** Source type */
-    type: 'buffer' | 'path' | 'string' | 'object';
+    type: 'buffer' | 'path' | 'string' | 'object' | 'url';
     /** Source data */
     data: Buffer | string | object;
     /** Optional filename hint */
     filename?: string;
+    /** HTTP headers for URL imports (v4.2.0) */
+    headers?: Record<string, string>;
+    /** Basic authentication for URL imports (v4.2.0) */
+    auth?: {
+        username: string;
+        password: string;
+    };
 }
-export interface ImportOptions {
+/**
+ * Valid import options for v4.x
+ */
+export interface ValidImportOptions {
     /** Force specific format (skip auto-detection) */
     format?: SupportedFormat;
     /** VFS root path for imported files */
@@ -52,9 +62,81 @@ export interface ImportOptions {
     enableHistory?: boolean;
     /** Chunk size for streaming large imports (0 = no streaming) */
     chunkSize?: number;
-    /** Progress callback */
-    onProgress?: (progress: ImportProgress) => void;
+    /**
+     * Progress callback for tracking import progress (v4.2.0+)
+     *
+     * **Streaming Architecture** (always enabled):
+     * - Indexes are flushed periodically during import (adaptive intervals)
+     * - Data is queryable progressively as import proceeds
+     * - `progress.queryable` is `true` after each flush
+     * - Provides crash resilience and live monitoring
+     *
+     * **Adaptive Flush Intervals**:
+     * - <1K entities: Flush every 100 entities (max 10 flushes)
+     * - 1K-10K entities: Flush every 1000 entities (10-100 flushes)
+     * - >10K entities: Flush every 5000 entities (low overhead)
+     *
+     * **Performance**:
+     * - Flush overhead: ~5-50ms per flush (~0.3% total time)
+     * - No configuration needed - works optimally out of the box
+     *
+     * @example
+     * ```typescript
+     * // Monitor import progress with live queries
+     * await brain.import(file, {
+     *   onProgress: async (progress) => {
+     *     console.log(`${progress.processed}/${progress.total}`)
+     *
+     *     // Query data as it's imported!
+     *     if (progress.queryable) {
+     *       const count = await brain.count({ type: 'Product' })
+     *       console.log(`${count} products imported so far`)
+     *     }
+     *   }
+     * })
+     * ```
+     */
+    onProgress?: (progress: ImportProgress) => void | Promise<void>;
 }
+/**
+ * Deprecated import options from v3.x
+ * Using these will cause TypeScript compile errors
+ *
+ * @deprecated These options are no longer supported in v4.x
+ * @see {@link https://brainy.dev/docs/guides/migrating-to-v4 Migration Guide}
+ */
+export interface DeprecatedImportOptions {
+    /**
+     * @deprecated Use `enableRelationshipInference` instead
+     * @see {@link https://brainy.dev/docs/guides/migrating-to-v4 Migration Guide}
+     */
+    extractRelationships?: never;
+    /**
+     * @deprecated Removed in v4.x - auto-detection is now always enabled
+     * @see {@link https://brainy.dev/docs/guides/migrating-to-v4 Migration Guide}
+     */
+    autoDetect?: never;
+    /**
+     * @deprecated Use `vfsPath` to specify the directory path instead
+     * @see {@link https://brainy.dev/docs/guides/migrating-to-v4 Migration Guide}
+     */
+    createFileStructure?: never;
+    /**
+     * @deprecated Removed in v4.x - all sheets are now processed automatically
+     * @see {@link https://brainy.dev/docs/guides/migrating-to-v4 Migration Guide}
+     */
+    excelSheets?: never;
+    /**
+     * @deprecated Removed in v4.x - table extraction is now automatic for PDF imports
+     * @see {@link https://brainy.dev/docs/guides/migrating-to-v4 Migration Guide}
+     */
+    pdfExtractTables?: never;
+}
+/**
+ * Complete import options interface
+ * Combines valid v4.x options with deprecated v3.x options (which cause TypeScript errors)
+ */
+export type ImportOptions = ValidImportOptions & DeprecatedImportOptions;
 export interface ImportProgress {
     stage: 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'relationships' | 'complete';
     /** Phase of import - extraction or relationship building (v3.49.0) */
@@ -70,6 +152,15 @@ export interface ImportProgress {
     throughput?: number;
     /** Estimated time remaining in ms (v3.38.0) */
     eta?: number;
+    /**
+     * Whether data is queryable at this point (v4.2.0+)
+     *
+     * When true, indexes have been flushed and queries will return up-to-date results.
+     * When false, data exists in storage but indexes may not be current (queries may be slower/incomplete).
+     *
+     * Only present during streaming imports with flushInterval > 0.
+     */
+    queryable?: boolean;
 }
 export interface ImportResult {
     /** Import ID for history tracking */
@@ -127,6 +218,8 @@ export declare class ImportCoordinator {
     private csvImporter;
     private jsonImporter;
     private markdownImporter;
+    private yamlImporter;
+    private docxImporter;
     private vfsGenerator;
     constructor(brain: Brainy);
     /**
@@ -139,12 +232,27 @@ export declare class ImportCoordinator {
     getHistory(): ImportHistory;
     /**
      * Import from any source with auto-detection
+     * v4.2.0: Now supports URL imports with authentication
      */
-    import(source: Buffer | string | object, options?: ImportOptions): Promise<ImportResult>;
+    import(source: Buffer | string | object | ImportSource, options?: ImportOptions): Promise<ImportResult>;
     /**
      * Normalize source to ImportSource
+     * v4.2.0: Now async to support URL fetching
      */
     private normalizeSource;
+    /**
+     * Check if value is an ImportSource object
+     */
+    private isImportSource;
+    /**
+     * Check if string is a URL
+     */
+    private isUrl;
+    /**
+     * Fetch content from URL
+     * v4.2.0: Supports authentication and custom headers
+     */
+    private fetchUrl;
     /**
      * Check if string is a file path
      */
@@ -165,4 +273,46 @@ export declare class ImportCoordinator {
      * Normalize extraction result to unified format (Excel-like structure)
      */
     private normalizeExtractionResult;
+    /**
+     * Validate options and reject deprecated v3.x options (v4.0.0+)
+     * Throws clear errors with migration guidance
+     */
+    private validateOptions;
+    /**
+     * Build detailed error message for invalid options
+     * Respects LOG_LEVEL for verbosity (detailed in dev, concise in prod)
+     */
+    private buildValidationErrorMessage;
+    /**
+     * Get progressive flush interval based on CURRENT entity count (v4.2.0+)
+     *
+     * Unlike adaptive intervals (which require knowing total count upfront),
+     * progressive intervals adjust dynamically as import proceeds.
+     *
+     * Thresholds:
+     * - 0-999 entities:   Flush every 100   (frequent updates for better UX)
+     * - 1K-9.9K entities: Flush every 1000  (balanced performance/responsiveness)
+     * - 10K+ entities:    Flush every 5000  (performance focused, minimal overhead)
+     *
+     * Benefits:
+     * - Works with known totals (file imports)
+     * - Works with unknown totals (streaming APIs, database cursors)
+     * - Frequent updates early when user is watching
+     * - Efficient processing later when performance matters
+     * - Low overhead (~0.3% for large imports)
+     * - No configuration required
+     *
+     * Example:
+     * - Import with 50K entities:
+     *   - Flushes at: 100, 200, ..., 900 (9 flushes with interval=100)
+     *   - Interval increases to 1000 at entity #1000
+     *   - Flushes at: 1000, 2000, ..., 9000 (9 more flushes)
+     *   - Interval increases to 5000 at entity #10000
+     *   - Flushes at: 10000, 15000, ..., 50000 (8 more flushes)
+     *   - Total: ~26 flushes = ~1.3s overhead = 0.026% of import time
+     *
+     * @param currentEntityCount - Current number of entities imported so far
+     * @returns Current optimal flush interval
+     */
+    private getProgressiveFlushInterval;
 }