jexidb 2.0.3 → 2.1.1

package/docs/README.md

@@ -1,184 +1,194 @@
-# JexiDB - High-Performance Local JSONL Database
-
-[![Tests](https://img.shields.io/badge/tests-71%2F71%20passing-brightgreen)](https://github.com/jexidb/jexidb)
-[![Performance](https://img.shields.io/badge/performance-10k%20ops%2Fsec-brightgreen)](https://github.com/jexidb/jexidb)
-[![License](https://img.shields.io/badge/license-MIT-blue)](LICENSE)
-
-JexiDB is a high-performance, local JSONL database with intelligent optimizations, real compression, and comprehensive error handling. Perfect for Electron apps, Node.js applications, and any scenario requiring fast local data storage.
-
-## ✨ Features
-
-- 🚀 **High Performance**: 10,000+ ops/sec for bulk operations
-- 🧠 **Intelligent Optimizations**: Adaptive mode switching and query optimization
-- 📦 **Real Compression**: LZ4 and Gzip compression with automatic fallback
-- 🛡️ **Error Recovery**: Comprehensive error handling with automatic recovery
-- 🔍 **Advanced Queries**: MongoDB-style query operators and nested field support
-- 📊 **Performance Monitoring**: Built-in statistics and optimization recommendations
-- 🎯 **Zero Dependencies**: Pure JavaScript implementation
-- 🔧 **Production Ready**: 100% test coverage and comprehensive documentation
-
-## 🚀 Quick Start
-
-### Installation
-
-```bash
-npm install jexidb
-```
-
-### Basic Usage
-
-```javascript
-const { Database } = require('jexidb');
-
-// Create database
-const db = new Database('./data/users.jsonl', {
-  indexes: { id: true, email: true }
-});
-
-// Initialize
-await db.init();
-
-// Insert data
-await db.insert({ id: '1', name: 'John', email: 'john@example.com' });
-
-// Find data
-const user = await db.findOne({ id: '1' });
-const users = await db.find({ name: { $regex: /john/i } });
-
-// Update data
-await db.update({ id: '1' }, { name: 'John Updated' });
-
-// Delete data
-await db.delete({ id: '1' });
-
-// Save and cleanup
-await db.save();
-await db.destroy();
-```
-
-## 📚 Documentation
-
-- [API Reference](docs/API.md) - Complete API documentation
-- [Usage Examples](docs/EXAMPLES.md) - Practical examples and patterns
-- [Performance Guide](docs/PERFORMANCE.md) - Optimization strategies
-- [Migration Guide](docs/MIGRATION.md) - Migrating from other databases
-
-## 🏗️ Architecture
-
-JexiDB is built with a modular architecture:
-
-- **Database**: Main database class with CRUD operations
-- **FileHandler**: Efficient file I/O with batch operations
-- **IndexManager**: Intelligent indexing with multiple strategies
-- **QueryOptimizer**: Query optimization and execution planning
-- **CompressionManager**: Real compression with LZ4 and Gzip
-- **CacheManager**: Intelligent caching with adaptive eviction
-- **ErrorHandler**: Comprehensive error recovery and logging
-- **BackgroundMaintenance**: Non-blocking maintenance operations
-
-## 🎯 Performance
-
-### Benchmarks
-
-| Operation | Throughput | Latency |
-|-----------|------------|---------|
-| Bulk Insert | 10,000 ops/sec | <1ms |
-| Single Insert | 1,000 ops/sec | <5ms |
-| Indexed Query | 5,000 ops/sec | <2ms |
-| Update | 1,000 ops/sec | <10ms |
-| Delete | 1,000 ops/sec | <10ms |
-
-### Compression
-
-- **LZ4**: 30-50% size reduction, very fast
-- **Gzip**: 20-40% size reduction, good compression
-- **Automatic**: Age-based compression strategy
-
-## 🔧 Configuration
-
-```javascript
-const db = new Database('./data.jsonl', {
-  // Indexes
-  indexes: {
-    id: true,
-    email: true,
-    'metadata.created': true
-  },
-  
-  // Behavior
-  markDeleted: true,
-  autoSave: true,
-  validateOnInit: false,
-  backgroundMaintenance: true,
-  
-  // Performance
-  cache: {
-    maxSize: 1000,
-    ttl: 300000
-  },
-  
-  // Compression
-  compression: {
-    hot: { type: 'none', threshold: 7 },
-    warm: { type: 'lz4', threshold: 30 },
-    cold: { type: 'gzip', threshold: Infinity }
-  },
-  
-  // Error handling
-  errorHandler: {
-    logLevel: 'info',
-    enableRecovery: true
-  }
-});
-```
-
-## 🧪 Testing
-
-```bash
-# Run all tests
-npm test
-
-# Run benchmarks
-npm run benchmark-performance
-npm run benchmark-compression
-
-# Generate test data
-npm run generate-test-db
-```
-
-## 📊 Monitoring
-
-```javascript
-// Get comprehensive statistics
-const stats = await db.getStats();
-console.log(`Records: ${stats.recordCount}`);
-console.log(`File size: ${stats.fileSize} bytes`);
-console.log(`Cache hit rate: ${stats.cacheStats.hitRate}%`);
-
-// Get optimization recommendations
-const recommendations = db.getOptimizationRecommendations();
-console.log('Optimization tips:', recommendations);
-```
-
-## 🤝 Contributing
-
-1. Fork the repository
-2. Create a feature branch
-3. Make your changes
-4. Add tests for new functionality
-5. Run the test suite
-6. Submit a pull request
-
-## 📄 License
-
-MIT License - see [LICENSE](LICENSE) for details.
-
-## 🙏 Acknowledgments
-
-- Inspired by JexiDB but with significant improvements
-- Built with performance and reliability in mind
-- Designed for real-world production use
-
----
-
-**JexiDB** - Fast, reliable, and intelligent local database storage.
+# JexiDB Documentation
+
+Welcome to the JexiDB documentation! This directory contains comprehensive guides and references for using JexiDB effectively.
+
+## 📚 Documentation Structure
+
+### 🚀 [API Reference](API.md)
+Complete API documentation covering all methods, options, and features:
+- Database constructor and configuration
+- Core methods (insert, update, delete, find)
+- Advanced features (term mapping, bulk operations)
+- Query operators and complex queries
+- Performance optimization tips
+
+### 💡 [Examples](EXAMPLES.md)
+Practical examples and real-world use cases:
+- Basic usage patterns
+- User management systems
+- Product catalogs
+- Blog systems
+- Analytics dashboards
+- Performance optimization techniques
+- Error handling strategies
+
+## 🎯 Quick Start
+
+If you're new to JexiDB, start with these resources:
+
+1. **Installation**: See the main [README.md](../README.md) for installation instructions
+2. **Basic Usage**: Check out the [Basic Usage](EXAMPLES.md#basic-usage) section in Examples
+3. **API Reference**: Browse the [API Reference](API.md) for detailed method documentation
+4. **Advanced Features**: Explore [Term Mapping](API.md#term-mapping) and [Bulk Operations](API.md#bulk-operations)
+
+## 🔍 What's New
+
+### Recent Features
+- **Term Mapping**: Reduce database size by up to 77% for repetitive data
+- **Bulk Operations**: High-performance `iterate()` method for large datasets
+- **Advanced Querying**: Support for complex queries with logical operators
+- **Indexed Query Modes**: Strict and permissive query modes for performance control
+
+### Performance Improvements
+- Streaming operations for memory efficiency
+- Automatic term cleanup
+- Optimized indexing strategies
+- Batch processing capabilities
+
+## 📖 Key Concepts
+
+### Indexes
+Define which fields to keep in memory for fast queries:
+```javascript
+const db = new Database('app.jdb', {
+  fields: {                    // REQUIRED - Define schema
+    id: 'number',
+    name: 'string',
+    email: 'string'
+  },
+  indexes: {                    // OPTIONAL - Only fields you query frequently
+    email: 'string'            // ✅ Login queries
+  }
+})
+```
+
+### Term Mapping
+Optimize storage for repetitive string data:
+```javascript
+const db = new Database('app.jdb', {
+  fields: {                    // REQUIRED - Define schema
+    id: 'number',
+    name: 'string',
+    tags: 'array:string',
+    categories: 'array:string'
+  }
+  // termMapping is now auto-enabled for array:string fields
+})
+```
+
+### Bulk Operations
+Process large datasets efficiently:
+```javascript
+for await (const record of db.iterate({ status: 'pending' })) {
+  record.status = 'processed'
+  record.updatedAt = Date.now()
+}
+```
+
+## 🛠️ Common Patterns
+
+### Database Lifecycle
+```javascript
+const db = new Database('my-app.jdb', options)
+await db.init()        // Initialize
+// ... use database ...
+await db.save()        // Save changes
+await db.destroy()     // Clean up
+```
+
+### Error Handling
+```javascript
+try {
+  await db.init()
+  await db.insert(data)
+  await db.save()
+} catch (error) {
+  console.error('Database error:', error)
+} finally {
+  await db.destroy()
+}
+```
+
+### Query Patterns
+```javascript
+// Simple queries
+const users = await db.find({ status: 'active' })
+
+// Complex queries
+const results = await db.find({
+  age: { '>': 18, '<': 65 },
+  $or: [
+    { role: 'admin' },
+    { role: 'moderator' }
+  ]
+})
+
+// Case-insensitive search
+const results = await db.find(
+  { name: 'john doe' },
+  { caseInsensitive: true }
+)
+```
+
+## 🔧 Configuration Options
+
+### Basic Configuration
+```javascript
+const db = new Database('database.jdb', {
+  create: true,           // Create file if doesn't exist
+  clear: false,           // Clear existing data
+  fields: {                // REQUIRED - Define schema
+    id: 'number',
+    name: 'string'
+  },
+  indexes: {              // OPTIONAL - Only fields you query frequently
+    name: 'string'        // ✅ Search by name
+  },
+  indexedQueryMode: 'permissive'  // Query mode
+})
+```
+
+### Advanced Configuration
+```javascript
+const db = new Database('database.jdb', {
+  fields: {                        // REQUIRED - Define schema
+    id: 'number',
+    name: 'string',
+    tags: 'array:string'
+  },
+  indexes: {                      // OPTIONAL - Only fields you query frequently
+    name: 'string',              // ✅ Search by name
+    tags: 'array:string'         // ✅ Search by tags
+  }
+  // termMapping is now auto-enabled for array:string fields
+})
+```
+
+## 📊 Performance Tips
+
+1. **Use indexed fields** in your queries for best performance
+2. **Enable term mapping** for datasets with repetitive strings
+3. **Use `iterate()`** for bulk operations on large datasets
+4. **Specify fewer indexes** to reduce memory usage
+5. **Use `save()`** strategically to persist changes
+
+## 🆘 Getting Help
+
+- **Issues**: Report bugs and request features on [GitHub Issues](https://github.com/EdenwareApps/jexidb/issues)
+- **Documentation**: Browse this documentation for detailed guides
+- **Examples**: Check out the [Examples](EXAMPLES.md) for practical use cases
+- **API Reference**: Consult the [API Reference](API.md) for method details
+
+## 📝 Contributing
+
+Found an issue with the documentation? Want to add an example?
+
+1. Fork the repository
+2. Make your changes
+3. Submit a pull request
+
+We welcome contributions to improve the documentation!
+
+---
+
+**Happy coding with JexiDB! 🚀**

package/examples/iterate-usage-example.js

@@ -0,0 +1,157 @@
+/**
+ * Example usage of the new iterate() method
+ * Demonstrates bulk update capabilities with streaming performance
+ */
+
+import { Database } from '../src/Database.mjs'
+
+async function demonstrateIterate() {
+  console.log('🚀 JexiDB iterate() Method Demo\n')
+  
+  // Create database with indexing
+  const db = new Database('iterate-demo.jdb', {
+    debugMode: true,
+    termMapping: true,
+    indexedFields: ['category', 'price', 'status']
+  })
+  
+  await db.init()
+  
+  try {
+    // 1. Insert sample data
+    console.log('📝 Inserting sample data...')
+    const sampleData = [
+      { id: 1, name: 'Apple', category: 'fruits', price: 1.50, status: 'active' },
+      { id: 2, name: 'Banana', category: 'fruits', price: 0.80, status: 'active' },
+      { id: 3, name: 'Orange', category: 'fruits', price: 1.20, status: 'inactive' },
+      { id: 4, name: 'Carrot', category: 'vegetables', price: 0.60, status: 'active' },
+      { id: 5, name: 'Broccoli', category: 'vegetables', price: 1.80, status: 'active' },
+      { id: 6, name: 'Lettuce', category: 'vegetables', price: 0.90, status: 'inactive' },
+      { id: 7, name: 'Milk', category: 'dairy', price: 2.50, status: 'active' },
+      { id: 8, name: 'Cheese', category: 'dairy', price: 3.20, status: 'active' },
+      { id: 9, name: 'Yogurt', category: 'dairy', price: 1.80, status: 'inactive' },
+      { id: 10, name: 'Bread', category: 'bakery', price: 2.00, status: 'active' }
+    ]
+    
+    for (const item of sampleData) {
+      await db.insert(item)
+    }
+    
+    console.log(`✅ Inserted ${sampleData.length} records\n`)
+    
+    // 2. Basic iteration without modifications
+    console.log('🔍 Basic iteration - listing all fruits:')
+    for await (const entry of db.iterate({ category: 'fruits' })) {
+      console.log(`  - ${entry.name}: $${entry.price} (${entry.status})`)
+    }
+    console.log()
+    
+    // 3. Bulk price update with progress tracking
+    console.log('💰 Bulk price update - 10% increase for all active items:')
+    let processedCount = 0
+    let modifiedCount = 0
+    
+    for await (const entry of db.iterate(
+      { status: 'active' }, 
+      { 
+        chunkSize: 5,
+        progressCallback: (progress) => {
+          if (progress.processed % 5 === 0) {
+            console.log(`  📊 Progress: ${progress.processed} processed, ${progress.modified} modified`)
+          }
+        }
+      }
+    )) {
+      const oldPrice = entry.price
+      entry.price = Math.round(entry.price * 1.1 * 100) / 100 // 10% increase, rounded to 2 decimals
+      entry.lastUpdated = new Date().toISOString()
+      
+      if (oldPrice !== entry.price) {
+        console.log(`  💵 ${entry.name}: $${oldPrice} → $${entry.price}`)
+      }
+    }
+    
+    console.log(`✅ Price update completed: ${processedCount} processed, ${modifiedCount} modified\n`)
+    
+    // 4. Bulk status change with deletions
+    console.log('🗑️ Bulk operations - removing inactive items and updating status:')
+    const inactiveItems = await db.find({ status: 'inactive' })
+    for (const item of inactiveItems) {
+      console.log(`  🗑️ Deleting inactive item: ${item.name}`)
+      await db.delete({ id: item.id })
+    }
+    
+    // Update remaining items
+    for await (const entry of db.iterate({})) {
+      if (entry.status === 'active') {
+        entry.status = 'available'
+        entry.updatedAt = new Date().toISOString()
+      }
+    }
+    
+    console.log('✅ Bulk operations completed\n')
+    
+    // 5. Verify results
+    console.log('📋 Final results:')
+    const allItems = await db.find({})
+    console.log(`Total items remaining: ${allItems.length}`)
+    
+    for (const item of allItems) {
+      console.log(`  - ${item.name} (${item.category}): $${item.price} [${item.status}]`)
+    }
+    
+    // 6. Performance demonstration with larger dataset
+    console.log('\n⚡ Performance test with larger dataset...')
+    
+    // Insert more data for performance test
+    const startTime = Date.now()
+    for (let i = 11; i <= 1000; i++) {
+      await db.insert({
+        id: i,
+        name: `Product${i}`,
+        category: ['electronics', 'clothing', 'books', 'home'][i % 4],
+        price: Math.random() * 100,
+        status: i % 3 === 0 ? 'inactive' : 'active'
+      })
+    }
+    
+    const insertTime = Date.now() - startTime
+    console.log(`📝 Inserted 990 additional records in ${insertTime}ms`)
+    
+    // Bulk update with performance tracking
+    const updateStartTime = Date.now()
+    let totalProcessed = 0
+    
+    for await (const entry of db.iterate(
+      { status: 'active' },
+      { 
+        chunkSize: 100,
+        progressCallback: (progress) => {
+          if (progress.completed) {
+            totalProcessed = progress.processed
+            console.log(`📊 Final stats: ${progress.processed} processed, ${progress.modified} modified in ${progress.elapsed}ms`)
+          }
+        }
+      }
+    )) {
+      // Add a processing timestamp
+      entry.processedAt = Date.now()
+    }
+    
+    const updateTime = Date.now() - updateStartTime
+    console.log(`⚡ Bulk update completed: ${totalProcessed} records processed in ${updateTime}ms`)
+    console.log(`📈 Performance: ${Math.round(totalProcessed / (updateTime / 1000))} records/second`)
+    
+  } finally {
+    // Clean up
+    await db.close()
+    console.log('\n🧹 Database closed and cleaned up')
+  }
+}
+
+// Run the demonstration
+if (import.meta.url === `file://${process.argv[1]}`) {
+  demonstrateIterate().catch(console.error)
+}
+
+export { demonstrateIterate }

package/examples/simple-iterate-example.js

@@ -0,0 +1,115 @@
+/**
+ * Simple example of the iterate() method
+ * Demonstrates bulk updates with streaming performance
+ */
+
+import { Database } from '../src/Database.mjs'
+
+async function simpleIterateExample() {
+  console.log('🚀 JexiDB iterate() Method - Simple Example\n')
+  
+  // Create database
+  const db = new Database('simple-iterate.jdb', {
+    debugMode: false,
+    termMapping: true,
+    indexedFields: ['category', 'price']
+  })
+  
+  await db.init()
+  
+  try {
+    // 1. Insert sample data
+    console.log('📝 Inserting sample data...')
+    const products = [
+      { id: 1, name: 'Apple', category: 'fruits', price: 1.50, inStock: true },
+      { id: 2, name: 'Banana', category: 'fruits', price: 0.80, inStock: true },
+      { id: 3, name: 'Orange', category: 'fruits', price: 1.20, inStock: false },
+      { id: 4, name: 'Carrot', category: 'vegetables', price: 0.60, inStock: true },
+      { id: 5, name: 'Broccoli', category: 'vegetables', price: 1.80, inStock: true }
+    ]
+    
+    for (const product of products) {
+      await db.insert(product)
+    }
+    
+    console.log(`✅ Inserted ${products.length} products\n`)
+    
+    // 2. Basic iteration - list all fruits
+    console.log('🍎 Listing all fruits:')
+    for await (const entry of db.iterate({ category: 'fruits' })) {
+      console.log(`  - ${entry.name}: $${entry.price} (${entry.inStock ? 'in stock' : 'out of stock'})`)
+    }
+    console.log()
+    
+    // 3. Bulk price update - 10% increase for in-stock items
+    console.log('💰 Bulk price update - 10% increase for in-stock items:')
+    let updatedCount = 0
+    
+    for await (const entry of db.iterate({ inStock: true })) {
+      const oldPrice = entry.price
+      entry.price = Math.round(entry.price * 1.1 * 100) / 100 // 10% increase
+      entry.lastUpdated = new Date().toISOString()
+      
+      if (oldPrice !== entry.price) {
+        console.log(`  💵 ${entry.name}: $${oldPrice} → $${entry.price}`)
+        updatedCount++
+      }
+    }
+    
+    console.log(`✅ Updated ${updatedCount} products\n`)
+    
+    // 4. Bulk status update - mark all vegetables as organic
+    console.log('🌱 Marking all vegetables as organic:')
+    let organicCount = 0
+    
+    for await (const entry of db.iterate({ category: 'vegetables' })) {
+      entry.organic = true
+      entry.certifiedAt = new Date().toISOString()
+      console.log(`  🌱 ${entry.name} is now organic`)
+      organicCount++
+    }
+    
+    console.log(`✅ Marked ${organicCount} vegetables as organic\n`)
+    
+    // 5. Performance test with progress tracking
+    console.log('⚡ Performance test with progress tracking:')
+    let processedCount = 0
+    
+    for await (const entry of db.iterate(
+      {}, // All records
+      { 
+        chunkSize: 2,
+        progressCallback: (progress) => {
+          if (progress.completed) {
+            console.log(`📊 Final stats: ${progress.processed} processed, ${progress.modified} modified in ${progress.elapsed}ms`)
+          }
+        }
+      }
+    )) {
+      // Add processing timestamp
+      entry.processedAt = Date.now()
+      processedCount++
+    }
+    
+    console.log(`⚡ Processed ${processedCount} records\n`)
+    
+    // 6. Verify all changes
+    console.log('📋 Final results:')
+    const allProducts = await db.find({})
+    for (const product of allProducts) {
+      console.log(`  - ${product.name} (${product.category}): $${product.price} ${product.organic ? '🌱' : ''} ${product.inStock ? '✅' : '❌'}`)
+    }
+    
+  } finally {
+    // Clean up
+    await db.close()
+    console.log('\n🧹 Database closed and cleaned up')
+  }
+}
+
+// Run the example
+if (import.meta.url === `file://${process.argv[1]}`) {
+  simpleIterateExample().catch(console.error)
+}
+
+export { simpleIterateExample }