jexidb 2.0.2 → 2.1.0

package/docs/README.md

@@ -0,0 +1,194 @@
+# JexiDB Documentation
+
+Welcome to the JexiDB documentation! This directory contains comprehensive guides and references for using JexiDB effectively.
+
+## 📚 Documentation Structure
+
+### 🚀 [API Reference](API.md)
+Complete API documentation covering all methods, options, and features:
+- Database constructor and configuration
+- Core methods (insert, update, delete, find)
+- Advanced features (term mapping, bulk operations)
+- Query operators and complex queries
+- Performance optimization tips
+
+### 💡 [Examples](EXAMPLES.md)
+Practical examples and real-world use cases:
+- Basic usage patterns
+- User management systems
+- Product catalogs
+- Blog systems
+- Analytics dashboards
+- Performance optimization techniques
+- Error handling strategies
+
+## 🎯 Quick Start
+
+If you're new to JexiDB, start with these resources:
+
+1. **Installation**: See the main [README.md](../README.md) for installation instructions
+2. **Basic Usage**: Check out the [Basic Usage](EXAMPLES.md#basic-usage) section in Examples
+3. **API Reference**: Browse the [API Reference](API.md) for detailed method documentation
+4. **Advanced Features**: Explore [Term Mapping](API.md#term-mapping) and [Bulk Operations](API.md#bulk-operations)
+
+## 🔍 What's New
+
+### Recent Features
+- **Term Mapping**: Reduce database size by up to 77% for repetitive data
+- **Bulk Operations**: High-performance `iterate()` method for large datasets
+- **Advanced Querying**: Support for complex queries with logical operators
+- **Indexed Query Modes**: Strict and permissive query modes for performance control
+
+### Performance Improvements
+- Streaming operations for memory efficiency
+- Automatic term cleanup
+- Optimized indexing strategies
+- Batch processing capabilities
+
+## 📖 Key Concepts
+
+### Indexes
+Define which fields to keep in memory for fast queries:
+```javascript
+const db = new Database('app.jdb', {
+  fields: {                    // REQUIRED - Define schema
+    id: 'number',
+    name: 'string',
+    email: 'string'
+  },
+  indexes: {                    // OPTIONAL - Only fields you query frequently
+    email: 'string'            // ✅ Login queries
+  }
+})
+```
+
+### Term Mapping
+Optimize storage for repetitive string data:
+```javascript
+const db = new Database('app.jdb', {
+  fields: {                    // REQUIRED - Define schema
+    id: 'number',
+    name: 'string',
+    tags: 'array:string',
+    categories: 'array:string'
+  }
+  // termMapping is now auto-enabled for array:string fields
+})
+```
+
+### Bulk Operations
+Process large datasets efficiently:
+```javascript
+for await (const record of db.iterate({ status: 'pending' })) {
+  record.status = 'processed'
+  record.updatedAt = Date.now()
+}
+```
+
+## 🛠️ Common Patterns
+
+### Database Lifecycle
+```javascript
+const db = new Database('my-app.jdb', options)
+await db.init()        // Initialize
+// ... use database ...
+await db.save()        // Save changes
+await db.destroy()     // Clean up
+```
+
+### Error Handling
+```javascript
+try {
+  await db.init()
+  await db.insert(data)
+  await db.save()
+} catch (error) {
+  console.error('Database error:', error)
+} finally {
+  await db.destroy()
+}
+```
+
+### Query Patterns
+```javascript
+// Simple queries
+const users = await db.find({ status: 'active' })
+
+// Complex queries
+const results = await db.find({
+  age: { '>': 18, '<': 65 },
+  $or: [
+    { role: 'admin' },
+    { role: 'moderator' }
+  ]
+})
+
+// Case-insensitive search
+const results = await db.find(
+  { name: 'john doe' },
+  { caseInsensitive: true }
+)
+```
+
+## 🔧 Configuration Options
+
+### Basic Configuration
+```javascript
+const db = new Database('database.jdb', {
+  create: true,           // Create file if doesn't exist
+  clear: false,           // Clear existing data
+  fields: {                // REQUIRED - Define schema
+    id: 'number',
+    name: 'string'
+  },
+  indexes: {              // OPTIONAL - Only fields you query frequently
+    name: 'string'        // ✅ Search by name
+  },
+  indexedQueryMode: 'permissive'  // Query mode
+})
+```
+
+### Advanced Configuration
+```javascript
+const db = new Database('database.jdb', {
+  fields: {                        // REQUIRED - Define schema
+    id: 'number',
+    name: 'string',
+    tags: 'array:string'
+  },
+  indexes: {                      // OPTIONAL - Only fields you query frequently
+    name: 'string',              // ✅ Search by name
+    tags: 'array:string'         // ✅ Search by tags
+  }
+  // termMapping is now auto-enabled for array:string fields
+})
+```
+
+## 📊 Performance Tips
+
+1. **Use indexed fields** in your queries for best performance
+2. **Enable term mapping** for datasets with repetitive strings
+3. **Use `iterate()`** for bulk operations on large datasets
+4. **Specify fewer indexes** to reduce memory usage
+5. **Use `save()`** strategically to persist changes
+
+## 🆘 Getting Help
+
+- **Issues**: Report bugs and request features on [GitHub Issues](https://github.com/EdenwareApps/jexidb/issues)
+- **Documentation**: Browse this documentation for detailed guides
+- **Examples**: Check out the [Examples](EXAMPLES.md) for practical use cases
+- **API Reference**: Consult the [API Reference](API.md) for method details
+
+## 📝 Contributing
+
+Found an issue with the documentation? Want to add an example?
+
+1. Fork the repository
+2. Make your changes
+3. Submit a pull request
+
+We welcome contributions to improve the documentation!
+
+---
+
+**Happy coding with JexiDB! 🚀**

package/examples/iterate-usage-example.js

@@ -0,0 +1,157 @@
+/**
+ * Example usage of the new iterate() method
+ * Demonstrates bulk update capabilities with streaming performance
+ */
+
+import { Database } from '../src/Database.mjs'
+
+async function demonstrateIterate() {
+  console.log('🚀 JexiDB iterate() Method Demo\n')
+  
+  // Create database with indexing
+  const db = new Database('iterate-demo.jdb', {
+    debugMode: true,
+    termMapping: true,
+    indexedFields: ['category', 'price', 'status']
+  })
+  
+  await db.init()
+  
+  try {
+    // 1. Insert sample data
+    console.log('📝 Inserting sample data...')
+    const sampleData = [
+      { id: 1, name: 'Apple', category: 'fruits', price: 1.50, status: 'active' },
+      { id: 2, name: 'Banana', category: 'fruits', price: 0.80, status: 'active' },
+      { id: 3, name: 'Orange', category: 'fruits', price: 1.20, status: 'inactive' },
+      { id: 4, name: 'Carrot', category: 'vegetables', price: 0.60, status: 'active' },
+      { id: 5, name: 'Broccoli', category: 'vegetables', price: 1.80, status: 'active' },
+      { id: 6, name: 'Lettuce', category: 'vegetables', price: 0.90, status: 'inactive' },
+      { id: 7, name: 'Milk', category: 'dairy', price: 2.50, status: 'active' },
+      { id: 8, name: 'Cheese', category: 'dairy', price: 3.20, status: 'active' },
+      { id: 9, name: 'Yogurt', category: 'dairy', price: 1.80, status: 'inactive' },
+      { id: 10, name: 'Bread', category: 'bakery', price: 2.00, status: 'active' }
+    ]
+    
+    for (const item of sampleData) {
+      await db.insert(item)
+    }
+    
+    console.log(`✅ Inserted ${sampleData.length} records\n`)
+    
+    // 2. Basic iteration without modifications
+    console.log('🔍 Basic iteration - listing all fruits:')
+    for await (const entry of db.iterate({ category: 'fruits' })) {
+      console.log(`  - ${entry.name}: $${entry.price} (${entry.status})`)
+    }
+    console.log()
+    
+    // 3. Bulk price update with progress tracking
+    console.log('💰 Bulk price update - 10% increase for all active items:')
+    let processedCount = 0
+    let modifiedCount = 0
+    
+    for await (const entry of db.iterate(
+      { status: 'active' }, 
+      { 
+        chunkSize: 5,
+        progressCallback: (progress) => {
+          if (progress.processed % 5 === 0) {
+            console.log(`  📊 Progress: ${progress.processed} processed, ${progress.modified} modified`)
+          }
+        }
+      }
+    )) {
+      const oldPrice = entry.price
+      entry.price = Math.round(entry.price * 1.1 * 100) / 100 // 10% increase, rounded to 2 decimals
+      entry.lastUpdated = new Date().toISOString()
+      
+      if (oldPrice !== entry.price) {
+        console.log(`  💵 ${entry.name}: $${oldPrice} → $${entry.price}`)
+      }
+    }
+    
+    console.log(`✅ Price update completed: ${processedCount} processed, ${modifiedCount} modified\n`)
+    
+    // 4. Bulk status change with deletions
+    console.log('🗑️ Bulk operations - removing inactive items and updating status:')
+    const inactiveItems = await db.find({ status: 'inactive' })
+    for (const item of inactiveItems) {
+      console.log(`  🗑️ Deleting inactive item: ${item.name}`)
+      await db.delete({ id: item.id })
+    }
+    
+    // Update remaining items
+    for await (const entry of db.iterate({})) {
+      if (entry.status === 'active') {
+        entry.status = 'available'
+        entry.updatedAt = new Date().toISOString()
+      }
+    }
+    
+    console.log('✅ Bulk operations completed\n')
+    
+    // 5. Verify results
+    console.log('📋 Final results:')
+    const allItems = await db.find({})
+    console.log(`Total items remaining: ${allItems.length}`)
+    
+    for (const item of allItems) {
+      console.log(`  - ${item.name} (${item.category}): $${item.price} [${item.status}]`)
+    }
+    
+    // 6. Performance demonstration with larger dataset
+    console.log('\n⚡ Performance test with larger dataset...')
+    
+    // Insert more data for performance test
+    const startTime = Date.now()
+    for (let i = 11; i <= 1000; i++) {
+      await db.insert({
+        id: i,
+        name: `Product${i}`,
+        category: ['electronics', 'clothing', 'books', 'home'][i % 4],
+        price: Math.random() * 100,
+        status: i % 3 === 0 ? 'inactive' : 'active'
+      })
+    }
+    
+    const insertTime = Date.now() - startTime
+    console.log(`📝 Inserted 990 additional records in ${insertTime}ms`)
+    
+    // Bulk update with performance tracking
+    const updateStartTime = Date.now()
+    let totalProcessed = 0
+    
+    for await (const entry of db.iterate(
+      { status: 'active' },
+      { 
+        chunkSize: 100,
+        progressCallback: (progress) => {
+          if (progress.completed) {
+            totalProcessed = progress.processed
+            console.log(`📊 Final stats: ${progress.processed} processed, ${progress.modified} modified in ${progress.elapsed}ms`)
+          }
+        }
+      }
+    )) {
+      // Add a processing timestamp
+      entry.processedAt = Date.now()
+    }
+    
+    const updateTime = Date.now() - updateStartTime
+    console.log(`⚡ Bulk update completed: ${totalProcessed} records processed in ${updateTime}ms`)
+    console.log(`📈 Performance: ${Math.round(totalProcessed / (updateTime / 1000))} records/second`)
+    
+  } finally {
+    // Clean up
+    await db.close()
+    console.log('\n🧹 Database closed and cleaned up')
+  }
+}
+
+// Run the demonstration
+if (import.meta.url === `file://${process.argv[1]}`) {
+  demonstrateIterate().catch(console.error)
+}
+
+export { demonstrateIterate }

package/examples/simple-iterate-example.js

@@ -0,0 +1,115 @@
+/**
+ * Simple example of the iterate() method
+ * Demonstrates bulk updates with streaming performance
+ */
+
+import { Database } from '../src/Database.mjs'
+
+async function simpleIterateExample() {
+  console.log('🚀 JexiDB iterate() Method - Simple Example\n')
+  
+  // Create database
+  const db = new Database('simple-iterate.jdb', {
+    debugMode: false,
+    termMapping: true,
+    indexedFields: ['category', 'price']
+  })
+  
+  await db.init()
+  
+  try {
+    // 1. Insert sample data
+    console.log('📝 Inserting sample data...')
+    const products = [
+      { id: 1, name: 'Apple', category: 'fruits', price: 1.50, inStock: true },
+      { id: 2, name: 'Banana', category: 'fruits', price: 0.80, inStock: true },
+      { id: 3, name: 'Orange', category: 'fruits', price: 1.20, inStock: false },
+      { id: 4, name: 'Carrot', category: 'vegetables', price: 0.60, inStock: true },
+      { id: 5, name: 'Broccoli', category: 'vegetables', price: 1.80, inStock: true }
+    ]
+    
+    for (const product of products) {
+      await db.insert(product)
+    }
+    
+    console.log(`✅ Inserted ${products.length} products\n`)
+    
+    // 2. Basic iteration - list all fruits
+    console.log('🍎 Listing all fruits:')
+    for await (const entry of db.iterate({ category: 'fruits' })) {
+      console.log(`  - ${entry.name}: $${entry.price} (${entry.inStock ? 'in stock' : 'out of stock'})`)
+    }
+    console.log()
+    
+    // 3. Bulk price update - 10% increase for in-stock items
+    console.log('💰 Bulk price update - 10% increase for in-stock items:')
+    let updatedCount = 0
+    
+    for await (const entry of db.iterate({ inStock: true })) {
+      const oldPrice = entry.price
+      entry.price = Math.round(entry.price * 1.1 * 100) / 100 // 10% increase
+      entry.lastUpdated = new Date().toISOString()
+      
+      if (oldPrice !== entry.price) {
+        console.log(`  💵 ${entry.name}: $${oldPrice} → $${entry.price}`)
+        updatedCount++
+      }
+    }
+    
+    console.log(`✅ Updated ${updatedCount} products\n`)
+    
+    // 4. Bulk status update - mark all vegetables as organic
+    console.log('🌱 Marking all vegetables as organic:')
+    let organicCount = 0
+    
+    for await (const entry of db.iterate({ category: 'vegetables' })) {
+      entry.organic = true
+      entry.certifiedAt = new Date().toISOString()
+      console.log(`  🌱 ${entry.name} is now organic`)
+      organicCount++
+    }
+    
+    console.log(`✅ Marked ${organicCount} vegetables as organic\n`)
+    
+    // 5. Performance test with progress tracking
+    console.log('⚡ Performance test with progress tracking:')
+    let processedCount = 0
+    
+    for await (const entry of db.iterate(
+      {}, // All records
+      { 
+        chunkSize: 2,
+        progressCallback: (progress) => {
+          if (progress.completed) {
+            console.log(`📊 Final stats: ${progress.processed} processed, ${progress.modified} modified in ${progress.elapsed}ms`)
+          }
+        }
+      }
+    )) {
+      // Add processing timestamp
+      entry.processedAt = Date.now()
+      processedCount++
+    }
+    
+    console.log(`⚡ Processed ${processedCount} records\n`)
+    
+    // 6. Verify all changes
+    console.log('📋 Final results:')
+    const allProducts = await db.find({})
+    for (const product of allProducts) {
+      console.log(`  - ${product.name} (${product.category}): $${product.price} ${product.organic ? '🌱' : ''} ${product.inStock ? '✅' : '❌'}`)
+    }
+    
+  } finally {
+    // Clean up
+    await db.close()
+    console.log('\n🧹 Database closed and cleaned up')
+  }
+}
+
+// Run the example
+if (import.meta.url === `file://${process.argv[1]}`) {
+  simpleIterateExample().catch(console.error)
+}
+
+export { simpleIterateExample }

package/jest.config.js

@@ -0,0 +1,24 @@
+export default {
+  testEnvironment: 'node',
+  testMatch: ['**/test/**/*.test.js', '**/test/**/*.test.mjs'],
+  collectCoverage: true,
+  coverageDirectory: 'coverage',
+  coverageReporters: ['text', 'lcov', 'html'],
+  coveragePathIgnorePatterns: [
+    '/node_modules/',
+    '/test/',
+    '/dist/'
+  ],
+  verbose: true,
+  testTimeout: 10000,
+  maxWorkers: '50%', // Allow parallel test execution for better performance
+  detectOpenHandles: true, // Detect open handles to identify resource leaks
+  transform: {
+    '^.+\\.mjs$': 'babel-jest',
+    '^.+\\.js$': 'babel-jest'
+  },
+  transformIgnorePatterns: [
+    'node_modules/(?!(.*\\.mjs$|p-limit|yocto-queue))'
+  ],
+  setupFilesAfterEnv: ['<rootDir>/test/setup.js']
+};

package/package.json

@@ -1,65 +1,77 @@
 {
   "name": "jexidb",
-  "version": "2.0.2",
-  "description": "JexiDB - A fast and reliable local CRUD database for Electron apps with pure JavaScript JSONL architecture",
-  "main": "./dist/index.js",
-  "module": "./src/index.js",
-  "types": "./dist/index.d.ts",
+  "version": "2.1.0",
+  "type": "module",
+  "description": "JexiDB is a pure JS NPM library for managing data on disk efficiently, without the need for a server.",
+  "main": "./dist/Database.cjs",
+  "module": "./src/Database.mjs",
   "exports": {
     ".": {
-      "import": "./src/index.js",
-      "require": "./dist/index.js",
-      "types": "./dist/index.d.ts"
-    },
-    "./package.json": "./package.json"
+      "require": "./dist/Database.cjs",
+      "import": "./src/Database.mjs"
+    }
   },
   "scripts": {
-    "build": "babel src -d dist",
-    "dev": "babel src -d dist --watch",
-    "test": "jest && node scripts/cleanup-after-tests.js",
+    "test": "node scripts/run-tests.js",
     "test:watch": "jest --watch",
-    "test:clean": "node scripts/cleanup-after-tests.js",
-    "test:optimized": "node scripts/optimize-tests.js --performance",
-    "test:parallel": "node scripts/optimize-tests.js --parallel --workers 4",
-    "test:fast": "node scripts/optimize-tests.js --parallel --workers 4 --performance --no-isolation",
-    "prepublishOnly": "npm run build",
-    "generate-test-db": "node scripts/run-test-db.js",
-    "generate-test-db:multiple": "node scripts/run-test-db.js --multiple",
-    "list-test-dbs": "node scripts/list-test-dbs.js",
-    "verify-test-db": "node scripts/verify-test-db.js",
-    "test-optimization": "node scripts/test-optimization.js",
-    "benchmark-compression": "node scripts/benchmark-compression.js",
-    "benchmark-performance": "node scripts/benchmark-performance.js",
-    "generate-docs": "node scripts/generate-docs.js"
+    "test:coverage": "jest --coverage && npm run clean:test-files",
+    "test:legacy": "node --expose-gc test/test.mjs",
+    "clean:test-files": "node scripts/clean-test-files.js",
+    "build": "npx babel src/Database.mjs --plugins @babel/plugin-transform-async-generator-functions --out-file-extension .cjs --out-dir dist",
+    "prepare": "node scripts/prepare.js"
   },
-  "keywords": [
-    "database",
-    "crud",
-    "local",
-    "electron",
-    "javascript",
-    "json",
-    "storage",
-    "jexidb",
-    "jsonl",
-    "nosql"
-  ],
-  "author": "Edenware",
+  "author": "EdenwareApps",
   "license": "MIT",
   "devDependencies": {
-    "@babel/cli": "^7.23.0",
-    "@babel/core": "^7.23.0",
-    "@babel/preset-env": "^7.23.0",
-    "jest": "^29.7.0"
+    "@babel/cli": "^7.25.6",
+    "@babel/core": "^7.25.2",
+    "@babel/plugin-transform-async-generator-functions": "^7.25.4",
+    "@babel/preset-env": "^7.28.3",
+    "@rollup/plugin-commonjs": "^28.0.6",
+    "@rollup/plugin-node-resolve": "^16.0.1",
+    "babel-jest": "^30.0.5",
+    "jest": "^30.1.3",
+    "jest-extended": "^6.0.0",
+    "rollup": "^4.48.1"
+  },
+  "dependencies": {
+    "@valentech/sializer": "^0.3.9",
+    "async-mutex": "^0.5.0",
+    "p-limit": "^6.1.0"
+  },
+  "optionalDependencies": {
+    "fast-deep-equal": "^3.1.3",
+    "fast-json-stringify": "^5.7.0",
+    "msgpack": "^1.0.3"
   },
-  "engines": {
-    "node": ">=14.0.0"
+  "directories": {
+    "test": "test"
   },
-  "files": [
-    "dist",
-    "src",
-    "README.md"
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/EdenwareApps/jexidb.git"
+  },
+  "keywords": [
+    "couchdb",
+    "database",
+    "nosql",
+    "pouchdb",
+    "local-storage",
+    "db",
+    "persistent-storage",
+    "dexiejs",
+    "embedded-database",
+    "data-management",
+    "nedb",
+    "lowdb",
+    "dexie",
+    "offline-database",
+    "simple-database",
+    "fast-database",
+    "jexidb"
   ],
-  "dependencies": {
-  }
+  "bugs": {
+    "url": "https://github.com/EdenwareApps/jexidb/issues"
+  },
+  "homepage": "https://github.com/EdenwareApps/jexidb#readme"
 }

package/scripts/README.md

@@ -0,0 +1,47 @@
+# Scripts
+
+This directory contains utility scripts for the JexiDB project.
+
+## clean-test-files.js
+
+A Node.js script that automatically cleans up test files generated during test execution.
+
+### Usage
+
+```bash
+# Run manually
+node scripts/clean-test-files.js
+
+# Run via npm script
+npm run clean:test-files
+```
+
+### What it cleans
+
+The script removes files matching these patterns:
+- `test-db-*` (with or without extensions)
+- `test-db-*.jdb`
+- `test-db-*.idx.jdb` (contains both index and offsets data)
+- `test-normalize*` (with or without extensions)
+- `test-confusion*` (with or without extensions)
+- `debug-*` (with or without extensions)
+- `test-simple-*`
+- `test-count.jdb`
+
+### Integration with npm test
+
+The cleanup script is automatically executed after running tests:
+
+```bash
+npm test  # Runs jest && npm run clean:test-files
+```
+
+This ensures that test files are automatically cleaned up after each test run, keeping the project directory clean.
+
+### Features
+
+- **Pattern-based cleanup**: Uses regex patterns to identify test files
+- **Safe deletion**: Only deletes files, not directories
+- **Error handling**: Continues execution even if some files can't be deleted
+- **Verbose output**: Shows which files were cleaned
+- **Summary**: Reports total number of files cleaned