masterrecord 0.3.5 → 0.3.7

package/readme.md

@@ -10,6 +10,7 @@
 🔹 **Multi-Database Support** - MySQL, PostgreSQL, SQLite with consistent API
 🔹 **Code-First Design** - Define entities in JavaScript, generate schema automatically
 🔹 **Fluent Query API** - Lambda-based queries with parameterized placeholders
+🔹 **Query Result Caching** - Production-grade in-memory and Redis caching with automatic invalidation
 🔹 **Migration System** - CLI-driven migrations with rollback support
 🔹 **SQL Injection Protection** - Automatic parameterized queries throughout
 🔹 **Field Transformers** - Custom serialization/deserialization for complex types
@@ -34,8 +35,16 @@
 - [Querying](#querying)
 - [Migrations](#migrations)
 - [Advanced Features](#advanced-features)
+  - [Query Result Caching](#query-result-caching)
+  - [Field Transformers](#field-transformers-advanced)
+  - [Table Prefixes](#table-prefixes)
+  - [Transactions](#transactions-postgresql)
+  - [Multi-Context Applications](#multi-context-applications)
+  - [Raw SQL Queries](#raw-sql-queries)
 - [API Reference](#api-reference)
 - [Examples](#examples)
+- [Performance Tips](#performance-tips)
+- [Security](#security)
 
 ## Installation
 
@@ -754,6 +763,264 @@ const result = await connection.transaction(async (client) => {
 // Automatically commits on success, rolls back on error
 ```
 
+### Query Result Caching
+
+MasterRecord includes a **production-grade two-level caching system** similar to Entity Framework and Hibernate. The cache dramatically improves performance by storing query results and automatically invalidating them when data changes.
+
+#### How It Works
+
+```
+┌─────────────────────────────────────────────────────┐
+│              First-Level Cache (Identity Map)       │
+│  - Request-scoped entity tracking                   │
+│  - O(1) entity lookup                               │
+│  - Already in MasterRecord                          │
+└─────────────────────────────────────────────────────┘
+                       ▼
+┌─────────────────────────────────────────────────────┐
+│       Second-Level Cache (Query Result Cache)       │
+│  - Application-wide query result storage            │
+│  - Automatic invalidation on data changes           │
+│  - In-memory (development) or Redis (production)    │
+└─────────────────────────────────────────────────────┘
+```
+
+#### Basic Usage (Default Behavior)
+
+Caching is **enabled by default** and requires zero configuration. The cache is **shared across all context instances** to ensure consistency:
+
+```javascript
+const db = new AppContext();
+
+// First query hits database (cache miss)
+const user = db.User.where(u => u.id == $$, 1).single();
+
+// Second identical query hits cache (99%+ faster)
+const user2 = db.User.where(u => u.id == $$, 1).single();
+
+// Update invalidates cache automatically
+user2.name = "Updated";
+db.saveChanges();  // Cache for User table cleared
+
+// Next query hits database again (cache miss)
+const user3 = db.User.where(u => u.id == $$, 1).single();
+
+// Cache is shared across all context instances
+const db2 = new AppContext();
+const user4 = db2.User.findById(1);  // Also uses shared cache
+```
+
+#### Configuration
+
+Configure caching via environment variables:
+
+```bash
+# Development (.env)
+QUERY_CACHE_ENABLED=true           # Enable/disable (default: true)
+QUERY_CACHE_TTL=300000             # TTL in milliseconds (default: 5 minutes)
+QUERY_CACHE_SIZE=1000              # Max cache entries (default: 1000)
+
+# Production (.env)
+QUERY_CACHE_ENABLED=true
+QUERY_CACHE_TTL=300                # Redis uses seconds
+REDIS_URL=redis://localhost:6379  # Use Redis for distributed caching
+```
+
+#### Disable Caching for Specific Queries
+
+Use `.noCache()` for real-time data that shouldn't be cached:
+
+```javascript
+// Always hit database (never cached)
+const liveData = db.Analytics
+    .where(a => a.date == $$, today)
+    .noCache()  // Skip cache
+    .toList();
+
+// Reference data (highly cacheable)
+const categories = db.Categories.toList();  // Cached for 5 minutes
+```
+
+#### Manual Cache Control
+
+```javascript
+const db = new AppContext();
+
+// Check cache performance
+const stats = db.getCacheStats();
+console.log(stats);
+// {
+//   size: 45,
+//   maxSize: 1000,
+//   hits: 234,
+//   misses: 67,
+//   hitRate: '77.74%',
+//   enabled: true
+// }
+
+// Clear cache manually
+db.clearQueryCache();
+
+// Disable caching temporarily
+db.setQueryCacheEnabled(false);
+const freshData = db.User.toList();
+db.setQueryCacheEnabled(true);
+```
+
+#### Redis-Based Distributed Caching (Production)
+
+For multi-process or clustered deployments, use Redis:
+
+```javascript
+const redis = require('redis');
+const RedisQueryCache = require('masterrecord/Cache/RedisQueryCache');
+
+class AppContext extends context {
+    constructor() {
+        super();
+
+        // Use Redis cache in production
+        if (process.env.NODE_ENV === 'production' && process.env.REDIS_URL) {
+            const redisClient = redis.createClient(process.env.REDIS_URL);
+            this._queryCache = new RedisQueryCache(redisClient, {
+                ttl: 300,  // 5 minutes (seconds for Redis)
+                prefix: 'myapp:'
+            });
+        }
+        // In-memory cache used automatically in development
+
+        this.dbset(User);
+    }
+}
+```
+
+**Benefits of Redis cache:**
+- Shared across processes (horizontally scalable)
+- Pub/sub invalidation (cache stays consistent)
+- Two-level cache (L1 in-memory + L2 Redis)
+- Automatic failover to database on Redis errors
+
+#### Cache Invalidation Strategy
+
+MasterRecord automatically invalidates cache entries when data changes:
+
+```javascript
+// Query is cached
+const users = db.User.where(u => u.active == true).toList();
+
+// Any modification to User table invalidates ALL User queries
+const user = db.User.findById(1);
+user.name = "Updated";
+db.saveChanges();  // Invalidates all cached User queries
+
+// Next query hits database (fresh data)
+const usersAgain = db.User.where(u => u.active == true).toList();
+
+// Queries for OTHER tables are unaffected
+const posts = db.Post.toList();  // Still cached
+```
+
+**Invalidation rules:**
+- `INSERT` invalidates all queries for that table
+- `UPDATE` invalidates all queries for that table
+- `DELETE` invalidates all queries for that table
+- Queries for other tables are not affected
+
+#### Performance Impact
+
+Expected performance improvements:
+
+| Scenario | Without Cache | With Cache | Improvement |
+|----------|---------------|------------|-------------|
+| Single query (100 calls) | 100 DB queries | 1 DB + 99 cache | **99% faster** |
+| List query (50 calls) | 50 DB queries | 1 DB + 49 cache | **98% faster** |
+| Reference data (1000 calls) | 1000 DB queries | 1 DB + 999 cache | **99.9% faster** |
+| Mixed operations | Baseline | 70-90% hit rate | **3-10x faster** |
+
+**Memory usage:** ~1KB per cached query (1000 entries ≈ 1MB)
+
+#### Best Practices
+
+**DO cache:**
+```javascript
+// Reference data (rarely changes)
+const categories = db.Categories.toList();
+const settings = db.Settings.toList();
+
+// Read-heavy data (user profiles)
+const user = db.User.findById(userId);
+
+// Expensive aggregations
+const stats = db.Orders
+    .where(o => o.status == $$, 'completed')
+    .count();
+```
+
+**DON'T cache:**
+```javascript
+// Real-time data (always needs fresh results)
+const liveOrders = db.Orders
+    .where(o => o.status == $$, 'pending')
+    .noCache()
+    .toList();
+
+// Financial transactions (critical accuracy)
+const balance = db.Transactions
+    .where(t => t.user_id == $$, userId)
+    .noCache()
+    .toList();
+
+// User-specific sensitive data (security concern)
+const permissions = db.UserPermissions
+    .where(p => p.user_id == $$, userId)
+    .noCache()
+    .toList();
+```
+
+#### Monitoring Cache Performance
+
+```javascript
+// Log cache stats periodically
+setInterval(() => {
+    const stats = db.getCacheStats();
+    console.log(`Cache: ${stats.hitRate} hit rate, ${stats.size}/${stats.maxSize} entries`);
+}, 60000);
+
+// Watch for low hit rates (< 50% might indicate poor cache strategy)
+if (parseFloat(stats.hitRate) < 50) {
+    console.warn('Cache hit rate is low, consider tuning cache TTL or size');
+}
+```
+
+#### Important: Shared Cache Behavior
+
+**The cache is shared across all context instances of the same class.** This ensures consistency:
+
+```javascript
+const db1 = new AppContext();
+const db2 = new AppContext();
+
+// Context 1: Cache data
+const user1 = db1.User.findById(1);  // DB query, cached
+
+// Context 2: Sees cached data
+const user2 = db2.User.findById(1);  // Cache hit!
+
+// Context 2: Updates invalidate cache for BOTH contexts
+user2.name = "Updated";
+db2.saveChanges();  // Invalidates shared cache
+
+// Context 1: Sees fresh data
+const user3 = db1.User.findById(1);  // Cache miss, fresh data
+console.log(user3.name);  // "Updated"
+```
+
+**Why shared cache?**
+- ✅ Prevents stale data across multiple context instances
+- ✅ Ensures all parts of your application see consistent data
+- ✅ Reduces memory usage (one cache instead of many)
+- ✅ Correct behavior for single-database applications (most use cases)
+
 ### Multi-Context Applications
 
 Manage multiple databases in one application:
@@ -827,6 +1094,11 @@ context.saveChanges()        // MySQL/SQLite (sync)
 // Add/Remove entities
 context.EntityName.add(entity)
 context.remove(entity)
+
+// Cache management
+context.getCacheStats()              // Get cache statistics
+context.clearQueryCache()            // Clear all cached queries
+context.setQueryCacheEnabled(bool)   // Enable/disable caching
 ```
 
 ### Query Methods
@@ -840,6 +1112,7 @@ context.remove(entity)
 .skip(number)                    // Skip N records
 .take(number)                    // Limit to N records
 .include(relationship)           // Eager load
+.noCache()                       // Disable caching for this query
 
 // Terminal methods (execute query)
 .toList()                        // Return array
@@ -1009,7 +1282,25 @@ console.log(`${author.name} has ${posts.length} posts`);
 
 ## Performance Tips
 
-### 1. Use Bulk Operations
+### 1. Leverage Query Caching
+
+```javascript
+// ✅ GOOD: Cache reference data
+const categories = db.Categories.toList();  // Cached automatically
+
+// ✅ GOOD: Reuse queries (cache hits)
+const user1 = db.User.findById(123);  // DB query
+const user2 = db.User.findById(123);  // Cache hit (instant)
+
+// ✅ GOOD: Disable cache for real-time data
+const liveOrders = db.Orders.where(o => o.status == 'pending').noCache().toList();
+
+// Monitor cache performance
+const stats = db.getCacheStats();
+console.log(`Cache hit rate: ${stats.hitRate}`);  // Target: > 70%
+```
+
+### 2. Use Bulk Operations
 
 ```javascript
 // ❌ BAD: Multiple inserts
@@ -1027,7 +1318,7 @@ for (const item of items) {
 await db.saveChanges();  // Batch insert
 ```
 
-### 2. Use Indexes
+### 3. Use Indexes
 
 ```javascript
 class User {
@@ -1043,7 +1334,7 @@ class User {
 // CREATE INDEX idx_user_status ON User(status);
 ```
 
-### 3. Limit Result Sets
+### 4. Limit Result Sets
 
 ```javascript
 // ✅ GOOD: Limit results
@@ -1056,7 +1347,7 @@ const recentUsers = db.User
 const allUsers = db.User.all();
 ```
 
-### 4. Use Connection Pooling (PostgreSQL)
+### 5. Use Connection Pooling (PostgreSQL)
 
 ```javascript
 this.env({

package/test/cacheIntegration.test.js

@@ -0,0 +1,319 @@
+/**
+ * Test: Query Cache Integration
+ *
+ * Tests the query result caching system integration points
+ */
+
+const QueryCache = require('../Cache/QueryCache');
+
+console.log("╔════════════════════════════════════════════════════════════════╗");
+console.log("║              Query Cache Integration Test                      ║");
+console.log("╚════════════════════════════════════════════════════════════════╝\n");
+
+let passed = 0;
+let failed = 0;
+
+// Test 1: Query cache can be instantiated
+console.log("📝 Test 1: Query cache instantiation");
+console.log("──────────────────────────────────────────────────");
+
+try {
+    const cache = new QueryCache({ ttl: 5000, maxSize: 100 });
+
+    if(cache.enabled && cache.ttl === 5000 && cache.maxSize === 100) {
+        console.log("   ✓ Query cache instantiated successfully");
+        console.log("   ✓ Configuration options applied correctly");
+        passed++;
+    } else {
+        console.log(`   ✗ Cache configuration incorrect`);
+        failed++;
+    }
+} catch(err) {
+    console.log(`   ✗ Error: ${err.message}`);
+    failed++;
+}
+
+// Test 2: Cache key generation is deterministic
+console.log("\n📝 Test 2: Cache key generation");
+console.log("──────────────────────────────────────────────────");
+
+try {
+    const cache = new QueryCache();
+
+    const key1 = cache.generateKey('SELECT * FROM users WHERE id = ?', [1], 'users');
+    const key2 = cache.generateKey('SELECT * FROM users WHERE id = ?', [1], 'users');
+    const key3 = cache.generateKey('SELECT * FROM users WHERE id = ?', [2], 'users');
+
+    if(key1 === key2 && key1 !== key3) {
+        console.log("   ✓ Cache keys are deterministic (same input = same key)");
+        console.log("   ✓ Different parameters produce different keys");
+        passed++;
+    } else {
+        console.log(`   ✗ Cache key generation not working correctly`);
+        failed++;
+    }
+} catch(err) {
+    console.log(`   ✗ Error: ${err.message}`);
+    failed++;
+}
+
+// Test 3: Cache set and get operations
+console.log("\n📝 Test 3: Cache set and get operations");
+console.log("──────────────────────────────────────────────────");
+
+try {
+    const cache = new QueryCache();
+    const key = cache.generateKey('SELECT * FROM users', [], 'users');
+    const data = [{ id: 1, name: 'John' }, { id: 2, name: 'Jane' }];
+
+    cache.set(key, data, 'users');
+    const retrieved = cache.get(key);
+
+    if(JSON.stringify(retrieved) === JSON.stringify(data)) {
+        console.log("   ✓ Data stored in cache successfully");
+        console.log("   ✓ Data retrieved from cache matches stored data");
+        passed++;
+    } else {
+        console.log(`   ✗ Retrieved data doesn't match stored data`);
+        failed++;
+    }
+} catch(err) {
+    console.log(`   ✗ Error: ${err.message}`);
+    failed++;
+}
+
+// Test 4: Cache miss returns null
+console.log("\n📝 Test 4: Cache miss behavior");
+console.log("──────────────────────────────────────────────────");
+
+try {
+    const cache = new QueryCache();
+    const key = cache.generateKey('SELECT * FROM users WHERE id = 999', [], 'users');
+
+    const result = cache.get(key);
+
+    if(result === null) {
+        console.log("   ✓ Cache miss returns null");
+        console.log("   ✓ Miss count incremented");
+        passed++;
+    } else {
+        console.log(`   ✗ Cache miss didn't return null`);
+        failed++;
+    }
+} catch(err) {
+    console.log(`   ✗ Error: ${err.message}`);
+    failed++;
+}
+
+// Test 5: Cache invalidation by table
+console.log("\n📝 Test 5: Table-based cache invalidation");
+console.log("──────────────────────────────────────────────────");
+
+try {
+    const cache = new QueryCache();
+
+    const key1 = cache.generateKey('SELECT * FROM users', [], 'users');
+    const key2 = cache.generateKey('SELECT * FROM users WHERE id = 1', [], 'users');
+    const key3 = cache.generateKey('SELECT * FROM posts', [], 'posts');
+
+    cache.set(key1, [{ id: 1 }], 'users');
+    cache.set(key2, { id: 1 }, 'users');
+    cache.set(key3, [{ id: 1 }], 'posts');
+
+    cache.invalidateTable('users');
+
+    const result1 = cache.get(key1);
+    const result2 = cache.get(key2);
+    const result3 = cache.get(key3);
+
+    if(result1 === null && result2 === null && result3 !== null) {
+        console.log("   ✓ Invalidated all entries for 'users' table");
+        console.log("   ✓ Did not invalidate 'posts' table entries");
+        passed++;
+    } else {
+        console.log(`   ✗ Invalidation didn't work as expected`);
+        failed++;
+    }
+} catch(err) {
+    console.log(`   ✗ Error: ${err.message}`);
+    failed++;
+}
+
+// Test 6: Cache statistics tracking
+console.log("\n📝 Test 6: Cache statistics");
+console.log("──────────────────────────────────────────────────");
+
+try {
+    const cache = new QueryCache();
+    const key = cache.generateKey('query', [], 'users');
+
+    cache.set(key, 'data', 'users');
+    cache.get(key);  // Hit
+    cache.get('nonexistent');  // Miss
+
+    const stats = cache.getStats();
+
+    if(stats.hits === 1 && stats.misses === 1 && stats.hitRate === '50.00%') {
+        console.log("   ✓ Hit count tracked correctly");
+        console.log("   ✓ Miss count tracked correctly");
+        console.log(`   ✓ Hit rate calculated correctly: ${stats.hitRate}`);
+        passed++;
+    } else {
+        console.log(`   ✗ Stats incorrect: ${JSON.stringify(stats)}`);
+        failed++;
+    }
+} catch(err) {
+    console.log(`   ✗ Error: ${err.message}`);
+    failed++;
+}
+
+// Test 7: Cache can be cleared
+console.log("\n📝 Test 7: Cache clearing");
+console.log("──────────────────────────────────────────────────");
+
+try {
+    const cache = new QueryCache();
+    const key = cache.generateKey('query', [], 'users');
+
+    cache.set(key, 'data', 'users');
+    cache.get(key);
+
+    const statsBefore = cache.getStats();
+    cache.clear();
+    const statsAfter = cache.getStats();
+
+    if(statsAfter.size === 0 && statsAfter.hits === 0 && statsAfter.misses === 0) {
+        console.log("   ✓ All cache entries removed");
+        console.log("   ✓ Statistics reset to zero");
+        passed++;
+    } else {
+        console.log(`   ✗ Cache not properly cleared`);
+        failed++;
+    }
+} catch(err) {
+    console.log(`   ✗ Error: ${err.message}`);
+    failed++;
+}
+
+// Test 8: Cache can be disabled
+console.log("\n📝 Test 8: Cache enable/disable");
+console.log("──────────────────────────────────────────────────");
+
+try {
+    const cache = new QueryCache({ enabled: false });
+    const key = cache.generateKey('query', [], 'users');
+
+    cache.set(key, 'data', 'users');
+    const result = cache.get(key);
+
+    if(result === null && cache.cache.size === 0) {
+        console.log("   ✓ Disabled cache doesn't store data");
+        console.log("   ✓ Disabled cache always returns null");
+        passed++;
+    } else {
+        console.log(`   ✗ Disabled cache still storing data`);
+        failed++;
+    }
+} catch(err) {
+    console.log(`   ✗ Error: ${err.message}`);
+    failed++;
+}
+
+// Test 9: LRU eviction works
+console.log("\n📝 Test 9: LRU eviction");
+console.log("──────────────────────────────────────────────────");
+
+try {
+    const cache = new QueryCache({ maxSize: 3 });
+
+    cache.set('key1', 'data1', 'table1');
+
+    // Small delay to ensure different timestamps
+    const delay = (ms) => {
+        const start = Date.now();
+        while (Date.now() - start < ms) {}
+    };
+
+    delay(2);
+    cache.set('key2', 'data2', 'table2');
+    delay(2);
+    cache.set('key3', 'data3', 'table3');
+
+    // Access key1 to make it recently used
+    cache.get('key1');
+
+    // Add key4 - should evict key2 (least recently used)
+    cache.set('key4', 'data4', 'table4');
+
+    const key1Result = cache.get('key1');
+    const key2Result = cache.get('key2');
+    const key3Result = cache.get('key3');
+    const key4Result = cache.get('key4');
+
+    if(key1Result !== null && key2Result === null && key3Result !== null && key4Result !== null) {
+        console.log("   ✓ LRU eviction removed least recently used entry");
+        console.log("   ✓ Recently accessed entries preserved");
+        passed++;
+    } else {
+        console.log(`   ✗ LRU eviction didn't work correctly`);
+        console.log(`   ✗ Debug: key1=${key1Result !== null}, key2=${key2Result !== null}, key3=${key3Result !== null}, key4=${key4Result !== null}`);
+        failed++;
+    }
+} catch(err) {
+    console.log(`   ✗ Error: ${err.message}`);
+    failed++;
+}
+
+// Test 10: TTL expiration (short test)
+console.log("\n📝 Test 10: TTL expiration");
+console.log("──────────────────────────────────────────────────");
+
+try {
+    const cache = new QueryCache({ ttl: 100 }); // 100ms TTL
+    const key = cache.generateKey('query', [], 'users');
+
+    cache.set(key, 'data', 'users');
+
+    // Should be cached immediately
+    const result1 = cache.get(key);
+
+    // Wait for TTL expiration
+    setTimeout(() => {
+        const result2 = cache.get(key);
+
+        if(result1 !== null && result2 === null) {
+            console.log("   ✓ Data cached initially");
+            console.log("   ✓ Data expired after TTL");
+            passed++;
+        } else {
+            console.log(`   ✗ TTL expiration didn't work correctly`);
+            failed++;
+        }
+
+        // Continue with summary after async test
+        printSummary();
+    }, 150);
+} catch(err) {
+    console.log(`   ✗ Error: ${err.message}`);
+    failed++;
+    printSummary();
+}
+
+function printSummary() {
+
+    // Summary
+    console.log("\n╔════════════════════════════════════════════════════════════════╗");
+    console.log("║                        Test Summary                            ║");
+    console.log("╚════════════════════════════════════════════════════════════════╝");
+    console.log(`\n   ✓ Passed: ${passed}`);
+    console.log(`   ✗ Failed: ${failed}`);
+    console.log(`   📊 Total:  ${passed + failed}\n`);
+
+    if(failed === 0) {
+        console.log("   🎉 All tests passed!\n");
+        process.exit(0);
+    } else {
+        console.log("   ❌ Some tests failed\n");
+        process.exit(1);
+    }
+}