masterrecord 0.3.26 → 0.3.29

package/readme.md

@@ -10,6 +10,11 @@
 🔹 **Multi-Database Support** - MySQL, PostgreSQL, SQLite with consistent API
 🔹 **Code-First Design** - Define entities in JavaScript, generate schema automatically
 🔹 **Fluent Query API** - Lambda-based queries with parameterized placeholders
+🔹 **Active Record Pattern** - Entities with `.save()`, `.delete()`, `.reload()` methods
+🔹 **Entity Serialization** - `.toObject()` and `.toJSON()` with circular reference protection
+🔹 **Lifecycle Hooks** - `beforeSave`, `afterSave`, `beforeDelete`, `afterDelete` hooks
+🔹 **Business Validation** - Built-in validators (required, email, length, pattern, custom)
+🔹 **Bulk Operations** - Efficient `bulkCreate`, `bulkUpdate`, `bulkDelete` APIs
 🔹 **Query Result Caching** - Production-grade in-memory and Redis caching with automatic invalidation
 🔹 **Migration System** - CLI-driven migrations with rollback support
 🔹 **SQL Injection Protection** - Automatic parameterized queries throughout
@@ -33,6 +38,24 @@
 - [Database Configuration](#database-configuration)
 - [Entity Definitions](#entity-definitions)
 - [Querying](#querying)
+- [Entity Serialization](#entity-serialization)
+  - [.toObject()](#toobjectoptions)
+  - [.toJSON()](#tojson)
+- [Entity Instance Methods](#entity-instance-methods)
+  - [.delete()](#delete)
+  - [.reload()](#reload)
+  - [.clone()](#clone)
+- [Query Helper Methods](#query-helper-methods)
+  - [.first()](#first)
+  - [.last()](#last)
+  - [.exists()](#exists)
+  - [.pluck()](#pluckfieldname)
+- [Lifecycle Hooks](#lifecycle-hooks)
+- [Business Logic Validation](#business-logic-validation)
+- [Bulk Operations API](#bulk-operations-api)
+  - [bulkCreate()](#bulkcreateentityname-data)
+  - [bulkUpdate()](#bulkupdateentityname-updates)
+  - [bulkDelete()](#bulkdeleteentityname-ids)
 - [Migrations](#migrations)
 - [Advanced Features](#advanced-features)
   - [Query Result Caching](#query-result-caching)
@@ -45,6 +68,113 @@
 - [Examples](#examples)
 - [Performance Tips](#performance-tips)
 - [Security](#security)
+- [Best Practices](#best-practices-critical)
+
+---
+
+## ⚠️ Best Practices (CRITICAL)
+
+### 1. Creating Entity Instances
+
+**ALWAYS** use `context.Entity.new()` to create new entity instances:
+
+```javascript
+// ✅ CORRECT - Creates proper data instance with getters/setters
+const task = this._qaContext.QaTask.new();
+const annotation = this._qaContext.QaAnnotation.new();
+const project = this._qaContext.QaProject.new();
+
+task.name = "My Task";
+task.status = "active";
+await db.saveChanges();  // ✅ Saves correctly
+
+// ❌ WRONG - Creates schema definition object with function properties
+const task = new QaTask();  // task.name is a FUNCTION, not a property!
+
+task.name = "My Task";  // ❌ Doesn't work - name is a function
+await db.saveChanges();  // ❌ Error: "Type mismatch: Expected string, got function"
+```
+
+**Why?**
+- `new Entity()` creates a **schema definition object** where properties are methods that define the schema
+- `context.Entity.new()` creates a **data instance** with proper getters/setters for storing values
+- Using `new Entity()` causes runtime errors: `"Type mismatch for Entity.field: Expected integer, got function with value undefined"`
+
+**Error Example:**
+```
+Error: INSERT failed: Type mismatch for QaTask.name: Expected string, got function with value undefined
+    at SQLLiteEngine._buildSQLInsertObjectParameterized
+```
+
+**This error means:** You used `new Entity()` instead of `context.Entity.new()`
+
+### 2. Saving Changes - ALWAYS use `await`
+
+**ALWAYS** use `await` when calling `saveChanges()`:
+
+```javascript
+// ✅ CORRECT - Waits for database write to complete
+await this._qaContext.saveChanges();
+
+// ❌ WRONG - Returns immediately without waiting for database write
+this._qaContext.saveChanges();  // Promise never completes!
+```
+
+**Why?**
+- `saveChanges()` is **async** and returns a Promise
+- Without `await`, code continues before database write completes
+- Causes **data loss** - appears successful but nothing saves to database
+- Results in "phantom saves" - data in memory but not persisted
+
+**Symptoms of missing `await`:**
+- API returns success but data not in database
+- Queries after save return old/missing data
+- Intermittent save failures
+- Race conditions
+
+**Repository Pattern - Make Methods Async:**
+```javascript
+// ✅ CORRECT - Async method with await
+async create(entity) {
+    this._qaContext.Entity.add(entity);
+    await this._qaContext.saveChanges();
+    return entity;
+}
+
+// ❌ WRONG - Synchronous method calling async saveChanges
+create(entity) {
+    this._qaContext.Entity.add(entity);
+    this._qaContext.saveChanges();  // No await - returns before save completes!
+    return entity;  // Returns entity with undefined ID
+}
+```
+
+### 3. Quick Reference Card
+
+```javascript
+// Entity Creation
+✅ const user = db.User.new();           // CORRECT
+❌ const user = new User();              // WRONG - creates schema object
+
+// Saving Data
+✅ await db.saveChanges();               // CORRECT - waits for completion
+❌ db.saveChanges();                     // WRONG - fire and forget
+
+// Repository Methods
+✅ async create(entity) {                // CORRECT - async method
+      await db.saveChanges();
+   }
+❌ create(entity) {                      // WRONG - sync method
+      db.saveChanges();                  // No await!
+   }
+
+// Querying (all require await)
+✅ const users = await db.User.toList();  // CORRECT
+✅ const user = await db.User.findById(1); // CORRECT
+❌ const users = db.User.toList();         // WRONG - returns Promise
+```
+
+---
 
 ## Installation
 
@@ -416,7 +546,7 @@ const user = db.User.new();
 user.tags = ['admin', 'moderator'];  // Assign array
 await db.saveChanges();  // Stored as '["admin","moderator"]'
 
-const loaded = db.User.findById(user.id);
+const loaded = await db.User.findById(user.id);
 console.log(loaded.tags);  // ['admin', 'moderator'] - JavaScript array!
 ```
 
@@ -425,19 +555,19 @@ console.log(loaded.tags);  // ['admin', 'moderator'] - JavaScript array!
 ### Basic Queries
 
 ```javascript
-// Find all
-const users = db.User.toList();
+// Find all (requires await)
+const users = await db.User.toList();
 
-// Find by primary key
-const user = db.User.findById(123);
+// Find by primary key (requires await)
+const user = await db.User.findById(123);
 
-// Find single with where clause
-const alice = db.User
+// Find single with where clause (requires await)
+const alice = await db.User
     .where(u => u.email == $$, 'alice@example.com')
     .single();
 
-// Find multiple with conditions
-const adults = db.User
+// Find multiple with conditions (requires await)
+const adults = await db.User
     .where(u => u.age >= $$, 18)
     .toList();
 ```
@@ -447,16 +577,16 @@ const adults = db.User
 **Always use `$$` placeholders** for SQL injection protection:
 
 ```javascript
-// Single parameter
-const user = db.User.where(u => u.id == $$, 123).single();
+// Single parameter (requires await)
+const user = await db.User.where(u => u.id == $$, 123).single();
 
-// Multiple parameters
-const results = db.User
+// Multiple parameters (requires await)
+const results = await db.User
     .where(u => u.age > $$ && u.status == $$, 25, 'active')
     .toList();
 
-// Single $ for OR conditions
-const results = db.User
+// Single $ for OR conditions (requires await)
+const results = await db.User
     .where(u => u.status == $ || u.status == null, 'active')
     .toList();
 ```
@@ -464,22 +594,22 @@ const results = db.User
 ### IN Clauses
 
 ```javascript
-// Array parameter with .includes()
+// Array parameter with .includes() (requires await)
 const ids = [1, 2, 3, 4, 5];
-const users = db.User
+const users = await db.User
     .where(u => $$.includes(u.id), ids)
     .toList();
 
 // Generated SQL: WHERE id IN ($1, $2, $3, $4, $5)
 // PostgreSQL parameters: [1, 2, 3, 4, 5]
 
-// Alternative .any() syntax
-const users = db.User
+// Alternative .any() syntax (requires await)
+const users = await db.User
     .where(u => u.id.any($$), [1, 2, 3])
     .toList();
 
-// Comma-separated strings (auto-splits)
-const users = db.User
+// Comma-separated strings (auto-splits) (requires await)
+const users = await db.User
     .where(u => u.id.any($$), "1,2,3,4,5")
     .toList();
 ```
@@ -498,8 +628,8 @@ if (minAge) {
     query = query.where(u => u.age >= $$, minAge);
 }
 
-// Add sorting and pagination
-const users = query
+// Add sorting and pagination (requires await)
+const users = await query
     .orderBy(u => u.created_at)
     .skip(offset)
     .take(limit)
@@ -509,13 +639,13 @@ const users = query
 ### Ordering
 
 ```javascript
-// Ascending
-const users = db.User
+// Ascending (requires await)
+const users = await db.User
     .orderBy(u => u.name)
     .toList();
 
-// Descending
-const users = db.User
+// Descending (requires await)
+const users = await db.User
     .orderByDescending(u => u.created_at)
     .toList();
 ```
@@ -523,17 +653,17 @@ const users = db.User
 ### Pagination
 
 ```javascript
-// Skip 20, take 10
-const users = db.User
+// Skip 20, take 10 (requires await)
+const users = await db.User
     .orderBy(u => u.id)
     .skip(20)
     .take(10)
     .toList();
 
-// Page-based pagination
+// Page-based pagination (requires await)
 const page = 2;
 const pageSize = 10;
-const users = db.User
+const users = await db.User
     .skip(page * pageSize)
     .take(pageSize)
     .toList();
@@ -542,11 +672,11 @@ const users = db.User
 ### Counting
 
 ```javascript
-// Count all
-const total = db.User.count();
+// Count all (requires await)
+const total = await db.User.count();
 
-// Count with conditions
-const activeCount = db.User
+// Count with conditions (requires await)
+const activeCount = await db.User
     .where(u => u.status == $$, 'active')
     .count();
 ```
@@ -554,19 +684,19 @@ const activeCount = db.User
 ### Complex Queries
 
 ```javascript
-// Multiple conditions with OR
-const results = db.User
+// Multiple conditions with OR (requires await)
+const results = await db.User
     .where(u => (u.status == 'active' || u.status == 'pending') && u.age >= $$, 18)
     .orderBy(u => u.name)
     .toList();
 
-// Nullable checks
-const usersWithoutEmail = db.User
+// Nullable checks (requires await)
+const usersWithoutEmail = await db.User
     .where(u => u.email == null)
     .toList();
 
-// LIKE queries
-const matching = db.User
+// LIKE queries (requires await)
+const matching = await db.User
     .where(u => u.name.like($$), '%john%')
     .toList();
 ```
@@ -838,13 +968,13 @@ const user = db.User.findById(1);  // DB query
 const user2 = db.User.findById(1);  // DB query again (no cache)
 
 // OPT-IN: Enable caching with .cache()
-const categories = db.Categories.cache().toList();  // DB query, cached
-const categories2 = db.Categories.cache().toList();  // Cache hit! (instant)
+const categories = await db.Categories.cache().toList();  // DB query, cached
+const categories2 = await db.Categories.cache().toList();  // Cache hit! (instant)
 
 // Update invalidates cache automatically
-const cat = db.Categories.findById(1);
+const cat = await db.Categories.findById(1);
 cat.name = "Updated";
-db.saveChanges();  // Cache for Categories table cleared
+await db.saveChanges();  // Cache for Categories table cleared
 
 // End request (clears cache - like Active Record)
 db.endRequest();  // Cache cleared for next request
@@ -865,9 +995,9 @@ app.use((req, res, next) => {
 });
 
 // In your routes
-app.get('/categories', (req, res) => {
+app.get('/categories', async (req, res) => {
     // Cache is fresh for this request
-    const categories = req.db.Categories.cache().toList();
+    const categories = await req.db.Categories.cache().toList();
     res.json(categories);
     // Cache auto-cleared after response
 });
@@ -900,14 +1030,14 @@ Use `.cache()` for frequently accessed, rarely changed data:
 
 ```javascript
 // DEFAULT: Always hits database (safe)
-const liveData = db.Analytics
+const liveData = await db.Analytics
     .where(a => a.date == $$, today)
     .toList();  // No caching (default)
 
 // OPT-IN: Cache reference data
-const categories = db.Categories.cache().toList();  // Cached for 5 seconds (default TTL)
-const settings = db.Settings.cache().toList();  // Cached
-const countries = db.Countries.cache().toList();  // Cached
+const categories = await db.Categories.cache().toList();  // Cached for 5 seconds (default TTL)
+const settings = await db.Settings.cache().toList();  // Cached
+const countries = await db.Countries.cache().toList();  // Cached
 
 // When to use .cache():
 // ✅ Reference data (categories, settings, countries)
@@ -940,7 +1070,7 @@ db.clearQueryCache();
 
 // Disable caching temporarily
 db.setQueryCacheEnabled(false);
-const freshData = db.User.toList();
+const freshData = await db.User.toList();
 db.setQueryCacheEnabled(true);
 ```
 
@@ -983,21 +1113,21 @@ MasterRecord automatically invalidates cache entries when data changes:
 
 ```javascript
 // Query with caching enabled
-const categories = db.Categories.cache().toList();  // DB query, cached
+const categories = await db.Categories.cache().toList();  // DB query, cached
 
 // Any modification to Categories table invalidates ALL cached Category queries
-const cat = db.Categories.findById(1);
+const cat = await db.Categories.findById(1);
 cat.name = "Updated";
-db.saveChanges();  // Invalidates all cached Categories queries
+await db.saveChanges();  // Invalidates all cached Categories queries
 
 // Next cached query hits database (fresh data)
-const categoriesAgain = db.Categories.cache().toList();  // DB query (cache cleared)
+const categoriesAgain = await db.Categories.cache().toList();  // DB query (cache cleared)
 
 // Non-cached queries are unaffected (always fresh)
-const users = db.User.toList();  // No .cache() = always DB query
+const users = await db.User.toList();  // No .cache() = always DB query
 
 // Queries for OTHER tables' caches are unaffected
-const settings = db.Settings.cache().toList();  // Still cached (different table)
+const settings = await db.Settings.cache().toList();  // Still cached (different table)
 ```
 
 **Invalidation rules:**
@@ -1024,12 +1154,12 @@ Expected performance improvements:
 **DO use .cache():**
 ```javascript
 // Reference data (rarely changes)
-const categories = db.Categories.cache().toList();
-const settings = db.Settings.cache().toList();
-const countries = db.Countries.cache().toList();
+const categories = await db.Categories.cache().toList();
+const settings = await db.Settings.cache().toList();
+const countries = await db.Countries.cache().toList();
 
 // Expensive aggregations (stable results)
-const totalRevenue = db.Orders
+const totalRevenue = await db.Orders
     .where(o => o.year == $$, 2024)
     .cache()
     .count();
@@ -1038,20 +1168,20 @@ const totalRevenue = db.Orders
 **DON'T use .cache():**
 ```javascript
 // User-specific data (default is safe - no caching)
-const user = db.User.findById(userId);  // Always fresh
+const user = await db.User.findById(userId);  // Always fresh
 
 // Real-time data (default is safe)
-const liveOrders = db.Orders
+const liveOrders = await db.Orders
     .where(o => o.status == $$, 'pending')
     .toList();  // Always fresh
 
 // Financial transactions (default is safe)
-const balance = db.Transactions
+const balance = await db.Transactions
     .where(t => t.user_id == $$, userId)
     .toList();  // Always fresh
 
 // User-specific sensitive data (default is safe)
-const permissions = db.UserPermissions
+const permissions = await db.UserPermissions
     .where(p => p.user_id == $$, userId)
     .toList();  // Always fresh
 ```
@@ -1089,12 +1219,12 @@ app.use((req, res, next) => {
 });
 
 // In routes - cache is fresh per request
-app.get('/api/categories', (req, res) => {
+app.get('/api/categories', async (req, res) => {
     // First call in this request - DB query
-    const categories = req.db.Categories.cache().toList();
+    const categories = await req.db.Categories.cache().toList();
 
     // Second call in same request - cache hit
-    const categoriesAgain = req.db.Categories.cache().toList();
+    const categoriesAgain = await req.db.Categories.cache().toList();
 
     res.json(categories);
     // After response, cache is automatically cleared
@@ -1118,18 +1248,18 @@ const db1 = new AppContext();
 const db2 = new AppContext();
 
 // Context 1: Cache data with .cache()
-const categories1 = db1.Categories.cache().toList();  // DB query, cached
+const categories1 = await db1.Categories.cache().toList();  // DB query, cached
 
 // Context 2: Sees cached data
-const categories2 = db2.Categories.cache().toList();  // Cache hit!
+const categories2 = await db2.Categories.cache().toList();  // Cache hit!
 
 // Context 2: Updates invalidate cache for BOTH contexts
-const cat = db2.Categories.findById(1);
+const cat = await db2.Categories.findById(1);
 cat.name = "Updated";
-db2.saveChanges();  // Invalidates shared cache
+await db2.saveChanges();  // Invalidates shared cache
 
 // Context 1: Sees fresh data
-const categories3 = db1.Categories.cache().toList();  // Cache miss, fresh data
+const categories3 = await db1.Categories.cache().toList();  // Cache miss, fresh data
 console.log(categories3[0].name);  // "Updated"
 ```
 
@@ -1168,8 +1298,9 @@ class AnalyticsContext extends context {
 const userDb = new UserContext();
 const analyticsDb = new AnalyticsContext();
 
-const user = userDb.User.findById(123);
-analyticsDb.Event.new().log('user_login', user.id);
+const user = await userDb.User.findById(123);
+const event = analyticsDb.Event.new();
+event.log('user_login', user.id);
 await analyticsDb.saveChanges();
 ```
 
@@ -1232,7 +1363,7 @@ context.setQueryCacheEnabled(bool)   // Enable/disable caching
 ### Query Methods
 
 ```javascript
-// Chainable query builders
+// Chainable query builders (do not execute query)
 .where(query, ...params)         // Add WHERE condition
 .and(query, ...params)           // Add AND condition
 .orderBy(field)                  // Sort ascending
@@ -1242,21 +1373,737 @@ context.setQueryCacheEnabled(bool)   // Enable/disable caching
 .include(relationship)           // Eager load
 .cache()                         // Enable caching for this query (opt-in)
 
-// Terminal methods (execute query)
-.toList()                        // Return array of all records
-.single()                        // Return one or null
-.first()                         // Return first or null
-.count()                         // Return count
-.any()                           // Return boolean
+// Terminal methods (execute query - ALL REQUIRE AWAIT)
+await .toList()                  // Return array of all records
+await .single()                  // Return one or null
+await .first()                   // Return first or null
+await .count()                   // Return count
+await .any()                     // Return boolean
 
-// Convenience methods
-.findById(id)                    // Find by primary key
-.new()                           // Create new entity instance
+// Convenience methods (REQUIRE AWAIT)
+await .findById(id)              // Find by primary key
+.new()                           // Create new entity instance (synchronous)
 
-// Entity methods (Active Record style)
+// Entity methods (Active Record style - REQUIRE AWAIT)
 await entity.save()              // Save this entity (and all tracked changes)
+await entity.delete()            // Delete this entity
+await entity.reload()            // Reload from database, discarding changes
+entity.clone()                   // Create a copy for duplication (synchronous)
+entity.toObject(options)         // Convert to plain JavaScript object (synchronous)
+entity.toJSON()                  // JSON.stringify compatibility (synchronous)
+```
+
+---
+
+## Entity Serialization
+
+### .toObject(options)
+
+Convert a MasterRecord entity to a plain JavaScript object, removing all internal properties and handling circular references automatically.
+
+**Parameters:**
+- `options.includeRelationships` (boolean, default: `true`) - Include related entities
+- `options.depth` (number, default: `1`) - Maximum depth for relationship traversal
+
+**Examples:**
+
+```javascript
+// Basic usage - get plain object
+const user = await db.User.findById(1);
+const plain = user.toObject();
+console.log(plain);
+// { id: 1, name: 'Alice', email: 'alice@example.com', age: 28 }
+
+// Include relationships
+const userWithPosts = user.toObject({ includeRelationships: true });
+console.log(userWithPosts);
+// {
+//   id: 1,
+//   name: 'Alice',
+//   Posts: [
+//     { id: 10, title: 'First Post', content: '...' },
+//     { id: 11, title: 'Second Post', content: '...' }
+//   ]
+// }
+
+// Control relationship depth
+const deep = user.toObject({ includeRelationships: true, depth: 3 });
+
+// Exclude relationships
+const shallow = user.toObject({ includeRelationships: false });
+```
+
+**Circular Reference Protection:**
+
+`.toObject()` automatically prevents infinite loops from circular references:
+
+```javascript
+// Scenario: User → Posts → User creates a cycle
+const user = await db.User.findById(1);
+await user.Posts;  // Load posts relationship
+
+const plain = user.toObject({ includeRelationships: true, depth: 2 });
+// Circular references marked as:
+// { __circular: true, __entityName: 'User', id: 1 }
+```
+
+**Why It's Needed:**
+
+MasterRecord entities have internal properties that cause `JSON.stringify()` to fail:
+
+```javascript
+const user = await db.User.findById(1);
+
+// ❌ FAILS: TypeError: Converting circular structure to JSON
+JSON.stringify(user);
+
+// ✅ WORKS: Use toObject() or toJSON()
+const plain = user.toObject();
+JSON.stringify(plain);  // Success!
+```
+
+### .toJSON()
+
+Used automatically by `JSON.stringify()` and Express `res.json()`. Returns the same as `.toObject({ includeRelationships: false })`.
+
+**Examples:**
+
+```javascript
+// JSON.stringify automatically calls toJSON()
+const user = await db.User.findById(1);
+const json = JSON.stringify(user);
+console.log(json);
+// '{"id":1,"name":"Alice","email":"alice@example.com"}'
+
+// Express automatically uses toJSON()
+app.get('/api/users/:id', async (req, res) => {
+    const user = await db.User.findById(req.params.id);
+    res.json(user);  // ✅ Works automatically!
+});
+
+// Array of entities
+app.get('/api/users', async (req, res) => {
+    const users = await db.User.toList();
+    res.json(users);  // ✅ Each entity's toJSON() called automatically
+});
+```
+
+---
+
+## Entity Instance Methods
+
+### .delete()
+
+Delete an entity without manually calling `context.remove()` and `context.saveChanges()`.
+
+**Example:**
+
+```javascript
+// Before
+const user = await db.User.findById(1);
+db.remove(user);
+await db.saveChanges();
+
+// After (Active Record style)
+const user = await db.User.findById(1);
+await user.delete();  // ✅ Entity deletes itself
+```
+
+**Cascade Deletion:**
+
+If your entity has cascade delete rules, they will be applied automatically:
+
+```javascript
+class User {
+    constructor() {
+        this.id = { type: 'integer', primary: true, auto: true };
+
+        // Posts will be deleted when user is deleted
+        this.Posts = {
+            type: 'hasMany',
+            model: 'Post',
+            foreignKey: 'user_id',
+            cascade: true  // Enable cascade delete
+        };
+    }
+}
+
+const user = await db.User.findById(1);
+await user.delete();  // ✅ Also deletes related Posts automatically
+```
+
+### .reload()
+
+Refresh an entity from the database, discarding any unsaved changes.
+
+**Example:**
+
+```javascript
+const user = await db.User.findById(1);
+console.log(user.name);  // 'Alice'
+
+user.name = 'Modified';
+console.log(user.name);  // 'Modified'
+
+await user.reload();  // ✅ Fetch fresh data from database
+console.log(user.name);  // 'Alice' - changes discarded
+```
+
+**Use Cases:**
+- Discard unsaved changes
+- Refresh stale data after external updates
+- Synchronize after concurrent modifications
+- Reset entity to clean state
+
+### .clone()
+
+Create a copy of an entity for duplication (primary key excluded).
+
+**Example:**
+
+```javascript
+const user = await db.User.findById(1);
+const duplicate = user.clone();
+
+duplicate.name = 'Copy of ' + user.name;
+duplicate.email = 'copy@example.com';
+
+await duplicate.save();
+console.log(duplicate.id);  // ✅ New ID (different from original)
+```
+
+**Notes:**
+- Primary key is automatically excluded
+- Relationships are not cloned (set manually if needed)
+- Useful for templates and duplicating records
+
+---
+
+## Query Helper Methods
+
+### .first()
+
+Get the first record ordered by primary key.
+
+**Example:**
+
+```javascript
+// Automatically orders by primary key
+const firstUser = await db.User.first();
+
+// With custom order (respects existing orderBy)
+const newestUser = await db.User
+    .orderByDescending(u => u.created_at)
+    .first();
+
+// With conditions
+const firstActive = await db.User
+    .where(u => u.status == $$, 'active')
+    .first();
+```
+
+### .last()
+
+Get the last record ordered by primary key (descending).
+
+**Example:**
+
+```javascript
+const lastUser = await db.User.last();
+
+// With custom order
+const oldestUser = await db.User
+    .orderBy(u => u.created_at)
+    .last();
+```
+
+### .exists()
+
+Check if any records match the query (returns boolean).
+
+**Example:**
+
+```javascript
+// Before
+const count = await db.User
+    .where(u => u.email == $$, 'test@example.com')
+    .count();
+const exists = count > 0;
+
+// After
+const exists = await db.User
+    .where(u => u.email == $$, 'test@example.com')
+    .exists();
+
+if (exists) {
+    throw new Error('Email already registered');
+}
+
+// Check if any users exist
+const hasUsers = await db.User.exists();
+if (!hasUsers) {
+    // Create default admin user
+}
+```
+
+### .pluck(fieldName)
+
+Extract a single column as an array.
+
+**Example:**
+
+```javascript
+// Get all active user emails
+const emails = await db.User
+    .where(u => u.status == $$, 'active')
+    .pluck('email');
+console.log(emails);
+// ['alice@example.com', 'bob@example.com', 'charlie@example.com']
+
+// Get all user IDs
+const ids = await db.User.pluck('id');
+console.log(ids);  // [1, 2, 3, 4, 5]
+
+// With sorting
+const recentEmails = await db.User
+    .orderByDescending(u => u.created_at)
+    .take(10)
+    .pluck('email');
+```
+
+---
+
+## Lifecycle Hooks
+
+Add lifecycle hooks to your entity definitions to execute logic before/after database operations.
+
+**Available Hooks:**
+- `beforeSave()` - Execute before insert or update
+- `afterSave()` - Execute after insert or update
+- `beforeDelete()` - Execute before deletion
+- `afterDelete()` - Execute after deletion
+
+**Example:**
+
+```javascript
+const bcrypt = require('bcrypt');
+
+class User {
+    constructor() {
+        this.id = { type: 'integer', primary: true, auto: true };
+        this.email = { type: 'string' };
+        this.password = { type: 'string' };
+        this.created_at = { type: 'timestamp' };
+        this.updated_at = { type: 'timestamp' };
+        this.role = { type: 'string' };
+    }
+
+    // Hash password before saving
+    beforeSave() {
+        // Only hash if password was changed
+        if (this.__dirtyFields.includes('password')) {
+            this.password = bcrypt.hashSync(this.password, 10);
+        }
+    }
+
+    // Set timestamps automatically
+    beforeSave() {
+        if (this.__state === 'insert') {
+            this.created_at = new Date();
+        }
+        this.updated_at = new Date();
+    }
+
+    // Log after successful save
+    afterSave() {
+        console.log(`User ${this.id} saved successfully`);
+    }
+
+    // Prevent deleting admin users
+    beforeDelete() {
+        if (this.role === 'admin') {
+            throw new Error('Cannot delete admin user');
+        }
+    }
+
+    // Cleanup related data after deletion
+    async afterDelete() {
+        console.log(`User ${this.id} deleted, cleaning up related data...`);
+        // Cleanup logic here (e.g., delete user files, clear cache)
+    }
+}
+```
+
+**Usage:**
+
+```javascript
+// Hooks execute automatically during save
+const user = db.User.new();
+user.email = 'alice@example.com';
+user.password = 'plain-text-password';
+await user.save();
+// ✅ beforeSave() hashes password automatically
+// ✅ afterSave() logs success message
+
+// Load and update
+const user = await db.User.findById(1);
+user.email = 'newemail@example.com';
+await user.save();
+// ✅ beforeSave() sets updated_at timestamp
+// ✅ Password not re-hashed (not in dirtyFields)
+
+// Hooks can prevent operations
+const admin = await db.User.where(u => u.role == $$, 'admin').single();
+try {
+    await admin.delete();
+} catch (error) {
+    console.log(error.message);  // "Cannot delete admin user"
+}
+// ✅ beforeDelete() prevented deletion
+```
+
+**Hook Execution Order:**
+
+```javascript
+// Insert:
+// 1. beforeSave()
+// 2. SQL INSERT
+// 3. afterSave()
+
+// Update:
+// 1. beforeSave()
+// 2. SQL UPDATE
+// 3. afterSave()
+
+// Delete:
+// 1. beforeDelete()
+// 2. SQL DELETE
+// 3. afterDelete()
+```
+
+**Notes:**
+- Hooks can be async (use `async` keyword)
+- Exceptions in `before*` hooks prevent the operation
+- Hooks execute for each entity during batch operations
+- Access entity state via `this.__state` ('insert', 'modified', 'delete')
+- Access changed fields via `this.__dirtyFields` array
+
+---
+
+## Business Logic Validation
+
+Add validators to your entity definitions for automatic validation on property assignment.
+
+**Built-in Validators:**
+- `required(message)` - Field must have a value
+- `email(message)` - Must be valid email format
+- `minLength(length, message)` - Minimum string length
+- `maxLength(length, message)` - Maximum string length
+- `pattern(regex, message)` - Must match regex pattern
+- `min(value, message)` - Minimum numeric value
+- `max(value, message)` - Maximum numeric value
+- `custom(fn, message)` - Custom validation function
+
+**Example:**
+
+```javascript
+class User {
+    id(db) {
+        db.integer().primary().auto();
+    }
+
+    name(db) {
+        db.string()
+          .required('Name is required')
+          .minLength(3, 'Name must be at least 3 characters')
+          .maxLength(50, 'Name cannot exceed 50 characters');
+    }
+
+    email(db) {
+        db.string()
+          .required('Email is required')
+          .email('Must be a valid email address');
+    }
+
+    password(db) {
+        db.string()
+          .required('Password is required')
+          .minLength(8, 'Password must be at least 8 characters')
+          .maxLength(100);
+    }
+
+    username(db) {
+        db.string()
+          .required()
+          .pattern(/^[a-zA-Z0-9_]+$/, 'Username can only contain letters, numbers, and underscores');
+    }
+
+    age(db) {
+        db.integer()
+          .min(18, 'Must be at least 18 years old')
+          .max(120, 'Age cannot exceed 120');
+    }
+
+    status(db) {
+        db.string()
+          .custom((value) => {
+              return ['active', 'inactive', 'pending'].includes(value);
+          }, 'Status must be active, inactive, or pending');
+    }
+}
+```
+
+**Validation Execution:**
+
+Validators run automatically when you assign values:
+
+```javascript
+const user = db.User.new();
+
+// ❌ Validation fails immediately
+try {
+    user.email = 'invalid-email';
+} catch (error) {
+    console.log(error.message);
+    // "Validation failed: Must be a valid email address"
+}
+
+// ✅ Valid value accepted
+user.email = 'valid@example.com';  // OK
+
+// ❌ Length validation
+try {
+    user.password = 'short';
+} catch (error) {
+    console.log(error.message);
+    // "Validation failed: Password must be at least 8 characters"
+}
+
+// ✅ Valid password
+user.password = 'secure-password-123';  // OK
+
+// ❌ Custom validation
+try {
+    user.status = 'invalid-status';
+} catch (error) {
+    console.log(error.message);
+    // "Validation failed: Status must be active, inactive, or pending"
+}
+
+// ✅ Valid status
+user.status = 'active';  // OK
+
+// Save (all fields already validated)
+await user.save();
+```
+
+**Validator Chaining:**
+
+Validators can be chained together:
+
+```javascript
+email(db) {
+    db.string()
+      .required('Email is required')      // ← First validator
+      .email('Invalid email format')      // ← Second validator
+      .minLength(5, 'Email too short')    // ← Third validator
+      .maxLength(100, 'Email too long');  // ← Fourth validator
+}
+```
+
+**Custom Validation:**
+
+```javascript
+discount(db) {
+    db.integer()
+      .min(0, 'Discount cannot be negative')
+      .max(100, 'Discount cannot exceed 100%')
+      .custom((value) => {
+          // Only allow multiples of 5
+          return value % 5 === 0;
+      }, 'Discount must be a multiple of 5');
+}
+
+// Usage
+product.discount = 7;   // ❌ Throws: "Discount must be a multiple of 5"
+product.discount = 10;  // ✅ OK
+```
+
+**Nullable Fields:**
+
+Required validation respects nullable fields:
+
+```javascript
+bio(db) {
+    db.string()
+      .maxLength(500, 'Bio cannot exceed 500 characters');
+    // No .required() = field is optional
+}
+
+// Both are valid
+user.bio = null;  // ✅ OK (nullable)
+user.bio = 'Short bio';  // ✅ OK (with value)
+user.bio = 'a'.repeat(501);  // ❌ Throws: "Bio cannot exceed 500 characters"
 ```
 
+---
+
+## Bulk Operations API
+
+Efficiently create, update, or delete multiple entities in a single operation.
+
+**Available Methods:**
+- `context.bulkCreate(entityName, data)` - Create multiple entities
+- `context.bulkUpdate(entityName, updates)` - Update multiple entities
+- `context.bulkDelete(entityName, ids)` - Delete multiple entities
+
+### bulkCreate(entityName, data)
+
+Create multiple entities efficiently in a batch operation.
+
+**Example:**
+
+```javascript
+// Create 5 users at once
+const users = await db.bulkCreate('User', [
+    { name: 'Alice', email: 'alice@example.com', status: 'active' },
+    { name: 'Bob', email: 'bob@example.com', status: 'active' },
+    { name: 'Charlie', email: 'charlie@example.com', status: 'inactive' },
+    { name: 'Dave', email: 'dave@example.com', status: 'active' },
+    { name: 'Eve', email: 'eve@example.com', status: 'pending' }
+]);
+
+console.log(users.length);  // 5
+console.log(users[0].id);   // 1 (auto-increment IDs assigned)
+console.log(users[4].id);   // 5
+
+// Entities are returned in the same order
+console.log(users.map(u => u.name));
+// ['Alice', 'Bob', 'Charlie', 'Dave', 'Eve']
+```
+
+**Performance:**
+
+```javascript
+// ❌ SLOW: Multiple individual inserts
+for (const data of users) {
+    const user = db.User.new();
+    user.name = data.name;
+    user.email = data.email;
+    await user.save();  // Separate database call
+}
+
+// ✅ FAST: Single bulk insert
+await db.bulkCreate('User', users);  // One database call
+```
+
+### bulkUpdate(entityName, updates)
+
+Update multiple entities by their primary keys.
+
+**Example:**
+
+```javascript
+// Update multiple users' status
+await db.bulkUpdate('User', [
+    { id: 1, status: 'inactive' },
+    { id: 2, status: 'inactive' },
+    { id: 4, status: 'inactive' }
+]);
+
+// Verify updates
+const user1 = await db.User.findById(1);
+console.log(user1.status);  // 'inactive'
+
+// Other fields unchanged
+console.log(user1.name);   // Original name preserved
+console.log(user1.email);  // Original email preserved
+```
+
+**Partial Updates:**
+
+Only the fields you specify are updated:
+
+```javascript
+// Update only email for multiple users
+await db.bulkUpdate('User', [
+    { id: 1, email: 'newemail1@example.com' },
+    { id: 2, email: 'newemail2@example.com' }
+]);
+
+// name, status, age, etc. remain unchanged
+```
+
+### bulkDelete(entityName, ids)
+
+Delete multiple entities by their primary keys.
+
+**Example:**
+
+```javascript
+// Delete multiple users by ID
+await db.bulkDelete('User', [3, 5, 7]);
+
+// Verify deletion
+const user3 = await db.User.findById(3);
+console.log(user3);  // null
+
+const remaining = await db.User.toList();
+console.log(remaining.length);  // Total users minus 3
+```
+
+**Non-Existent IDs:**
+
+Bulk delete handles non-existent IDs gracefully:
+
+```javascript
+// Some IDs don't exist
+await db.bulkDelete('User', [999, 1000, 1001]);
+// ✅ No error thrown - operation completes successfully
+```
+
+**Error Handling:**
+
+```javascript
+// Empty array throws error
+try {
+    await db.bulkCreate('User', []);
+} catch (error) {
+    console.log(error.message);
+    // "bulkCreate requires a non-empty array of data"
+}
+
+// Invalid entity name throws error
+try {
+    await db.bulkUpdate('NonExistentEntity', [{ id: 1 }]);
+} catch (error) {
+    console.log(error.message);
+    // "Entity NonExistentEntity not found"
+}
+```
+
+**Lifecycle Hooks:**
+
+Bulk operations execute lifecycle hooks for each entity:
+
+```javascript
+class User {
+    beforeSave() {
+        console.log(`Saving user: ${this.name}`);
+    }
+}
+
+await db.bulkCreate('User', [
+    { name: 'Alice' },
+    { name: 'Bob' }
+]);
+// Console output:
+// Saving user: Alice
+// Saving user: Bob
+```
+
+---
+
 ### Migration Methods
 
 ```javascript
@@ -1315,14 +2162,14 @@ demo();
 async function getUsers(page = 0, pageSize = 10) {
     const db = new AppContext();
 
-    const users = db.User
+    const users = await db.User
         .where(u => u.status == $$, 'active')
         .orderBy(u => u.created_at)
         .skip(page * pageSize)
         .take(pageSize)
         .toList();
 
-    const total = db.User
+    const total = await db.User
         .where(u => u.status == $$, 'active')
         .count();
 
@@ -1373,7 +2220,7 @@ async function searchUsers(filters) {
             .take(filters.pageSize);
     }
 
-    return query.toList();
+    return await query.toList();
 }
 ```
 
@@ -1403,7 +2250,7 @@ post.author_id = author.id;
 await db.saveChanges();
 
 // Query with relationships
-const posts = db.Post
+const posts = await db.Post
     .where(p => p.author_id == $$, author.id)
     .toList();
 
@@ -1416,15 +2263,15 @@ console.log(`${author.name} has ${posts.length} posts`);
 
 ```javascript
 // ✅ GOOD: Cache reference data that rarely changes
-const categories = db.Categories.cache().toList();  // Opt-in caching
-const settings = db.Settings.cache().toList();
+const categories = await db.Categories.cache().toList();  // Opt-in caching
+const settings = await db.Settings.cache().toList();
 
 // ✅ GOOD: Queries without .cache() are always fresh (safe default)
-const user1 = db.User.findById(123);  // Always DB query (no cache)
-const user2 = db.User.findById(123);  // Always DB query (no cache)
+const user1 = await db.User.findById(123);  // Always DB query (no cache)
+const user2 = await db.User.findById(123);  // Always DB query (no cache)
 
 // ✅ GOOD: Cache expensive queries with stable results
-const revenue2024 = db.Orders
+const revenue2024 = await db.Orders
     .where(o => o.year == $$, 2024)
     .cache()  // Historical data doesn't change
     .count();
@@ -1472,13 +2319,13 @@ class User {
 
 ```javascript
 // ✅ GOOD: Limit results
-const recentUsers = db.User
+const recentUsers = await db.User
     .orderByDescending(u => u.created_at)
     .take(100)
     .toList();
 
 // ❌ BAD: Load everything
-const allUsers = db.User.toList();
+const allUsers = await db.User.toList();
 ```
 
 ### 5. Use Connection Pooling (PostgreSQL)
@@ -1500,7 +2347,7 @@ MasterRecord uses **parameterized queries throughout** to prevent SQL injection:
 
 ```javascript
 // ✅ SAFE: Parameterized
-const user = db.User.where(u => u.name == $$, userInput).single();
+const user = await db.User.where(u => u.name == $$, userInput).single();
 
 // ❌ UNSAFE: Never do this
 // const query = `SELECT * FROM User WHERE name = '${userInput}'`;
@@ -1520,12 +2367,12 @@ While SQL injection is prevented, always validate business logic:
 
 ```javascript
 // Validate input before querying
-function getUser(userId) {
+async function getUser(userId) {
     if (!Number.isInteger(userId) || userId <= 0) {
         throw new Error('Invalid user ID');
     }
 
-    return db.User.findById(userId);
+    return await db.User.findById(userId);
 }
 ```
 
@@ -1652,7 +2499,128 @@ Created by Alexander Rich
 
 ---
 
-## Recent Improvements (v0.3.13)
+## Recent Improvements
+
+### v0.3.30 - Mature ORM Features (Latest)
+
+MasterRecord is now feature-complete with lifecycle hooks, validation, and bulk operations - matching the capabilities of mature ORMs like Sequelize, TypeORM, and Prisma.
+
+**🎯 Entity Serialization:**
+- ✅ **`.toObject()`** - Convert entities to plain JavaScript objects with circular reference protection
+- ✅ **`.toJSON()`** - Automatic JSON.stringify() compatibility for Express responses
+- ✅ **Circular Reference Handling** - Prevents infinite loops from bidirectional relationships
+- ✅ **Depth Control** - Configurable relationship traversal depth
+
+**🎯 Active Record Pattern:**
+- ✅ **`.delete()`** - Entities can delete themselves (`await user.delete()`)
+- ✅ **`.reload()`** - Refresh entity from database, discard unsaved changes
+- ✅ **`.clone()`** - Create entity copies for duplication (excludes primary key)
+- ✅ **`.save()`** - Already existed, now part of complete Active Record pattern
+
+**🎯 Query Helpers:**
+- ✅ **`.first()`** - Get first record ordered by primary key
+- ✅ **`.last()`** - Get last record ordered by primary key descending
+- ✅ **`.exists()`** - Check if any records match query (returns boolean)
+- ✅ **`.pluck(field)`** - Extract single column values as array
+
+**🎯 Lifecycle Hooks:**
+- ✅ **`beforeSave()`** - Execute before insert or update (e.g., hash passwords)
+- ✅ **`afterSave()`** - Execute after successful save (e.g., logging)
+- ✅ **`beforeDelete()`** - Execute before deletion (can prevent deletion)
+- ✅ **`afterDelete()`** - Execute after deletion (e.g., cleanup)
+- ✅ **Hook Execution Order** - Guaranteed execution order with error handling
+- ✅ **Async Support** - Hooks can be async for database operations
+
+**🎯 Business Logic Validation:**
+- ✅ **`.required()`** - Field must have a value
+- ✅ **`.email()`** - Must be valid email format
+- ✅ **`.minLength()` / `.maxLength()`** - String length constraints
+- ✅ **`.min()` / `.max()`** - Numeric value constraints
+- ✅ **`.pattern()`** - Must match regex pattern
+- ✅ **`.custom()`** - Custom validation functions
+- ✅ **Chainable Validators** - Multiple validators per field
+- ✅ **Immediate Validation** - Errors thrown on property assignment
+
+**🎯 Bulk Operations API:**
+- ✅ **`bulkCreate()`** - Create multiple entities efficiently in one transaction
+- ✅ **`bulkUpdate()`** - Update multiple entities by primary key
+- ✅ **`bulkDelete()`** - Delete multiple entities by primary key
+- ✅ **Lifecycle Hook Support** - Hooks execute for each entity in bulk operations
+- ✅ **Auto-Increment IDs** - IDs properly assigned after bulk inserts
+
+**🎯 Critical Bug Fixes:**
+- ✅ **Auto-Increment ID Bug Fixed** - IDs now correctly set on entities after insert (SQLite, MySQL, PostgreSQL)
+- ✅ **Lifecycle Hook Isolation** - Hooks excluded from SQL queries and INSERT/UPDATE operations
+- ✅ **Circular Reference Prevention** - WeakSet-based tracking prevents infinite loops
+
+**Example Usage:**
+
+```javascript
+// Entity serialization
+const user = await db.User.findById(1);
+const plain = user.toObject({ includeRelationships: true, depth: 2 });
+res.json(user);  // Works automatically with toJSON()
+
+// Active Record pattern
+await user.delete();    // Entity deletes itself
+await user.reload();    // Discard changes
+const copy = user.clone();  // Duplicate entity
+
+// Query helpers
+const first = await db.User.first();
+const exists = await db.User.where(u => u.email == $$, 'test@test.com').exists();
+const emails = await db.User.where(u => u.status == $$, 'active').pluck('email');
+
+// Lifecycle hooks
+class User {
+    beforeSave() {
+        if (this.__dirtyFields.includes('password')) {
+            this.password = bcrypt.hashSync(this.password, 10);
+        }
+        this.updated_at = new Date();
+    }
+
+    beforeDelete() {
+        if (this.role === 'admin') {
+            throw new Error('Cannot delete admin user');
+        }
+    }
+}
+
+// Business validation
+class User {
+    email(db) {
+        db.string()
+          .required('Email is required')
+          .email('Must be a valid email address');
+    }
+
+    age(db) {
+        db.integer()
+          .min(18, 'Must be at least 18 years old')
+          .max(120);
+    }
+}
+
+// Bulk operations
+const users = await db.bulkCreate('User', [
+    { name: 'Alice', email: 'alice@example.com' },
+    { name: 'Bob', email: 'bob@example.com' },
+    { name: 'Charlie', email: 'charlie@example.com' }
+]);
+console.log(users.map(u => u.id));  // [1, 2, 3] - IDs assigned
+
+await db.bulkUpdate('User', [
+    { id: 1, status: 'inactive' },
+    { id: 2, status: 'inactive' }
+]);
+
+await db.bulkDelete('User', [3, 5, 7]);
+```
+
+---
+
+### v0.3.13 - FAANG Engineering Standards
 
 MasterRecord has been upgraded to meet **FAANG engineering standards** (Google/Meta/Amazon) with critical bug fixes and performance improvements: