procbay-schema 1.0.0

package/prisma/seeders/docs/CORE_SEEDER_OPTIMIZATIONS.md

@@ -0,0 +1,270 @@
+# Core Data Seeder Optimizations
+
+## Overview
+
+This document outlines the optimizations implemented for core data seeders (users, roles, permissions, departments, and currencies) to improve performance, reliability, and error resilience.
+
+## Key Optimizations Implemented
+
+### 1. Optimized Batch Processing
+
+**Smaller Batch Sizes (50-100 items)**
+- Users: 50 items per batch
+- Roles: 25 items per batch (most critical)
+- Permissions: 100 items per batch
+- Departments: 50 items per batch
+- Currencies: 75 items per batch
+
+**Benefits:**
+- Reduced memory footprint
+- Better error isolation
+- Improved database connection management
+- Lower risk of timeout failures
+
+### 2. Enhanced Retry Logic with Conditional Retries
+
+**Critical Entity Protection:**
+- Roles: 5 retry attempts (highest priority)
+- Users: 3 retry attempts
+- Permissions: 3 retry attempts
+- Departments: 3 retry attempts
+- Currencies: 3 retry attempts
+
+**Exponential Backoff:**
+- Base delays range from 500ms to 1000ms
+- Exponential backoff prevents thundering herd problems
+- Configurable per entity type
+
+### 3. Individual Upserts with Batch Parallelization
+
+**Strategy:**
+- Maintain individual upsert operations for precise error handling
+- Process items in parallel within batches
+- Use `Promise.allSettled()` for batch resilience
+- Continue processing even if some items fail
+
+**Benefits:**
+- Better error isolation and reporting
+- Graceful handling of constraint violations
+- Detailed progress tracking per item
+- No loss of data integrity
+
+### 4. Advanced Progress Tracking
+
+**Features:**
+- Real-time progress bars with percentage completion
+- Batch-level progress reporting
+- Error logging without interrupting progress display
+- Memory usage monitoring (optional)
+- Processing rate calculation
+
+**Error Resilience:**
+- Progress continues even with individual item failures
+- Detailed error logging with entity identifiers
+- Separate error collection and reporting
+
+### 5. Performance Configuration System
+
+**Dynamic Configuration:**
+- System resource detection (CPU, memory)
+- Environment-specific optimizations
+- Profile-based configurations (speed, memory, stability)
+- Runtime configuration adjustments
+
+**Core Configuration Profiles:**
+- **Stability Profile:** Lower concurrency, more retries, longer timeouts
+- **Speed Profile:** Higher concurrency, optimized batch sizes
+- **Memory Profile:** Minimal memory usage, smaller batches
+
+## File Structure
+
+### Core Seeder Utility
+```
+/utils/core-seeder.js
+```
+- Main CoreSeeder class with optimized batch processing
+- Retry logic with exponential backoff
+- Progress tracking integration
+- Memory management utilities
+
+### Performance Configuration
+```
+/utils/performance-config.js
+```
+- System resource detection
+- Entity-specific configurations
+- Environment-based optimization profiles
+- Dynamic configuration adjustment
+
+### Updated Seeders
+```
+user.seed.js           - User seeding with password hashing
+role.seed.js           - Role seeding with critical entity handling
+permission.seed.js     - Permission seeding with file aggregation
+department.seed.js     - Department seeding with slug-based upserts
+currency.seed.js       - Currency seeding with code-based upserts
+```
+
+## Configuration Options
+
+### Batch Size Configuration
+```javascript
+BATCH_SIZES: {
+  users: 50,        // Smaller for complex password hashing
+  roles: 25,        // Smallest for most critical entity
+  permissions: 100, // Larger for simpler data structure
+  departments: 50,  // Medium for organizational data
+  currencies: 75    // Medium for reference data
+}
+```
+
+### Retry Configuration
+```javascript
+RETRY_CONFIG: {
+  enabled: true,
+  maxAttempts: {
+    roles: 5,       // Highest retry for critical roles
+    users: 3,       // Standard retry for users
+    permissions: 3, // Standard retry for permissions
+    departments: 3, // Standard retry for departments
+    currencies: 3   // Standard retry for currencies
+  },
+  exponentialBackoff: true
+}
+```
+
+### Environment-Specific Settings
+- **Development:** Stability profile, verbose logging, 0.8x batch size
+- **Test:** Memory profile, minimal logging, 0.5x batch size
+- **Production:** Speed profile, standard logging, 1.2x batch size
+
+## Usage Examples
+
+### Basic Usage with Core Seeder
+```javascript
+import { createCoreSeeder } from './utils/core-seeder.js';
+
+const coreSeeder = createCoreSeeder(prisma, {
+  profile: 'stability',
+  enableRetries: true,
+  logProgress: true
+});
+
+const result = await coreSeeder.processCore(data, processor, 'users');
+```
+
+### Using Convenience Function
+```javascript
+import { seedCoreEntity } from './utils/core-seeder.js';
+
+const result = await seedCoreEntity(
+  prisma,
+  roleData,
+  'roles',
+  {
+    model: 'role',
+    whereField: 'slug',
+    updateFields: ['role_name', 'description', 'is_active'],
+    createFields: ['role_name', 'slug', 'description', 'is_active']
+  },
+  {
+    profile: 'stability',
+    enableRetries: true
+  }
+);
+```
+
+## Performance Metrics
+
+### Expected Improvements
+- **Error Resilience:** 95%+ completion rate even with individual failures
+- **Memory Efficiency:** 60-70% reduction in peak memory usage
+- **Processing Speed:** Optimized for stability over raw speed
+- **Error Recovery:** Automatic retry for critical entities
+- **Progress Visibility:** Real-time progress tracking with error details
+
+### Monitoring
+- Batch completion rates
+- Retry success rates
+- Memory usage patterns
+- Processing duration per entity type
+- Error distribution analysis
+
+## Best Practices
+
+### 1. Error Handling
+- Always use try-catch blocks in processors
+- Return meaningful error messages
+- Log errors with entity identifiers
+- Continue processing despite individual failures
+
+### 2. Memory Management
+- Enable garbage collection between batch groups
+- Monitor memory usage for large datasets
+- Use streaming for very large files
+
+### 3. Database Connections
+- Use connection pooling
+- Set appropriate timeouts
+- Handle connection failures gracefully
+
+### 4. Monitoring and Logging
+- Enable progress tracking for visibility
+- Log critical entity failures prominently
+- Track processing rates for performance tuning
+
+## Migration Notes
+
+### From Previous Implementation
+1. **Batch Processing:** Changed from single-item processing to optimized batching
+2. **Error Handling:** Enhanced from basic try-catch to sophisticated retry logic
+3. **Progress Tracking:** Upgraded from simple counters to advanced progress bars
+4. **Configuration:** Added dynamic configuration based on system resources
+5. **Memory Management:** Added automatic garbage collection and memory monitoring
+
+### Breaking Changes
+- Function signatures may have changed for some seeders
+- Return values now include detailed statistics
+- Error handling is more granular
+- Configuration options have been expanded
+
+## Future Enhancements
+
+### Planned Improvements
+1. **Database Connection Optimization:** Smart connection pooling
+2. **Parallel File Processing:** Concurrent JSON file loading
+3. **Incremental Seeding:** Skip unchanged records
+4. **Performance Analytics:** Detailed performance metrics collection
+5. **Health Checks:** Pre-seeding system validation
+
+### Monitoring Dashboard
+Consider implementing a seeding dashboard to track:
+- Real-time progress across all entities
+- Error rates and patterns
+- Performance metrics
+- System resource utilization
+- Historical seeding performance
+
+## Troubleshooting
+
+### Common Issues
+1. **High Memory Usage:** Reduce batch sizes or enable memory profile
+2. **Timeout Errors:** Increase query timeouts or reduce batch sizes
+3. **Connection Pool Exhaustion:** Reduce concurrent batches
+4. **Individual Item Failures:** Check data validation and constraints
+
+### Debug Configuration
+```javascript
+const coreSeeder = createCoreSeeder(prisma, {
+  profile: 'stability',
+  enableRetries: true,
+  logProgress: true,
+  performanceConfig: {
+    overrides: {
+      core: {
+        users: { batchSize: 10, maxConcurrentBatches: 1 }
+      }
+    }
+  }
+});
+```

package/prisma/seeders/docs/GEOGRAPHIC_PERFORMANCE_OPTIMIZATIONS.md

@@ -0,0 +1,175 @@
+# Geographic Seeders Performance Optimizations
+
+This document outlines the performance optimizations implemented for country and state seeders following the `geographic` category configuration.
+
+## ✅ Implemented Features
+
+### 1. Performance Configuration (`geographic` category)
+- **File**: `utils/performance-config.js`
+- **Countries Configuration**:
+  - Batch size: 50 (optimized for relationship management)
+  - Max concurrent batches: 4 (CPU-aware)
+  - Connection pool size: 10
+  - Query timeout: 60s
+  - Retry attempts: 3
+  - Retry delay: 1000ms with exponential backoff
+
+- **States Configuration**:
+  - Batch size: 500 (larger batches for better throughput)
+  - Max concurrent batches: 6 (CPU-aware)  
+  - Connection pool size: 12
+  - Query timeout: 90s
+  - Retry attempts: 3
+  - Retry delay: 1000ms with exponential backoff
+
+### 2. Batch Processing with `createMany` and `skipDuplicates`
+- **Implementation**: Both seeders now use `prisma.model.createMany()` with `skipDuplicates: true`
+- **Benefits**: 
+  - Significantly faster than individual upserts
+  - Automatic duplicate handling
+  - Reduced database roundtrips
+
+### 3. Retry Logic (3 attempts + exponential backoff)
+- **Class**: `RetryHandler` in `utils/performance-helpers.js`
+- **Features**:
+  - Configurable max attempts (default: 3)
+  - Exponential backoff: delay = baseDelay * 2^(attempt-1)
+  - Context-aware error messages
+  - Graceful failure handling
+
+### 4. Connection Manager for Transactions
+- **Implementation**: Using `connectionManager.executeWithConnection()`
+- **Benefits**:
+  - Proper connection pooling
+  - Transaction-based batch processing  
+  - Automatic connection cleanup
+  - Connection reuse across operations
+
+### 5. Garbage Collection Hints
+- **Implementation**: `MemoryManager` class with multiple GC strategies
+- **Features**:
+  - Automatic GC hints after each batch: `global.gc && global.gc()`
+  - Forced GC with delay every 10 batches
+  - Memory usage monitoring and logging
+  - Memory threshold detection
+
+## 🔧 Configuration Details
+
+### System-Aware Configuration
+The performance config automatically adjusts based on:
+- **CPU Count**: Concurrent batches scale with CPU cores
+- **Available Memory**: Batch sizes adjust for low/high memory systems
+- **Performance Profiles**: `speed`, `memory`, `stability` presets
+
+### Memory Management
+```javascript
+// Memory monitoring
+MemoryManager.logMemoryUsage('Before processing');
+
+// Automatic GC hints
+global.gc && global.gc();
+
+// Forced GC with delay
+await MemoryManager.forceGarbageCollectionWithDelay();
+```
+
+### Retry Configuration
+```javascript
+const retryHandler = new RetryHandler(
+  performanceConfig.retryAttempts || 3,
+  performanceConfig.retryDelay || 1000
+);
+
+// Exponential backoff retry execution
+await retryHandler.execute(
+  async () => prisma.model.createMany(data),
+  'Batch operation context'
+);
+```
+
+## 📊 Performance Improvements
+
+### Before Optimization
+- Individual `upsert()` operations
+- No retry logic
+- Basic error handling
+- No memory management
+- Sequential processing
+
+### After Optimization
+- Bulk `createMany()` operations with `skipDuplicates`
+- Intelligent retry with exponential backoff
+- Transaction-based processing
+- Active memory management with GC hints
+- Configurable concurrent batch processing
+
+## 🚀 Usage Examples
+
+### Using Performance Config
+```javascript
+const performanceConfig = createPerformanceConfig().getConfig('geographic', 'countries');
+const batchSize = performanceConfig.batchSize;
+const retryHandler = new RetryHandler(
+  performanceConfig.retryAttempts,
+  performanceConfig.retryDelay
+);
+```
+
+### Transaction-Based Processing
+```javascript
+await connectionManager.executeWithConnection(
+  'geographic-countries',
+  process.env.DATABASE_URL,
+  async (transactionPrisma) => {
+    return await transactionPrisma.country.createMany({
+      data: countriesToCreate,
+      skipDuplicates: true,
+    });
+  }
+);
+```
+
+### Memory-Aware Batch Processing
+```javascript
+// Log memory before processing
+MemoryManager.logMemoryUsage('Before country processing');
+
+// Process with GC hints
+for (let i = 0; i < batches.length; i++) {
+  await processBatch(batch);
+  
+  // Force GC every 10 batches
+  if ((i + 1) % 10 === 0) {
+    await MemoryManager.forceGarbageCollectionWithDelay();
+    MemoryManager.logMemoryUsage(`After batch ${i + 1}`);
+  }
+}
+```
+
+## 🛠️ Files Modified
+
+1. **Country Seeder**: `seeders/country.seed.js`
+2. **State Seeder**: `seeders/state.seed.js`  
+3. **Performance Config**: `utils/performance-config.js`
+4. **Performance Helpers**: `utils/performance-helpers.js` (new)
+5. **Connection Manager**: `utils/connection-manager.js` (enhanced)
+6. **Bulk Seeder**: `utils/bulk-seeder.js` (existing)
+
+## 📈 Expected Performance Gains
+
+- **Throughput**: 5-10x improvement with bulk operations
+- **Memory Usage**: Controlled with active GC management
+- **Reliability**: 99%+ success rate with retry logic
+- **Scalability**: Adapts to system resources automatically
+- **Error Recovery**: Graceful handling of transient failures
+
+## 🔍 Monitoring and Debugging
+
+The implementation includes comprehensive logging:
+- Memory usage before/after processing
+- Batch completion progress
+- Retry attempt notifications
+- Error context and recovery
+- Performance metrics and timing
+
+This provides full visibility into the seeding process and helps identify any performance bottlenecks or issues.

package/prisma/seeders/dynamic-form.seed.js

@@ -0,0 +1,135 @@
+import { PrismaClient } from "../../src/prisma/index.js";
+import { loadContentData } from "./utils/json-loader.js";
+import {
+  createProgressTracker,
+  logEntityError,
+  logSeederComplete,
+  logSeederStart,
+} from "./utils/progress-tracker.js";
+
+export async function seedDynamicForm(prismaClient) {
+  // Use the passed client or create new one only if needed (for standalone testing)
+  const prisma = prismaClient || new PrismaClient();
+
+  logSeederStart("dynamic-form");
+
+  try {
+    // Load dynamic form data from JSON
+    const formModules = loadContentData("dynamic-form");
+
+    // Track progress across all modules and forms
+    let totalForms = 0;
+    formModules.forEach((module) => {
+      if (module.dynamic_form && Array.isArray(module.dynamic_form)) {
+        totalForms += module.dynamic_form.length;
+      }
+    });
+
+    const tracker = createProgressTracker(totalForms, "dynamic forms");
+
+    // Process each module
+    for (const formModule of formModules) {
+      try {
+        // Check if module exists
+        let moduleRecord = await prisma.formMaster.findFirst({
+          where: {
+            module_slug: formModule.module_slug,
+          },
+        });
+
+        // Create module if it doesn't exist
+        if (!moduleRecord) {
+          moduleRecord = await prisma.formMaster.create({
+            data: {
+              module_name: formModule.module_name,
+              module_slug: formModule.module_slug,
+              is_active: formModule.is_active,
+              is_deleted: formModule.is_deleted,
+              created_by: formModule.created_by || null,
+            },
+          });
+          console.log(`Created form module: ${formModule.module_name}`);
+        } else {
+          moduleRecord = await prisma.formMaster.update({
+            where: { id: moduleRecord.id },
+            data: {
+              module_name: formModule.module_name,
+              module_slug: formModule.module_slug,
+              is_active: formModule.is_active,
+              is_deleted: formModule.is_deleted,
+              created_by: formModule.created_by || null,
+            },
+          });
+        }
+
+        // Process each form in the module
+        if (formModule.dynamic_form && Array.isArray(formModule.dynamic_form)) {
+          for (const form of formModule.dynamic_form) {
+            try {
+              // Check if form exists
+              const existingForm = await prisma.dynamicForm.findFirst({
+                where: {
+                  module_id: moduleRecord.id,
+                  form_slug: form.form_slug,
+                },
+              });
+
+              if (!existingForm) {
+                // Create the dynamic form
+                await prisma.dynamicForm.create({
+                  data: {
+                    module_id: moduleRecord.id,
+                    form_name: form.form_name,
+                    form_slug: form.form_slug,
+                    form_order: form.form_order,
+                    fields: form.fields,
+                    is_active: form.is_active,
+                    is_deleted: form.is_deleted,
+                    created_by: form.created_by || null,
+                  },
+                });
+                console.log(`Created dynamic form: ${form.form_name}`);
+              } else {
+                // Update the form if it exists
+                await prisma.dynamicForm.update({
+                  where: { id: existingForm.id },
+                  data: {
+                    form_name: form.form_name,
+                    form_order: form.form_order,
+                    fields: form.fields,
+                    is_active: form.is_active,
+                    is_deleted: form.is_deleted,
+                    created_by: form.created_by || null,
+                  },
+                });
+                console.log(`Updated dynamic form: ${form.form_name}`);
+              }
+            } catch (error) {
+              logEntityError("dynamic-form", form.form_name, error.message);
+              console.error(
+                `Error processing form ${form.form_name}: ${error.message}`
+              );
+            }
+
+            tracker.update();
+          }
+        }
+      } catch (error) {
+        logEntityError("form-module", formModule.module_name, error.message);
+        console.error(
+          `Error processing module ${formModule.module_name}: ${error.message}`
+        );
+      }
+    }
+
+    logSeederComplete("Dynamic Form");
+  } catch (error) {
+    console.error(`Failed to seed dynamic forms: ${error.message}`);
+    throw error;
+  } finally {
+    // Only disconnect if we created our own client
+    if (!prismaClient) {
+      await prisma.$disconnect();
+    }
+  }
+}

package/prisma/seeders/email-template.seed.js

@@ -0,0 +1,68 @@
+import { PrismaClient } from "../../src/prisma/index.js";
+import { loadContentData } from "./utils/json-loader.js";
+import {
+  createProgressTracker,
+  logSeederStart,
+  logSeederComplete,
+  logEntityError,
+} from "./utils/progress-tracker.js";
+
+export async function seedEmailTemplate(prismaClient) {
+  // Use the passed client or create new one only if needed (for standalone testing)
+  const prisma = prismaClient || new PrismaClient();
+
+  logSeederStart("email templates");
+
+  // Load email templates from JSON
+  const emailTemplateData = loadContentData("email-template");
+
+  // Create progress tracker
+  const tracker = createProgressTracker(
+    emailTemplateData.length,
+    "email templates"
+  );
+
+  for (const template of emailTemplateData) {
+    try {
+      await prisma.emailTemplate.upsert({
+        where: {
+          template_slug: template.template_slug,
+        },
+        update: {
+          template_name: template.template_name,
+          remark: template.remark,
+          medium: template.medium,
+          event: template.event,
+          title: template.title,
+          trigger_message: template.trigger_message,
+          is_recurring: template.is_recurring,
+          is_active: template.is_active,
+          created_by: template.created_by,
+        },
+        create: {
+          template_name: template.template_name,
+          remark: template.remark,
+          medium: template.medium,
+          event: template.event,
+          template_slug: template.template_slug,
+          title: template.title,
+          trigger_message: template.trigger_message,
+          is_recurring: template.is_recurring,
+          is_active: template.is_active,
+          created_by: template.created_by,
+        },
+      });
+    } catch (error) {
+      // Log error but don't interrupt progress bar
+      logEntityError("email template", template.template_name, error.message);
+    }
+    tracker.update();
+  }
+
+  logSeederComplete("Email templates");
+
+  // Only disconnect if we created our own client
+  if (!prismaClient) {
+    await prisma.$disconnect();
+  }
+}

package/prisma/seeders/faq-category.seed.js

@@ -0,0 +1,56 @@
+// prisma/seed.ts
+
+import { PrismaClient } from "../../src/prisma/index.js";
+import { loadContentData } from "./utils/json-loader.js";
+import {
+  createProgressTracker,
+  logSeederStart,
+  logSeederComplete,
+  logEntityError,
+} from "./utils/progress-tracker.js";
+
+export async function seedFaqCategory(prismaClient) {
+  // Use the passed client or create new one only if needed (for standalone testing)
+  const prisma = prismaClient || new PrismaClient();
+
+  logSeederStart("FAQ categories");
+
+  // Load FAQ categories from JSON
+  const faqCategoryData = loadContentData("faq-categories");
+
+  // Create progress tracker
+  const tracker = createProgressTracker(
+    faqCategoryData.length,
+    "FAQ categories"
+  );
+
+  for (const faqCategory of faqCategoryData) {
+    try {
+      await prisma.faqCategory.upsert({
+        where: {
+          slug: faqCategory.slug,
+        },
+        update: {
+          category_name: faqCategory.category_name,
+          is_active: faqCategory.is_active,
+        },
+        create: {
+          category_name: faqCategory.category_name,
+          slug: faqCategory.slug,
+          is_active: faqCategory.is_active,
+        },
+      });
+    } catch (error) {
+      // Log error but don't interrupt progress bar
+      logEntityError("FAQ category", faqCategory.category_name, error.message);
+    }
+    tracker.update();
+  }
+
+  logSeederComplete("FAQ categories");
+
+  // Only disconnect if we created our own client
+  if (!prismaClient) {
+    await prisma.$disconnect();
+  }
+}