@memberjunction/metadata-sync 2.48.0 → 2.50.0

package/README.md

@@ -98,6 +98,386 @@ The tool is intended for managing business-level metadata such as:
 
 For more information about how CodeGen reflects system-level data from the database into the MJ metadata layer, see the [CodeGen documentation](../CodeGen/README.md).
 
+## Creating Error-Free Entity Files
+
+### Quick Start Checklist
+
+Before creating entity JSON files, follow this checklist to avoid common mistakes:
+
+✅ **1. Find the Entity Definition**
+- Open `packages/MJCoreEntities/src/generated/entity_subclasses.ts` or `packages/GeneratedEntities/src/generated/entity_subclasses.ts`
+- Search for `class [EntityName]Entity` (e.g., `class TemplateEntity`)
+- Review JSDoc comments and property definitions to identify required vs optional fields
+
+✅ **2. Check Required Fields**
+- Look for JSDoc comments with `@required` annotations
+- Fields without `?` in TypeScript definitions are typically required
+- Always include `Name` (almost always required)
+- Always include `UserID` (use System User ID: `ECAFCCEC-6A37-EF11-86D4-000D3A4E707E`)
+
+✅ **3. Validate Field Names**
+- Use exact field names from the BaseEntity class definition
+- Field names are case-sensitive
+- Don't assume fields exist (e.g., not all entities have `Status`)
+
+✅ **4. Use Correct File Naming**
+- Configuration files (.mj-sync.json, .mj-folder.json) must start with dot
+- Metadata files follow the `filePattern` in your .mj-sync.json
+- Most common: `"filePattern": "*.json"` (matches any .json file)
+- Alternative: `"filePattern": ".*.json"` (matches dot-prefixed .json files)
+
+✅ **5. Set Up Directory Structure**
+- Create `.mj-sync.json` in the entity directory
+- Use glob patterns: `"filePattern": "*.json"` (not regex: `".*.json"`)
+
+### Discovering Entity Structure
+
+**CRITICAL**: Before creating entity files, you must understand the entity's field structure. Most errors occur because users are unfamiliar with the required fields, data types, and constraints.
+
+#### Finding Entity Definitions
+
+The approach depends on whether you're working inside or outside the MemberJunction monorepo:
+
+##### Working Inside MJ Monorepo
+
+Entity classes are located in:
+
+- **Core MJ Entities**: `packages/MJCoreEntities/src/generated/entity_subclasses.ts`
+  - System entities like Users, Roles, EntityFields, etc.
+  - AI-related entities like AI Prompts, AI Models, etc.
+  
+- **Custom Entities**: `packages/GeneratedEntities/src/generated/entity_subclasses.ts`  
+  - Your application-specific entities
+  - Business domain entities
+
+##### Working Outside MJ Monorepo (In Your Own Project)
+
+Entity classes are located in:
+
+- **Core MJ Entities**: `node_modules/@memberjunction/core-entities/dist/generated/entity_subclasses.js`
+  - Note: This is compiled JavaScript, but your IDE should provide IntelliSense
+  - For TypeScript definitions: `node_modules/@memberjunction/core-entities/dist/generated/entity_subclasses.d.ts`
+  
+- **Custom Entities**: Your project's generated entities location (varies by project structure)
+  - Common locations: `src/generated/`, `packages/entities/`, or similar
+  - Look for files containing your custom entity classes
+
+##### Best Practice: Use Your IDE's IntelliSense
+
+**Recommended approach for all scenarios:**
+
+1. **Import the entity class** in your IDE:
+   ```typescript
+   import { TemplateEntity } from '@memberjunction/core-entities';
+   ```
+
+2. **Create an instance and explore with IntelliSense**:
+   ```typescript
+   const template = new TemplateEntity();
+   // Type "template." and let your IDE show available properties
+   ```
+
+3. **Check the class definition** (F12 or "Go to Definition") to see:
+   - JSDoc comments with field descriptions
+   - Required vs optional fields
+   - Field types and validation rules
+   - Relationships and constraints
+
+#### How to Find Required Fields
+
+1. **Use IDE IntelliSense** (Recommended):
+   - Import the entity class
+   - Create an instance: `const entity = new TemplateEntity();`
+   - Use "Go to Definition" (F12) to see the BaseEntity class
+   - Look for JSDoc comments and field definitions
+
+2. **Examine the BaseEntity Class**:
+   - Find the entity class (e.g., `class TemplateEntity`)
+   - Look at property declarations with JSDoc comments
+   - Check for required vs optional field annotations
+   - Review any validation methods or constraints
+
+3. **Runtime Metadata Discovery**:
+   ```typescript
+   import { Metadata } from '@memberjunction/core';
+   
+   const md = new Metadata();
+   const entityInfo = md.EntityByName('Templates');
+   console.log('Required fields:', entityInfo.Fields.filter(f => !f.AllowsNull));
+   ```
+
+#### Example: Templates Entity Structure
+```typescript
+// BaseEntity class (accessible via IDE IntelliSense)
+export class TemplateEntity extends BaseEntity {
+  /**
+   * Primary key - auto-generated GUID
+   */
+  ID: string;
+  
+  /**
+   * Template name - REQUIRED
+   * @required
+   */
+  Name: string;
+  
+  /**
+   * Template description - optional
+   */
+  Description?: string;
+  
+  /**
+   * User who created this template - REQUIRED
+   * Must be a valid User ID
+   * @required
+   * @foreignKey Users.ID
+   */
+  UserID: string;
+  
+  /**
+   * Category for organizing templates - optional
+   * @foreignKey TemplateCategories.ID
+   */
+  CategoryID?: string;
+  
+  // Note: Status field may not exist on all entities!
+}
+```
+
+#### Common Required Fields Pattern
+
+Most MJ entities follow these patterns:
+
+**Always Required:**
+- `ID` - Primary key (GUID) - auto-generated if not provided
+- `Name` - Human-readable name
+- `UserID` - Creator/owner (use System User: `ECAFCCEC-6A37-EF11-86D4-000D3A4E707E`)
+
+**Often Required:**
+- `Description` - Usually optional but recommended
+- Foreign key fields ending in `ID` - Check if they have `.optional()`
+
+**Be Careful With:**
+- `Status` fields - Some entities have them, others don't
+- Enum fields - Must match exact values from database
+- DateTime fields - Use ISO format: `2024-01-15T10:30:00Z`
+
+### Common Mistakes and Solutions
+
+#### ❌ Mistake 1: Using Non-Existent Fields
+```json
+{
+  "fields": {
+    "Name": "My Template",
+    "Status": "Active"  // ❌ Templates entity may not have Status field
+  }
+}
+```
+
+**✅ Solution**: Check the BaseEntity class first
+```typescript
+// In entity_subclasses.ts - if you don't see Status here, don't use it
+export class TemplateEntity extends BaseEntity {
+  Name: string;  // Required
+  Description?: string;  // Optional (note the ?)
+  // No Status field defined
+}
+```
+
+#### ❌ Mistake 2: Missing Required Fields
+```json
+{
+  "fields": {
+    "Name": "My Template"
+    // ❌ Missing required UserID
+  }
+}
+```
+
+**✅ Solution**: Include all required fields
+```json
+{
+  "fields": {
+    "Name": "My Template",
+    "UserID": "ECAFCCEC-6A37-EF11-86D4-000D3A4E707E"
+  }
+}
+```
+
+#### ❌ Mistake 3: Wrong File Pattern in .mj-sync.json
+```json
+{
+  "entity": "Templates",
+  "filePattern": ".*.json"  // ❌ This is regex, not glob
+}
+```
+
+**✅ Solution**: Use glob patterns
+```json
+{
+  "entity": "Templates", 
+  "filePattern": "*.json"  // ✅ Correct glob pattern
+}
+```
+
+#### ❌ Mistake 4: Incorrect Data Types
+```json
+{
+  "fields": {
+    "Name": "My Template",
+    "CreatedAt": "2024-01-15",  // ❌ Wrong datetime format
+    "Priority": "1"  // ❌ Should be number, not string
+  }
+}
+```
+
+**✅ Solution**: Use correct data types
+```json
+{
+  "fields": {
+    "Name": "My Template",
+    "CreatedAt": "2024-01-15T10:30:00Z",  // ✅ ISO format
+    "Priority": 1  // ✅ Number type
+  }
+}
+```
+
+#### ❌ Mistake 5: Files Not Being Detected
+```
+mydir/
+├── .mj-sync.json (with "filePattern": "*.json")
+├── template1.txt  // ❌ Wrong extension
+└── .template2.json  // ❌ Dot prefix when pattern is "*.json"
+```
+
+**✅ Solution**: Match your filePattern
+```
+mydir/
+├── .mj-sync.json (with "filePattern": "*.json")
+├── template1.json  // ✅ Matches *.json pattern
+└── template2.json  // ✅ Matches *.json pattern
+```
+
+### Step-by-Step Entity File Creation
+
+#### Step 1: Research the Entity
+```bash
+# Open in your IDE:
+packages/MJCoreEntities/src/generated/entity_subclasses.ts
+
+# Search for your entity class (Ctrl+F):
+class TemplateEntity
+
+# Note the required vs optional fields:
+Name: string;           // Required (no ?)
+UserID: string;         // Required (no ?)  
+Description?: string;   // Optional (note the ?)
+```
+
+#### Step 2: Create Directory Structure
+```bash
+mkdir templates
+cd templates
+
+# Create entity config (dot-prefixed configuration file)
+echo '{
+  "entity": "Templates",
+  "filePattern": "*.json"
+}' > .mj-sync.json
+```
+
+#### Step 3: Create Your First Entity File
+```bash
+# Create metadata file (follows filePattern from .mj-sync.json)
+echo '{
+  "fields": {
+    "Name": "My First Template",
+    "Description": "A test template",
+    "UserID": "ECAFCCEC-6A37-EF11-86D4-000D3A4E707E"
+  }
+}' > my-first-template.json
+```
+
+#### Step 4: Test and Validate
+```bash
+# Dry run to check for errors
+mj-sync push --dir="templates" --dry-run
+
+# If successful, do actual push
+mj-sync push --dir="templates"
+```
+
+### AI/LLM Guidelines
+
+When using AI tools (like Claude, ChatGPT, etc.) to generate entity files:
+
+**🤖 For AI Assistants:**
+
+1. **Always check entity definitions first** - Never assume field names or requirements
+2. **Look up the exact BaseEntity class** in the generated entity files
+3. **Use the System User ID** (`ECAFCCEC-6A37-EF11-86D4-000D3A4E707E`) for UserID fields
+4. **Include only fields that exist** in the entity definition
+5. **Use proper data types** as defined in the BaseEntity class
+6. **Remember file naming rules**:
+   - Configuration files (.mj-sync.json) must have dot prefix
+   - Metadata files follow the filePattern in .mj-sync.json
+7. **Use glob patterns** in .mj-sync.json, not regex patterns
+
+**📝 Prompt Template for AI:**
+```
+I need to create entity files for the [EntityName] entity in MemberJunction.
+
+Please:
+1. First, check the entity definition in packages/MJCoreEntities/src/generated/entity_subclasses.ts
+2. Find the class [EntityName]Entity (e.g., class TemplateEntity)
+3. Review JSDoc comments and property definitions to identify required vs optional fields
+4. Create a .mj-sync.json file with correct glob pattern
+5. Create sample metadata JSON files following the filePattern
+6. Use UserID: "ECAFCCEC-6A37-EF11-86D4-000D3A4E707E" for required UserID fields
+7. Follow the exact field names and data types from the BaseEntity class definition
+
+CRITICAL: Configuration files (.mj-sync.json) must start with dot, but metadata files follow the filePattern specified in the configuration.
+```
+
+### Understanding File Naming Rules
+
+**Configuration Files (Always Dot-Prefixed):**
+- ✅ `.mj-sync.json` - Entity configuration
+- ✅ `.mj-folder.json` - Folder defaults
+- ❌ `mj-sync.json` - Won't be recognized
+
+**Metadata Files (Follow filePattern):**
+With `"filePattern": "*.json"`:
+- ✅ `my-template.json` - Will be processed
+- ✅ `greeting.json` - Will be processed  
+- ❌ `.my-template.json` - Won't match pattern
+- ❌ `package.json` - Will be ignored (add to ignore list if needed)
+
+With `"filePattern": ".*.json"`:
+- ✅ `.my-template.json` - Will be processed
+- ✅ `.greeting.json` - Will be processed
+- ❌ `my-template.json` - Won't match pattern
+- ❌ `package.json` - Won't match pattern
+
+### Troubleshooting Quick Reference
+
+| Error Message | Cause | Solution |
+|---------------|-------|----------|
+| `No entity directories found` | Missing .mj-sync.json or wrong filePattern | Check .mj-sync.json exists and uses `"*.json"` |
+| `Field 'X' does not exist on entity 'Y'` | Using non-existent field | Check BaseEntity class in entity_subclasses.ts |
+| `User ID cannot be null` | Missing required UserID | Add `"UserID": "ECAFCCEC-6A37-EF11-86D4-000D3A4E707E"` |
+| `Processing 0 records` | Files don't match filePattern | Check files match pattern in .mj-sync.json |
+| Failed validation | Wrong data type or format | Check BaseEntity class for field types |
+
+### System User ID Reference
+
+**Always use this GUID for UserID fields:**
+```
+ECAFCCEC-6A37-EF11-86D4-000D3A4E707E
+```
+
+This is the System User ID that should be used when creating entity records through the MetadataSync tool. Using any other ID or leaving it null will cause validation errors.
+
 ## File Structure
 
 The tool uses a hierarchical directory structure with cascading defaults:
@@ -471,14 +851,129 @@ Configuration follows a hierarchical structure:
 - **Entity configs**: Each entity directory has its own config defining the entity type
 - **Inheritance**: All files within an entity directory are treated as records of that entity type
 
+### Directory Processing Order (NEW)
+
+The MetadataSync tool now supports custom directory processing order to handle dependencies between entity types. This feature ensures that dependent entities are processed in the correct order.
+
+#### Directory Order Configuration
+
+Directory order is configured in the root-level `.mj-sync.json` file only (not inherited by subdirectories):
+
+```json
+{
+  "version": "1.0.0",
+  "directoryOrder": [
+    "prompts",
+    "agent-types"
+  ]
+}
+```
+
+#### How It Works
+
+- **Ordered Processing**: Directories listed in `directoryOrder` are processed first, in the specified order
+- **Remaining Directories**: Any directories not listed are processed after the ordered ones, in alphabetical order
+- **Dependency Management**: Ensures prompts are created before agent types that reference them
+- **Flexible**: Only specify the directories that have order requirements
+
+#### Example Use Cases
+
+1. **AI Prompts → Agent Types**: Create prompts before agent types that reference them
+2. **Categories → Items**: Create category records before items that reference them
+3. **Parent → Child**: Process parent entities before child entities with foreign key dependencies
+
+### SQL Logging (NEW)
+
+The MetadataSync tool now supports SQL logging for capturing all database operations during push commands. This feature is useful for:
+- Creating migration files from MetadataSync operations
+- Debugging database changes
+- Understanding what SQL operations occur during push
+- Creating migration scripts for deployment to other environments
+
+#### SQL Logging Configuration
+
+SQL logging is configured in the root-level `.mj-sync.json` file only (not inherited by subdirectories):
+
+```json
+{
+  "version": "1.0.0",
+  "sqlLogging": {
+    "enabled": true,
+    "outputDirectory": "./sql_logging",
+    "formatAsMigration": true
+  }
+}
+```
+
+#### SQL Logging Options
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `enabled` | boolean | false | Whether to enable SQL logging during push operations |
+| `outputDirectory` | string | "./sql_logging" | Directory to output SQL log files (relative to command execution directory) |
+| `formatAsMigration` | boolean | false | Whether to format SQL as migration-ready files with Flyway schema placeholders |
+
+#### SQL Log File Format
+
+When `formatAsMigration` is `false`, log files are named:
+```
+metadatasync-push-YYYY-MM-DDTHH-MM-SS.sql
+```
+
+When `formatAsMigration` is `true`, log files are named as Flyway migrations:
+```
+VYYYYMMDDHHMMSS__MetadataSync_Push.sql
+```
+
+Migration files include:
+- Header comments with timestamp and description
+- Schema placeholders that can be replaced during deployment
+- Properly formatted SQL statements with parameters
+
+#### Example Usage
+
+1. **Enable SQL logging** in your root `.mj-sync.json`:
+   ```json
+   {
+     "version": "1.0.0",
+     "sqlLogging": {
+       "enabled": true,
+       "outputDirectory": "./migrations",
+       "formatAsMigration": true
+     }
+   }
+   ```
+
+2. **Run push command** as normal:
+   ```bash
+   mj-sync push
+   ```
+
+3. **Review generated SQL** in the output directory:
+   ```
+   migrations/
+   └── V20241215103045__MetadataSync_Push.sql
+   ```
+
+The SQL logging runs in parallel with the actual database operations, ensuring minimal performance impact while capturing all SQL statements for review and potential migration use.
+
 ### Root Configuration (metadata/.mj-sync.json)
 ```json
 {
   "version": "1.0.0",
+  "directoryOrder": [
+    "prompts",
+    "agent-types"
+  ],
   "push": {
     "validateBeforePush": true,
     "requireConfirmation": true
   },
+  "sqlLogging": {
+    "enabled": true,
+    "outputDirectory": "./sql_logging",
+    "formatAsMigration": false
+  },
   "watch": {
     "debounceMs": 1000,
     "ignorePatterns": ["*.tmp", "*.bak"]
@@ -552,6 +1047,108 @@ The tool now supports managing related entities as embedded collections within p
 - **Cleaner Organization**: Fewer files to manage
 - **Relationship Clarity**: Visual representation of data relationships
 
+## Recursive Patterns (NEW)
+
+The tool now supports automatic recursive patterns for self-referencing entities, eliminating the need to manually define each nesting level for hierarchical data structures.
+
+### Benefits
+- **Simplified Configuration**: No need to manually define each hierarchy level
+- **Automatic Depth Handling**: Adapts to actual data depth dynamically
+- **Reduced Maintenance**: Configuration stays simple regardless of data changes
+- **Safeguards**: Built-in protection against infinite loops and excessive memory usage
+
+### Recursive Configuration
+
+Enable recursive patterns for self-referencing entities:
+
+```json
+{
+  "pull": {
+    "entities": {
+      "AI Agents": {
+        "relatedEntities": {
+          "AI Agents": {
+            "entity": "AI Agents",
+            "foreignKey": "ParentID",
+            "recursive": true,        // Enable recursive fetching
+            "maxDepth": 10,          // Optional depth limit (omit for default of 10)
+            "filter": "Status = 'Active'"
+          }
+        }
+      }
+    }
+  }
+}
+```
+
+### How It Works
+
+When `recursive: true` is set:
+
+1. **Automatic Child Fetching**: The tool automatically fetches child records at each level
+2. **Dynamic Depth**: Continues until no more children are found or max depth is reached
+3. **Circular Reference Protection**: Prevents infinite loops by tracking processed record IDs
+4. **Consistent Configuration**: All recursive levels use the same `lookupFields`, `externalizeFields`, etc.
+
+### Before vs After
+
+**Before (Manual Configuration):**
+```json
+{
+  "pull": {
+    "relatedEntities": {
+      "AI Agents": {
+        "entity": "AI Agents", 
+        "foreignKey": "ParentID",
+        "relatedEntities": {
+          "AI Agents": {
+            "entity": "AI Agents",
+            "foreignKey": "ParentID",
+            "relatedEntities": {
+              "AI Agents": {
+                "entity": "AI Agents",
+                "foreignKey": "ParentID"
+                // Must manually add more levels...
+              }
+            }
+          }
+        }
+      }
+    }
+  }
+}
+```
+
+**After (Recursive Configuration):**
+```json
+{
+  "pull": {
+    "relatedEntities": {
+      "AI Agents": {
+        "entity": "AI Agents",
+        "foreignKey": "ParentID", 
+        "recursive": true,
+        "maxDepth": 10
+      }
+    }
+  }
+}
+```
+
+### Configuration Options
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `recursive` | boolean | false | Enable automatic recursive fetching |
+| `maxDepth` | number | 10 | Maximum recursion depth to prevent infinite loops |
+
+### Safeguards
+
+- **Circular Reference Detection**: Tracks processed record IDs to prevent infinite loops
+- **Maximum Depth Limit**: Configurable depth limit (default: 10) prevents excessive memory usage
+- **Performance Monitoring**: Verbose mode shows recursion depth and skipped circular references
+- **Backward Compatibility**: Existing configurations continue to work unchanged
+
 ### Configuration for Pull
 
 The pull command now supports smart update capabilities with extensive configuration options:

package/dist/commands/init/index.js

@@ -90,6 +90,31 @@ class Init extends core_1.Command {
         }
         catch (error) {
             spinner.fail('Initialization failed');
+            // Enhanced error logging for debugging
+            this.log('\n=== Initialization Error Details ===');
+            this.log(`Error type: ${error?.constructor?.name || 'Unknown'}`);
+            this.log(`Error message: ${error instanceof Error ? error.message : String(error)}`);
+            if (error instanceof Error && error.stack) {
+                this.log(`\nStack trace:`);
+                this.log(error.stack);
+            }
+            // Log context information
+            this.log(`\nContext:`);
+            this.log(`- Working directory: ${process.cwd()}`);
+            // Check if error is related to common issues
+            const errorMessage = error instanceof Error ? error.message : String(error);
+            if (errorMessage.includes('permission') || errorMessage.includes('EACCES')) {
+                this.log(`\nHint: This appears to be a file permission issue.`);
+                this.log(`Make sure you have write permissions in the current directory.`);
+            }
+            else if (errorMessage.includes('ENOENT') || errorMessage.includes('no such file')) {
+                this.log(`\nHint: This appears to be a file or directory access issue.`);
+                this.log(`Make sure the current directory exists and is accessible.`);
+            }
+            else if (errorMessage.includes('already exists') || errorMessage.includes('EEXIST')) {
+                this.log(`\nHint: Files or directories already exist.`);
+                this.log(`Try using the overwrite option or manually remove existing files.`);
+            }
             this.error(error);
         }
     }