@memberjunction/metadata-sync 2.47.0 → 2.49.0

package/README.md

@@ -98,16 +98,410 @@ The tool is intended for managing business-level metadata such as:
 
 For more information about how CodeGen reflects system-level data from the database into the MJ metadata layer, see the [CodeGen documentation](../CodeGen/README.md).
 
+## Creating Error-Free Entity Files
+
+### Quick Start Checklist
+
+Before creating entity JSON files, follow this checklist to avoid common mistakes:
+
+✅ **1. Find the Entity Definition**
+- Open `packages/MJCoreEntities/src/generated/entity_subclasses.ts` or `packages/GeneratedEntities/src/generated/entity_subclasses.ts`
+- Search for `class [EntityName]Entity` (e.g., `class TemplateEntity`)
+- Review JSDoc comments and property definitions to identify required vs optional fields
+
+✅ **2. Check Required Fields**
+- Look for JSDoc comments with `@required` annotations
+- Fields without `?` in TypeScript definitions are typically required
+- Always include `Name` (almost always required)
+- Always include `UserID` (use System User ID: `ECAFCCEC-6A37-EF11-86D4-000D3A4E707E`)
+
+✅ **3. Validate Field Names**
+- Use exact field names from the BaseEntity class definition
+- Field names are case-sensitive
+- Don't assume fields exist (e.g., not all entities have `Status`)
+
+✅ **4. Use Correct File Naming**
+- Configuration files (.mj-sync.json, .mj-folder.json) must start with dot
+- Metadata files follow the `filePattern` in your .mj-sync.json
+- Most common: `"filePattern": "*.json"` (matches any .json file)
+- Alternative: `"filePattern": ".*.json"` (matches dot-prefixed .json files)
+
+✅ **5. Set Up Directory Structure**
+- Create `.mj-sync.json` in the entity directory
+- Use glob patterns: `"filePattern": "*.json"` (not regex: `".*.json"`)
+
+### Discovering Entity Structure
+
+**CRITICAL**: Before creating entity files, you must understand the entity's field structure. Most errors occur because users are unfamiliar with the required fields, data types, and constraints.
+
+#### Finding Entity Definitions
+
+The approach depends on whether you're working inside or outside the MemberJunction monorepo:
+
+##### Working Inside MJ Monorepo
+
+Entity classes are located in:
+
+- **Core MJ Entities**: `packages/MJCoreEntities/src/generated/entity_subclasses.ts`
+  - System entities like Users, Roles, EntityFields, etc.
+  - AI-related entities like AI Prompts, AI Models, etc.
+  
+- **Custom Entities**: `packages/GeneratedEntities/src/generated/entity_subclasses.ts`  
+  - Your application-specific entities
+  - Business domain entities
+
+##### Working Outside MJ Monorepo (In Your Own Project)
+
+Entity classes are located in:
+
+- **Core MJ Entities**: `node_modules/@memberjunction/core-entities/dist/generated/entity_subclasses.js`
+  - Note: This is compiled JavaScript, but your IDE should provide IntelliSense
+  - For TypeScript definitions: `node_modules/@memberjunction/core-entities/dist/generated/entity_subclasses.d.ts`
+  
+- **Custom Entities**: Your project's generated entities location (varies by project structure)
+  - Common locations: `src/generated/`, `packages/entities/`, or similar
+  - Look for files containing your custom entity classes
+
+##### Best Practice: Use Your IDE's IntelliSense
+
+**Recommended approach for all scenarios:**
+
+1. **Import the entity class** in your IDE:
+   ```typescript
+   import { TemplateEntity } from '@memberjunction/core-entities';
+   ```
+
+2. **Create an instance and explore with IntelliSense**:
+   ```typescript
+   const template = new TemplateEntity();
+   // Type "template." and let your IDE show available properties
+   ```
+
+3. **Check the class definition** (F12 or "Go to Definition") to see:
+   - JSDoc comments with field descriptions
+   - Required vs optional fields
+   - Field types and validation rules
+   - Relationships and constraints
+
+#### How to Find Required Fields
+
+1. **Use IDE IntelliSense** (Recommended):
+   - Import the entity class
+   - Create an instance: `const entity = new TemplateEntity();`
+   - Use "Go to Definition" (F12) to see the BaseEntity class
+   - Look for JSDoc comments and field definitions
+
+2. **Examine the BaseEntity Class**:
+   - Find the entity class (e.g., `class TemplateEntity`)
+   - Look at property declarations with JSDoc comments
+   - Check for required vs optional field annotations
+   - Review any validation methods or constraints
+
+3. **Runtime Metadata Discovery**:
+   ```typescript
+   import { Metadata } from '@memberjunction/core';
+   
+   const md = new Metadata();
+   const entityInfo = md.EntityByName('Templates');
+   console.log('Required fields:', entityInfo.Fields.filter(f => !f.AllowsNull));
+   ```
+
+#### Example: Templates Entity Structure
+```typescript
+// BaseEntity class (accessible via IDE IntelliSense)
+export class TemplateEntity extends BaseEntity {
+  /**
+   * Primary key - auto-generated GUID
+   */
+  ID: string;
+  
+  /**
+   * Template name - REQUIRED
+   * @required
+   */
+  Name: string;
+  
+  /**
+   * Template description - optional
+   */
+  Description?: string;
+  
+  /**
+   * User who created this template - REQUIRED
+   * Must be a valid User ID
+   * @required
+   * @foreignKey Users.ID
+   */
+  UserID: string;
+  
+  /**
+   * Category for organizing templates - optional
+   * @foreignKey TemplateCategories.ID
+   */
+  CategoryID?: string;
+  
+  // Note: Status field may not exist on all entities!
+}
+```
+
+#### Common Required Fields Pattern
+
+Most MJ entities follow these patterns:
+
+**Always Required:**
+- `ID` - Primary key (GUID) - auto-generated if not provided
+- `Name` - Human-readable name
+- `UserID` - Creator/owner (use System User: `ECAFCCEC-6A37-EF11-86D4-000D3A4E707E`)
+
+**Often Required:**
+- `Description` - Usually optional but recommended
+- Foreign key fields ending in `ID` - Check if they have `.optional()`
+
+**Be Careful With:**
+- `Status` fields - Some entities have them, others don't
+- Enum fields - Must match exact values from database
+- DateTime fields - Use ISO format: `2024-01-15T10:30:00Z`
+
+### Common Mistakes and Solutions
+
+#### ❌ Mistake 1: Using Non-Existent Fields
+```json
+{
+  "fields": {
+    "Name": "My Template",
+    "Status": "Active"  // ❌ Templates entity may not have Status field
+  }
+}
+```
+
+**✅ Solution**: Check the BaseEntity class first
+```typescript
+// In entity_subclasses.ts - if you don't see Status here, don't use it
+export class TemplateEntity extends BaseEntity {
+  Name: string;  // Required
+  Description?: string;  // Optional (note the ?)
+  // No Status field defined
+}
+```
+
+#### ❌ Mistake 2: Missing Required Fields
+```json
+{
+  "fields": {
+    "Name": "My Template"
+    // ❌ Missing required UserID
+  }
+}
+```
+
+**✅ Solution**: Include all required fields
+```json
+{
+  "fields": {
+    "Name": "My Template",
+    "UserID": "ECAFCCEC-6A37-EF11-86D4-000D3A4E707E"
+  }
+}
+```
+
+#### ❌ Mistake 3: Wrong File Pattern in .mj-sync.json
+```json
+{
+  "entity": "Templates",
+  "filePattern": ".*.json"  // ❌ This is regex, not glob
+}
+```
+
+**✅ Solution**: Use glob patterns
+```json
+{
+  "entity": "Templates", 
+  "filePattern": "*.json"  // ✅ Correct glob pattern
+}
+```
+
+#### ❌ Mistake 4: Incorrect Data Types
+```json
+{
+  "fields": {
+    "Name": "My Template",
+    "CreatedAt": "2024-01-15",  // ❌ Wrong datetime format
+    "Priority": "1"  // ❌ Should be number, not string
+  }
+}
+```
+
+**✅ Solution**: Use correct data types
+```json
+{
+  "fields": {
+    "Name": "My Template",
+    "CreatedAt": "2024-01-15T10:30:00Z",  // ✅ ISO format
+    "Priority": 1  // ✅ Number type
+  }
+}
+```
+
+#### ❌ Mistake 5: Files Not Being Detected
+```
+mydir/
+├── .mj-sync.json (with "filePattern": "*.json")
+├── template1.txt  // ❌ Wrong extension
+└── .template2.json  // ❌ Dot prefix when pattern is "*.json"
+```
+
+**✅ Solution**: Match your filePattern
+```
+mydir/
+├── .mj-sync.json (with "filePattern": "*.json")
+├── template1.json  // ✅ Matches *.json pattern
+└── template2.json  // ✅ Matches *.json pattern
+```
+
+### Step-by-Step Entity File Creation
+
+#### Step 1: Research the Entity
+```bash
+# Open in your IDE:
+packages/MJCoreEntities/src/generated/entity_subclasses.ts
+
+# Search for your entity class (Ctrl+F):
+class TemplateEntity
+
+# Note the required vs optional fields:
+Name: string;           // Required (no ?)
+UserID: string;         // Required (no ?)  
+Description?: string;   // Optional (note the ?)
+```
+
+#### Step 2: Create Directory Structure
+```bash
+mkdir templates
+cd templates
+
+# Create entity config (dot-prefixed configuration file)
+echo '{
+  "entity": "Templates",
+  "filePattern": "*.json"
+}' > .mj-sync.json
+```
+
+#### Step 3: Create Your First Entity File
+```bash
+# Create metadata file (follows filePattern from .mj-sync.json)
+echo '{
+  "fields": {
+    "Name": "My First Template",
+    "Description": "A test template",
+    "UserID": "ECAFCCEC-6A37-EF11-86D4-000D3A4E707E"
+  }
+}' > my-first-template.json
+```
+
+#### Step 4: Test and Validate
+```bash
+# Dry run to check for errors
+mj-sync push --dir="templates" --dry-run
+
+# If successful, do actual push
+mj-sync push --dir="templates"
+```
+
+### AI/LLM Guidelines
+
+When using AI tools (like Claude, ChatGPT, etc.) to generate entity files:
+
+**🤖 For AI Assistants:**
+
+1. **Always check entity definitions first** - Never assume field names or requirements
+2. **Look up the exact BaseEntity class** in the generated entity files
+3. **Use the System User ID** (`ECAFCCEC-6A37-EF11-86D4-000D3A4E707E`) for UserID fields
+4. **Include only fields that exist** in the entity definition
+5. **Use proper data types** as defined in the BaseEntity class
+6. **Remember file naming rules**:
+   - Configuration files (.mj-sync.json) must have dot prefix
+   - Metadata files follow the filePattern in .mj-sync.json
+7. **Use glob patterns** in .mj-sync.json, not regex patterns
+
+**📝 Prompt Template for AI:**
+```
+I need to create entity files for the [EntityName] entity in MemberJunction.
+
+Please:
+1. First, check the entity definition in packages/MJCoreEntities/src/generated/entity_subclasses.ts
+2. Find the class [EntityName]Entity (e.g., class TemplateEntity)
+3. Review JSDoc comments and property definitions to identify required vs optional fields
+4. Create a .mj-sync.json file with correct glob pattern
+5. Create sample metadata JSON files following the filePattern
+6. Use UserID: "ECAFCCEC-6A37-EF11-86D4-000D3A4E707E" for required UserID fields
+7. Follow the exact field names and data types from the BaseEntity class definition
+
+CRITICAL: Configuration files (.mj-sync.json) must start with dot, but metadata files follow the filePattern specified in the configuration.
+```
+
+### Understanding File Naming Rules
+
+**Configuration Files (Always Dot-Prefixed):**
+- ✅ `.mj-sync.json` - Entity configuration
+- ✅ `.mj-folder.json` - Folder defaults
+- ❌ `mj-sync.json` - Won't be recognized
+
+**Metadata Files (Follow filePattern):**
+With `"filePattern": "*.json"`:
+- ✅ `my-template.json` - Will be processed
+- ✅ `greeting.json` - Will be processed  
+- ❌ `.my-template.json` - Won't match pattern
+- ❌ `package.json` - Will be ignored (add to ignore list if needed)
+
+With `"filePattern": ".*.json"`:
+- ✅ `.my-template.json` - Will be processed
+- ✅ `.greeting.json` - Will be processed
+- ❌ `my-template.json` - Won't match pattern
+- ❌ `package.json` - Won't match pattern
+
+### Troubleshooting Quick Reference
+
+| Error Message | Cause | Solution |
+|---------------|-------|----------|
+| `No entity directories found` | Missing .mj-sync.json or wrong filePattern | Check .mj-sync.json exists and uses `"*.json"` |
+| `Field 'X' does not exist on entity 'Y'` | Using non-existent field | Check BaseEntity class in entity_subclasses.ts |
+| `User ID cannot be null` | Missing required UserID | Add `"UserID": "ECAFCCEC-6A37-EF11-86D4-000D3A4E707E"` |
+| `Processing 0 records` | Files don't match filePattern | Check files match pattern in .mj-sync.json |
+| Failed validation | Wrong data type or format | Check BaseEntity class for field types |
+
+### System User ID Reference
+
+**Always use this GUID for UserID fields:**
+```
+ECAFCCEC-6A37-EF11-86D4-000D3A4E707E
+```
+
+This is the System User ID that should be used when creating entity records through the MetadataSync tool. Using any other ID or leaving it null will cause validation errors.
+
 ## File Structure
 
 The tool uses a hierarchical directory structure with cascading defaults:
 - Each top-level directory represents an entity type
 - `.mj-sync.json` files define entities and base defaults
 - `.mj-folder.json` files define folder-specific defaults (optional)
-- All JSON files within are treated as records of that entity type
+- Only dot-prefixed JSON files (e.g., `.prompt-template.json`, `.category.json`) are treated as metadata records
+- Regular JSON files without the dot prefix are ignored, allowing package.json and other config files to coexist
 - External files (`.md`, `.html`, etc.) are referenced from the JSON files
 - Defaults cascade down through the folder hierarchy
 
+### File Naming Convention
+
+**Metadata files must be prefixed with a dot (.)** to be recognized by the sync tool. This convention:
+- Clearly distinguishes metadata files from regular configuration files
+- Allows `package.json`, `tsconfig.json` and other standard files to coexist without being processed
+- Follows established patterns like `.gitignore` and `.eslintrc.json`
+
+Examples:
+- ✅ `.greeting.json` - Will be processed as metadata
+- ✅ `.customer-prompt.json` - Will be processed as metadata  
+- ❌ `greeting.json` - Will be ignored
+- ❌ `package.json` - Will be ignored
+
 ### File Format Options
 
 #### Single Record per File (Default)
@@ -147,12 +541,12 @@ metadata/
 │   ├── .mj-sync.json               # Defines entity: "AI Prompts"
 │   ├── customer-service/
 │   │   ├── .mj-folder.json         # Folder metadata (CategoryID, etc.)
-│   │   ├── greeting.json           # AI Prompt record with embedded models
+│   │   ├── .greeting.json          # AI Prompt record with embedded models
 │   │   ├── greeting.prompt.md      # Prompt content (referenced)
 │   │   └── greeting.notes.md       # Notes field (referenced)
 │   └── analytics/
 │       ├── .mj-folder.json         # Folder metadata (CategoryID, etc.)
-│       ├── daily-report.json       # AI Prompt record
+│       ├── .daily-report.json      # AI Prompt record
 │       └── daily-report.prompt.md  # Prompt content (referenced)
 ├── templates/                       # Reusable JSON templates
 │   ├── standard-prompt-settings.json # Common prompt configurations
@@ -163,17 +557,17 @@ metadata/
     ├── .mj-sync.json               # Defines entity: "Templates"
     ├── email/
     │   ├── .mj-folder.json         # Folder metadata
-    │   ├── welcome.json            # Template record
+    │   ├── .welcome.json           # Template record (dot-prefixed)
     │   └── welcome.template.html   # Template content (referenced)
     └── reports/
         ├── .mj-folder.json         # Folder metadata
-        ├── invoice.json            # Template record
+        ├── .invoice.json           # Template record (dot-prefixed)
         └── invoice.template.html   # Template content (referenced)
 ```
 
 ## JSON Metadata Format
 
-### Individual Record (e.g., ai-prompts/customer-service/greeting.json)
+### Individual Record (e.g., ai-prompts/customer-service/.greeting.json)
 ```json
 {
   "fields": {
@@ -457,14 +851,129 @@ Configuration follows a hierarchical structure:
 - **Entity configs**: Each entity directory has its own config defining the entity type
 - **Inheritance**: All files within an entity directory are treated as records of that entity type
 
+### Directory Processing Order (NEW)
+
+The MetadataSync tool now supports custom directory processing order to handle dependencies between entity types. This feature ensures that dependent entities are processed in the correct order.
+
+#### Directory Order Configuration
+
+Directory order is configured in the root-level `.mj-sync.json` file only (not inherited by subdirectories):
+
+```json
+{
+  "version": "1.0.0",
+  "directoryOrder": [
+    "prompts",
+    "agent-types"
+  ]
+}
+```
+
+#### How It Works
+
+- **Ordered Processing**: Directories listed in `directoryOrder` are processed first, in the specified order
+- **Remaining Directories**: Any directories not listed are processed after the ordered ones, in alphabetical order
+- **Dependency Management**: Ensures prompts are created before agent types that reference them
+- **Flexible**: Only specify the directories that have order requirements
+
+#### Example Use Cases
+
+1. **AI Prompts → Agent Types**: Create prompts before agent types that reference them
+2. **Categories → Items**: Create category records before items that reference them
+3. **Parent → Child**: Process parent entities before child entities with foreign key dependencies
+
+### SQL Logging (NEW)
+
+The MetadataSync tool now supports SQL logging for capturing all database operations during push commands. This feature is useful for:
+- Creating migration files from MetadataSync operations
+- Debugging database changes
+- Understanding what SQL operations occur during push
+- Creating migration scripts for deployment to other environments
+
+#### SQL Logging Configuration
+
+SQL logging is configured in the root-level `.mj-sync.json` file only (not inherited by subdirectories):
+
+```json
+{
+  "version": "1.0.0",
+  "sqlLogging": {
+    "enabled": true,
+    "outputDirectory": "./sql_logging",
+    "formatAsMigration": true
+  }
+}
+```
+
+#### SQL Logging Options
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `enabled` | boolean | false | Whether to enable SQL logging during push operations |
+| `outputDirectory` | string | "./sql_logging" | Directory to output SQL log files (relative to command execution directory) |
+| `formatAsMigration` | boolean | false | Whether to format SQL as migration-ready files with Flyway schema placeholders |
+
+#### SQL Log File Format
+
+When `formatAsMigration` is `false`, log files are named:
+```
+metadatasync-push-YYYY-MM-DDTHH-MM-SS.sql
+```
+
+When `formatAsMigration` is `true`, log files are named as Flyway migrations:
+```
+VYYYYMMDDHHMMSS__MetadataSync_Push.sql
+```
+
+Migration files include:
+- Header comments with timestamp and description
+- Schema placeholders that can be replaced during deployment
+- Properly formatted SQL statements with parameters
+
+#### Example Usage
+
+1. **Enable SQL logging** in your root `.mj-sync.json`:
+   ```json
+   {
+     "version": "1.0.0",
+     "sqlLogging": {
+       "enabled": true,
+       "outputDirectory": "./migrations",
+       "formatAsMigration": true
+     }
+   }
+   ```
+
+2. **Run push command** as normal:
+   ```bash
+   mj-sync push
+   ```
+
+3. **Review generated SQL** in the output directory:
+   ```
+   migrations/
+   └── V20241215103045__MetadataSync_Push.sql
+   ```
+
+The SQL logging runs in parallel with the actual database operations, ensuring minimal performance impact while capturing all SQL statements for review and potential migration use.
+
 ### Root Configuration (metadata/.mj-sync.json)
 ```json
 {
   "version": "1.0.0",
+  "directoryOrder": [
+    "prompts",
+    "agent-types"
+  ],
   "push": {
     "validateBeforePush": true,
     "requireConfirmation": true
   },
+  "sqlLogging": {
+    "enabled": true,
+    "outputDirectory": "./sql_logging",
+    "formatAsMigration": false
+  },
   "watch": {
     "debounceMs": 1000,
     "ignorePatterns": ["*.tmp", "*.bak"]
@@ -476,7 +985,7 @@ Configuration follows a hierarchical structure:
 ```json
 {
   "entity": "AI Prompts",
-  "filePattern": "*.json",
+  "filePattern": ".*.json",
   "defaults": {
     "TypeID": "@lookup:AI Prompt Types.Name=Chat",
     "Temperature": 0.7,
@@ -484,11 +993,21 @@ Configuration follows a hierarchical structure:
     "Status": "Active"
   },
   "pull": {
+    "filePattern": ".*.json",
+    "updateExistingRecords": true,
+    "createNewFileIfNotFound": true,
+    "mergeStrategy": "merge",
     "filter": "Status = 'Active'",
+    "externalizeFields": [
+      {
+        "field": "Prompt",
+        "pattern": "@file:{Name}.prompt.md"
+      }
+    ],
     "relatedEntities": {
       "MJ: AI Prompt Models": {
         "entity": "MJ: AI Prompt Models",
-        "foreignKey": "ID",
+        "foreignKey": "PromptID",
         "filter": "Status = 'Active'"
       }
     }
@@ -528,27 +1047,307 @@ The tool now supports managing related entities as embedded collections within p
 - **Cleaner Organization**: Fewer files to manage
 - **Relationship Clarity**: Visual representation of data relationships
 
+## Recursive Patterns (NEW)
+
+The tool now supports automatic recursive patterns for self-referencing entities, eliminating the need to manually define each nesting level for hierarchical data structures.
+
+### Benefits
+- **Simplified Configuration**: No need to manually define each hierarchy level
+- **Automatic Depth Handling**: Adapts to actual data depth dynamically
+- **Reduced Maintenance**: Configuration stays simple regardless of data changes
+- **Safeguards**: Built-in protection against infinite loops and excessive memory usage
+
+### Recursive Configuration
+
+Enable recursive patterns for self-referencing entities:
+
+```json
+{
+  "pull": {
+    "entities": {
+      "AI Agents": {
+        "relatedEntities": {
+          "AI Agents": {
+            "entity": "AI Agents",
+            "foreignKey": "ParentID",
+            "recursive": true,        // Enable recursive fetching
+            "maxDepth": 10,          // Optional depth limit (omit for default of 10)
+            "filter": "Status = 'Active'"
+          }
+        }
+      }
+    }
+  }
+}
+```
+
+### How It Works
+
+When `recursive: true` is set:
+
+1. **Automatic Child Fetching**: The tool automatically fetches child records at each level
+2. **Dynamic Depth**: Continues until no more children are found or max depth is reached
+3. **Circular Reference Protection**: Prevents infinite loops by tracking processed record IDs
+4. **Consistent Configuration**: All recursive levels use the same `lookupFields`, `externalizeFields`, etc.
+
+### Before vs After
+
+**Before (Manual Configuration):**
+```json
+{
+  "pull": {
+    "relatedEntities": {
+      "AI Agents": {
+        "entity": "AI Agents", 
+        "foreignKey": "ParentID",
+        "relatedEntities": {
+          "AI Agents": {
+            "entity": "AI Agents",
+            "foreignKey": "ParentID",
+            "relatedEntities": {
+              "AI Agents": {
+                "entity": "AI Agents",
+                "foreignKey": "ParentID"
+                // Must manually add more levels...
+              }
+            }
+          }
+        }
+      }
+    }
+  }
+}
+```
+
+**After (Recursive Configuration):**
+```json
+{
+  "pull": {
+    "relatedEntities": {
+      "AI Agents": {
+        "entity": "AI Agents",
+        "foreignKey": "ParentID", 
+        "recursive": true,
+        "maxDepth": 10
+      }
+    }
+  }
+}
+```
+
+### Configuration Options
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `recursive` | boolean | false | Enable automatic recursive fetching |
+| `maxDepth` | number | 10 | Maximum recursion depth to prevent infinite loops |
+
+### Safeguards
+
+- **Circular Reference Detection**: Tracks processed record IDs to prevent infinite loops
+- **Maximum Depth Limit**: Configurable depth limit (default: 10) prevents excessive memory usage
+- **Performance Monitoring**: Verbose mode shows recursion depth and skipped circular references
+- **Backward Compatibility**: Existing configurations continue to work unchanged
+
 ### Configuration for Pull
-Configure which related entities to pull in `.mj-sync.json`:
+
+The pull command now supports smart update capabilities with extensive configuration options:
+
 ```json
 {
   "entity": "AI Prompts",
+  "filePattern": ".*.json",
   "pull": {
+    "filePattern": ".*.json",
+    "createNewFileIfNotFound": true,
+    "newFileName": ".all-new.json",
+    "appendRecordsToExistingFile": true,
+    "updateExistingRecords": true,
+    "preserveFields": ["customField", "localNotes"],
+    "mergeStrategy": "merge",
+    "backupBeforeUpdate": true,
+    "filter": "Status = 'Active'",
+    "externalizeFields": [
+      {
+        "field": "TemplateText",
+        "pattern": "@file:{Name}.template.md"
+      },
+      {
+        "field": "PromptText",
+        "pattern": "@file:prompts/{Name}.prompt.md"
+      }
+    ],
+    "excludeFields": ["InternalID", "TempField"],
+    "lookupFields": {
+      "CategoryID": {
+        "entity": "AI Prompt Categories",
+        "field": "Name"
+      },
+      "TypeID": {
+        "entity": "AI Prompt Types",
+        "field": "Name"
+      }
+    },
     "relatedEntities": {
       "MJ: AI Prompt Models": {
         "entity": "MJ: AI Prompt Models",
-        "foreignKey": "ID",
-        "filter": "Status = 'Active'"
-      },
-      "AI Prompt Parameters": {
-        "entity": "AI Prompt Parameters",
-        "foreignKey": "ID"
+        "foreignKey": "PromptID",
+        "filter": "Status = 'Active'",
+        "lookupFields": {
+          "ModelID": {
+            "entity": "AI Models",
+            "field": "Name"
+          }
+        }
       }
     }
   }
 }
 ```
 
+#### Pull Configuration Options
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `filePattern` | string | Entity filePattern | Pattern for finding existing files to update |
+| `createNewFileIfNotFound` | boolean | true | Create files for records not found locally |
+| `newFileName` | string | - | Filename for new records when appending (see warning below) |
+| `appendRecordsToExistingFile` | boolean | false | Append new records to a single file |
+| `updateExistingRecords` | boolean | true | Update existing records found in local files |
+| `preserveFields` | string[] | [] | Fields that retain local values during updates (see detailed explanation below) |
+| `mergeStrategy` | string | "merge" | How to merge updates: "merge", "overwrite", or "skip" |
+| `backupBeforeUpdate` | boolean | false | Create timestamped backups before updating files |
+| `backupDirectory` | string | ".backups" | Directory name for backup files (relative to entity directory) |
+| `filter` | string | - | SQL WHERE clause for filtering records |
+| `externalizeFields` | array/object | - | Fields to save as external files with optional patterns |
+| `excludeFields` | string[] | [] | Fields to completely omit from pulled data (see detailed explanation below) |
+| `lookupFields` | object | - | Foreign keys to convert to @lookup references |
+| `relatedEntities` | object | - | Related entities to pull as embedded collections |
+
+> **⚠️ Important Configuration Warning**
+> 
+> When both `appendRecordsToExistingFile: true` and `newFileName` are set, ALL new records will be appended to the single file specified by `newFileName`, effectively ignoring the standard per-record file pattern. This can lead to unexpected file organization:
+> 
+> ```json
+> // This configuration will put ALL new records in .all-new.json
+> "pull": {
+>   "appendRecordsToExistingFile": true,
+>   "newFileName": ".all-new.json"  // ⚠️ Overrides individual file creation
+> }
+> ```
+> 
+> **Recommended configurations:**
+> - For individual files per record: Set `appendRecordsToExistingFile: false` (or omit it)
+> - For grouped new records: Set both `appendRecordsToExistingFile: true` and `newFileName`
+> - For mixed approach: Omit `newFileName` to let new records follow the standard pattern
+
+#### Merge Strategies
+
+- **`merge`** (default): Combines fields from database and local file, with database values taking precedence for existing fields
+- **`overwrite`**: Completely replaces local record with database version (except preserved fields)
+- **`skip`**: Leaves existing records unchanged, only adds new records
+
+#### Understanding excludeFields vs preserveFields
+
+These two configuration options serve different purposes for managing fields during pull operations:
+
+##### excludeFields
+- **Purpose**: Completely omit specified fields from your local files
+- **Use Case**: Remove internal/system fields you don't want in version control
+- **Effect**: Fields never appear in the JSON files
+- **Example**: Excluding internal IDs, timestamps, or sensitive data
+
+##### preserveFields  
+- **Purpose**: Protect local customizations from being overwritten during updates
+- **Use Case**: Keep locally modified values while updating other fields
+- **Effect**: Fields exist in files but retain their local values during pull
+- **Example**: Preserving custom file paths, local notes, or environment-specific values
+- **Special Behavior for @file: references**: When a preserved field contains a `@file:` reference, the tool will update the content at the existing file path rather than creating a new file with a generated name
+
+##### Example Configuration
+```json
+{
+  "pull": {
+    "excludeFields": ["TemplateID", "InternalNotes", "CreatedAt"],
+    "preserveFields": ["TemplateText", "OutputExample", "LocalConfig"]
+  }
+}
+```
+
+With this configuration:
+- **TemplateID, InternalNotes, CreatedAt** → Never appear in local files
+- **TemplateText, OutputExample, LocalConfig** → Keep their local values during updates
+
+##### Common Scenario: Customized File References
+When you customize file paths (e.g., changing `@file:templates/skip-conductor.md` to `@file:templates/conductor.md`), use `preserveFields` to protect these customizations:
+
+```json
+{
+  "pull": {
+    "preserveFields": ["TemplateText", "OutputExample"],
+    "externalizeFields": [
+      {
+        "field": "TemplateText",
+        "pattern": "@file:templates/{Name}.template.md"
+      }
+    ]
+  }
+}
+```
+
+This ensures your custom paths aren't overwritten when pulling updates from the database.
+
+**How it works:**
+1. **Without preserveFields**: Pull would create a new file using the pattern (e.g., `templates/skip-conductor.template.md`) and update the JSON to point to it
+2. **With preserveFields**: Pull keeps your custom path (e.g., `@file:templates/conductor.md`) in the JSON and updates the content at that existing location
+
+This is particularly useful when:
+- You've reorganized your file structure after initial pull
+- You've renamed files to follow your own naming conventions
+- You want to maintain consistent paths across team members
+
+#### Backup Configuration
+
+When `backupBeforeUpdate` is enabled, the tool creates timestamped backups before updating existing files:
+
+- **Backup Location**: Files are backed up to the `backupDirectory` (default: `.backups`) within the entity directory
+- **Backup Naming**: Original filename + timestamp + `.backup` extension (e.g., `.greeting.json` → `.greeting.2024-03-15T10-30-45-123Z.backup`)
+- **Extension**: All backup files use the `.backup` extension, preventing them from being processed by push/pull/status commands
+- **Deduplication**: Only one backup is created per file per pull operation, even if the file contains multiple records
+
+Example configuration:
+```json
+"pull": {
+  "backupBeforeUpdate": true,
+  "backupDirectory": ".backups"  // Custom backup directory name
+}
+```
+
+#### Externalize Fields Patterns
+
+The `externalizeFields` configuration supports dynamic file naming with placeholders:
+
+```json
+"externalizeFields": [
+  {
+    "field": "TemplateText",
+    "pattern": "@file:{Name}.template.md"
+  },
+  {
+    "field": "SQLQuery", 
+    "pattern": "@file:queries/{CategoryName}/{Name}.sql"
+  }
+]
+```
+
+Supported placeholders:
+- `{Name}` - The entity's name field value
+- `{ID}` - The entity's primary key
+- `{FieldName}` - The field being externalized
+- `{AnyFieldName}` - Any field from the entity record
+
+All values are sanitized for filesystem compatibility (lowercase, spaces to hyphens, special characters removed).
+
 ### Nested Related Entities
 Support for multiple levels of nesting:
 ```json