@memberjunction/metadata-sync 2.52.0 → 2.54.0

package/README.md

@@ -718,14 +718,27 @@ Examples:
 ### @lookup: References (ENHANCED)
 Enable entity relationships using human-readable values:
 - Basic syntax: `@lookup:EntityName.FieldName=Value`
+- Multi-field syntax: `@lookup:EntityName.Field1=Value1&Field2=Value2`
 - Auto-create syntax: `@lookup:EntityName.FieldName=Value?create`
 - With additional fields: `@lookup:EntityName.FieldName=Value?create&Field2=Value2`
 
 Examples:
-- `@lookup:AI Prompt Types.Name=Chat` - Fails if not found
+- `@lookup:AI Prompt Types.Name=Chat` - Single field lookup, fails if not found
+- `@lookup:Users.Email=john@example.com&Department=Sales` - Multi-field lookup for precise matching
 - `@lookup:AI Prompt Categories.Name=Examples?create` - Creates if missing
 - `@lookup:AI Prompt Categories.Name=Examples?create&Description=Example prompts` - Creates with description
 
+#### Multi-Field Lookups (NEW)
+When you need to match records based on multiple criteria, use the multi-field syntax:
+```json
+{
+  "CategoryID": "@lookup:AI Prompt Categories.Name=Actions&Status=Active",
+  "ManagerID": "@lookup:Users.Email=manager@company.com&Department=Engineering&Status=Active"
+}
+```
+
+This ensures you get the exact record you want when multiple records might have the same value in a single field.
+
 ### @parent: References (NEW)
 Reference fields from the immediate parent entity in embedded collections:
 - `@parent:ID` - Get the parent's ID field
@@ -743,6 +756,84 @@ Support environment-specific values:
 - `@env:VARIABLE_NAME`
 - Useful for different environments (dev/staging/prod)
 
+### {@include} References in Files (NEW)
+Enable content composition within non-JSON files (like .md, .html, .txt) using JSDoc-style include syntax:
+- Pattern: `{@include path/to/file.ext}`
+- Supports relative paths from the containing file
+- Recursive includes (includes within includes)
+- Circular reference detection prevents infinite loops
+- Works seamlessly with `@file:` references
+
+#### How It Works
+When a JSON metadata file uses `@file:` to reference an external file, the MetadataSync tool:
+1. Loads the referenced file
+2. Scans for `{@include}` patterns
+3. Recursively resolves all includes
+4. Returns the fully composed content
+
+#### Example Usage
+```markdown
+# My Prompt Template
+
+## System Instructions
+{@include ./shared/system-instructions.md}
+
+## Context
+{@include ../common/context-header.md}
+
+## Task
+Please analyze the following...
+```
+
+#### Complex Example with Nested Includes
+Directory structure:
+```
+prompts/
+├── customer-service/
+│   ├── greeting.json          # Uses @file:greeting.md
+│   ├── greeting.md            # Contains {@include} references
+│   └── shared/
+│       ├── tone.md
+│       └── guidelines.md
+└── common/
+    ├── company-info.md
+    └── legal-disclaimer.md
+```
+
+greeting.json:
+```json
+{
+  "fields": {
+    "Name": "Customer Greeting",
+    "Prompt": "@file:greeting.md"
+  }
+}
+```
+
+greeting.md:
+```markdown
+# Customer Service Greeting
+
+{@include ./shared/tone.md}
+
+## Guidelines
+{@include ./shared/guidelines.md}
+
+## Company Information
+{@include ../common/company-info.md}
+
+## Legal
+{@include ../common/legal-disclaimer.md}
+```
+
+The final content pushed to the database will have all includes fully resolved.
+
+Benefits:
+- **DRY Principle**: Share common content across multiple files
+- **Maintainability**: Update shared content in one place
+- **Flexibility**: Build complex documents from modular parts
+- **Validation**: Automatic checking of included file existence and circular references
+
 ### @template: References (NEW)
 Enable JSON template composition for reusable configurations:
 
@@ -818,6 +909,9 @@ mj-sync validate --verbose
 # Validate with JSON output for CI/CD
 mj-sync validate --format=json
 
+# Save validation report to markdown file
+mj-sync validate --save-report
+
 # Initialize a directory for metadata sync
 mj-sync init
 
@@ -898,6 +992,53 @@ Directory order is configured in the root-level `.mj-sync.json` file only (not i
 2. **Categories → Items**: Create category records before items that reference them
 3. **Parent → Child**: Process parent entities before child entities with foreign key dependencies
 
+### Ignore Directories
+
+The MetadataSync tool supports ignoring specific directories during push/pull operations. This is useful for:
+- Excluding output or example directories from processing
+- Skipping temporary or build directories
+- Organizing support files without them being processed as metadata
+
+#### Configuration
+
+Ignore directories are configured in `.mj-sync.json` files and are **cumulative** through the directory hierarchy:
+
+```json
+{
+  "version": "1.0.0",
+  "ignoreDirectories": [
+    "output",
+    "examples",
+    "templates"
+  ]
+}
+```
+
+#### How It Works
+
+- **Cumulative Inheritance**: Each directory inherits ignore patterns from its parent directories
+- **Relative Paths**: Directory names are relative to the location of the `.mj-sync.json` file
+- **Simple Patterns**: Supports exact directory names (e.g., "output", "temp")
+- **Additive**: Child directories can add their own ignore patterns to parent patterns
+
+#### Example
+
+```
+metadata/.mj-sync.json            → ignoreDirectories: ["output", "temp"]
+├── prompts/.mj-sync.json         → ignoreDirectories: ["examples"]
+│   ├── output/                   → IGNORED (from root)
+│   ├── examples/                 → IGNORED (from prompts)
+│   └── production/.mj-sync.json  → ignoreDirectories: ["drafts"]
+│       ├── drafts/               → IGNORED (from production)
+│       └── output/               → IGNORED (inherited from root)
+```
+
+In this example:
+- Root level ignores "output" and "temp" everywhere
+- Prompts directory adds "examples" to the ignore list
+- Production subdirectory further adds "drafts"
+- All patterns are cumulative, so production inherits all parent ignores
+
 ### SQL Logging (NEW)
 
 The MetadataSync tool now supports SQL logging for capturing all database operations during push commands. This feature is useful for:
@@ -973,6 +1114,75 @@ Migration files include:
 
 The SQL logging runs in parallel with the actual database operations, ensuring minimal performance impact while capturing all SQL statements for review and potential migration use.
 
+### User Role Validation (NEW)
+
+MetadataSync now supports validating UserID fields against specific roles in the MemberJunction system. This ensures that only users with appropriate roles can be referenced in metadata files.
+
+#### Configuration
+
+Add the `userRoleValidation` configuration to your root `.mj-sync.json` file:
+
+```json
+{
+  "version": "1.0.0",
+  "userRoleValidation": {
+    "enabled": true,
+    "allowedRoles": [
+      "Administrator",
+      "Developer",
+      "Content Manager"
+    ],
+    "allowUsersWithoutRoles": false
+  }
+}
+```
+
+#### Configuration Options
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `enabled` | boolean | false | Enable user role validation for UserID fields |
+| `allowedRoles` | string[] | [] | List of role names that are allowed |
+| `allowUsersWithoutRoles` | boolean | false | Allow users without any assigned roles |
+
+#### How It Works
+
+1. During validation, all user roles are loaded from the database and cached
+2. For each UserID field in metadata files, the validator checks:
+   - If the user exists and has roles assigned
+   - If the user has at least one of the allowed roles
+3. Validation fails if:
+   - A UserID references a user without any roles (unless `allowUsersWithoutRoles` is true)
+   - A UserID references a user whose roles are not in the `allowedRoles` list
+
+#### Example
+
+Given a metadata file with a UserID field:
+
+```json
+{
+  "fields": {
+    "Name": "Admin Action",
+    "UserID": "user-123"
+  }
+}
+```
+
+The validation will:
+1. Check if user-123 exists in the system
+2. Verify that user-123 has one of the allowed roles
+3. Report an error if the user doesn't have appropriate roles
+
+#### Error Messages
+
+```
+✗ UserID 'user-123' does not have any assigned roles
+  Suggestion: User must have one of these roles: Administrator, Developer
+
+✗ UserID 'user-456' has roles [Viewer] but none are in allowed list
+  Suggestion: Allowed roles: Administrator, Developer, Content Manager
+```
+
 ### Root Configuration (metadata/.mj-sync.json)
 ```json
 {
@@ -990,6 +1200,11 @@ The SQL logging runs in parallel with the actual database operations, ensuring m
     "outputDirectory": "./sql_logging",
     "formatAsMigration": false
   },
+  "userRoleValidation": {
+    "enabled": true,
+    "allowedRoles": ["Administrator", "Developer"],
+    "allowUsersWithoutRoles": false
+  },
   "watch": {
     "debounceMs": 1000,
     "ignorePatterns": ["*.tmp", "*.bak"]

package/dist/commands/push/index.d.ts

@@ -26,6 +26,10 @@ export default class Push extends Command {
      * Check if a record has already been processed and warn if duplicate
      */
     private checkAndTrackRecord;
+    /**
+     * Format field value for console display
+     */
+    private formatFieldValue;
     /**
      * Parse JSON file and track line numbers for array elements
      */

package/dist/commands/push/index.js

@@ -137,7 +137,7 @@ class Push extends core_1.Command {
                 }
             }
             // Find entity directories to process
-            const entityDirs = (0, provider_utils_1.findEntityDirectories)(config_manager_1.configManager.getOriginalCwd(), flags.dir, syncConfig?.directoryOrder);
+            const entityDirs = (0, provider_utils_1.findEntityDirectories)(config_manager_1.configManager.getOriginalCwd(), flags.dir, syncConfig?.directoryOrder, syncConfig?.ignoreDirectories);
             if (entityDirs.length === 0) {
                 this.error('No entity directories found');
             }
@@ -168,7 +168,9 @@ class Push extends core_1.Command {
                             default: false
                         });
                         if (!shouldContinue) {
-                            this.error('Push cancelled due to validation errors.');
+                            this.log(chalk_1.default.yellow('\n⚠️  Push cancelled due to validation errors.'));
+                            // Exit cleanly without throwing an error
+                            return;
                         }
                     }
                 }
@@ -249,7 +251,12 @@ class Push extends core_1.Command {
                 if (flags.verbose) {
                     this.log(`\nProcessing ${entityConfig.entity} in ${entityDir}`);
                 }
-                const result = await this.processEntityDirectory(entityDir, entityConfig, syncEngine, flags, syncConfig, fileBackupManager);
+                // Combine root ignoreDirectories with entity-level ignoreDirectories
+                const initialIgnoreDirectories = [
+                    ...(syncConfig?.ignoreDirectories || []),
+                    ...(entityConfig?.ignoreDirectories || [])
+                ];
+                const result = await this.processEntityDirectory(entityDir, entityConfig, syncEngine, flags, syncConfig, fileBackupManager, initialIgnoreDirectories);
                 // Show per-directory summary
                 const dirName = path_1.default.relative(process.cwd(), entityDir) || '.';
                 const dirTotal = result.created + result.updated + result.unchanged;
@@ -444,7 +451,7 @@ class Push extends core_1.Command {
             process.exit(0);
         }
     }
-    async processEntityDirectory(entityDir, entityConfig, syncEngine, flags, syncConfig, fileBackupManager) {
+    async processEntityDirectory(entityDir, entityConfig, syncEngine, flags, syncConfig, fileBackupManager, parentIgnoreDirectories) {
         const result = { created: 0, updated: 0, unchanged: 0, errors: 0 };
         // Find files matching the configured pattern
         const pattern = entityConfig.filePattern || '*.json';
@@ -454,6 +461,56 @@ class Push extends core_1.Command {
             dot: true, // Include dotfiles (files starting with .)
             onlyFiles: true
         });
+        // Check if no JSON files were found
+        if (jsonFiles.length === 0) {
+            const relativePath = path_1.default.relative(process.cwd(), entityDir) || '.';
+            const parentPath = path_1.default.dirname(entityDir);
+            const dirName = path_1.default.basename(entityDir);
+            // Check if this is a subdirectory (not a top-level entity directory)
+            const isSubdirectory = parentPath !== path_1.default.resolve(config_manager_1.configManager.getOriginalCwd(), flags.dir || '.');
+            if (isSubdirectory) {
+                // For subdirectories, make it a warning instead of an error
+                let warningMessage = `No JSON files found in ${relativePath} matching pattern: ${pattern}`;
+                // Try to be more helpful by checking what files do exist
+                const allFiles = await (0, fast_glob_1.default)('*', {
+                    cwd: entityDir,
+                    onlyFiles: true,
+                    dot: true
+                });
+                if (allFiles.length > 0) {
+                    warningMessage += `\n  Files found: ${allFiles.slice(0, 3).join(', ')}`;
+                    if (allFiles.length > 3) {
+                        warningMessage += ` (and ${allFiles.length - 3} more)`;
+                    }
+                }
+                const rootConfigPath = path_1.default.join(config_manager_1.configManager.getOriginalCwd(), flags.dir || '.', '.mj-sync.json');
+                warningMessage += `\n  💡 If this directory should be ignored, add "${dirName}" to the "ignoreDirectories" array in:\n     ${rootConfigPath}`;
+                this.warn(warningMessage);
+                return result; // Return early without processing further
+            }
+            else {
+                // For top-level entity directories, this is still an error
+                const configFile = path_1.default.join(entityDir, '.mj-sync.json');
+                let errorMessage = `No JSON files found in ${relativePath} matching pattern: ${pattern}\n`;
+                errorMessage += `\nPlease check:\n`;
+                errorMessage += `  1. Files exist with the expected extension (.json)\n`;
+                errorMessage += `  2. The filePattern in ${configFile} matches your files\n`;
+                errorMessage += `  3. Files are not in ignored patterns: .mj-sync.json, .mj-folder.json, *.backup\n`;
+                // Try to be more helpful by checking what files do exist
+                const allFiles = await (0, fast_glob_1.default)('*', {
+                    cwd: entityDir,
+                    onlyFiles: true,
+                    dot: true
+                });
+                if (allFiles.length > 0) {
+                    errorMessage += `\nFiles found in directory: ${allFiles.slice(0, 5).join(', ')}`;
+                    if (allFiles.length > 5) {
+                        errorMessage += ` (and ${allFiles.length - 5} more)`;
+                    }
+                }
+                throw new Error(errorMessage);
+            }
+        }
         if (flags.verbose) {
             this.log(`Processing ${jsonFiles.length} records in ${path_1.default.relative(process.cwd(), entityDir) || '.'}`);
         }
@@ -463,6 +520,24 @@ class Push extends core_1.Command {
         const entries = await fs_extra_1.default.readdir(entityDir, { withFileTypes: true });
         for (const entry of entries) {
             if (entry.isDirectory() && !entry.name.startsWith('.')) {
+                // Build cumulative ignore list: parent + current directory's ignores
+                const currentDirConfig = await (0, config_1.loadSyncConfig)(entityDir);
+                const currentEntityConfig = await (0, config_1.loadEntityConfig)(entityDir);
+                const cumulativeIgnoreDirectories = [
+                    ...(parentIgnoreDirectories || []),
+                    ...(currentDirConfig?.ignoreDirectories || []),
+                    ...(currentEntityConfig?.ignoreDirectories || [])
+                ];
+                // Check if this directory should be ignored
+                if (cumulativeIgnoreDirectories.some((pattern) => {
+                    // Simple pattern matching: exact name or ends with pattern
+                    return entry.name === pattern || entry.name.endsWith(pattern);
+                })) {
+                    if (flags.verbose) {
+                        this.log(`  Ignoring directory: ${entry.name} (matched ignore pattern)`);
+                    }
+                    continue;
+                }
                 const subDir = path_1.default.join(entityDir, entry.name);
                 // Load subdirectory config and merge with parent config
                 let subEntityConfig = { ...entityConfig };
@@ -483,8 +558,8 @@ class Push extends core_1.Command {
                         }
                     };
                 }
-                // Process subdirectory with merged config
-                const subResult = await this.processEntityDirectory(subDir, subEntityConfig, syncEngine, flags, syncConfig, fileBackupManager);
+                // Process subdirectory with merged config and cumulative ignore directories
+                const subResult = await this.processEntityDirectory(subDir, subEntityConfig, syncEngine, flags, syncConfig, fileBackupManager, cumulativeIgnoreDirectories);
                 result.created += subResult.created;
                 result.updated += subResult.updated;
                 result.unchanged += subResult.unchanged;
@@ -558,6 +633,7 @@ class Push extends core_1.Command {
                 const fullErrorMessage = `Failed to process ${file}: ${errorMessage}`;
                 this.errors.push(fullErrorMessage);
                 this.error(fullErrorMessage, { exit: false });
+                this.log('   ⚠️  This error will cause all changes to be rolled back at the end of processing');
             }
         }
         if (flags.verbose) {
@@ -640,7 +716,7 @@ class Push extends core_1.Command {
                 try {
                     const processedValue = await syncEngine.processFieldValue(value, baseDir, null, null);
                     if (verbose) {
-                        this.log(`  Setting ${field}: ${JSON.stringify(value)} -> ${JSON.stringify(processedValue)}`);
+                        this.log(`  Setting ${field}: ${this.formatFieldValue(value)} -> ${this.formatFieldValue(processedValue)}`);
                     }
                     entity[field] = processedValue;
                 }
@@ -688,7 +764,7 @@ class Push extends core_1.Command {
                     const field = entity.GetFieldByName(fieldName);
                     const oldValue = field ? field.OldValue : undefined;
                     const newValue = changes[fieldName];
-                    this.log(`     ${fieldName}: ${oldValue} → ${newValue}`);
+                    this.log(`     ${fieldName}: ${this.formatFieldValue(oldValue)} → ${this.formatFieldValue(newValue)}`);
                 }
             }
         }
@@ -814,7 +890,7 @@ class Push extends core_1.Command {
                             try {
                                 const processedValue = await syncEngine.processFieldValue(value, baseDir, parentEntity, rootEntity);
                                 if (verbose) {
-                                    this.log(`${indent}  Setting ${field}: ${JSON.stringify(value)} -> ${JSON.stringify(processedValue)}`);
+                                    this.log(`${indent}  Setting ${field}: ${this.formatFieldValue(value)} -> ${this.formatFieldValue(processedValue)}`);
                                 }
                                 entity[field] = processedValue;
                             }
@@ -859,7 +935,7 @@ class Push extends core_1.Command {
                                 const field = entity.GetFieldByName(fieldName);
                                 const oldValue = field ? field.OldValue : undefined;
                                 const newValue = changes[fieldName];
-                                this.log(`${indent}     ${fieldName}: ${oldValue} → ${newValue}`);
+                                this.log(`${indent}     ${fieldName}: ${this.formatFieldValue(oldValue)} → ${this.formatFieldValue(newValue)}`);
                             }
                         }
                     }
@@ -973,6 +1049,20 @@ class Push extends core_1.Command {
         });
         return false; // not duplicate
     }
+    /**
+     * Format field value for console display
+     */
+    formatFieldValue(value, maxLength = 50) {
+        // Convert value to string representation
+        let strValue = JSON.stringify(value);
+        // Trim the string
+        strValue = strValue.trim();
+        // If it's longer than maxLength, truncate and add ellipsis
+        if (strValue.length > maxLength) {
+            return strValue.substring(0, maxLength) + '...';
+        }
+        return strValue;
+    }
     /**
      * Parse JSON file and track line numbers for array elements
      */