@memberjunction/metadata-sync 2.46.0 → 2.47.0

package/README.md

@@ -74,18 +74,29 @@ The Metadata Sync tool bridges the gap between database-stored metadata and file
 
 ## Supported Entities
 
-### Phase 1: AI Prompts (Current)
-- Full support for all AI Prompt fields
-- Markdown files for prompt content
-- Category-based organization
-- AI Prompt Models as embedded collections
-
-### Future Phases
-- Templates
-- AI Models
-- AI Vendors
-- Query definitions
-- Any MJ entity with metadata
+The tool works with any MemberJunction entity - both core system entities and user-created entities. Each entity type can have its own directory structure, file naming conventions, and related entity configurations.
+
+### Important Limitation: Database-Reflected Metadata
+
+**This tool should NOT be used to modify metadata that is reflected from the underlying database catalog.** Examples include:
+- Entity field data types
+- Column lengths/precision
+- Primary key definitions
+- Foreign key relationships
+- Table/column existence
+
+These properties are designed to flow **from** the database catalog **up** into MJ metadata, not the other way around. Attempting to modify these via file sync could create inconsistencies between the metadata and actual database schema.
+
+The tool is intended for managing business-level metadata such as:
+- Descriptions and documentation
+- Display names and user-facing text
+- Categories and groupings
+- Custom properties and settings
+- AI prompts, templates, and other content
+- Permissions and security settings
+- Any other data that is not reflected **up** from the underlying system database catalogs
+
+For more information about how CodeGen reflects system-level data from the database into the MJ metadata layer, see the [CodeGen documentation](../CodeGen/README.md).
 
 ## File Structure
 
@@ -97,6 +108,37 @@ The tool uses a hierarchical directory structure with cascading defaults:
 - External files (`.md`, `.html`, etc.) are referenced from the JSON files
 - Defaults cascade down through the folder hierarchy
 
+### File Format Options
+
+#### Single Record per File (Default)
+Each JSON file contains one record:
+```json
+{
+  "fields": { ... },
+  "relatedEntities": { ... }
+}
+```
+
+#### Multiple Records per File (NEW)
+JSON files can contain arrays of records:
+```json
+[
+  {
+    "fields": { ... },
+    "relatedEntities": { ... }
+  },
+  {
+    "fields": { ... },
+    "relatedEntities": { ... }
+  }
+]
+```
+
+This is useful for:
+- Grouping related records in a single file
+- Reducing file clutter for entities with many small records
+- Maintaining logical groupings while using `@file:` references for large content
+
 ### Example Structure
 ```
 metadata/
@@ -112,7 +154,12 @@ metadata/
 │       ├── .mj-folder.json         # Folder metadata (CategoryID, etc.)
 │       ├── daily-report.json       # AI Prompt record
 │       └── daily-report.prompt.md  # Prompt content (referenced)
-└── templates/
+├── templates/                       # Reusable JSON templates
+│   ├── standard-prompt-settings.json # Common prompt configurations
+│   ├── standard-ai-models.json     # Standard model configurations
+│   ├── high-performance-models.json # High-power model configurations
+│   └── customer-service-defaults.json # CS-specific defaults
+└── template-entities/
     ├── .mj-sync.json               # Defines entity: "Templates"
     ├── email/
     │   ├── .mj-folder.json         # Folder metadata
@@ -302,6 +349,66 @@ Support environment-specific values:
 - `@env:VARIABLE_NAME`
 - Useful for different environments (dev/staging/prod)
 
+### @template: References (NEW)
+Enable JSON template composition for reusable configurations:
+
+#### String Template Reference
+Use `@template:` to replace any value with template content:
+```json
+{
+  "relatedEntities": {
+    "MJ: AI Prompt Models": "@template:templates/standard-ai-models.json"
+  }
+}
+```
+
+#### Object Template Merging
+Use `@template` field within objects to merge template content:
+```json
+{
+  "fields": {
+    "Name": "My Prompt",
+    "@template": "templates/standard-prompt-settings.json",
+    "Temperature": 0.9  // Overrides template value
+  }
+}
+```
+
+#### Multiple Template Merging
+Merge multiple templates in order (later templates override earlier ones):
+```json
+{
+  "fields": {
+    "@template": [
+      "templates/base-settings.json",
+      "templates/customer-service-defaults.json"
+    ],
+    "Name": "Customer Bot"  // Local fields override all templates
+  }
+}
+```
+
+#### Nested Templates
+Templates can reference other templates:
+```json
+// templates/high-performance-models.json
+[
+  {
+    "fields": {
+      "@template": "../templates/model-defaults.json",
+      "ModelID": "@lookup:AI Models.Name=GPT 4o"
+    }
+  }
+]
+```
+
+#### Template Benefits
+- **DRY Principle**: Define configurations once, use everywhere
+- **Maintainability**: Update template to affect all uses
+- **Flexibility**: Use at any JSON level
+- **Composability**: Build complex configurations from simple parts
+- **Override Support**: Local values always override template values
+
 ## CLI Commands
 
 ```bash
@@ -314,6 +421,10 @@ mj-sync pull --entity="AI Prompts"
 # Pull specific records by filter
 mj-sync pull --entity="AI Prompts" --filter="CategoryID='customer-service-id'"
 
+# Pull multiple records into a single file (NEW)
+mj-sync pull --entity="AI Prompts" --multi-file="all-prompts"
+mj-sync pull --entity="AI Prompts" --filter="Status='Active'" --multi-file="active-prompts.json"
+
 # Push all changes from current directory and subdirectories
 mj-sync push
 

package/dist/commands/pull/index.d.ts

@@ -1,4 +1,34 @@
+/**
+ * @fileoverview Pull command implementation for MetadataSync
+ * @module commands/pull
+ *
+ * This module implements the pull command which retrieves metadata records from
+ * the MemberJunction database and saves them as local JSON files. It supports:
+ * - Filtering records with SQL expressions
+ * - Pulling related entities with foreign key relationships
+ * - Externalizing large text fields to separate files
+ * - Creating multi-record JSON files
+ * - Recursive directory search for entity configurations
+ */
 import { Command } from '@oclif/core';
+/**
+ * Pull metadata records from database to local files
+ *
+ * @class Pull
+ * @extends Command
+ *
+ * @example
+ * ```bash
+ * # Pull all records for an entity
+ * mj-sync pull --entity="AI Prompts"
+ *
+ * # Pull with filter
+ * mj-sync pull --entity="AI Prompts" --filter="CategoryID='123'"
+ *
+ * # Pull to multi-record file
+ * mj-sync pull --entity="AI Prompts" --multi-file="all-prompts.json"
+ * ```
+ */
 export default class Pull extends Command {
     static description: string;
     static examples: string[];
@@ -6,13 +36,118 @@ export default class Pull extends Command {
         entity: import("@oclif/core/lib/interfaces").OptionFlag<string, import("@oclif/core/lib/interfaces").CustomOptions>;
         filter: import("@oclif/core/lib/interfaces").OptionFlag<string | undefined, import("@oclif/core/lib/interfaces").CustomOptions>;
         'dry-run': import("@oclif/core/lib/interfaces").BooleanFlag<boolean>;
+        'multi-file': import("@oclif/core/lib/interfaces").OptionFlag<string | undefined, import("@oclif/core/lib/interfaces").CustomOptions>;
     };
     run(): Promise<void>;
+    /**
+     * Find directories containing configuration for the specified entity
+     *
+     * Recursively searches the current working directory for .mj-sync.json files
+     * that specify the given entity name. Returns all matching directories to
+     * allow user selection when multiple locations exist.
+     *
+     * @param entityName - Name of the entity to search for
+     * @returns Promise resolving to array of directory paths
+     * @private
+     */
     private findEntityDirectories;
+    /**
+     * Process a single record and save to file
+     *
+     * Converts a database record into the file format and writes it to disk.
+     * This is a wrapper around processRecordData that handles file writing.
+     *
+     * @param record - Raw database record
+     * @param primaryKey - Primary key fields and values
+     * @param targetDir - Directory to save the file
+     * @param entityConfig - Entity configuration with pull settings
+     * @param syncEngine - Sync engine instance
+     * @returns Promise that resolves when file is written
+     * @private
+     */
     private processRecord;
+    /**
+     * Process record data for storage
+     *
+     * Transforms a raw database record into the RecordData format used for file storage.
+     * Handles field externalization, related entity pulling, and checksum calculation.
+     *
+     * @param record - Raw database record
+     * @param primaryKey - Primary key fields and values
+     * @param targetDir - Directory where files will be saved
+     * @param entityConfig - Entity configuration with defaults and settings
+     * @param syncEngine - Sync engine for checksum calculation
+     * @returns Promise resolving to formatted RecordData
+     * @private
+     */
+    private processRecordData;
+    /**
+     * Determine if a field should be saved to an external file
+     *
+     * Checks if a field contains substantial text content that would be better
+     * stored in a separate file rather than inline in the JSON. Uses heuristics
+     * based on field name and content length.
+     *
+     * @param fieldName - Name of the field to check
+     * @param fieldValue - Value of the field
+     * @param entityConfig - Entity configuration (for future extension)
+     * @returns Promise resolving to true if field should be externalized
+     * @private
+     */
     private shouldExternalizeField;
+    /**
+     * Create an external file for a field value
+     *
+     * Saves large text content to a separate file and returns the filename.
+     * Automatically determines appropriate file extension based on field name
+     * and content type (e.g., .md for prompts, .html for templates).
+     *
+     * @param targetDir - Directory to save the file
+     * @param primaryKey - Primary key for filename generation
+     * @param fieldName - Name of the field being externalized
+     * @param content - Content to write to the file
+     * @returns Promise resolving to the created filename
+     * @private
+     */
     private createExternalFile;
+    /**
+     * Build a filename from primary key values
+     *
+     * Creates a safe filename based on the entity's primary key values.
+     * Handles GUIDs by using first 8 characters, sanitizes special characters,
+     * and creates composite names for multi-field keys.
+     *
+     * @param primaryKey - Primary key fields and values
+     * @param entityConfig - Entity configuration (for future extension)
+     * @returns Filename with .json extension
+     * @private
+     */
     private buildFileName;
+    /**
+     * Pull related entities for a parent record
+     *
+     * Retrieves child records that have foreign key relationships to the parent.
+     * Converts foreign key values to @parent references and supports nested
+     * related entities for deep object graphs.
+     *
+     * @param parentRecord - Parent entity record
+     * @param relatedConfig - Configuration for related entities to pull
+     * @param syncEngine - Sync engine instance
+     * @returns Promise resolving to map of entity names to related records
+     * @private
+     */
     private pullRelatedEntities;
+    /**
+     * Find which field in the parent record contains a specific value
+     *
+     * Used to convert foreign key references to @parent references by finding
+     * the parent field that contains the foreign key value. Typically finds
+     * the primary key field but can match any field.
+     *
+     * @param parentRecord - Parent record to search
+     * @param value - Value to search for
+     * @returns Field name containing the value, or null if not found
+     * @private
+     */
     private findParentField;
 }

package/dist/commands/pull/index.js

@@ -1,4 +1,16 @@
 "use strict";
+/**
+ * @fileoverview Pull command implementation for MetadataSync
+ * @module commands/pull
+ *
+ * This module implements the pull command which retrieves metadata records from
+ * the MemberJunction database and saves them as local JSON files. It supports:
+ * - Filtering records with SQL expressions
+ * - Pulling related entities with foreign key relationships
+ * - Externalizing large text fields to separate files
+ * - Creating multi-record JSON files
+ * - Recursive directory search for entity configurations
+ */
 var __importDefault = (this && this.__importDefault) || function (mod) {
     return (mod && mod.__esModule) ? mod : { "default": mod };
 };
@@ -12,6 +24,24 @@ const config_1 = require("../../config");
 const sync_engine_1 = require("../../lib/sync-engine");
 const core_2 = require("@memberjunction/core");
 const provider_utils_1 = require("../../lib/provider-utils");
+/**
+ * Pull metadata records from database to local files
+ *
+ * @class Pull
+ * @extends Command
+ *
+ * @example
+ * ```bash
+ * # Pull all records for an entity
+ * mj-sync pull --entity="AI Prompts"
+ *
+ * # Pull with filter
+ * mj-sync pull --entity="AI Prompts" --filter="CategoryID='123'"
+ *
+ * # Pull to multi-record file
+ * mj-sync pull --entity="AI Prompts" --multi-file="all-prompts.json"
+ * ```
+ */
 class Pull extends core_1.Command {
     static description = 'Pull metadata from database to local files';
     static examples = [
@@ -22,6 +52,7 @@ class Pull extends core_1.Command {
         entity: core_1.Flags.string({ description: 'Entity name to pull', required: true }),
         filter: core_1.Flags.string({ description: 'Additional filter for pulling specific records' }),
         'dry-run': core_1.Flags.boolean({ description: 'Show what would be pulled without actually pulling' }),
+        'multi-file': core_1.Flags.string({ description: 'Create a single file with multiple records (provide filename)' }),
     };
     async run() {
         const { flags } = await this.parse(Pull);
@@ -88,23 +119,54 @@ class Pull extends core_1.Command {
             }
             spinner.start('Processing records');
             let processed = 0;
-            for (const record of result.Results) {
-                try {
-                    // Build primary key
-                    const primaryKey = {};
-                    for (const pk of entityInfo.PrimaryKeys) {
-                        primaryKey[pk.Name] = record[pk.Name];
+            // If multi-file flag is set, collect all records
+            if (flags['multi-file']) {
+                const allRecords = [];
+                for (const record of result.Results) {
+                    try {
+                        // Build primary key
+                        const primaryKey = {};
+                        for (const pk of entityInfo.PrimaryKeys) {
+                            primaryKey[pk.Name] = record[pk.Name];
+                        }
+                        // Process record for multi-file
+                        const recordData = await this.processRecordData(record, primaryKey, targetDir, entityConfig, syncEngine);
+                        allRecords.push(recordData);
+                        processed++;
+                        spinner.text = `Processing records (${processed}/${result.Results.length})`;
+                    }
+                    catch (error) {
+                        this.warn(`Failed to process record: ${error.message || error}`);
                     }
-                    // Process record
-                    await this.processRecord(record, primaryKey, targetDir, entityConfig, syncEngine);
-                    processed++;
-                    spinner.text = `Processing records (${processed}/${result.Results.length})`;
                 }
-                catch (error) {
-                    this.warn(`Failed to process record: ${error.message || error}`);
+                // Write all records to single file
+                if (allRecords.length > 0) {
+                    const fileName = flags['multi-file'].endsWith('.json') ? flags['multi-file'] : `${flags['multi-file']}.json`;
+                    const filePath = path_1.default.join(targetDir, fileName);
+                    await fs_extra_1.default.writeJson(filePath, allRecords, { spaces: 2 });
+                    spinner.succeed(`Pulled ${processed} records to ${filePath}`);
+                }
+            }
+            else {
+                // Original single-file-per-record logic
+                for (const record of result.Results) {
+                    try {
+                        // Build primary key
+                        const primaryKey = {};
+                        for (const pk of entityInfo.PrimaryKeys) {
+                            primaryKey[pk.Name] = record[pk.Name];
+                        }
+                        // Process record
+                        await this.processRecord(record, primaryKey, targetDir, entityConfig, syncEngine);
+                        processed++;
+                        spinner.text = `Processing records (${processed}/${result.Results.length})`;
+                    }
+                    catch (error) {
+                        this.warn(`Failed to process record: ${error.message || error}`);
+                    }
                 }
+                spinner.succeed(`Pulled ${processed} records to ${targetDir}`);
             }
-            spinner.succeed(`Pulled ${processed} records to ${targetDir}`);
         }
         catch (error) {
             spinner.fail('Pull failed');
@@ -115,6 +177,17 @@ class Pull extends core_1.Command {
             await (0, provider_utils_1.cleanupProvider)();
         }
     }
+    /**
+     * Find directories containing configuration for the specified entity
+     *
+     * Recursively searches the current working directory for .mj-sync.json files
+     * that specify the given entity name. Returns all matching directories to
+     * allow user selection when multiple locations exist.
+     *
+     * @param entityName - Name of the entity to search for
+     * @returns Promise resolving to array of directory paths
+     * @private
+     */
     async findEntityDirectories(entityName) {
         const dirs = [];
         // Search for directories with matching entity config
@@ -137,7 +210,43 @@ class Pull extends core_1.Command {
         await searchDirs(process.cwd());
         return dirs;
     }
+    /**
+     * Process a single record and save to file
+     *
+     * Converts a database record into the file format and writes it to disk.
+     * This is a wrapper around processRecordData that handles file writing.
+     *
+     * @param record - Raw database record
+     * @param primaryKey - Primary key fields and values
+     * @param targetDir - Directory to save the file
+     * @param entityConfig - Entity configuration with pull settings
+     * @param syncEngine - Sync engine instance
+     * @returns Promise that resolves when file is written
+     * @private
+     */
     async processRecord(record, primaryKey, targetDir, entityConfig, syncEngine) {
+        const recordData = await this.processRecordData(record, primaryKey, targetDir, entityConfig, syncEngine);
+        // Determine file path
+        const fileName = this.buildFileName(primaryKey, entityConfig);
+        const filePath = path_1.default.join(targetDir, fileName);
+        // Write JSON file
+        await fs_extra_1.default.writeJson(filePath, recordData, { spaces: 2 });
+    }
+    /**
+     * Process record data for storage
+     *
+     * Transforms a raw database record into the RecordData format used for file storage.
+     * Handles field externalization, related entity pulling, and checksum calculation.
+     *
+     * @param record - Raw database record
+     * @param primaryKey - Primary key fields and values
+     * @param targetDir - Directory where files will be saved
+     * @param entityConfig - Entity configuration with defaults and settings
+     * @param syncEngine - Sync engine for checksum calculation
+     * @returns Promise resolving to formatted RecordData
+     * @private
+     */
+    async processRecordData(record, primaryKey, targetDir, entityConfig, syncEngine) {
         // Build record data
         const recordData = {
             primaryKey: primaryKey,
@@ -172,12 +281,21 @@ class Pull extends core_1.Command {
         }
         // Calculate checksum
         recordData.sync.checksum = syncEngine.calculateChecksum(recordData.fields);
-        // Determine file path
-        const fileName = this.buildFileName(primaryKey, entityConfig);
-        const filePath = path_1.default.join(targetDir, fileName);
-        // Write JSON file
-        await fs_extra_1.default.writeJson(filePath, recordData, { spaces: 2 });
+        return recordData;
     }
+    /**
+     * Determine if a field should be saved to an external file
+     *
+     * Checks if a field contains substantial text content that would be better
+     * stored in a separate file rather than inline in the JSON. Uses heuristics
+     * based on field name and content length.
+     *
+     * @param fieldName - Name of the field to check
+     * @param fieldValue - Value of the field
+     * @param entityConfig - Entity configuration (for future extension)
+     * @returns Promise resolving to true if field should be externalized
+     * @private
+     */
     async shouldExternalizeField(fieldName, fieldValue, entityConfig) {
         // Only externalize string fields with significant content
         if (typeof fieldValue !== 'string') {
@@ -192,6 +310,20 @@ class Pull extends core_1.Command {
         }
         return false;
     }
+    /**
+     * Create an external file for a field value
+     *
+     * Saves large text content to a separate file and returns the filename.
+     * Automatically determines appropriate file extension based on field name
+     * and content type (e.g., .md for prompts, .html for templates).
+     *
+     * @param targetDir - Directory to save the file
+     * @param primaryKey - Primary key for filename generation
+     * @param fieldName - Name of the field being externalized
+     * @param content - Content to write to the file
+     * @returns Promise resolving to the created filename
+     * @private
+     */
     async createExternalFile(targetDir, primaryKey, fieldName, content) {
         // Determine file extension based on field name and content
         let extension = '.txt';
@@ -218,6 +350,18 @@ class Pull extends core_1.Command {
         await fs_extra_1.default.writeFile(filePath, content, 'utf-8');
         return fileName;
     }
+    /**
+     * Build a filename from primary key values
+     *
+     * Creates a safe filename based on the entity's primary key values.
+     * Handles GUIDs by using first 8 characters, sanitizes special characters,
+     * and creates composite names for multi-field keys.
+     *
+     * @param primaryKey - Primary key fields and values
+     * @param entityConfig - Entity configuration (for future extension)
+     * @returns Filename with .json extension
+     * @private
+     */
     buildFileName(primaryKey, entityConfig) {
         // Use primary key values to build filename
         const keys = Object.values(primaryKey);
@@ -236,6 +380,19 @@ class Pull extends core_1.Command {
         // Multiple keys or numeric - create composite name
         return keys.map(k => String(k).replace(/[^a-zA-Z0-9-_]/g, '_')).join('-') + '.json';
     }
+    /**
+     * Pull related entities for a parent record
+     *
+     * Retrieves child records that have foreign key relationships to the parent.
+     * Converts foreign key values to @parent references and supports nested
+     * related entities for deep object graphs.
+     *
+     * @param parentRecord - Parent entity record
+     * @param relatedConfig - Configuration for related entities to pull
+     * @param syncEngine - Sync engine instance
+     * @returns Promise resolving to map of entity names to related records
+     * @private
+     */
     async pullRelatedEntities(parentRecord, relatedConfig, syncEngine) {
         const relatedEntities = {};
         for (const [key, config] of Object.entries(relatedConfig)) {
@@ -298,6 +455,18 @@ class Pull extends core_1.Command {
         }
         return relatedEntities;
     }
+    /**
+     * Find which field in the parent record contains a specific value
+     *
+     * Used to convert foreign key references to @parent references by finding
+     * the parent field that contains the foreign key value. Typically finds
+     * the primary key field but can match any field.
+     *
+     * @param parentRecord - Parent record to search
+     * @param value - Value to search for
+     * @returns Field name containing the value, or null if not found
+     * @private
+     */
     findParentField(parentRecord, value) {
         // Find which field in the parent contains this value
         // Typically this will be the primary key field