@memberjunction/metadata-sync 2.88.0 → 2.90.0

package/README.md

@@ -711,11 +711,13 @@ When a field value starts with `@file:`, the tool will:
 1. Read content from the specified file for push operations
 2. Write content to the specified file for pull operations
 3. Track both files for change detection
+4. **For JSON files**: Automatically process any `@include` directives within them
 
 Examples:
 - `@file:greeting.prompt.md` - File in same directory as JSON
 - `@file:./shared/common-prompt.md` - Relative path
 - `@file:../templates/standard-header.md` - Parent directory reference
+- `@file:spec.json` - JSON file with `@include` directives (processed automatically)
 
 ### @url: References
 When a field value starts with `@url:`, the tool will:
@@ -798,7 +800,7 @@ Examples:
 
 The `Configuration`, `Tags`, and `Metadata` fields will automatically be converted to JSON strings when pushed to the database, while maintaining their structured format in your source files.
 
-### {@include} References in Files (NEW)
+### {@include} References in Files
 Enable content composition within non-JSON files (like .md, .html, .txt) using JSDoc-style include syntax:
 - Pattern: `{@include path/to/file.ext}`
 - Supports relative paths from the containing file
@@ -876,6 +878,250 @@ Benefits:
 - **Flexibility**: Build complex documents from modular parts
 - **Validation**: Automatic checking of included file existence and circular references
 
+### @include References in JSON Files
+
+Enable modular JSON composition by including external JSON files directly into your metadata files. This feature allows you to break large JSON configurations into smaller, reusable components.
+
+#### Syntax Options
+
+**Object Context - Property Spreading (Default)**
+```json
+{
+  "name": "Parent Record",
+  "@include": "child.json",
+  "description": "Additional fields"
+}
+```
+The included file's properties are spread into the parent object.
+
+**Multiple Includes with Dot Notation (Eliminates VS Code Warnings)**
+```json
+{
+  "name": "Parent Record",
+  "@include.data": "shared/data-fields.json",
+  "description": "Middle field",
+  "@include.config": "shared/config-fields.json",
+  "status": "Active"
+}
+```
+Use dot notation (`@include.anything`) to include multiple files at different positions in your object. The part after the dot is ignored by the processor but makes each key unique, eliminating VS Code's duplicate key warnings. The includes are processed in the order they appear, allowing precise control over property ordering.
+
+**Array Context - Element Insertion**
+```json
+[
+  {"name": "First item"},
+  "@include:child.json",
+  {"name": "Last item"}
+]
+```
+The included file's content is inserted as array element(s).
+
+**Explicit Mode Control**
+```json
+{
+  "@include": {
+    "file": "child.json",
+    "mode": "spread"  // or "element"
+  }
+}
+```
+
+#### Modes Explained
+
+**"spread" mode**: 
+- Merges all properties from the included file into the parent object
+- Only works when including an object into an object
+- Parent properties override child properties if there are conflicts
+- Default mode for objects
+
+**"element" mode**:
+- Directly inserts the JSON content at that position
+- Works with any JSON type (object, array, string, number, etc.)
+- Replaces the @include directive with the actual content
+- Default mode for arrays when using string syntax
+
+#### Path Resolution
+- All paths are relative to the file containing the @include
+- Supports: `"child.json"`, `"./child.json"`, `"../shared/base.json"`, `"subfolder/config.json"`
+- Circular references are detected and prevented
+
+#### Complex Example
+
+Directory structure:
+```
+metadata/
+├── components/
+│   ├── dashboard.json
+│   ├── base-props.json
+│   └── items/
+│       └── dashboard-items.json
+└── shared/
+    └── common-settings.json
+```
+
+dashboard.json:
+```json
+{
+  "fields": {
+    "Name": "Analytics Dashboard",
+    "@include.common": "../shared/common-settings.json",
+    "Type": "Dashboard",
+    "@include.defaults": "../shared/default-values.json",
+    "Configuration": {
+      "@include": {"file": "./base-props.json", "mode": "element"}
+    }
+  },
+  "relatedEntities": {
+    "Dashboard Items": [
+      "@include:./items/dashboard-items.json"
+    ]
+  }
+}
+```
+
+common-settings.json:
+```json
+{
+  "CategoryID": "@lookup:Categories.Name=Analytics",
+  "Status": "Active",
+  "Priority": 1
+}
+```
+
+base-props.json:
+```json
+{
+  "refreshInterval": 60,
+  "theme": "dark",
+  "layout": "grid"
+}
+```
+
+dashboard-items.json:
+```json
+[
+  {"name": "Revenue Chart", "type": "chart"},
+  {"name": "User Stats", "type": "stats"},
+  {"name": "Activity Feed", "type": "feed"}
+]
+```
+
+Result after processing:
+```json
+{
+  "fields": {
+    "Name": "Analytics Dashboard",
+    "CategoryID": "@lookup:Categories.Name=Analytics",
+    "Status": "Active", 
+    "Priority": 1,
+    "Type": "Dashboard",
+    "Configuration": {
+      "refreshInterval": 60,
+      "theme": "dark",
+      "layout": "grid"
+    }
+  },
+  "relatedEntities": {
+    "Dashboard Items": [
+      {"name": "Revenue Chart", "type": "chart"},
+      {"name": "User Stats", "type": "stats"},
+      {"name": "Activity Feed", "type": "feed"}
+    ]
+  }
+}
+```
+
+#### Use Cases
+- **Shared Configurations**: Reuse common settings across multiple entities
+- **Modular Records**: Build complex records from smaller components
+- **Template Libraries**: Create libraries of reusable JSON fragments
+- **Environment Configs**: Include environment-specific settings
+- **Large Data Sets**: Break up large JSON files for better maintainability
+- **VS Code Compatibility**: Use dot notation to avoid duplicate key warnings when including multiple files
+
+#### Practical Example: Component with Multiple Includes
+```json
+{
+  "name": "DashboardComponent",
+  "type": "dashboard",
+  "@include.dataRequirements": "../shared/data-requirements.json",
+  "functionalRequirements": "Dashboard displays real-time metrics...",
+  "@include.libraries": "../shared/chart-libraries.json",
+  "technicalDesign": "Component uses React hooks for state...",
+  "@include.eventHandlers": "../shared/event-handlers.json",
+  "code": "const Dashboard = () => { ... }"
+}
+```
+In this example, data requirements, libraries, and event handlers are spread into the component definition at their specific positions, maintaining a logical property order while avoiding VS Code warnings about duplicate `@include` keys.
+
+#### @include in Referenced JSON Files
+When using `@file:` to reference a JSON file, any `@include` directives within that JSON file are automatically processed:
+
+#### @file References in Included JSON Files
+The system now automatically resolves `@file` references found within JSON files that are pulled in via `@include`. This allows for complete nesting of references:
+
+```json
+// main-entity.json
+{
+  "fields": {
+    "Name": "MyComponent",
+    "Specification": "@file:files/component-spec.json"
+  }
+}
+
+// files/component-spec.json
+{
+  "name": "ComponentSpec",
+  "@include.base": "../shared/base-spec.json",
+  "customFields": {
+    "feature": "advanced"
+  },
+  "@include.libs": "../shared/libraries.json"
+}
+```
+
+The `component-spec.json` file's `@include` directives are processed before the content is returned to the `Specification` field, ensuring all includes are resolved.
+
+#### Nested @file References in JSON Files
+The system now recursively processes `@file` references within JSON files loaded via `@file`. This enables powerful composition patterns:
+
+```json
+// components.json
+{
+  "fields": {
+    "Name": "RecentDealsList",
+    "Specification": "@file:spec/recent-deals-list.spec.json"
+  }
+}
+
+// spec/recent-deals-list.spec.json
+{
+  "name": "RecentDealsList",
+  "description": "List of recent deals",
+  "code": "@file:../code/recent-deals-list.js",  // This nested @file is now resolved!
+  "style": "@file:../styles/deals.css",
+  "config": {
+    "template": "@file:../templates/deal-row.html"  // Even deeply nested @file references work
+  }
+}
+```
+
+All `@file` references are recursively resolved, regardless of nesting depth. The final result will have all file contents properly loaded and embedded.
+
+#### Processing Order
+1. @include directives are processed first (recursively)
+2. @file references are recursively resolved (including nested ones in JSON)
+3. Then @template references
+4. Finally, other @ references (@lookup, etc.)
+
+This ensures that included content can contain other special references that will be properly resolved.
+
+**New Feature**: @file references within @included JSON files are now automatically resolved. This means you can have:
+- A main JSON file with `@include` directives
+- The included JSON files can have `@file` references to load code, templates, etc.
+- Those @file references are resolved to their actual content
+- If the @file points to a JSON file with @include directives, those are also processed
+
 ### @template: References (NEW)
 Enable JSON template composition for reusable configurations:
 
@@ -982,6 +1228,9 @@ mj sync push --verbose
 # Dry run to see what would change
 mj sync push --dry-run
 
+# Push with parallel processing (NEW)
+mj sync push --parallel-batch-size=20  # Process 20 records in parallel (default: 10, max: 50)
+
 # Show status of local vs database
 mj sync status
 
@@ -1008,6 +1257,56 @@ Configuration follows a hierarchical structure:
 - **Entity configs**: Each entity directory has its own config defining the entity type
 - **Inheritance**: All files within an entity directory are treated as records of that entity type
 
+### Parallel Processing (NEW)
+
+MetadataSync now supports parallel processing of records during push operations, significantly improving performance for large datasets.
+
+#### How It Works
+
+Records are automatically grouped into dependency levels:
+- **Level 0**: Records with no dependencies
+- **Level 1**: Records that depend only on Level 0 records
+- **Level 2**: Records that depend on Level 0 or Level 1 records
+- And so on...
+
+Records within the same dependency level can be safely processed in parallel since they have no dependencies on each other.
+
+#### Configuration
+
+Use the `--parallel-batch-size` flag to control parallelism:
+
+```bash
+# Default: 10 records in parallel
+mj sync push
+
+# Process 20 records in parallel
+mj sync push --parallel-batch-size=20
+
+# Maximum parallelism (50 records)
+mj sync push --parallel-batch-size=50
+
+# Conservative approach for debugging
+mj sync push --parallel-batch-size=1
+```
+
+#### Performance Benefits
+
+- **2-3x faster** for typical metadata pushes
+- **5-10x faster** for records with many file references (@file) or lookups (@lookup)
+- Most beneficial when processing large numbers of independent records
+
+#### When to Use
+
+**Recommended for:**
+- Large initial data imports
+- Bulk metadata updates
+- CI/CD pipelines with time constraints
+
+**Use conservative settings for:**
+- Debugging sync issues
+- Working with complex dependencies
+- Limited database connection pools
+
 ### Directory Processing Order (NEW)
 
 The MetadataSync tool now supports custom directory processing order to handle dependencies between entity types. This feature ensures that dependent entities are processed in the correct order.

package/dist/lib/json-preprocessor.d.ts

@@ -0,0 +1,67 @@
+/**
+ * Include directive configuration
+ */
+export interface IncludeDirective {
+    file: string;
+    mode?: 'spread' | 'element';
+}
+/**
+ * Preprocesses JSON files to handle @include directives
+ * Supports including external JSON files with spreading or element insertion
+ */
+export declare class JsonPreprocessor {
+    private visitedPaths;
+    /**
+     * Process a JSON file and resolve all @include directives
+     * @param filePath - Path to the JSON file to process
+     * @returns The processed JSON data with all includes resolved
+     */
+    processFile(filePath: string): Promise<any>;
+    /**
+     * Recursively process @include directives in JSON data
+     * @param data - The JSON data to process
+     * @param currentFilePath - Path of the current file for resolving relative paths
+     * @returns The processed data with includes resolved
+     */
+    private processIncludesInternal;
+    /**
+     * Process an array, handling @include directives
+     * @param arr - The array to process
+     * @param currentFilePath - Current file path for resolving relative paths
+     * @returns Processed array with includes resolved
+     */
+    private processArray;
+    /**
+     * Process an object, handling @include directives
+     * @param obj - The object to process
+     * @param currentFilePath - Current file path for resolving relative paths
+     * @returns Processed object with includes resolved
+     */
+    private processObject;
+    /**
+     * Load and process an included file
+     * @param filePath - Path to the file to include
+     * @returns The processed content of the included file
+     */
+    private loadAndProcessInclude;
+    /**
+     * Load file content and process it if it's JSON with @include directives
+     * @param filePath - Path to the file to load
+     * @returns The file content (processed if JSON with @includes)
+     */
+    private loadFileContent;
+    /**
+     * Resolve a potentially relative path to an absolute path
+     * @param includePath - The path specified in the @include
+     * @param currentFilePath - The current file's path
+     * @returns Absolute path to the included file
+     */
+    private resolvePath;
+    /**
+     * Process JSON data that's already loaded (for integration with existing code)
+     * @param data - The JSON data to process
+     * @param filePath - The file path (for resolving relative includes)
+     * @returns Processed data with includes resolved
+     */
+    processJsonData(data: any, filePath: string): Promise<any>;
+}

package/dist/lib/json-preprocessor.js

@@ -0,0 +1,241 @@
+"use strict";
+var __importDefault = (this && this.__importDefault) || function (mod) {
+    return (mod && mod.__esModule) ? mod : { "default": mod };
+};
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.JsonPreprocessor = void 0;
+const fs_extra_1 = __importDefault(require("fs-extra"));
+const path_1 = __importDefault(require("path"));
+/**
+ * Preprocesses JSON files to handle @include directives
+ * Supports including external JSON files with spreading or element insertion
+ */
+class JsonPreprocessor {
+    visitedPaths = new Set();
+    /**
+     * Process a JSON file and resolve all @include directives
+     * @param filePath - Path to the JSON file to process
+     * @returns The processed JSON data with all includes resolved
+     */
+    async processFile(filePath) {
+        this.visitedPaths.clear();
+        const fileContent = await fs_extra_1.default.readJson(filePath);
+        return this.processIncludesInternal(fileContent, filePath);
+    }
+    /**
+     * Recursively process @include directives in JSON data
+     * @param data - The JSON data to process
+     * @param currentFilePath - Path of the current file for resolving relative paths
+     * @returns The processed data with includes resolved
+     */
+    async processIncludesInternal(data, currentFilePath) {
+        // Process based on data type
+        if (Array.isArray(data)) {
+            return this.processArray(data, currentFilePath);
+        }
+        else if (data && typeof data === 'object') {
+            return this.processObject(data, currentFilePath);
+        }
+        else {
+            return data;
+        }
+    }
+    /**
+     * Process an array, handling @include directives
+     * @param arr - The array to process
+     * @param currentFilePath - Current file path for resolving relative paths
+     * @returns Processed array with includes resolved
+     */
+    async processArray(arr, currentFilePath) {
+        const result = [];
+        for (const item of arr) {
+            // Check for string-based include in array (default element mode)
+            if (typeof item === 'string' && item.startsWith('@include:')) {
+                const includePath = item.substring(9).trim();
+                const resolvedPath = this.resolvePath(includePath, currentFilePath);
+                const includedContent = await this.loadAndProcessInclude(resolvedPath);
+                // If included content is an array, spread its elements
+                if (Array.isArray(includedContent)) {
+                    result.push(...includedContent);
+                }
+                else {
+                    // Otherwise add as single element
+                    result.push(includedContent);
+                }
+            }
+            else if (item && typeof item === 'object') {
+                // Process nested objects/arrays
+                result.push(await this.processIncludesInternal(item, currentFilePath));
+            }
+            else {
+                result.push(item);
+            }
+        }
+        return result;
+    }
+    /**
+     * Process an object, handling @include directives
+     * @param obj - The object to process
+     * @param currentFilePath - Current file path for resolving relative paths
+     * @returns Processed object with includes resolved
+     */
+    async processObject(obj, currentFilePath) {
+        const result = {};
+        const includeKeys = [];
+        const includeDirectives = new Map();
+        // First pass: identify all @include keys (both @include and @include.*)
+        for (const key of Object.keys(obj)) {
+            if (key === '@include' || key.startsWith('@include.')) {
+                includeKeys.push(key);
+                const includeValue = obj[key];
+                if (typeof includeValue === 'string') {
+                    // Simple string include - default to spread mode in objects
+                    includeDirectives.set(key, { file: includeValue, mode: 'spread' });
+                }
+                else if (includeValue && typeof includeValue === 'object') {
+                    // Explicit include configuration
+                    const directive = includeValue;
+                    // Default to spread mode if not specified
+                    if (!directive.mode) {
+                        directive.mode = 'spread';
+                    }
+                    includeDirectives.set(key, directive);
+                }
+            }
+        }
+        // Second pass: process all properties in order, spreading includes when encountered
+        for (const [key, value] of Object.entries(obj)) {
+            if (key === '@include' || key.startsWith('@include.')) {
+                // Process this include directive
+                const includeDirective = includeDirectives.get(key);
+                if (includeDirective) {
+                    const resolvedPath = this.resolvePath(includeDirective.file, currentFilePath);
+                    const includedContent = await this.loadAndProcessInclude(resolvedPath);
+                    if (includeDirective.mode === 'spread') {
+                        // Spread mode: merge included object properties at this position
+                        if (includedContent && typeof includedContent === 'object' && !Array.isArray(includedContent)) {
+                            Object.assign(result, includedContent);
+                        }
+                        else {
+                            throw new Error(`Cannot spread non-object content from ${includeDirective.file}. Use mode: "element" for non-object includes.`);
+                        }
+                    }
+                    else if (includeDirective.mode === 'element') {
+                        // Element mode: directly insert the content
+                        // For dot notation includes, we can't replace the whole object,
+                        // so we'll add it as a property instead (though this is unusual)
+                        if (key.includes('.')) {
+                            // Extract the part after the dot to use as property name
+                            const propName = key.split('.').slice(1).join('.');
+                            result[propName] = includedContent;
+                        }
+                        else {
+                            // For plain @include with element mode, replace the entire object
+                            return includedContent;
+                        }
+                    }
+                }
+            }
+            else {
+                // Regular property - process recursively and handle @file references
+                if (typeof value === 'string' && value.startsWith('@file:')) {
+                    // Process @file reference
+                    const filePath = value.substring(6); // Remove '@file:' prefix
+                    const resolvedPath = this.resolvePath(filePath, currentFilePath);
+                    result[key] = await this.loadFileContent(resolvedPath);
+                }
+                else if (value && typeof value === 'object') {
+                    result[key] = await this.processIncludesInternal(value, currentFilePath);
+                }
+                else {
+                    result[key] = value;
+                }
+            }
+        }
+        return result;
+    }
+    /**
+     * Load and process an included file
+     * @param filePath - Path to the file to include
+     * @returns The processed content of the included file
+     */
+    async loadAndProcessInclude(filePath) {
+        const absolutePath = path_1.default.resolve(filePath);
+        // Check if this file is already being processed (circular reference)
+        if (this.visitedPaths.has(absolutePath)) {
+            throw new Error(`Circular reference detected: ${absolutePath} is already being processed`);
+        }
+        if (!await fs_extra_1.default.pathExists(filePath)) {
+            throw new Error(`Include file not found: ${filePath}`);
+        }
+        // Add to visited paths before processing
+        this.visitedPaths.add(absolutePath);
+        try {
+            const content = await fs_extra_1.default.readJson(filePath);
+            // Process the content (visited tracking is handled in this method)
+            return this.processIncludesInternal(content, filePath);
+        }
+        finally {
+            // Remove from visited paths after processing
+            this.visitedPaths.delete(absolutePath);
+        }
+    }
+    /**
+     * Load file content and process it if it's JSON with @include directives
+     * @param filePath - Path to the file to load
+     * @returns The file content (processed if JSON with @includes)
+     */
+    async loadFileContent(filePath) {
+        if (!await fs_extra_1.default.pathExists(filePath)) {
+            // Return the original @file reference if file doesn't exist
+            // This matches the behavior of SyncEngine.processFieldValue
+            return `@file:${path_1.default.relative(path_1.default.dirname(filePath), filePath)}`;
+        }
+        try {
+            if (filePath.endsWith('.json')) {
+                // For JSON files, load and check for @include directives
+                const jsonContent = await fs_extra_1.default.readJson(filePath);
+                const jsonString = JSON.stringify(jsonContent);
+                const hasIncludes = jsonString.includes('"@include"') || jsonString.includes('"@include.');
+                if (hasIncludes) {
+                    // Process @include directives in the JSON file
+                    return await this.processIncludesInternal(jsonContent, filePath);
+                }
+                else {
+                    // Return the JSON content as-is
+                    return jsonContent;
+                }
+            }
+            else {
+                // For non-JSON files, return the text content
+                return await fs_extra_1.default.readFile(filePath, 'utf-8');
+            }
+        }
+        catch (error) {
+            // On error, return the original @file reference
+            return `@file:${path_1.default.relative(path_1.default.dirname(filePath), filePath)}`;
+        }
+    }
+    /**
+     * Resolve a potentially relative path to an absolute path
+     * @param includePath - The path specified in the @include
+     * @param currentFilePath - The current file's path
+     * @returns Absolute path to the included file
+     */
+    resolvePath(includePath, currentFilePath) {
+        const currentDir = path_1.default.dirname(currentFilePath);
+        return path_1.default.resolve(currentDir, includePath);
+    }
+    /**
+     * Process JSON data that's already loaded (for integration with existing code)
+     * @param data - The JSON data to process
+     * @param filePath - The file path (for resolving relative includes)
+     * @returns Processed data with includes resolved
+     */
+    async processJsonData(data, filePath) {
+        this.visitedPaths.clear();
+        return this.processIncludesInternal(data, filePath);
+    }
+}
+exports.JsonPreprocessor = JsonPreprocessor;
+//# sourceMappingURL=json-preprocessor.js.map

package/dist/lib/json-preprocessor.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"json-preprocessor.js","sourceRoot":"","sources":["../../src/lib/json-preprocessor.ts"],"names":[],"mappings":";;;;;;AAAA,wDAA0B;AAC1B,gDAAwB;AAUxB;;;GAGG;AACH,MAAa,gBAAgB;IACnB,YAAY,GAAgB,IAAI,GAAG,EAAE,CAAC;IAE9C;;;;OAIG;IACH,KAAK,CAAC,WAAW,CAAC,QAAgB;QAChC,IAAI,CAAC,YAAY,CAAC,KAAK,EAAE,CAAC;QAC1B,MAAM,WAAW,GAAG,MAAM,kBAAE,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAC;QAChD,OAAO,IAAI,CAAC,uBAAuB,CAAC,WAAW,EAAE,QAAQ,CAAC,CAAC;IAC7D,CAAC;IAED;;;;;OAKG;IACK,KAAK,CAAC,uBAAuB,CAAC,IAAS,EAAE,eAAuB;QACtE,6BAA6B;QAC7B,IAAI,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC,EAAE,CAAC;YACxB,OAAO,IAAI,CAAC,YAAY,CAAC,IAAI,EAAE,eAAe,CAAC,CAAC;QAClD,CAAC;aAAM,IAAI,IAAI,IAAI,OAAO,IAAI,KAAK,QAAQ,EAAE,CAAC;YAC5C,OAAO,IAAI,CAAC,aAAa,CAAC,IAAI,EAAE,eAAe,CAAC,CAAC;QACnD,CAAC;aAAM,CAAC;YACN,OAAO,IAAI,CAAC;QACd,CAAC;IACH,CAAC;IAED;;;;;OAKG;IACK,KAAK,CAAC,YAAY,CAAC,GAAU,EAAE,eAAuB;QAC5D,MAAM,MAAM,GAAU,EAAE,CAAC;QAEzB,KAAK,MAAM,IAAI,IAAI,GAAG,EAAE,CAAC;YACvB,iEAAiE;YACjE,IAAI,OAAO,IAAI,KAAK,QAAQ,IAAI,IAAI,CAAC,UAAU,CAAC,WAAW,CAAC,EAAE,CAAC;gBAC7D,MAAM,WAAW,GAAG,IAAI,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;gBAC7C,MAAM,YAAY,GAAG,IAAI,CAAC,WAAW,CAAC,WAAW,EAAE,eAAe,CAAC,CAAC;gBACpE,MAAM,eAAe,GAAG,MAAM,IAAI,CAAC,qBAAqB,CAAC,YAAY,CAAC,CAAC;gBAEvE,uDAAuD;gBACvD,IAAI,KAAK,CAAC,OAAO,CAAC,eAAe,CAAC,EAAE,CAAC;oBACnC,MAAM,CAAC,IAAI,CAAC,GAAG,eAAe,CAAC,CAAC;gBAClC,CAAC;qBAAM,CAAC;oBACN,kCAAkC;oBAClC,MAAM,CAAC,IAAI,CAAC,eAAe,CAAC,CAAC;gBAC/B,CAAC;YACH,CAAC;iBAAM,IAAI,IAAI,IAAI,OAAO,IAAI,KAAK,QAAQ,EAAE,CAAC;gBAC5C,gCAAgC;gBAChC,MAAM,CAAC,IAAI,CAAC,MAAM,IAAI,CAAC,uBAAuB,CAAC,IAAI,EAAE,eAAe,CAAC,CAAC,CAAC;YACzE,CAAC;iBAAM,CAAC;gBACN,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;YACpB,CAAC;QACH,CAAC;QAED,OAAO,MAAM,CAAC;IAChB,CAAC;IAED;;;;;OAKG;IACK,KAAK,CAAC,aAAa,CAAC,GAAQ,EAAE,eAAuB;QAC3D,MAAM,MAAM,GAAQ,EAAE,CAAC;QACvB,MAAM,WAAW,GAAa,EAAE,CAAC;QACjC,MAAM,iBAAiB,GAAkC,IAAI,GAAG,EAAE,CAAC;QAEnE,wEAAwE;QACxE,KAAK,MAAM,GAAG,IAAI,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC;YACnC,IAAI,GAAG,KAAK,UAAU,IAAI,GAAG,CAAC,UAAU,CAAC,WAAW,CAAC,EAAE,CAAC;gBACtD,WAAW,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;gBACtB,MAAM,YAAY,GAAG,GAAG,CAAC,GAAG,CAAC,CAAC;gBAE9B,IAAI,OAAO,YAAY,KAAK,QAAQ,EAAE,CAAC;oBACrC,4DAA4D;oBAC5D,iBAAiB,CAAC,GAAG,CAAC,GAAG,EAAE,EAAE,IAAI,EAAE,YAAY,EAAE,IAAI,EAAE,QAAQ,EAAE,CAAC,CAAC;gBACrE,CAAC;qBAAM,IAAI,YAAY,IAAI,OAAO,YAAY,KAAK,QAAQ,EAAE,CAAC;oBAC5D,iCAAiC;oBACjC,MAAM,SAAS,GAAG,YAAgC,CAAC;oBACnD,0CAA0C;oBAC1C,IAAI,CAAC,SAAS,CAAC,IAAI,EAAE,CAAC;wBACpB,SAAS,CAAC,IAAI,GAAG,QAAQ,CAAC;oBAC5B,CAAC;oBACD,iBAAiB,CAAC,GAAG,CAAC,GAAG,EAAE,SAAS,CAAC,CAAC;gBACxC,CAAC;YACH,CAAC;QACH,CAAC;QAED,oFAAoF;QACpF,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC;YAC/C,IAAI,GAAG,KAAK,UAAU,IAAI,GAAG,CAAC,UAAU,CAAC,WAAW,CAAC,EAAE,CAAC;gBACtD,iCAAiC;gBACjC,MAAM,gBAAgB,GAAG,iBAAiB,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;gBACpD,IAAI,gBAAgB,EAAE,CAAC;oBACrB,MAAM,YAAY,GAAG,IAAI,CAAC,WAAW,CAAC,gBAAgB,CAAC,IAAI,EAAE,eAAe,CAAC,CAAC;oBAC9E,MAAM,eAAe,GAAG,MAAM,IAAI,CAAC,qBAAqB,CAAC,YAAY,CAAC,CAAC;oBAEvE,IAAI,gBAAgB,CAAC,IAAI,KAAK,QAAQ,EAAE,CAAC;wBACvC,iEAAiE;wBACjE,IAAI,eAAe,IAAI,OAAO,eAAe,KAAK,QAAQ,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,eAAe,CAAC,EAAE,CAAC;4BAC9F,MAAM,CAAC,MAAM,CAAC,MAAM,EAAE,eAAe,CAAC,CAAC;wBACzC,CAAC;6BAAM,CAAC;4BACN,MAAM,IAAI,KAAK,CAAC,yCAAyC,gBAAgB,CAAC,IAAI,gDAAgD,CAAC,CAAC;wBAClI,CAAC;oBACH,CAAC;yBAAM,IAAI,gBAAgB,CAAC,IAAI,KAAK,SAAS,EAAE,CAAC;wBAC/C,4CAA4C;wBAC5C,gEAAgE;wBAChE,iEAAiE;wBACjE,IAAI,GAAG,CAAC,QAAQ,CAAC,GAAG,CAAC,EAAE,CAAC;4BACtB,yDAAyD;4BACzD,MAAM,QAAQ,GAAG,GAAG,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;4BACnD,MAAM,CAAC,QAAQ,CAAC,GAAG,eAAe,CAAC;wBACrC,CAAC;6BAAM,CAAC;4BACN,kEAAkE;4BAClE,OAAO,eAAe,CAAC;wBACzB,CAAC;oBACH,CAAC;gBACH,CAAC;YACH,CAAC;iBAAM,CAAC;gBACN,qEAAqE;gBACrE,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,CAAC,UAAU,CAAC,QAAQ,CAAC,EAAE,CAAC;oBAC5D,0BAA0B;oBAC1B,MAAM,QAAQ,GAAG,KAAK,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC,yBAAyB;oBAC9D,MAAM,YAAY,GAAG,IAAI,CAAC,WAAW,CAAC,QAAQ,EAAE,eAAe,CAAC,CAAC;oBACjE,MAAM,CAAC,GAAG,CAAC,GAAG,MAAM,IAAI,CAAC,eAAe,CAAC,YAAY,CAAC,CAAC;gBACzD,CAAC;qBAAM,IAAI,KAAK,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;oBAC9C,MAAM,CAAC,GAAG,CAAC,GAAG,MAAM,IAAI,CAAC,uBAAuB,CAAC,KAAK,EAAE,eAAe,CAAC,CAAC;gBAC3E,CAAC;qBAAM,CAAC;oBACN,MAAM,CAAC,GAAG,CAAC,GAAG,KAAK,CAAC;gBACtB,CAAC;YACH,CAAC;QACH,CAAC;QAED,OAAO,MAAM,CAAC;IAChB,CAAC;IAED;;;;OAIG;IACK,KAAK,CAAC,qBAAqB,CAAC,QAAgB;QAClD,MAAM,YAAY,GAAG,cAAI,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC;QAE5C,qEAAqE;QACrE,IAAI,IAAI,CAAC,YAAY,CAAC,GAAG,CAAC,YAAY,CAAC,EAAE,CAAC;YACxC,MAAM,IAAI,KAAK,CAAC,gCAAgC,YAAY,6BAA6B,CAAC,CAAC;QAC7F,CAAC;QAED,IAAI,CAAC,MAAM,kBAAE,CAAC,UAAU,CAAC,QAAQ,CAAC,EAAE,CAAC;YACnC,MAAM,IAAI,KAAK,CAAC,2BAA2B,QAAQ,EAAE,CAAC,CAAC;QACzD,CAAC;QAED,yCAAyC;QACzC,IAAI,CAAC,YAAY,CAAC,GAAG,CAAC,YAAY,CAAC,CAAC;QAEpC,IAAI,CAAC;YACH,MAAM,OAAO,GAAG,MAAM,kBAAE,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAC;YAC5C,mEAAmE;YACnE,OAAO,IAAI,CAAC,uBAAuB,CAAC,OAAO,EAAE,QAAQ,CAAC,CAAC;QACzD,CAAC;gBAAS,CAAC;YACT,6CAA6C;YAC7C,IAAI,CAAC,YAAY,CAAC,MAAM,CAAC,YAAY,CAAC,CAAC;QACzC,CAAC;IACH,CAAC;IAED;;;;OAIG;IACK,KAAK,CAAC,eAAe,CAAC,QAAgB;QAC5C,IAAI,CAAC,MAAM,kBAAE,CAAC,UAAU,CAAC,QAAQ,CAAC,EAAE,CAAC;YACnC,4DAA4D;YAC5D,4DAA4D;YAC5D,OAAO,SAAS,cAAI,CAAC,QAAQ,CAAC,cAAI,CAAC,OAAO,CAAC,QAAQ,CAAC,EAAE,QAAQ,CAAC,EAAE,CAAC;QACpE,CAAC;QAED,IAAI,CAAC;YACH,IAAI,QAAQ,CAAC,QAAQ,CAAC,OAAO,CAAC,EAAE,CAAC;gBAC/B,yDAAyD;gBACzD,MAAM,WAAW,GAAG,MAAM,kBAAE,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAC;gBAChD,MAAM,UAAU,GAAG,IAAI,CAAC,SAAS,CAAC,WAAW,CAAC,CAAC;gBAC/C,MAAM,WAAW,GAAG,UAAU,CAAC,QAAQ,CAAC,YAAY,CAAC,IAAI,UAAU,CAAC,QAAQ,CAAC,YAAY,CAAC,CAAC;gBAE3F,IAAI,WAAW,EAAE,CAAC;oBAChB,+CAA+C;oBAC/C,OAAO,MAAM,IAAI,CAAC,uBAAuB,CAAC,WAAW,EAAE,QAAQ,CAAC,CAAC;gBACnE,CAAC;qBAAM,CAAC;oBACN,gCAAgC;oBAChC,OAAO,WAAW,CAAC;gBACrB,CAAC;YACH,CAAC;iBAAM,CAAC;gBACN,8CAA8C;gBAC9C,OAAO,MAAM,kBAAE,CAAC,QAAQ,CAAC,QAAQ,EAAE,OAAO,CAAC,CAAC;YAC9C,CAAC;QACH,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,gDAAgD;YAChD,OAAO,SAAS,cAAI,CAAC,QAAQ,CAAC,cAAI,CAAC,OAAO,CAAC,QAAQ,CAAC,EAAE,QAAQ,CAAC,EAAE,CAAC;QACpE,CAAC;IACH,CAAC;IAED;;;;;OAKG;IACK,WAAW,CAAC,WAAmB,EAAE,eAAuB;QAC9D,MAAM,UAAU,GAAG,cAAI,CAAC,OAAO,CAAC,eAAe,CAAC,CAAC;QACjD,OAAO,cAAI,CAAC,OAAO,CAAC,UAAU,EAAE,WAAW,CAAC,CAAC;IAC/C,CAAC;IAED;;;;;OAKG;IACH,KAAK,CAAC,eAAe,CAAC,IAAS,EAAE,QAAgB;QAC/C,IAAI,CAAC,YAAY,CAAC,KAAK,EAAE,CAAC;QAC1B,OAAO,IAAI,CAAC,uBAAuB,CAAC,IAAI,EAAE,QAAQ,CAAC,CAAC;IACtD,CAAC;CACF;AAxOD,4CAwOC","sourcesContent":["import fs from 'fs-extra';\nimport path from 'path';\n\n/**\n * Include directive configuration\n */\nexport interface IncludeDirective {\n  file: string;\n  mode?: 'spread' | 'element';\n}\n\n/**\n * Preprocesses JSON files to handle @include directives\n * Supports including external JSON files with spreading or element insertion\n */\nexport class JsonPreprocessor {\n  private visitedPaths: Set<string> = new Set();\n\n  /**\n   * Process a JSON file and resolve all @include directives\n   * @param filePath - Path to the JSON file to process\n   * @returns The processed JSON data with all includes resolved\n   */\n  async processFile(filePath: string): Promise<any> {\n    this.visitedPaths.clear();\n    const fileContent = await fs.readJson(filePath);\n    return this.processIncludesInternal(fileContent, filePath);\n  }\n\n  /**\n   * Recursively process @include directives in JSON data\n   * @param data - The JSON data to process\n   * @param currentFilePath - Path of the current file for resolving relative paths\n   * @returns The processed data with includes resolved\n   */\n  private async processIncludesInternal(data: any, currentFilePath: string): Promise<any> {\n    // Process based on data type\n    if (Array.isArray(data)) {\n      return this.processArray(data, currentFilePath);\n    } else if (data && typeof data === 'object') {\n      return this.processObject(data, currentFilePath);\n    } else {\n      return data;\n    }\n  }\n\n  /**\n   * Process an array, handling @include directives\n   * @param arr - The array to process\n   * @param currentFilePath - Current file path for resolving relative paths\n   * @returns Processed array with includes resolved\n   */\n  private async processArray(arr: any[], currentFilePath: string): Promise<any[]> {\n    const result: any[] = [];\n    \n    for (const item of arr) {\n      // Check for string-based include in array (default element mode)\n      if (typeof item === 'string' && item.startsWith('@include:')) {\n        const includePath = item.substring(9).trim();\n        const resolvedPath = this.resolvePath(includePath, currentFilePath);\n        const includedContent = await this.loadAndProcessInclude(resolvedPath);\n        \n        // If included content is an array, spread its elements\n        if (Array.isArray(includedContent)) {\n          result.push(...includedContent);\n        } else {\n          // Otherwise add as single element\n          result.push(includedContent);\n        }\n      } else if (item && typeof item === 'object') {\n        // Process nested objects/arrays\n        result.push(await this.processIncludesInternal(item, currentFilePath));\n      } else {\n        result.push(item);\n      }\n    }\n    \n    return result;\n  }\n\n  /**\n   * Process an object, handling @include directives\n   * @param obj - The object to process\n   * @param currentFilePath - Current file path for resolving relative paths\n   * @returns Processed object with includes resolved\n   */\n  private async processObject(obj: any, currentFilePath: string): Promise<any> {\n    const result: any = {};\n    const includeKeys: string[] = [];\n    const includeDirectives: Map<string, IncludeDirective> = new Map();\n    \n    // First pass: identify all @include keys (both @include and @include.*)\n    for (const key of Object.keys(obj)) {\n      if (key === '@include' || key.startsWith('@include.')) {\n        includeKeys.push(key);\n        const includeValue = obj[key];\n        \n        if (typeof includeValue === 'string') {\n          // Simple string include - default to spread mode in objects\n          includeDirectives.set(key, { file: includeValue, mode: 'spread' });\n        } else if (includeValue && typeof includeValue === 'object') {\n          // Explicit include configuration\n          const directive = includeValue as IncludeDirective;\n          // Default to spread mode if not specified\n          if (!directive.mode) {\n            directive.mode = 'spread';\n          }\n          includeDirectives.set(key, directive);\n        }\n      }\n    }\n    \n    // Second pass: process all properties in order, spreading includes when encountered\n    for (const [key, value] of Object.entries(obj)) {\n      if (key === '@include' || key.startsWith('@include.')) {\n        // Process this include directive\n        const includeDirective = includeDirectives.get(key);\n        if (includeDirective) {\n          const resolvedPath = this.resolvePath(includeDirective.file, currentFilePath);\n          const includedContent = await this.loadAndProcessInclude(resolvedPath);\n          \n          if (includeDirective.mode === 'spread') {\n            // Spread mode: merge included object properties at this position\n            if (includedContent && typeof includedContent === 'object' && !Array.isArray(includedContent)) {\n              Object.assign(result, includedContent);\n            } else {\n              throw new Error(`Cannot spread non-object content from ${includeDirective.file}. Use mode: \"element\" for non-object includes.`);\n            }\n          } else if (includeDirective.mode === 'element') {\n            // Element mode: directly insert the content\n            // For dot notation includes, we can't replace the whole object,\n            // so we'll add it as a property instead (though this is unusual)\n            if (key.includes('.')) {\n              // Extract the part after the dot to use as property name\n              const propName = key.split('.').slice(1).join('.');\n              result[propName] = includedContent;\n            } else {\n              // For plain @include with element mode, replace the entire object\n              return includedContent;\n            }\n          }\n        }\n      } else {\n        // Regular property - process recursively and handle @file references\n        if (typeof value === 'string' && value.startsWith('@file:')) {\n          // Process @file reference\n          const filePath = value.substring(6); // Remove '@file:' prefix\n          const resolvedPath = this.resolvePath(filePath, currentFilePath);\n          result[key] = await this.loadFileContent(resolvedPath);\n        } else if (value && typeof value === 'object') {\n          result[key] = await this.processIncludesInternal(value, currentFilePath);\n        } else {\n          result[key] = value;\n        }\n      }\n    }\n    \n    return result;\n  }\n\n  /**\n   * Load and process an included file\n   * @param filePath - Path to the file to include\n   * @returns The processed content of the included file\n   */\n  private async loadAndProcessInclude(filePath: string): Promise<any> {\n    const absolutePath = path.resolve(filePath);\n    \n    // Check if this file is already being processed (circular reference)\n    if (this.visitedPaths.has(absolutePath)) {\n      throw new Error(`Circular reference detected: ${absolutePath} is already being processed`);\n    }\n    \n    if (!await fs.pathExists(filePath)) {\n      throw new Error(`Include file not found: ${filePath}`);\n    }\n    \n    // Add to visited paths before processing\n    this.visitedPaths.add(absolutePath);\n    \n    try {\n      const content = await fs.readJson(filePath);\n      // Process the content (visited tracking is handled in this method)\n      return this.processIncludesInternal(content, filePath);\n    } finally {\n      // Remove from visited paths after processing\n      this.visitedPaths.delete(absolutePath);\n    }\n  }\n\n  /**\n   * Load file content and process it if it's JSON with @include directives\n   * @param filePath - Path to the file to load\n   * @returns The file content (processed if JSON with @includes)\n   */\n  private async loadFileContent(filePath: string): Promise<any> {\n    if (!await fs.pathExists(filePath)) {\n      // Return the original @file reference if file doesn't exist\n      // This matches the behavior of SyncEngine.processFieldValue\n      return `@file:${path.relative(path.dirname(filePath), filePath)}`;\n    }\n\n    try {\n      if (filePath.endsWith('.json')) {\n        // For JSON files, load and check for @include directives\n        const jsonContent = await fs.readJson(filePath);\n        const jsonString = JSON.stringify(jsonContent);\n        const hasIncludes = jsonString.includes('\"@include\"') || jsonString.includes('\"@include.');\n        \n        if (hasIncludes) {\n          // Process @include directives in the JSON file\n          return await this.processIncludesInternal(jsonContent, filePath);\n        } else {\n          // Return the JSON content as-is\n          return jsonContent;\n        }\n      } else {\n        // For non-JSON files, return the text content\n        return await fs.readFile(filePath, 'utf-8');\n      }\n    } catch (error) {\n      // On error, return the original @file reference\n      return `@file:${path.relative(path.dirname(filePath), filePath)}`;\n    }\n  }\n\n  /**\n   * Resolve a potentially relative path to an absolute path\n   * @param includePath - The path specified in the @include\n   * @param currentFilePath - The current file's path\n   * @returns Absolute path to the included file\n   */\n  private resolvePath(includePath: string, currentFilePath: string): string {\n    const currentDir = path.dirname(currentFilePath);\n    return path.resolve(currentDir, includePath);\n  }\n\n  /**\n   * Process JSON data that's already loaded (for integration with existing code)\n   * @param data - The JSON data to process\n   * @param filePath - The file path (for resolving relative includes)\n   * @returns Processed data with includes resolved\n   */\n  async processJsonData(data: any, filePath: string): Promise<any> {\n    this.visitedPaths.clear();\n    return this.processIncludesInternal(data, filePath);\n  }\n}"]}