@memberjunction/metadata-sync 2.120.0 → 2.122.0

package/README.md

@@ -804,12 +804,15 @@ Enable entity relationships using human-readable values:
 - Multi-field syntax: `@lookup:EntityName.Field1=Value1&Field2=Value2`
 - Auto-create syntax: `@lookup:EntityName.FieldName=Value?create`
 - With additional fields: `@lookup:EntityName.FieldName=Value?create&Field2=Value2`
+- Deferred lookup syntax: `@lookup:EntityName.FieldName=Value?allowDefer`
+- Combined flags: `@lookup:EntityName.FieldName=Value?create&allowDefer`
 
 Examples:
 - `@lookup:AI Prompt Types.Name=Chat` - Single field lookup, fails if not found
 - `@lookup:Users.Email=john@example.com&Department=Sales` - Multi-field lookup for precise matching
 - `@lookup:AI Prompt Categories.Name=Examples?create` - Creates if missing
 - `@lookup:AI Prompt Categories.Name=Examples?create&Description=Example prompts` - Creates with description
+- `@lookup:Dashboards.Name=Data Explorer?allowDefer` - Defers lookup if not found, retries at end of push
 
 #### Multi-Field Lookups
 When you need to match records based on multiple criteria, use the multi-field syntax:
@@ -822,6 +825,115 @@ When you need to match records based on multiple criteria, use the multi-field s
 
 This ensures you get the exact record you want when multiple records might have the same value in a single field.
 
+#### Deferred Lookups (?allowDefer)
+
+The `?allowDefer` flag enables handling of circular dependencies between entities during push operations. Use this when Entity A references Entity B and Entity B references Entity A - or any situation where a lookup target might not exist yet during initial processing.
+
+**How it works:**
+
+The flag is permission-based, not imperative. The lookup is always attempted first, and only deferred if it fails:
+
+```mermaid
+flowchart TD
+    A["@lookup:Entity.Field=Value?allowDefer"] --> B{Try lookup now}
+    B -->|Found| C[Return ID immediately]
+    B -->|Not found| D{Has ?allowDefer?}
+    D -->|Yes| E[Skip this field, continue processing]
+    D -->|No| F[Fatal error - rollback transaction]
+    E --> G[Save record without deferred field]
+    G --> H[Queue record for re-processing]
+    H --> I[Phase 2.5: Re-process entire record]
+    I -->|Success| J[Update record with resolved field]
+    I -->|Failure| F
+```
+
+**When to use `?allowDefer`:**
+- When Entity A references Entity B, and Entity B references Entity A
+- When you're creating related records that need to reference each other
+- When the lookup target might not exist yet during initial processing
+
+**Processing phases:**
+1. During the initial push phase, if a lookup with `?allowDefer` fails (record not found), the **field is skipped** but the record still saves
+2. The record IS saved during the initial pass (without the deferred field value), allowing other records to reference it
+3. The record is queued for re-processing in Phase 2.5
+4. After all other records are processed, deferred records are re-processed using the exact same logic
+5. If retry succeeds, the record is updated with the resolved field; if it fails, an error is reported and the transaction rolls back
+
+**Example: Application ↔ Dashboard circular reference**
+
+The Applications entity can have `DefaultNavItems` (a JSON field) that contains nested references to Dashboards, while Dashboards have an `ApplicationID` that references Applications.
+
+Since Applications are processed before Dashboards (alphabetical order), the Dashboard lookup in `DefaultNavItems` needs `?allowDefer`:
+
+```json
+// .data-explorer-application.json
+{
+  "fields": {
+    "Name": "Data Explorer",
+    "DefaultNavItems": [
+      {
+        "Label": "Explorer",
+        "ResourceType": "Dashboard",
+        "RecordID": "@lookup:Dashboards.Name=Data Explorer?allowDefer"
+      }
+    ]
+  }
+}
+
+// .data-explorer-dashboard.json
+// Note: No ?allowDefer needed - Applications are processed first
+{
+  "fields": {
+    "Name": "Data Explorer",
+    "ApplicationID": "@lookup:Applications.Name=Data Explorer"
+  }
+}
+```
+
+**Processing order:**
+1. Applications are processed first (per `directoryOrder` in `.mj-sync.json`):
+   - The Dashboard lookup fails (Dashboard doesn't exist yet)
+   - Because `?allowDefer` is set, the `DefaultNavItems` field is skipped
+   - Application IS saved (without the `DefaultNavItems` value)
+   - Application record is queued for re-processing
+2. Dashboards are processed:
+   - Dashboard references Application via `ApplicationID` - this lookup succeeds because Application was saved in step 1
+   - Dashboard is created normally
+3. Deferred records are re-processed (Phase 2.5):
+   - The Application record is processed again using the exact same logic
+   - The Dashboard lookup now succeeds since Dashboard exists in the database
+   - Application is updated with the resolved `DefaultNavItems` field
+
+**Console output:**
+```
+Processing Applications...
+   ⏳ Deferring lookup for Applications.DefaultNavItems -> Dashboards
+   📋 Queued Applications for deferred processing (record saved, some fields pending)
+   ✓ Created: 1
+
+Processing Dashboards...
+   ✓ Created: 1
+
+⏳ Processing 1 deferred record...
+   ✓ Applications (ID=867CB743-...) - updated
+   ✓ Resolved 1 deferred record (0 created, 1 updated)
+```
+
+**Important:** The `?allowDefer` flag queues the entire record for re-processing, not just the failed field. This ensures the exact same processing logic is used on retry, including proper handling of nested lookups within JSON structures, `@parent` references, and all other field processing.
+
+**Combining flags:**
+You can combine `?allowDefer` with `?create`:
+```json
+"CategoryID": "@lookup:Categories.Name=New Category?create&allowDefer"
+```
+This means: "Look up the category, create if missing, and if the lookup still fails for some reason, defer it."
+
+**Important notes:**
+- Deferred records are processed before the final commit (Phase 2.5)
+- If any deferred record fails on retry, the entire push transaction is rolled back
+- Use sparingly - only for genuine circular dependencies
+- The record must have a primaryKey defined in the metadata file
+
 ### @parent: References 
 Reference fields from the immediate parent entity in embedded collections:
 - `@parent:ID` - Get the parent's ID field

package/dist/index.d.ts

@@ -1,5 +1,5 @@
 export { FileBackupManager } from './lib/file-backup-manager';
-export { SyncEngine } from './lib/sync-engine';
+export { SyncEngine, DeferrableLookupError } from './lib/sync-engine';
 export type { RecordData } from './lib/sync-engine';
 export { ConfigManager, configManager } from './lib/config-manager';
 export { getSyncEngine, resetSyncEngine } from './lib/singleton-manager';
@@ -7,6 +7,8 @@ export { SQLLogger } from './lib/sql-logger';
 export { TransactionManager } from './lib/transaction-manager';
 export { JsonWriteHelper } from './lib/json-write-helper';
 export { FileWriteBatch } from './lib/file-write-batch';
+export { JsonPreprocessor } from './lib/json-preprocessor';
+export type { IncludeDirective } from './lib/json-preprocessor';
 export { RecordDependencyAnalyzer } from './lib/record-dependency-analyzer';
 export type { FlattenedRecord, ReverseDependency, DependencyAnalysisResult } from './lib/record-dependency-analyzer';
 export { EntityForeignKeyHelper } from './lib/entity-foreign-key-helper';

package/dist/index.js

@@ -1,12 +1,13 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.getDataProvider = exports.findEntityDirectories = exports.getSystemUser = exports.initializeProvider = exports.loadFolderConfig = exports.loadEntityConfig = exports.loadSyncConfig = exports.loadMJConfig = exports.FormattingService = exports.ValidationService = exports.WatchService = exports.FileResetService = exports.StatusService = exports.PushService = exports.PullService = exports.InitService = exports.DeletionReportGenerator = exports.DeletionAuditor = exports.DatabaseReferenceScanner = exports.EntityForeignKeyHelper = exports.RecordDependencyAnalyzer = exports.FileWriteBatch = exports.JsonWriteHelper = exports.TransactionManager = exports.SQLLogger = exports.resetSyncEngine = exports.getSyncEngine = exports.configManager = exports.ConfigManager = exports.SyncEngine = exports.FileBackupManager = void 0;
+exports.getDataProvider = exports.findEntityDirectories = exports.getSystemUser = exports.initializeProvider = exports.loadFolderConfig = exports.loadEntityConfig = exports.loadSyncConfig = exports.loadMJConfig = exports.FormattingService = exports.ValidationService = exports.WatchService = exports.FileResetService = exports.StatusService = exports.PushService = exports.PullService = exports.InitService = exports.DeletionReportGenerator = exports.DeletionAuditor = exports.DatabaseReferenceScanner = exports.EntityForeignKeyHelper = exports.RecordDependencyAnalyzer = exports.JsonPreprocessor = exports.FileWriteBatch = exports.JsonWriteHelper = exports.TransactionManager = exports.SQLLogger = exports.resetSyncEngine = exports.getSyncEngine = exports.configManager = exports.ConfigManager = exports.DeferrableLookupError = exports.SyncEngine = exports.FileBackupManager = void 0;
 const core_entities_server_1 = require("@memberjunction/core-entities-server");
 // Core library exports
 var file_backup_manager_1 = require("./lib/file-backup-manager");
 Object.defineProperty(exports, "FileBackupManager", { enumerable: true, get: function () { return file_backup_manager_1.FileBackupManager; } });
 var sync_engine_1 = require("./lib/sync-engine");
 Object.defineProperty(exports, "SyncEngine", { enumerable: true, get: function () { return sync_engine_1.SyncEngine; } });
+Object.defineProperty(exports, "DeferrableLookupError", { enumerable: true, get: function () { return sync_engine_1.DeferrableLookupError; } });
 var config_manager_1 = require("./lib/config-manager");
 Object.defineProperty(exports, "ConfigManager", { enumerable: true, get: function () { return config_manager_1.ConfigManager; } });
 Object.defineProperty(exports, "configManager", { enumerable: true, get: function () { return config_manager_1.configManager; } });
@@ -21,6 +22,8 @@ var json_write_helper_1 = require("./lib/json-write-helper");
 Object.defineProperty(exports, "JsonWriteHelper", { enumerable: true, get: function () { return json_write_helper_1.JsonWriteHelper; } });
 var file_write_batch_1 = require("./lib/file-write-batch");
 Object.defineProperty(exports, "FileWriteBatch", { enumerable: true, get: function () { return file_write_batch_1.FileWriteBatch; } });
+var json_preprocessor_1 = require("./lib/json-preprocessor");
+Object.defineProperty(exports, "JsonPreprocessor", { enumerable: true, get: function () { return json_preprocessor_1.JsonPreprocessor; } });
 // Deletion audit exports
 var record_dependency_analyzer_1 = require("./lib/record-dependency-analyzer");
 Object.defineProperty(exports, "RecordDependencyAnalyzer", { enumerable: true, get: function () { return record_dependency_analyzer_1.RecordDependencyAnalyzer; } });

package/dist/index.js.map

@@ -1 +1 @@
-{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":";;;AAAA,+EAAgG;AAEhG,uBAAuB;AACvB,iEAA8D;AAArD,wHAAA,iBAAiB,OAAA;AAC1B,iDAA+C;AAAtC,yGAAA,UAAU,OAAA;AAEnB,uDAAoE;AAA3D,+GAAA,aAAa,OAAA;AAAE,+GAAA,aAAa,OAAA;AACrC,6DAAyE;AAAhE,kHAAA,aAAa,OAAA;AAAE,oHAAA,eAAe,OAAA;AACvC,+CAA6C;AAApC,uGAAA,SAAS,OAAA;AAClB,iEAA+D;AAAtD,yHAAA,kBAAkB,OAAA;AAC3B,6DAA0D;AAAjD,oHAAA,eAAe,OAAA;AACxB,2DAAwD;AAA/C,kHAAA,cAAc,OAAA;AAEvB,yBAAyB;AACzB,+EAA4E;AAAnE,sIAAA,wBAAwB,OAAA;AAEjC,6EAAyE;AAAhE,mIAAA,sBAAsB,OAAA;AAE/B,+EAA4E;AAAnE,sIAAA,wBAAwB,OAAA;AAEjC,2DAAyD;AAAhD,mHAAA,eAAe,OAAA;AAExB,6EAA0E;AAAjE,oIAAA,uBAAuB,OAAA;AAEhC,kBAAkB;AAClB,sDAAqD;AAA5C,0GAAA,WAAW,OAAA;AAGpB,sDAAqD;AAA5C,0GAAA,WAAW,OAAA;AAGpB,sDAAqD;AAA5C,0GAAA,WAAW,OAAA;AAGpB,0DAAyD;AAAhD,8GAAA,aAAa,OAAA;AAGtB,gEAA+D;AAAtD,oHAAA,gBAAgB,OAAA;AAGzB,wDAAuD;AAA9C,4GAAA,YAAY,OAAA;AAGrB,kEAAiE;AAAxD,sHAAA,iBAAiB,OAAA;AAC1B,kEAAiE;AAAxD,sHAAA,iBAAiB,OAAA;AAE1B,sBAAsB;AACtB,mCAQkB;AAPhB,sGAAA,YAAY,OAAA;AACZ,wGAAA,cAAc,OAAA;AACd,0GAAA,gBAAgB,OAAA;AAChB,0GAAA,gBAAgB,OAAA;AAMlB,qBAAqB;AACrB,uDAK8B;AAJ5B,oHAAA,kBAAkB,OAAA;AAClB,+GAAA,aAAa,OAAA;AACb,uHAAA,qBAAqB,OAAA;AACrB,iHAAA,eAAe,OAAA;AAejB,IAAA,+DAAwC,GAAE,CAAC","sourcesContent":["import { LoadAIPromptEntityExtendedServerSubClass } from '@memberjunction/core-entities-server';\n\n// Core library exports\nexport { FileBackupManager } from './lib/file-backup-manager';\nexport { SyncEngine } from './lib/sync-engine';\nexport type { RecordData } from './lib/sync-engine';\nexport { ConfigManager, configManager } from './lib/config-manager';\nexport { getSyncEngine, resetSyncEngine } from './lib/singleton-manager';\nexport { SQLLogger } from './lib/sql-logger';\nexport { TransactionManager } from './lib/transaction-manager';\nexport { JsonWriteHelper } from './lib/json-write-helper';\nexport { FileWriteBatch } from './lib/file-write-batch';\n\n// Deletion audit exports\nexport { RecordDependencyAnalyzer } from './lib/record-dependency-analyzer';\nexport type { FlattenedRecord, ReverseDependency, DependencyAnalysisResult } from './lib/record-dependency-analyzer';\nexport { EntityForeignKeyHelper } from './lib/entity-foreign-key-helper';\nexport type { ReverseFKInfo } from './lib/entity-foreign-key-helper';\nexport { DatabaseReferenceScanner } from './lib/database-reference-scanner';\nexport type { DatabaseReference } from './lib/database-reference-scanner';\nexport { DeletionAuditor } from './lib/deletion-auditor';\nexport type { DeletionAudit } from './lib/deletion-auditor';\nexport { DeletionReportGenerator } from './lib/deletion-report-generator';\n\n// Service exports\nexport { InitService } from './services/InitService';\nexport type { InitOptions, InitCallbacks } from './services/InitService';\n\nexport { PullService } from './services/PullService';\nexport type { PullOptions, PullCallbacks, PullResult } from './services/PullService';\n\nexport { PushService } from './services/PushService';\nexport type { PushOptions, PushCallbacks, PushResult } from './services/PushService';\n\nexport { StatusService } from './services/StatusService';\nexport type { StatusOptions, StatusCallbacks, StatusResult } from './services/StatusService';\n\nexport { FileResetService } from './services/FileResetService';\nexport type { FileResetOptions, FileResetCallbacks, FileResetResult } from './services/FileResetService';\n\nexport { WatchService } from './services/WatchService';\nexport type { WatchOptions, WatchCallbacks, WatchResult } from './services/WatchService';\n\nexport { ValidationService } from './services/ValidationService';\nexport { FormattingService } from './services/FormattingService';\n\n// Configuration types\nexport {\n  loadMJConfig,\n  loadSyncConfig,\n  loadEntityConfig,\n  loadFolderConfig,\n  type EntityConfig,\n  type FolderConfig,\n  type RelatedEntityConfig\n} from './config';\n\n// Provider utilities\nexport {\n  initializeProvider,\n  getSystemUser,\n  findEntityDirectories,\n  getDataProvider\n} from './lib/provider-utils';\n\n// Validation types\nexport type {\n  ValidationResult,\n  ValidationError,\n  ValidationWarning,\n  EntityDependency,\n  FileValidationResult,\n  ValidationOptions,\n  ReferenceType,\n  ParsedReference\n} from './types/validation';\n\nLoadAIPromptEntityExtendedServerSubClass();"]}
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":";;;AAAA,+EAAgG;AAEhG,uBAAuB;AACvB,iEAA8D;AAArD,wHAAA,iBAAiB,OAAA;AAC1B,iDAAsE;AAA7D,yGAAA,UAAU,OAAA;AAAE,oHAAA,qBAAqB,OAAA;AAE1C,uDAAoE;AAA3D,+GAAA,aAAa,OAAA;AAAE,+GAAA,aAAa,OAAA;AACrC,6DAAyE;AAAhE,kHAAA,aAAa,OAAA;AAAE,oHAAA,eAAe,OAAA;AACvC,+CAA6C;AAApC,uGAAA,SAAS,OAAA;AAClB,iEAA+D;AAAtD,yHAAA,kBAAkB,OAAA;AAC3B,6DAA0D;AAAjD,oHAAA,eAAe,OAAA;AACxB,2DAAwD;AAA/C,kHAAA,cAAc,OAAA;AACvB,6DAA2D;AAAlD,qHAAA,gBAAgB,OAAA;AAGzB,yBAAyB;AACzB,+EAA4E;AAAnE,sIAAA,wBAAwB,OAAA;AAEjC,6EAAyE;AAAhE,mIAAA,sBAAsB,OAAA;AAE/B,+EAA4E;AAAnE,sIAAA,wBAAwB,OAAA;AAEjC,2DAAyD;AAAhD,mHAAA,eAAe,OAAA;AAExB,6EAA0E;AAAjE,oIAAA,uBAAuB,OAAA;AAEhC,kBAAkB;AAClB,sDAAqD;AAA5C,0GAAA,WAAW,OAAA;AAGpB,sDAAqD;AAA5C,0GAAA,WAAW,OAAA;AAGpB,sDAAqD;AAA5C,0GAAA,WAAW,OAAA;AAGpB,0DAAyD;AAAhD,8GAAA,aAAa,OAAA;AAGtB,gEAA+D;AAAtD,oHAAA,gBAAgB,OAAA;AAGzB,wDAAuD;AAA9C,4GAAA,YAAY,OAAA;AAGrB,kEAAiE;AAAxD,sHAAA,iBAAiB,OAAA;AAC1B,kEAAiE;AAAxD,sHAAA,iBAAiB,OAAA;AAE1B,sBAAsB;AACtB,mCAQkB;AAPhB,sGAAA,YAAY,OAAA;AACZ,wGAAA,cAAc,OAAA;AACd,0GAAA,gBAAgB,OAAA;AAChB,0GAAA,gBAAgB,OAAA;AAMlB,qBAAqB;AACrB,uDAK8B;AAJ5B,oHAAA,kBAAkB,OAAA;AAClB,+GAAA,aAAa,OAAA;AACb,uHAAA,qBAAqB,OAAA;AACrB,iHAAA,eAAe,OAAA;AAejB,IAAA,+DAAwC,GAAE,CAAC","sourcesContent":["import { LoadAIPromptEntityExtendedServerSubClass } from '@memberjunction/core-entities-server';\n\n// Core library exports\nexport { FileBackupManager } from './lib/file-backup-manager';\nexport { SyncEngine, DeferrableLookupError } from './lib/sync-engine';\nexport type { RecordData } from './lib/sync-engine';\nexport { ConfigManager, configManager } from './lib/config-manager';\nexport { getSyncEngine, resetSyncEngine } from './lib/singleton-manager';\nexport { SQLLogger } from './lib/sql-logger';\nexport { TransactionManager } from './lib/transaction-manager';\nexport { JsonWriteHelper } from './lib/json-write-helper';\nexport { FileWriteBatch } from './lib/file-write-batch';\nexport { JsonPreprocessor } from './lib/json-preprocessor';\nexport type { IncludeDirective } from './lib/json-preprocessor';\n\n// Deletion audit exports\nexport { RecordDependencyAnalyzer } from './lib/record-dependency-analyzer';\nexport type { FlattenedRecord, ReverseDependency, DependencyAnalysisResult } from './lib/record-dependency-analyzer';\nexport { EntityForeignKeyHelper } from './lib/entity-foreign-key-helper';\nexport type { ReverseFKInfo } from './lib/entity-foreign-key-helper';\nexport { DatabaseReferenceScanner } from './lib/database-reference-scanner';\nexport type { DatabaseReference } from './lib/database-reference-scanner';\nexport { DeletionAuditor } from './lib/deletion-auditor';\nexport type { DeletionAudit } from './lib/deletion-auditor';\nexport { DeletionReportGenerator } from './lib/deletion-report-generator';\n\n// Service exports\nexport { InitService } from './services/InitService';\nexport type { InitOptions, InitCallbacks } from './services/InitService';\n\nexport { PullService } from './services/PullService';\nexport type { PullOptions, PullCallbacks, PullResult } from './services/PullService';\n\nexport { PushService } from './services/PushService';\nexport type { PushOptions, PushCallbacks, PushResult } from './services/PushService';\n\nexport { StatusService } from './services/StatusService';\nexport type { StatusOptions, StatusCallbacks, StatusResult } from './services/StatusService';\n\nexport { FileResetService } from './services/FileResetService';\nexport type { FileResetOptions, FileResetCallbacks, FileResetResult } from './services/FileResetService';\n\nexport { WatchService } from './services/WatchService';\nexport type { WatchOptions, WatchCallbacks, WatchResult } from './services/WatchService';\n\nexport { ValidationService } from './services/ValidationService';\nexport { FormattingService } from './services/FormattingService';\n\n// Configuration types\nexport {\n  loadMJConfig,\n  loadSyncConfig,\n  loadEntityConfig,\n  loadFolderConfig,\n  type EntityConfig,\n  type FolderConfig,\n  type RelatedEntityConfig\n} from './config';\n\n// Provider utilities\nexport {\n  initializeProvider,\n  getSystemUser,\n  findEntityDirectories,\n  getDataProvider\n} from './lib/provider-utils';\n\n// Validation types\nexport type {\n  ValidationResult,\n  ValidationError,\n  ValidationWarning,\n  EntityDependency,\n  FileValidationResult,\n  ValidationOptions,\n  ReferenceType,\n  ParsedReference\n} from './types/validation';\n\nLoadAIPromptEntityExtendedServerSubClass();"]}

package/dist/lib/json-preprocessor.js

@@ -51,8 +51,10 @@ class JsonPreprocessor {
         const result = [];
         for (const item of arr) {
             // Check for string-based include in array (default element mode)
-            if (typeof item === 'string' && item.startsWith(`${metadata_keywords_1.METADATA_KEYWORDS.INCLUDE}:`)) {
-                const includePath = (0, metadata_keywords_1.extractKeywordValue)(item).trim();
+            const includePrefix = `${metadata_keywords_1.METADATA_KEYWORDS.INCLUDE}:`;
+            if (typeof item === 'string' && item.startsWith(includePrefix)) {
+                // Extract path directly since @include: format isn't handled by extractKeywordValue
+                const includePath = item.substring(includePrefix.length).trim();
                 const resolvedPath = this.resolvePath(includePath, currentFilePath);
                 const includedContent = await this.loadAndProcessInclude(resolvedPath);
                 // If included content is an array, spread its elements

package/dist/lib/json-preprocessor.js.map

@@ -1 +1 @@
-{"version":3,"file":"json-preprocessor.js","sourceRoot":"","sources":["../../src/lib/json-preprocessor.ts"],"names":[],"mappings":";;;;;;AAAA,wDAA0B;AAC1B,gDAAwB;AACxB,sEAAwF;AAUxF;;;GAGG;AACH,MAAa,gBAAgB;IACnB,YAAY,GAAgB,IAAI,GAAG,EAAE,CAAC;IAE9C;;;;OAIG;IACH,KAAK,CAAC,WAAW,CAAC,QAAgB;QAChC,IAAI,CAAC,YAAY,CAAC,KAAK,EAAE,CAAC;QAC1B,MAAM,WAAW,GAAG,MAAM,kBAAE,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAC;QAChD,OAAO,IAAI,CAAC,uBAAuB,CAAC,WAAW,EAAE,QAAQ,CAAC,CAAC;IAC7D,CAAC;IAED;;;;;OAKG;IACK,KAAK,CAAC,uBAAuB,CAAC,IAAS,EAAE,eAAuB;QACtE,6BAA6B;QAC7B,IAAI,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC,EAAE,CAAC;YACxB,OAAO,IAAI,CAAC,YAAY,CAAC,IAAI,EAAE,eAAe,CAAC,CAAC;QAClD,CAAC;aAAM,IAAI,IAAI,IAAI,OAAO,IAAI,KAAK,QAAQ,EAAE,CAAC;YAC5C,OAAO,IAAI,CAAC,aAAa,CAAC,IAAI,EAAE,eAAe,CAAC,CAAC;QACnD,CAAC;aAAM,CAAC;YACN,OAAO,IAAI,CAAC;QACd,CAAC;IACH,CAAC;IAED;;;;;OAKG;IACK,KAAK,CAAC,YAAY,CAAC,GAAU,EAAE,eAAuB;QAC5D,MAAM,MAAM,GAAU,EAAE,CAAC;QAEzB,KAAK,MAAM,IAAI,IAAI,GAAG,EAAE,CAAC;YACvB,iEAAiE;YACjE,IAAI,OAAO,IAAI,KAAK,QAAQ,IAAI,IAAI,CAAC,UAAU,CAAC,GAAG,qCAAiB,CAAC,OAAO,GAAG,CAAC,EAAE,CAAC;gBACjF,MAAM,WAAW,GAAI,IAAA,uCAAmB,EAAC,IAAI,CAAY,CAAC,IAAI,EAAE,CAAC;gBACjE,MAAM,YAAY,GAAG,IAAI,CAAC,WAAW,CAAC,WAAW,EAAE,eAAe,CAAC,CAAC;gBACpE,MAAM,eAAe,GAAG,MAAM,IAAI,CAAC,qBAAqB,CAAC,YAAY,CAAC,CAAC;gBAEvE,uDAAuD;gBACvD,IAAI,KAAK,CAAC,OAAO,CAAC,eAAe,CAAC,EAAE,CAAC;oBACnC,MAAM,CAAC,IAAI,CAAC,GAAG,eAAe,CAAC,CAAC;gBAClC,CAAC;qBAAM,CAAC;oBACN,kCAAkC;oBAClC,MAAM,CAAC,IAAI,CAAC,eAAe,CAAC,CAAC;gBAC/B,CAAC;YACH,CAAC;iBAAM,IAAI,IAAI,IAAI,OAAO,IAAI,KAAK,QAAQ,EAAE,CAAC;gBAC5C,gCAAgC;gBAChC,MAAM,CAAC,IAAI,CAAC,MAAM,IAAI,CAAC,uBAAuB,CAAC,IAAI,EAAE,eAAe,CAAC,CAAC,CAAC;YACzE,CAAC;iBAAM,CAAC;gBACN,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;YACpB,CAAC;QACH,CAAC;QAED,OAAO,MAAM,CAAC;IAChB,CAAC;IAED;;;;;OAKG;IACK,KAAK,CAAC,aAAa,CAAC,GAAQ,EAAE,eAAuB;QAC3D,MAAM,MAAM,GAAQ,EAAE,CAAC;QACvB,MAAM,WAAW,GAAa,EAAE,CAAC;QACjC,MAAM,iBAAiB,GAAkC,IAAI,GAAG,EAAE,CAAC;QAEnE,wEAAwE;QACxE,KAAK,MAAM,GAAG,IAAI,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC;YACnC,IAAI,GAAG,KAAK,qCAAiB,CAAC,OAAO,IAAI,GAAG,CAAC,UAAU,CAAC,GAAG,qCAAiB,CAAC,OAAO,GAAG,CAAC,EAAE,CAAC;gBACzF,WAAW,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;gBACtB,MAAM,YAAY,GAAG,GAAG,CAAC,GAAG,CAAC,CAAC;gBAE9B,IAAI,OAAO,YAAY,KAAK,QAAQ,EAAE,CAAC;oBACrC,4DAA4D;oBAC5D,iBAAiB,CAAC,GAAG,CAAC,GAAG,EAAE,EAAE,IAAI,EAAE,YAAY,EAAE,IAAI,EAAE,QAAQ,EAAE,CAAC,CAAC;gBACrE,CAAC;qBAAM,IAAI,YAAY,IAAI,OAAO,YAAY,KAAK,QAAQ,EAAE,CAAC;oBAC5D,iCAAiC;oBACjC,MAAM,SAAS,GAAG,YAAgC,CAAC;oBACnD,0CAA0C;oBAC1C,IAAI,CAAC,SAAS,CAAC,IAAI,EAAE,CAAC;wBACpB,SAAS,CAAC,IAAI,GAAG,QAAQ,CAAC;oBAC5B,CAAC;oBACD,iBAAiB,CAAC,GAAG,CAAC,GAAG,EAAE,SAAS,CAAC,CAAC;gBACxC,CAAC;YACH,CAAC;QACH,CAAC;QAED,oFAAoF;QACpF,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC;YAC/C,IAAI,GAAG,KAAK,qCAAiB,CAAC,OAAO,IAAI,GAAG,CAAC,UAAU,CAAC,GAAG,qCAAiB,CAAC,OAAO,GAAG,CAAC,EAAE,CAAC;gBACzF,iCAAiC;gBACjC,MAAM,gBAAgB,GAAG,iBAAiB,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;gBACpD,IAAI,gBAAgB,EAAE,CAAC;oBACrB,MAAM,YAAY,GAAG,IAAI,CAAC,WAAW,CAAC,gBAAgB,CAAC,IAAI,EAAE,eAAe,CAAC,CAAC;oBAC9E,MAAM,eAAe,GAAG,MAAM,IAAI,CAAC,qBAAqB,CAAC,YAAY,CAAC,CAAC;oBAEvE,IAAI,gBAAgB,CAAC,IAAI,KAAK,QAAQ,EAAE,CAAC;wBACvC,iEAAiE;wBACjE,IAAI,eAAe,IAAI,OAAO,eAAe,KAAK,QAAQ,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,eAAe,CAAC,EAAE,CAAC;4BAC9F,MAAM,CAAC,MAAM,CAAC,MAAM,EAAE,eAAe,CAAC,CAAC;wBACzC,CAAC;6BAAM,CAAC;4BACN,MAAM,IAAI,KAAK,CAAC,yCAAyC,gBAAgB,CAAC,IAAI,gDAAgD,CAAC,CAAC;wBAClI,CAAC;oBACH,CAAC;yBAAM,IAAI,gBAAgB,CAAC,IAAI,KAAK,SAAS,EAAE,CAAC;wBAC/C,4CAA4C;wBAC5C,gEAAgE;wBAChE,iEAAiE;wBACjE,IAAI,GAAG,CAAC,QAAQ,CAAC,GAAG,CAAC,EAAE,CAAC;4BACtB,yDAAyD;4BACzD,MAAM,QAAQ,GAAG,GAAG,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;4BACnD,MAAM,CAAC,QAAQ,CAAC,GAAG,eAAe,CAAC;wBACrC,CAAC;6BAAM,CAAC;4BACN,kEAAkE;4BAClE,OAAO,eAAe,CAAC;wBACzB,CAAC;oBACH,CAAC;gBACH,CAAC;YACH,CAAC;iBAAM,CAAC;gBACN,qEAAqE;gBACrE,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,CAAC,UAAU,CAAC,qCAAiB,CAAC,IAAI,CAAC,EAAE,CAAC;oBAC1E,0BAA0B;oBAC1B,MAAM,QAAQ,GAAG,IAAA,uCAAmB,EAAC,KAAK,CAAW,CAAC;oBACtD,MAAM,YAAY,GAAG,IAAI,CAAC,WAAW,CAAC,QAAQ,EAAE,eAAe,CAAC,CAAC;oBACjE,MAAM,CAAC,GAAG,CAAC,GAAG,MAAM,IAAI,CAAC,eAAe,CAAC,YAAY,CAAC,CAAC;gBACzD,CAAC;qBAAM,IAAI,KAAK,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;oBAC9C,MAAM,CAAC,GAAG,CAAC,GAAG,MAAM,IAAI,CAAC,uBAAuB,CAAC,KAAK,EAAE,eAAe,CAAC,CAAC;gBAC3E,CAAC;qBAAM,CAAC;oBACN,MAAM,CAAC,GAAG,CAAC,GAAG,KAAK,CAAC;gBACtB,CAAC;YACH,CAAC;QACH,CAAC;QAED,OAAO,MAAM,CAAC;IAChB,CAAC;IAED;;;;OAIG;IACK,KAAK,CAAC,qBAAqB,CAAC,QAAgB;QAClD,MAAM,YAAY,GAAG,cAAI,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC;QAE5C,qEAAqE;QACrE,IAAI,IAAI,CAAC,YAAY,CAAC,GAAG,CAAC,YAAY,CAAC,EAAE,CAAC;YACxC,MAAM,IAAI,KAAK,CAAC,gCAAgC,YAAY,6BAA6B,CAAC,CAAC;QAC7F,CAAC;QAED,IAAI,CAAC,MAAM,kBAAE,CAAC,UAAU,CAAC,QAAQ,CAAC,EAAE,CAAC;YACnC,oCAAoC;YACpC,OAAO,CAAC,KAAK,CAAC,4BAA4B,CAAC,CAAC;YAC5C,OAAO,CAAC,KAAK,CAAC,uBAAuB,QAAQ,EAAE,CAAC,CAAC;YACjD,OAAO,CAAC,KAAK,CAAC,6BAA6B,CAAC,CAAC;YAC7C,OAAO,CAAC,KAAK,CAAC,8EAA8E,CAAC,CAAC;YAE9F,MAAM,IAAI,KAAK,CAAC,2BAA2B,QAAQ,EAAE,CAAC,CAAC;QACzD,CAAC;QAED,yCAAyC;QACzC,IAAI,CAAC,YAAY,CAAC,GAAG,CAAC,YAAY,CAAC,CAAC;QAEpC,IAAI,CAAC;YACH,MAAM,OAAO,GAAG,MAAM,kBAAE,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAC;YAC5C,mEAAmE;YACnE,OAAO,IAAI,CAAC,uBAAuB,CAAC,OAAO,EAAE,QAAQ,CAAC,CAAC;QACzD,CAAC;gBAAS,CAAC;YACT,6CAA6C;YAC7C,IAAI,CAAC,YAAY,CAAC,MAAM,CAAC,YAAY,CAAC,CAAC;QACzC,CAAC;IACH,CAAC;IAED;;;;OAIG;IACK,KAAK,CAAC,eAAe,CAAC,QAAgB;QAC5C,IAAI,CAAC,MAAM,kBAAE,CAAC,UAAU,CAAC,QAAQ,CAAC,EAAE,CAAC;YACnC,oCAAoC;YACpC,OAAO,CAAC,KAAK,CAAC,oBAAoB,CAAC,CAAC;YACpC,OAAO,CAAC,KAAK,CAAC,uBAAuB,QAAQ,EAAE,CAAC,CAAC;YACjD,OAAO,CAAC,KAAK,CAAC,2BAA2B,CAAC,CAAC;YAC3C,OAAO,CAAC,KAAK,CAAC,mEAAmE,CAAC,CAAC;YAEnF,MAAM,IAAI,KAAK,CAAC,mBAAmB,QAAQ,0BAA0B,CAAC,CAAC;QACzE,CAAC;QAED,IAAI,CAAC;YACH,IAAI,QAAQ,CAAC,QAAQ,CAAC,OAAO,CAAC,EAAE,CAAC;gBAC/B,yDAAyD;gBACzD,MAAM,WAAW,GAAG,MAAM,kBAAE,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAC;gBAChD,MAAM,UAAU,GAAG,IAAI,CAAC,SAAS,CAAC,WAAW,CAAC,CAAC;gBAC/C,MAAM,WAAW,GAAG,UAAU,CAAC,QAAQ,CAAC,YAAY,CAAC,IAAI,UAAU,CAAC,QAAQ,CAAC,YAAY,CAAC,CAAC;gBAE3F,IAAI,WAAW,EAAE,CAAC;oBAChB,+CAA+C;oBAC/C,OAAO,MAAM,IAAI,CAAC,uBAAuB,CAAC,WAAW,EAAE,QAAQ,CAAC,CAAC;gBACnE,CAAC;qBAAM,CAAC;oBACN,gCAAgC;oBAChC,OAAO,WAAW,CAAC;gBACrB,CAAC;YACH,CAAC;iBAAM,CAAC;gBACN,8CAA8C;gBAC9C,OAAO,MAAM,kBAAE,CAAC,QAAQ,CAAC,QAAQ,EAAE,OAAO,CAAC,CAAC;YAC9C,CAAC;QACH,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,uCAAuC;YACvC,OAAO,CAAC,KAAK,CAAC,qBAAqB,CAAC,CAAC;YACrC,OAAO,CAAC,KAAK,CAAC,2BAA2B,QAAQ,EAAE,CAAC,CAAC;YACrD,OAAO,CAAC,KAAK,CAAC,aAAa,KAAK,EAAE,CAAC,CAAC;YACpC,OAAO,CAAC,KAAK,CAAC,qEAAqE,CAAC,CAAC;YAErF,iCAAiC;YACjC,MAAM,IAAI,KAAK,CAAC,oCAAoC,QAAQ,KAAK,KAAK,EAAE,CAAC,CAAC;QAC5E,CAAC;IACH,CAAC;IAED;;;;;OAKG;IACK,WAAW,CAAC,WAAmB,EAAE,eAAuB;QAC9D,MAAM,UAAU,GAAG,cAAI,CAAC,OAAO,CAAC,eAAe,CAAC,CAAC;QACjD,OAAO,cAAI,CAAC,OAAO,CAAC,UAAU,EAAE,WAAW,CAAC,CAAC;IAC/C,CAAC;IAED;;;;;OAKG;IACH,KAAK,CAAC,eAAe,CAAC,IAAS,EAAE,QAAgB;QAC/C,IAAI,CAAC,YAAY,CAAC,KAAK,EAAE,CAAC;QAC1B,OAAO,IAAI,CAAC,uBAAuB,CAAC,IAAI,EAAE,QAAQ,CAAC,CAAC;IACtD,CAAC;CACF;AAxPD,4CAwPC","sourcesContent":["import fs from 'fs-extra';\nimport path from 'path';\nimport { METADATA_KEYWORDS, extractKeywordValue } from '../constants/metadata-keywords';\n\n/**\n * Include directive configuration\n */\nexport interface IncludeDirective {\n  file: string;\n  mode?: 'spread' | 'element';\n}\n\n/**\n * Preprocesses JSON files to handle @include directives\n * Supports including external JSON files with spreading or element insertion\n */\nexport class JsonPreprocessor {\n  private visitedPaths: Set<string> = new Set();\n\n  /**\n   * Process a JSON file and resolve all @include directives\n   * @param filePath - Path to the JSON file to process\n   * @returns The processed JSON data with all includes resolved\n   */\n  async processFile(filePath: string): Promise<any> {\n    this.visitedPaths.clear();\n    const fileContent = await fs.readJson(filePath);\n    return this.processIncludesInternal(fileContent, filePath);\n  }\n\n  /**\n   * Recursively process @include directives in JSON data\n   * @param data - The JSON data to process\n   * @param currentFilePath - Path of the current file for resolving relative paths\n   * @returns The processed data with includes resolved\n   */\n  private async processIncludesInternal(data: any, currentFilePath: string): Promise<any> {\n    // Process based on data type\n    if (Array.isArray(data)) {\n      return this.processArray(data, currentFilePath);\n    } else if (data && typeof data === 'object') {\n      return this.processObject(data, currentFilePath);\n    } else {\n      return data;\n    }\n  }\n\n  /**\n   * Process an array, handling @include directives\n   * @param arr - The array to process\n   * @param currentFilePath - Current file path for resolving relative paths\n   * @returns Processed array with includes resolved\n   */\n  private async processArray(arr: any[], currentFilePath: string): Promise<any[]> {\n    const result: any[] = [];\n    \n    for (const item of arr) {\n      // Check for string-based include in array (default element mode)\n      if (typeof item === 'string' && item.startsWith(`${METADATA_KEYWORDS.INCLUDE}:`)) {\n        const includePath = (extractKeywordValue(item) as string).trim();\n        const resolvedPath = this.resolvePath(includePath, currentFilePath);\n        const includedContent = await this.loadAndProcessInclude(resolvedPath);\n        \n        // If included content is an array, spread its elements\n        if (Array.isArray(includedContent)) {\n          result.push(...includedContent);\n        } else {\n          // Otherwise add as single element\n          result.push(includedContent);\n        }\n      } else if (item && typeof item === 'object') {\n        // Process nested objects/arrays\n        result.push(await this.processIncludesInternal(item, currentFilePath));\n      } else {\n        result.push(item);\n      }\n    }\n    \n    return result;\n  }\n\n  /**\n   * Process an object, handling @include directives\n   * @param obj - The object to process\n   * @param currentFilePath - Current file path for resolving relative paths\n   * @returns Processed object with includes resolved\n   */\n  private async processObject(obj: any, currentFilePath: string): Promise<any> {\n    const result: any = {};\n    const includeKeys: string[] = [];\n    const includeDirectives: Map<string, IncludeDirective> = new Map();\n    \n    // First pass: identify all @include keys (both @include and @include.*)\n    for (const key of Object.keys(obj)) {\n      if (key === METADATA_KEYWORDS.INCLUDE || key.startsWith(`${METADATA_KEYWORDS.INCLUDE}.`)) {\n        includeKeys.push(key);\n        const includeValue = obj[key];\n        \n        if (typeof includeValue === 'string') {\n          // Simple string include - default to spread mode in objects\n          includeDirectives.set(key, { file: includeValue, mode: 'spread' });\n        } else if (includeValue && typeof includeValue === 'object') {\n          // Explicit include configuration\n          const directive = includeValue as IncludeDirective;\n          // Default to spread mode if not specified\n          if (!directive.mode) {\n            directive.mode = 'spread';\n          }\n          includeDirectives.set(key, directive);\n        }\n      }\n    }\n    \n    // Second pass: process all properties in order, spreading includes when encountered\n    for (const [key, value] of Object.entries(obj)) {\n      if (key === METADATA_KEYWORDS.INCLUDE || key.startsWith(`${METADATA_KEYWORDS.INCLUDE}.`)) {\n        // Process this include directive\n        const includeDirective = includeDirectives.get(key);\n        if (includeDirective) {\n          const resolvedPath = this.resolvePath(includeDirective.file, currentFilePath);\n          const includedContent = await this.loadAndProcessInclude(resolvedPath);\n          \n          if (includeDirective.mode === 'spread') {\n            // Spread mode: merge included object properties at this position\n            if (includedContent && typeof includedContent === 'object' && !Array.isArray(includedContent)) {\n              Object.assign(result, includedContent);\n            } else {\n              throw new Error(`Cannot spread non-object content from ${includeDirective.file}. Use mode: \"element\" for non-object includes.`);\n            }\n          } else if (includeDirective.mode === 'element') {\n            // Element mode: directly insert the content\n            // For dot notation includes, we can't replace the whole object,\n            // so we'll add it as a property instead (though this is unusual)\n            if (key.includes('.')) {\n              // Extract the part after the dot to use as property name\n              const propName = key.split('.').slice(1).join('.');\n              result[propName] = includedContent;\n            } else {\n              // For plain @include with element mode, replace the entire object\n              return includedContent;\n            }\n          }\n        }\n      } else {\n        // Regular property - process recursively and handle @file references\n        if (typeof value === 'string' && value.startsWith(METADATA_KEYWORDS.FILE)) {\n          // Process @file reference\n          const filePath = extractKeywordValue(value) as string;\n          const resolvedPath = this.resolvePath(filePath, currentFilePath);\n          result[key] = await this.loadFileContent(resolvedPath);\n        } else if (value && typeof value === 'object') {\n          result[key] = await this.processIncludesInternal(value, currentFilePath);\n        } else {\n          result[key] = value;\n        }\n      }\n    }\n    \n    return result;\n  }\n\n  /**\n   * Load and process an included file\n   * @param filePath - Path to the file to include\n   * @returns The processed content of the included file\n   */\n  private async loadAndProcessInclude(filePath: string): Promise<any> {\n    const absolutePath = path.resolve(filePath);\n    \n    // Check if this file is already being processed (circular reference)\n    if (this.visitedPaths.has(absolutePath)) {\n      throw new Error(`Circular reference detected: ${absolutePath} is already being processed`);\n    }\n    \n    if (!await fs.pathExists(filePath)) {\n      // Log error details before throwing\n      console.error(`\\n❌ INCLUDE FILE NOT FOUND`);\n      console.error(`   Referenced file: ${filePath}`);\n      console.error(`   Reference type: @include`);\n      console.error(`   Tip: Check that the file path is correct relative to the including file\\n`);\n      \n      throw new Error(`Include file not found: ${filePath}`);\n    }\n    \n    // Add to visited paths before processing\n    this.visitedPaths.add(absolutePath);\n    \n    try {\n      const content = await fs.readJson(filePath);\n      // Process the content (visited tracking is handled in this method)\n      return this.processIncludesInternal(content, filePath);\n    } finally {\n      // Remove from visited paths after processing\n      this.visitedPaths.delete(absolutePath);\n    }\n  }\n\n  /**\n   * Load file content and process it if it's JSON with @include directives\n   * @param filePath - Path to the file to load\n   * @returns The file content (processed if JSON with @includes)\n   */\n  private async loadFileContent(filePath: string): Promise<any> {\n    if (!await fs.pathExists(filePath)) {\n      // Log error details before throwing\n      console.error(`\\n❌ FILE NOT FOUND`);\n      console.error(`   Referenced file: ${filePath}`);\n      console.error(`   Reference type: @file:`);\n      console.error(`   Tip: Check that the file path is correct and the file exists\\n`);\n      \n      throw new Error(`File not found: ${filePath} (referenced via @file:)`);\n    }\n\n    try {\n      if (filePath.endsWith('.json')) {\n        // For JSON files, load and check for @include directives\n        const jsonContent = await fs.readJson(filePath);\n        const jsonString = JSON.stringify(jsonContent);\n        const hasIncludes = jsonString.includes('\"@include\"') || jsonString.includes('\"@include.');\n        \n        if (hasIncludes) {\n          // Process @include directives in the JSON file\n          return await this.processIncludesInternal(jsonContent, filePath);\n        } else {\n          // Return the JSON content as-is\n          return jsonContent;\n        }\n      } else {\n        // For non-JSON files, return the text content\n        return await fs.readFile(filePath, 'utf-8');\n      }\n    } catch (error) {\n      // Log error details before re-throwing\n      console.error(`\\n❌ FILE LOAD ERROR`);\n      console.error(`   Failed to load file: ${filePath}`);\n      console.error(`   Error: ${error}`);\n      console.error(`   Tip: Check file permissions and that the file is not corrupted\\n`);\n      \n      // Re-throw with enhanced context\n      throw new Error(`Failed to load file content from ${filePath}: ${error}`);\n    }\n  }\n\n  /**\n   * Resolve a potentially relative path to an absolute path\n   * @param includePath - The path specified in the @include\n   * @param currentFilePath - The current file's path\n   * @returns Absolute path to the included file\n   */\n  private resolvePath(includePath: string, currentFilePath: string): string {\n    const currentDir = path.dirname(currentFilePath);\n    return path.resolve(currentDir, includePath);\n  }\n\n  /**\n   * Process JSON data that's already loaded (for integration with existing code)\n   * @param data - The JSON data to process\n   * @param filePath - The file path (for resolving relative includes)\n   * @returns Processed data with includes resolved\n   */\n  async processJsonData(data: any, filePath: string): Promise<any> {\n    this.visitedPaths.clear();\n    return this.processIncludesInternal(data, filePath);\n  }\n}"]}
+{"version":3,"file":"json-preprocessor.js","sourceRoot":"","sources":["../../src/lib/json-preprocessor.ts"],"names":[],"mappings":";;;;;;AAAA,wDAA0B;AAC1B,gDAAwB;AACxB,sEAAwF;AAUxF;;;GAGG;AACH,MAAa,gBAAgB;IACnB,YAAY,GAAgB,IAAI,GAAG,EAAE,CAAC;IAE9C;;;;OAIG;IACH,KAAK,CAAC,WAAW,CAAC,QAAgB;QAChC,IAAI,CAAC,YAAY,CAAC,KAAK,EAAE,CAAC;QAC1B,MAAM,WAAW,GAAG,MAAM,kBAAE,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAC;QAChD,OAAO,IAAI,CAAC,uBAAuB,CAAC,WAAW,EAAE,QAAQ,CAAC,CAAC;IAC7D,CAAC;IAED;;;;;OAKG;IACK,KAAK,CAAC,uBAAuB,CAAC,IAAS,EAAE,eAAuB;QACtE,6BAA6B;QAC7B,IAAI,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC,EAAE,CAAC;YACxB,OAAO,IAAI,CAAC,YAAY,CAAC,IAAI,EAAE,eAAe,CAAC,CAAC;QAClD,CAAC;aAAM,IAAI,IAAI,IAAI,OAAO,IAAI,KAAK,QAAQ,EAAE,CAAC;YAC5C,OAAO,IAAI,CAAC,aAAa,CAAC,IAAI,EAAE,eAAe,CAAC,CAAC;QACnD,CAAC;aAAM,CAAC;YACN,OAAO,IAAI,CAAC;QACd,CAAC;IACH,CAAC;IAED;;;;;OAKG;IACK,KAAK,CAAC,YAAY,CAAC,GAAU,EAAE,eAAuB;QAC5D,MAAM,MAAM,GAAU,EAAE,CAAC;QAEzB,KAAK,MAAM,IAAI,IAAI,GAAG,EAAE,CAAC;YACvB,iEAAiE;YACjE,MAAM,aAAa,GAAG,GAAG,qCAAiB,CAAC,OAAO,GAAG,CAAC;YACtD,IAAI,OAAO,IAAI,KAAK,QAAQ,IAAI,IAAI,CAAC,UAAU,CAAC,aAAa,CAAC,EAAE,CAAC;gBAC/D,oFAAoF;gBACpF,MAAM,WAAW,GAAG,IAAI,CAAC,SAAS,CAAC,aAAa,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,CAAC;gBAChE,MAAM,YAAY,GAAG,IAAI,CAAC,WAAW,CAAC,WAAW,EAAE,eAAe,CAAC,CAAC;gBACpE,MAAM,eAAe,GAAG,MAAM,IAAI,CAAC,qBAAqB,CAAC,YAAY,CAAC,CAAC;gBAEvE,uDAAuD;gBACvD,IAAI,KAAK,CAAC,OAAO,CAAC,eAAe,CAAC,EAAE,CAAC;oBACnC,MAAM,CAAC,IAAI,CAAC,GAAG,eAAe,CAAC,CAAC;gBAClC,CAAC;qBAAM,CAAC;oBACN,kCAAkC;oBAClC,MAAM,CAAC,IAAI,CAAC,eAAe,CAAC,CAAC;gBAC/B,CAAC;YACH,CAAC;iBAAM,IAAI,IAAI,IAAI,OAAO,IAAI,KAAK,QAAQ,EAAE,CAAC;gBAC5C,gCAAgC;gBAChC,MAAM,CAAC,IAAI,CAAC,MAAM,IAAI,CAAC,uBAAuB,CAAC,IAAI,EAAE,eAAe,CAAC,CAAC,CAAC;YACzE,CAAC;iBAAM,CAAC;gBACN,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;YACpB,CAAC;QACH,CAAC;QAED,OAAO,MAAM,CAAC;IAChB,CAAC;IAED;;;;;OAKG;IACK,KAAK,CAAC,aAAa,CAAC,GAAQ,EAAE,eAAuB;QAC3D,MAAM,MAAM,GAAQ,EAAE,CAAC;QACvB,MAAM,WAAW,GAAa,EAAE,CAAC;QACjC,MAAM,iBAAiB,GAAkC,IAAI,GAAG,EAAE,CAAC;QAEnE,wEAAwE;QACxE,KAAK,MAAM,GAAG,IAAI,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC;YACnC,IAAI,GAAG,KAAK,qCAAiB,CAAC,OAAO,IAAI,GAAG,CAAC,UAAU,CAAC,GAAG,qCAAiB,CAAC,OAAO,GAAG,CAAC,EAAE,CAAC;gBACzF,WAAW,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;gBACtB,MAAM,YAAY,GAAG,GAAG,CAAC,GAAG,CAAC,CAAC;gBAE9B,IAAI,OAAO,YAAY,KAAK,QAAQ,EAAE,CAAC;oBACrC,4DAA4D;oBAC5D,iBAAiB,CAAC,GAAG,CAAC,GAAG,EAAE,EAAE,IAAI,EAAE,YAAY,EAAE,IAAI,EAAE,QAAQ,EAAE,CAAC,CAAC;gBACrE,CAAC;qBAAM,IAAI,YAAY,IAAI,OAAO,YAAY,KAAK,QAAQ,EAAE,CAAC;oBAC5D,iCAAiC;oBACjC,MAAM,SAAS,GAAG,YAAgC,CAAC;oBACnD,0CAA0C;oBAC1C,IAAI,CAAC,SAAS,CAAC,IAAI,EAAE,CAAC;wBACpB,SAAS,CAAC,IAAI,GAAG,QAAQ,CAAC;oBAC5B,CAAC;oBACD,iBAAiB,CAAC,GAAG,CAAC,GAAG,EAAE,SAAS,CAAC,CAAC;gBACxC,CAAC;YACH,CAAC;QACH,CAAC;QAED,oFAAoF;QACpF,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC;YAC/C,IAAI,GAAG,KAAK,qCAAiB,CAAC,OAAO,IAAI,GAAG,CAAC,UAAU,CAAC,GAAG,qCAAiB,CAAC,OAAO,GAAG,CAAC,EAAE,CAAC;gBACzF,iCAAiC;gBACjC,MAAM,gBAAgB,GAAG,iBAAiB,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;gBACpD,IAAI,gBAAgB,EAAE,CAAC;oBACrB,MAAM,YAAY,GAAG,IAAI,CAAC,WAAW,CAAC,gBAAgB,CAAC,IAAI,EAAE,eAAe,CAAC,CAAC;oBAC9E,MAAM,eAAe,GAAG,MAAM,IAAI,CAAC,qBAAqB,CAAC,YAAY,CAAC,CAAC;oBAEvE,IAAI,gBAAgB,CAAC,IAAI,KAAK,QAAQ,EAAE,CAAC;wBACvC,iEAAiE;wBACjE,IAAI,eAAe,IAAI,OAAO,eAAe,KAAK,QAAQ,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,eAAe,CAAC,EAAE,CAAC;4BAC9F,MAAM,CAAC,MAAM,CAAC,MAAM,EAAE,eAAe,CAAC,CAAC;wBACzC,CAAC;6BAAM,CAAC;4BACN,MAAM,IAAI,KAAK,CAAC,yCAAyC,gBAAgB,CAAC,IAAI,gDAAgD,CAAC,CAAC;wBAClI,CAAC;oBACH,CAAC;yBAAM,IAAI,gBAAgB,CAAC,IAAI,KAAK,SAAS,EAAE,CAAC;wBAC/C,4CAA4C;wBAC5C,gEAAgE;wBAChE,iEAAiE;wBACjE,IAAI,GAAG,CAAC,QAAQ,CAAC,GAAG,CAAC,EAAE,CAAC;4BACtB,yDAAyD;4BACzD,MAAM,QAAQ,GAAG,GAAG,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;4BACnD,MAAM,CAAC,QAAQ,CAAC,GAAG,eAAe,CAAC;wBACrC,CAAC;6BAAM,CAAC;4BACN,kEAAkE;4BAClE,OAAO,eAAe,CAAC;wBACzB,CAAC;oBACH,CAAC;gBACH,CAAC;YACH,CAAC;iBAAM,CAAC;gBACN,qEAAqE;gBACrE,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,CAAC,UAAU,CAAC,qCAAiB,CAAC,IAAI,CAAC,EAAE,CAAC;oBAC1E,0BAA0B;oBAC1B,MAAM,QAAQ,GAAG,IAAA,uCAAmB,EAAC,KAAK,CAAW,CAAC;oBACtD,MAAM,YAAY,GAAG,IAAI,CAAC,WAAW,CAAC,QAAQ,EAAE,eAAe,CAAC,CAAC;oBACjE,MAAM,CAAC,GAAG,CAAC,GAAG,MAAM,IAAI,CAAC,eAAe,CAAC,YAAY,CAAC,CAAC;gBACzD,CAAC;qBAAM,IAAI,KAAK,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;oBAC9C,MAAM,CAAC,GAAG,CAAC,GAAG,MAAM,IAAI,CAAC,uBAAuB,CAAC,KAAK,EAAE,eAAe,CAAC,CAAC;gBAC3E,CAAC;qBAAM,CAAC;oBACN,MAAM,CAAC,GAAG,CAAC,GAAG,KAAK,CAAC;gBACtB,CAAC;YACH,CAAC;QACH,CAAC;QAED,OAAO,MAAM,CAAC;IAChB,CAAC;IAED;;;;OAIG;IACK,KAAK,CAAC,qBAAqB,CAAC,QAAgB;QAClD,MAAM,YAAY,GAAG,cAAI,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC;QAE5C,qEAAqE;QACrE,IAAI,IAAI,CAAC,YAAY,CAAC,GAAG,CAAC,YAAY,CAAC,EAAE,CAAC;YACxC,MAAM,IAAI,KAAK,CAAC,gCAAgC,YAAY,6BAA6B,CAAC,CAAC;QAC7F,CAAC;QAED,IAAI,CAAC,MAAM,kBAAE,CAAC,UAAU,CAAC,QAAQ,CAAC,EAAE,CAAC;YACnC,oCAAoC;YACpC,OAAO,CAAC,KAAK,CAAC,4BAA4B,CAAC,CAAC;YAC5C,OAAO,CAAC,KAAK,CAAC,uBAAuB,QAAQ,EAAE,CAAC,CAAC;YACjD,OAAO,CAAC,KAAK,CAAC,6BAA6B,CAAC,CAAC;YAC7C,OAAO,CAAC,KAAK,CAAC,8EAA8E,CAAC,CAAC;YAE9F,MAAM,IAAI,KAAK,CAAC,2BAA2B,QAAQ,EAAE,CAAC,CAAC;QACzD,CAAC;QAED,yCAAyC;QACzC,IAAI,CAAC,YAAY,CAAC,GAAG,CAAC,YAAY,CAAC,CAAC;QAEpC,IAAI,CAAC;YACH,MAAM,OAAO,GAAG,MAAM,kBAAE,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAC;YAC5C,mEAAmE;YACnE,OAAO,IAAI,CAAC,uBAAuB,CAAC,OAAO,EAAE,QAAQ,CAAC,CAAC;QACzD,CAAC;gBAAS,CAAC;YACT,6CAA6C;YAC7C,IAAI,CAAC,YAAY,CAAC,MAAM,CAAC,YAAY,CAAC,CAAC;QACzC,CAAC;IACH,CAAC;IAED;;;;OAIG;IACK,KAAK,CAAC,eAAe,CAAC,QAAgB;QAC5C,IAAI,CAAC,MAAM,kBAAE,CAAC,UAAU,CAAC,QAAQ,CAAC,EAAE,CAAC;YACnC,oCAAoC;YACpC,OAAO,CAAC,KAAK,CAAC,oBAAoB,CAAC,CAAC;YACpC,OAAO,CAAC,KAAK,CAAC,uBAAuB,QAAQ,EAAE,CAAC,CAAC;YACjD,OAAO,CAAC,KAAK,CAAC,2BAA2B,CAAC,CAAC;YAC3C,OAAO,CAAC,KAAK,CAAC,mEAAmE,CAAC,CAAC;YAEnF,MAAM,IAAI,KAAK,CAAC,mBAAmB,QAAQ,0BAA0B,CAAC,CAAC;QACzE,CAAC;QAED,IAAI,CAAC;YACH,IAAI,QAAQ,CAAC,QAAQ,CAAC,OAAO,CAAC,EAAE,CAAC;gBAC/B,yDAAyD;gBACzD,MAAM,WAAW,GAAG,MAAM,kBAAE,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAC;gBAChD,MAAM,UAAU,GAAG,IAAI,CAAC,SAAS,CAAC,WAAW,CAAC,CAAC;gBAC/C,MAAM,WAAW,GAAG,UAAU,CAAC,QAAQ,CAAC,YAAY,CAAC,IAAI,UAAU,CAAC,QAAQ,CAAC,YAAY,CAAC,CAAC;gBAE3F,IAAI,WAAW,EAAE,CAAC;oBAChB,+CAA+C;oBAC/C,OAAO,MAAM,IAAI,CAAC,uBAAuB,CAAC,WAAW,EAAE,QAAQ,CAAC,CAAC;gBACnE,CAAC;qBAAM,CAAC;oBACN,gCAAgC;oBAChC,OAAO,WAAW,CAAC;gBACrB,CAAC;YACH,CAAC;iBAAM,CAAC;gBACN,8CAA8C;gBAC9C,OAAO,MAAM,kBAAE,CAAC,QAAQ,CAAC,QAAQ,EAAE,OAAO,CAAC,CAAC;YAC9C,CAAC;QACH,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,uCAAuC;YACvC,OAAO,CAAC,KAAK,CAAC,qBAAqB,CAAC,CAAC;YACrC,OAAO,CAAC,KAAK,CAAC,2BAA2B,QAAQ,EAAE,CAAC,CAAC;YACrD,OAAO,CAAC,KAAK,CAAC,aAAa,KAAK,EAAE,CAAC,CAAC;YACpC,OAAO,CAAC,KAAK,CAAC,qEAAqE,CAAC,CAAC;YAErF,iCAAiC;YACjC,MAAM,IAAI,KAAK,CAAC,oCAAoC,QAAQ,KAAK,KAAK,EAAE,CAAC,CAAC;QAC5E,CAAC;IACH,CAAC;IAED;;;;;OAKG;IACK,WAAW,CAAC,WAAmB,EAAE,eAAuB;QAC9D,MAAM,UAAU,GAAG,cAAI,CAAC,OAAO,CAAC,eAAe,CAAC,CAAC;QACjD,OAAO,cAAI,CAAC,OAAO,CAAC,UAAU,EAAE,WAAW,CAAC,CAAC;IAC/C,CAAC;IAED;;;;;OAKG;IACH,KAAK,CAAC,eAAe,CAAC,IAAS,EAAE,QAAgB;QAC/C,IAAI,CAAC,YAAY,CAAC,KAAK,EAAE,CAAC;QAC1B,OAAO,IAAI,CAAC,uBAAuB,CAAC,IAAI,EAAE,QAAQ,CAAC,CAAC;IACtD,CAAC;CACF;AA1PD,4CA0PC","sourcesContent":["import fs from 'fs-extra';\nimport path from 'path';\nimport { METADATA_KEYWORDS, extractKeywordValue } from '../constants/metadata-keywords';\n\n/**\n * Include directive configuration\n */\nexport interface IncludeDirective {\n  file: string;\n  mode?: 'spread' | 'element';\n}\n\n/**\n * Preprocesses JSON files to handle @include directives\n * Supports including external JSON files with spreading or element insertion\n */\nexport class JsonPreprocessor {\n  private visitedPaths: Set<string> = new Set();\n\n  /**\n   * Process a JSON file and resolve all @include directives\n   * @param filePath - Path to the JSON file to process\n   * @returns The processed JSON data with all includes resolved\n   */\n  async processFile(filePath: string): Promise<any> {\n    this.visitedPaths.clear();\n    const fileContent = await fs.readJson(filePath);\n    return this.processIncludesInternal(fileContent, filePath);\n  }\n\n  /**\n   * Recursively process @include directives in JSON data\n   * @param data - The JSON data to process\n   * @param currentFilePath - Path of the current file for resolving relative paths\n   * @returns The processed data with includes resolved\n   */\n  private async processIncludesInternal(data: any, currentFilePath: string): Promise<any> {\n    // Process based on data type\n    if (Array.isArray(data)) {\n      return this.processArray(data, currentFilePath);\n    } else if (data && typeof data === 'object') {\n      return this.processObject(data, currentFilePath);\n    } else {\n      return data;\n    }\n  }\n\n  /**\n   * Process an array, handling @include directives\n   * @param arr - The array to process\n   * @param currentFilePath - Current file path for resolving relative paths\n   * @returns Processed array with includes resolved\n   */\n  private async processArray(arr: any[], currentFilePath: string): Promise<any[]> {\n    const result: any[] = [];\n    \n    for (const item of arr) {\n      // Check for string-based include in array (default element mode)\n      const includePrefix = `${METADATA_KEYWORDS.INCLUDE}:`;\n      if (typeof item === 'string' && item.startsWith(includePrefix)) {\n        // Extract path directly since @include: format isn't handled by extractKeywordValue\n        const includePath = item.substring(includePrefix.length).trim();\n        const resolvedPath = this.resolvePath(includePath, currentFilePath);\n        const includedContent = await this.loadAndProcessInclude(resolvedPath);\n        \n        // If included content is an array, spread its elements\n        if (Array.isArray(includedContent)) {\n          result.push(...includedContent);\n        } else {\n          // Otherwise add as single element\n          result.push(includedContent);\n        }\n      } else if (item && typeof item === 'object') {\n        // Process nested objects/arrays\n        result.push(await this.processIncludesInternal(item, currentFilePath));\n      } else {\n        result.push(item);\n      }\n    }\n    \n    return result;\n  }\n\n  /**\n   * Process an object, handling @include directives\n   * @param obj - The object to process\n   * @param currentFilePath - Current file path for resolving relative paths\n   * @returns Processed object with includes resolved\n   */\n  private async processObject(obj: any, currentFilePath: string): Promise<any> {\n    const result: any = {};\n    const includeKeys: string[] = [];\n    const includeDirectives: Map<string, IncludeDirective> = new Map();\n    \n    // First pass: identify all @include keys (both @include and @include.*)\n    for (const key of Object.keys(obj)) {\n      if (key === METADATA_KEYWORDS.INCLUDE || key.startsWith(`${METADATA_KEYWORDS.INCLUDE}.`)) {\n        includeKeys.push(key);\n        const includeValue = obj[key];\n        \n        if (typeof includeValue === 'string') {\n          // Simple string include - default to spread mode in objects\n          includeDirectives.set(key, { file: includeValue, mode: 'spread' });\n        } else if (includeValue && typeof includeValue === 'object') {\n          // Explicit include configuration\n          const directive = includeValue as IncludeDirective;\n          // Default to spread mode if not specified\n          if (!directive.mode) {\n            directive.mode = 'spread';\n          }\n          includeDirectives.set(key, directive);\n        }\n      }\n    }\n    \n    // Second pass: process all properties in order, spreading includes when encountered\n    for (const [key, value] of Object.entries(obj)) {\n      if (key === METADATA_KEYWORDS.INCLUDE || key.startsWith(`${METADATA_KEYWORDS.INCLUDE}.`)) {\n        // Process this include directive\n        const includeDirective = includeDirectives.get(key);\n        if (includeDirective) {\n          const resolvedPath = this.resolvePath(includeDirective.file, currentFilePath);\n          const includedContent = await this.loadAndProcessInclude(resolvedPath);\n          \n          if (includeDirective.mode === 'spread') {\n            // Spread mode: merge included object properties at this position\n            if (includedContent && typeof includedContent === 'object' && !Array.isArray(includedContent)) {\n              Object.assign(result, includedContent);\n            } else {\n              throw new Error(`Cannot spread non-object content from ${includeDirective.file}. Use mode: \"element\" for non-object includes.`);\n            }\n          } else if (includeDirective.mode === 'element') {\n            // Element mode: directly insert the content\n            // For dot notation includes, we can't replace the whole object,\n            // so we'll add it as a property instead (though this is unusual)\n            if (key.includes('.')) {\n              // Extract the part after the dot to use as property name\n              const propName = key.split('.').slice(1).join('.');\n              result[propName] = includedContent;\n            } else {\n              // For plain @include with element mode, replace the entire object\n              return includedContent;\n            }\n          }\n        }\n      } else {\n        // Regular property - process recursively and handle @file references\n        if (typeof value === 'string' && value.startsWith(METADATA_KEYWORDS.FILE)) {\n          // Process @file reference\n          const filePath = extractKeywordValue(value) as string;\n          const resolvedPath = this.resolvePath(filePath, currentFilePath);\n          result[key] = await this.loadFileContent(resolvedPath);\n        } else if (value && typeof value === 'object') {\n          result[key] = await this.processIncludesInternal(value, currentFilePath);\n        } else {\n          result[key] = value;\n        }\n      }\n    }\n    \n    return result;\n  }\n\n  /**\n   * Load and process an included file\n   * @param filePath - Path to the file to include\n   * @returns The processed content of the included file\n   */\n  private async loadAndProcessInclude(filePath: string): Promise<any> {\n    const absolutePath = path.resolve(filePath);\n    \n    // Check if this file is already being processed (circular reference)\n    if (this.visitedPaths.has(absolutePath)) {\n      throw new Error(`Circular reference detected: ${absolutePath} is already being processed`);\n    }\n    \n    if (!await fs.pathExists(filePath)) {\n      // Log error details before throwing\n      console.error(`\\n❌ INCLUDE FILE NOT FOUND`);\n      console.error(`   Referenced file: ${filePath}`);\n      console.error(`   Reference type: @include`);\n      console.error(`   Tip: Check that the file path is correct relative to the including file\\n`);\n      \n      throw new Error(`Include file not found: ${filePath}`);\n    }\n    \n    // Add to visited paths before processing\n    this.visitedPaths.add(absolutePath);\n    \n    try {\n      const content = await fs.readJson(filePath);\n      // Process the content (visited tracking is handled in this method)\n      return this.processIncludesInternal(content, filePath);\n    } finally {\n      // Remove from visited paths after processing\n      this.visitedPaths.delete(absolutePath);\n    }\n  }\n\n  /**\n   * Load file content and process it if it's JSON with @include directives\n   * @param filePath - Path to the file to load\n   * @returns The file content (processed if JSON with @includes)\n   */\n  private async loadFileContent(filePath: string): Promise<any> {\n    if (!await fs.pathExists(filePath)) {\n      // Log error details before throwing\n      console.error(`\\n❌ FILE NOT FOUND`);\n      console.error(`   Referenced file: ${filePath}`);\n      console.error(`   Reference type: @file:`);\n      console.error(`   Tip: Check that the file path is correct and the file exists\\n`);\n      \n      throw new Error(`File not found: ${filePath} (referenced via @file:)`);\n    }\n\n    try {\n      if (filePath.endsWith('.json')) {\n        // For JSON files, load and check for @include directives\n        const jsonContent = await fs.readJson(filePath);\n        const jsonString = JSON.stringify(jsonContent);\n        const hasIncludes = jsonString.includes('\"@include\"') || jsonString.includes('\"@include.');\n        \n        if (hasIncludes) {\n          // Process @include directives in the JSON file\n          return await this.processIncludesInternal(jsonContent, filePath);\n        } else {\n          // Return the JSON content as-is\n          return jsonContent;\n        }\n      } else {\n        // For non-JSON files, return the text content\n        return await fs.readFile(filePath, 'utf-8');\n      }\n    } catch (error) {\n      // Log error details before re-throwing\n      console.error(`\\n❌ FILE LOAD ERROR`);\n      console.error(`   Failed to load file: ${filePath}`);\n      console.error(`   Error: ${error}`);\n      console.error(`   Tip: Check file permissions and that the file is not corrupted\\n`);\n      \n      // Re-throw with enhanced context\n      throw new Error(`Failed to load file content from ${filePath}: ${error}`);\n    }\n  }\n\n  /**\n   * Resolve a potentially relative path to an absolute path\n   * @param includePath - The path specified in the @include\n   * @param currentFilePath - The current file's path\n   * @returns Absolute path to the included file\n   */\n  private resolvePath(includePath: string, currentFilePath: string): string {\n    const currentDir = path.dirname(currentFilePath);\n    return path.resolve(currentDir, includePath);\n  }\n\n  /**\n   * Process JSON data that's already loaded (for integration with existing code)\n   * @param data - The JSON data to process\n   * @param filePath - The file path (for resolving relative includes)\n   * @returns Processed data with includes resolved\n   */\n  async processJsonData(data: any, filePath: string): Promise<any> {\n    this.visitedPaths.clear();\n    return this.processIncludesInternal(data, filePath);\n  }\n}"]}

package/dist/lib/sync-engine.d.ts

@@ -9,6 +9,28 @@
  */
 import { EntityInfo, BaseEntity, UserInfo } from '@memberjunction/core';
 import { EntityConfig } from '../config';
+/**
+ * Custom error class for lookup failures that can be deferred.
+ * When a lookup with `?allowDefer` flag fails, this error is thrown instead of a regular Error.
+ * The calling code can catch this specific error type and queue the lookup for later retry.
+ */
+export declare class DeferrableLookupError extends Error {
+    /** The entity name being looked up */
+    readonly entityName: string;
+    /** The lookup fields and values that failed */
+    readonly lookupFields: Array<{
+        fieldName: string;
+        fieldValue: string;
+    }>;
+    /** The original lookup string value */
+    readonly originalValue: string;
+    /** The field name where this lookup was used */
+    readonly targetFieldName?: string;
+    constructor(message: string, entityName: string, lookupFields: Array<{
+        fieldName: string;
+        fieldValue: string;
+    }>, originalValue: string, targetFieldName?: string);
+}
 /**
  * Represents the structure of a metadata record with optional sync tracking
  */
@@ -105,25 +127,39 @@ export declare class SyncEngine {
      * @param fieldValue - Value to search for
      * @param autoCreate - Whether to create the record if not found
      * @param createFields - Additional fields to set when creating
+     * @param batchContext - Optional batch context for in-memory entity resolution
+     * @param allowDefer - If true, throws DeferrableLookupError on failure instead of regular Error
+     * @param originalValue - Original lookup string (for error context in deferred lookups)
      * @returns The ID of the found or created record
-     * @throws Error if lookup fails and autoCreate is false
+     * @throws Error if lookup fails and autoCreate is false and allowDefer is false
+     * @throws DeferrableLookupError if lookup fails and allowDefer is true
      *
      * @example
      * ```typescript
      * // Simple lookup
-     * const categoryId = await resolveLookup('Categories', 'Name', 'Technology');
+     * const categoryId = await resolveLookup('Categories', [{ fieldName: 'Name', fieldValue: 'Technology' }]);
      *
      * // Lookup with auto-create
-     * const tagId = await resolveLookup('Tags', 'Name', 'New Tag', true, {
+     * const tagId = await resolveLookup('Tags', [{ fieldName: 'Name', fieldValue: 'New Tag' }], true, {
      *   Description: 'Auto-created tag',
      *   Status: 'Active'
      * });
+     *
+     * // Lookup with allowDefer - will throw DeferrableLookupError on failure
+     * try {
+     *   const id = await resolveLookup('Dashboards', [{ fieldName: 'Name', fieldValue: 'My Dashboard' }],
+     *     false, {}, batchContext, true, '@lookup:Dashboards.Name=My Dashboard?allowDefer');
+     * } catch (e) {
+     *   if (e instanceof DeferrableLookupError) {
+     *     // Queue for later retry
+     *   }
+     * }
      * ```
      */
     resolveLookup(entityName: string, lookupFields: Array<{
         fieldName: string;
         fieldValue: string;
-    }>, autoCreate?: boolean, createFields?: Record<string, any>, batchContext?: Map<string, BaseEntity>): Promise<string>;
+    }>, autoCreate?: boolean, createFields?: Record<string, any>, batchContext?: Map<string, BaseEntity>, allowDefer?: boolean, originalValue?: string): Promise<string>;
     /**
      * Build cascading defaults for a file path and process field values
      *

package/dist/lib/sync-engine.js

@@ -12,7 +12,7 @@ var __importDefault = (this && this.__importDefault) || function (mod) {
     return (mod && mod.__esModule) ? mod : { "default": mod };
 };
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.SyncEngine = void 0;
+exports.SyncEngine = exports.DeferrableLookupError = void 0;
 const path_1 = __importDefault(require("path"));
 const fs_extra_1 = __importDefault(require("fs-extra"));
 const crypto_1 = __importDefault(require("crypto"));
@@ -20,6 +20,32 @@ const axios_1 = __importDefault(require("axios"));
 const core_1 = require("@memberjunction/core");
 const json_preprocessor_1 = require("./json-preprocessor");
 const metadata_keywords_1 = require("../constants/metadata-keywords");
+/**
+ * Custom error class for lookup failures that can be deferred.
+ * When a lookup with `?allowDefer` flag fails, this error is thrown instead of a regular Error.
+ * The calling code can catch this specific error type and queue the lookup for later retry.
+ */
+class DeferrableLookupError extends Error {
+    /** The entity name being looked up */
+    entityName;
+    /** The lookup fields and values that failed */
+    lookupFields;
+    /** The original lookup string value */
+    originalValue;
+    /** The field name where this lookup was used */
+    targetFieldName;
+    constructor(message, entityName, lookupFields, originalValue, targetFieldName) {
+        super(message);
+        this.name = 'DeferrableLookupError';
+        this.entityName = entityName;
+        this.lookupFields = lookupFields;
+        this.originalValue = originalValue;
+        this.targetFieldName = targetFieldName;
+        // Maintains proper prototype chain for instanceof checks
+        Object.setPrototypeOf(this, DeferrableLookupError.prototype);
+    }
+}
+exports.DeferrableLookupError = DeferrableLookupError;
 /**
  * Core engine for synchronizing MemberJunction metadata between database and files
  *
@@ -191,17 +217,22 @@ class SyncEngine {
         // Check for @lookup: reference
         if (value.startsWith(metadata_keywords_1.METADATA_KEYWORDS.LOOKUP)) {
             const lookupStr = (0, metadata_keywords_1.extractKeywordValue)(value);
-            // Parse lookup with optional create syntax
-            // Format: EntityName.Field1=Value1&Field2=Value2?create&OtherField=Value
+            // Parse lookup with optional flags: ?create, ?allowDefer
+            // Format: EntityName.Field1=Value1&Field2=Value2?create&allowDefer&OtherField=Value
             const entityMatch = lookupStr.match(/^([^.]+)\./);
             if (!entityMatch) {
                 throw new Error(`Invalid lookup format: ${value}`);
             }
             const entityName = entityMatch[1];
             const remaining = lookupStr.substring(entityName.length + 1);
-            // Check if this has ?create syntax
-            const hasCreate = remaining.includes('?create');
-            const lookupPart = hasCreate ? remaining.split('?')[0] : remaining;
+            // Split into lookup part and flags part
+            const questionMarkIndex = remaining.indexOf('?');
+            const lookupPart = questionMarkIndex >= 0 ? remaining.substring(0, questionMarkIndex) : remaining;
+            const flagsPart = questionMarkIndex >= 0 ? remaining.substring(questionMarkIndex + 1) : '';
+            // Parse flags from the query string portion
+            const flags = new Set(flagsPart.split('&').map(f => f.split('=')[0].toLowerCase()));
+            const hasCreate = flags.has('create');
+            const allowDefer = flags.has('allowdefer');
             // Parse all lookup fields (can be multiple with &)
             const lookupFields = [];
             const lookupPairs = lookupPart.split('&');
@@ -219,11 +250,15 @@ class SyncEngine {
                 throw new Error(`No lookup fields specified: ${value}`);
             }
             // Parse additional fields for creation if ?create is present
+            // These are key=value pairs after the flags (e.g., ?create&Description=Some%20Value)
             let createFields = {};
-            if (hasCreate && remaining.includes('?create&')) {
-                const createPart = remaining.split('?create&')[1];
-                const pairs = createPart.split('&');
-                for (const pair of pairs) {
+            if (hasCreate && flagsPart) {
+                const flagPairs = flagsPart.split('&');
+                for (const pair of flagPairs) {
+                    // Skip known flags
+                    if (pair.toLowerCase() === 'create' || pair.toLowerCase() === 'allowdefer') {
+                        continue;
+                    }
                     const [key, val] = pair.split('=');
                     if (key && val) {
                         const decodedVal = decodeURIComponent(val);
@@ -232,7 +267,7 @@ class SyncEngine {
                     }
                 }
             }
-            return await this.resolveLookup(entityName, lookupFields, hasCreate, createFields, batchContext);
+            return await this.resolveLookup(entityName, lookupFields, hasCreate, createFields, batchContext, allowDefer, value);
         }
         // Check for @env: reference
         if (value.startsWith(metadata_keywords_1.METADATA_KEYWORDS.ENV)) {
@@ -253,22 +288,36 @@ class SyncEngine {
      * @param fieldValue - Value to search for
      * @param autoCreate - Whether to create the record if not found
      * @param createFields - Additional fields to set when creating
+     * @param batchContext - Optional batch context for in-memory entity resolution
+     * @param allowDefer - If true, throws DeferrableLookupError on failure instead of regular Error
+     * @param originalValue - Original lookup string (for error context in deferred lookups)
      * @returns The ID of the found or created record
-     * @throws Error if lookup fails and autoCreate is false
+     * @throws Error if lookup fails and autoCreate is false and allowDefer is false
+     * @throws DeferrableLookupError if lookup fails and allowDefer is true
      *
      * @example
      * ```typescript
      * // Simple lookup
-     * const categoryId = await resolveLookup('Categories', 'Name', 'Technology');
+     * const categoryId = await resolveLookup('Categories', [{ fieldName: 'Name', fieldValue: 'Technology' }]);
      *
      * // Lookup with auto-create
-     * const tagId = await resolveLookup('Tags', 'Name', 'New Tag', true, {
+     * const tagId = await resolveLookup('Tags', [{ fieldName: 'Name', fieldValue: 'New Tag' }], true, {
      *   Description: 'Auto-created tag',
      *   Status: 'Active'
      * });
+     *
+     * // Lookup with allowDefer - will throw DeferrableLookupError on failure
+     * try {
+     *   const id = await resolveLookup('Dashboards', [{ fieldName: 'Name', fieldValue: 'My Dashboard' }],
+     *     false, {}, batchContext, true, '@lookup:Dashboards.Name=My Dashboard?allowDefer');
+     * } catch (e) {
+     *   if (e instanceof DeferrableLookupError) {
+     *     // Queue for later retry
+     *   }
+     * }
      * ```
      */
-    async resolveLookup(entityName, lookupFields, autoCreate = false, createFields = {}, batchContext) {
+    async resolveLookup(entityName, lookupFields, autoCreate = false, createFields = {}, batchContext, allowDefer = false, originalValue) {
         // First check batch context for in-memory entities
         if (batchContext) {
             // Try to find the entity in batch context
@@ -378,7 +427,12 @@ class SyncEngine {
             }
         }
         const filterDesc = lookupFields.map(({ fieldName, fieldValue }) => `${fieldName}='${fieldValue}'`).join(' AND ');
-        throw new Error(`Lookup failed: No record found in '${entityName}' where ${filterDesc}`);
+        const errorMessage = `Lookup failed: No record found in '${entityName}' where ${filterDesc}`;
+        // If allowDefer is true, throw DeferrableLookupError so caller can queue for retry
+        if (allowDefer) {
+            throw new DeferrableLookupError(errorMessage, entityName, lookupFields, originalValue || `@lookup:${entityName}.${filterDesc}`);
+        }
+        throw new Error(errorMessage);
     }
     /**
      * Build cascading defaults for a file path and process field values