@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (495) hide show
  1. package/CHANGELOG.md +30 -2
  2. package/README.md +39 -0
  3. package/dist/cjs/auth/index.d.ts +3 -0
  4. package/dist/cjs/auth/index.js +13 -0
  5. package/dist/cjs/auth/profile-loader.d.ts +18 -0
  6. package/dist/cjs/auth/profile-loader.js +208 -0
  7. package/dist/cjs/client-factory.d.ts +4 -0
  8. package/dist/cjs/client-factory.js +10 -0
  9. package/dist/cjs/clients/fluent-client.js +13 -6
  10. package/dist/cjs/index.d.ts +3 -1
  11. package/dist/cjs/index.js +8 -2
  12. package/dist/cjs/utils/pagination-helpers.js +38 -2
  13. package/dist/cjs/versori/fluent-versori-client.js +11 -5
  14. package/dist/esm/auth/index.d.ts +3 -0
  15. package/dist/esm/auth/index.js +2 -0
  16. package/dist/esm/auth/profile-loader.d.ts +18 -0
  17. package/dist/esm/auth/profile-loader.js +169 -0
  18. package/dist/esm/client-factory.d.ts +4 -0
  19. package/dist/esm/client-factory.js +9 -0
  20. package/dist/esm/clients/fluent-client.js +13 -6
  21. package/dist/esm/index.d.ts +3 -1
  22. package/dist/esm/index.js +2 -1
  23. package/dist/esm/utils/pagination-helpers.js +38 -2
  24. package/dist/esm/versori/fluent-versori-client.js +11 -5
  25. package/dist/tsconfig.esm.tsbuildinfo +1 -1
  26. package/dist/tsconfig.tsbuildinfo +1 -1
  27. package/dist/tsconfig.types.tsbuildinfo +1 -1
  28. package/dist/types/auth/index.d.ts +3 -0
  29. package/dist/types/auth/profile-loader.d.ts +18 -0
  30. package/dist/types/client-factory.d.ts +4 -0
  31. package/dist/types/index.d.ts +3 -1
  32. package/docs/00-START-HERE/EXPORT-VALIDATION.md +158 -158
  33. package/docs/00-START-HERE/cli-analyze-source-structure-guide.md +655 -655
  34. package/docs/00-START-HERE/cli-documentation-index.md +202 -202
  35. package/docs/00-START-HERE/cli-quick-reference.md +252 -252
  36. package/docs/00-START-HERE/decision-tree.md +552 -552
  37. package/docs/00-START-HERE/getting-started.md +1070 -1070
  38. package/docs/00-START-HERE/mapper-quick-decision-guide.md +235 -235
  39. package/docs/00-START-HERE/readme.md +237 -237
  40. package/docs/00-START-HERE/retailerid-configuration.md +404 -404
  41. package/docs/00-START-HERE/sdk-philosophy.md +794 -794
  42. package/docs/00-START-HERE/troubleshooting-quick-reference.md +1086 -1086
  43. package/docs/01-TEMPLATES/faq.md +686 -686
  44. package/docs/01-TEMPLATES/patterns/pattern-templates-guide.md +68 -68
  45. package/docs/01-TEMPLATES/patterns/patterns-csv-schema-validation-and-rejection-report.md +233 -233
  46. package/docs/01-TEMPLATES/patterns/patterns-custom-resolvers.md +407 -407
  47. package/docs/01-TEMPLATES/patterns/patterns-error-handling-retry.md +511 -511
  48. package/docs/01-TEMPLATES/patterns/patterns-field-mapping-universal.md +701 -701
  49. package/docs/01-TEMPLATES/patterns/patterns-large-file-splitting.md +1430 -1430
  50. package/docs/01-TEMPLATES/patterns/patterns-master-data-etl.md +2399 -2399
  51. package/docs/01-TEMPLATES/patterns/patterns-pagination-streaming.md +447 -447
  52. package/docs/01-TEMPLATES/patterns/patterns-state-duplicate-prevention.md +385 -385
  53. package/docs/01-TEMPLATES/readme.md +957 -957
  54. package/docs/01-TEMPLATES/standalone/standalone-asn-inbound-processing.md +1209 -1209
  55. package/docs/01-TEMPLATES/standalone/standalone-graphql-query-export.md +1140 -1140
  56. package/docs/01-TEMPLATES/standalone/standalone-graphql-to-parquet-partitioned-s3.md +432 -432
  57. package/docs/01-TEMPLATES/standalone/standalone-multi-channel-inventory-sync.md +1185 -1185
  58. package/docs/01-TEMPLATES/standalone/standalone-multi-source-aggregation.md +1462 -1462
  59. package/docs/01-TEMPLATES/standalone/standalone-s3-csv-batch-api.md +1390 -1390
  60. package/docs/01-TEMPLATES/standalone/standalone-s3-csv-inventory-to-batch.md +330 -330
  61. package/docs/01-TEMPLATES/standalone/standalone-scripts-guide.md +87 -87
  62. package/docs/01-TEMPLATES/standalone/standalone-sftp-xml-graphql.md +1444 -1444
  63. package/docs/01-TEMPLATES/standalone/standalone-webhook-payload-processing.md +688 -688
  64. package/docs/01-TEMPLATES/versori/business-examples/business-examples-dropship-order-routing.md +193 -193
  65. package/docs/01-TEMPLATES/versori/business-examples/business-examples-graphql-parquet-extraction.md +518 -518
  66. package/docs/01-TEMPLATES/versori/business-examples/business-examples-inter-location-transfers.md +2162 -2162
  67. package/docs/01-TEMPLATES/versori/business-examples/business-examples-pre-order-allocation.md +2226 -2226
  68. package/docs/01-TEMPLATES/versori/business-examples/business-scenarios-guide.md +87 -87
  69. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-connection-validation-pattern.md +656 -656
  70. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-dual-workflow-connector.md +835 -835
  71. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-guide.md +108 -108
  72. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-kv-state-management.md +1533 -1533
  73. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-xml-response-patterns.md +1160 -1160
  74. package/docs/01-TEMPLATES/versori/versori-platform-guide.md +201 -201
  75. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-asn-purchase-order.md +1906 -1906
  76. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-dropship-routing.md +1074 -1074
  77. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-flash-sale-reserve.md +1395 -1395
  78. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-generic-xml-order.md +888 -888
  79. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-payment-gateway-integration.md +2478 -2478
  80. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-rma-returns-comprehensive.md +2240 -2240
  81. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-xml-order-ingestion.md +2029 -2029
  82. package/docs/01-TEMPLATES/versori/webhooks/webhook-templates-guide.md +140 -140
  83. package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/inventory-mapping.json +20 -20
  84. package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/products_2025-01-22.csv +11 -11
  85. package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/sample-data-guide.md +34 -34
  86. package/docs/01-TEMPLATES/versori/workflows/_examples/workflow-examples-guide.md +36 -36
  87. package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-modes-guide.md +1038 -1038
  88. package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-workflows-guide.md +138 -138
  89. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/graphql-extraction-guide.md +63 -63
  90. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-csv.md +2062 -2062
  91. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-xml.md +2294 -2294
  92. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-s3-csv.md +2461 -2461
  93. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-sftp-xml.md +2529 -2529
  94. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-csv.md +2464 -2464
  95. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-json.md +1959 -1959
  96. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-s3-csv.md +1953 -1953
  97. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-sftp-xml.md +2541 -2541
  98. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-s3-json.md +2384 -2384
  99. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-sftp-xml.md +2445 -2445
  100. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-csv.md +2355 -2355
  101. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-json.md +2042 -2042
  102. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-sftp-xml.md +2726 -2726
  103. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/batch-api-guide.md +206 -206
  104. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-cycle-count-reconciliation.md +2030 -2030
  105. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-multi-channel-inventory-sync.md +1882 -1882
  106. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-csv-inventory-batch.md +2827 -2827
  107. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-json-inventory-batch.md +1952 -1952
  108. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-xml-inventory-batch.md +3289 -3289
  109. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-csv-inventory-batch.md +3064 -3064
  110. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-json-inventory-batch.md +3238 -3238
  111. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-xml-inventory-batch.md +2977 -2977
  112. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/event-api-guide.md +321 -321
  113. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-json-order-cancel-event.md +959 -959
  114. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-xml-order-cancel-event.md +1170 -1170
  115. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-csv-product-event.md +2312 -2312
  116. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-json-product-event.md +2999 -2999
  117. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-parquet-product-event.md +2836 -2836
  118. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-xml-product-event.md +2395 -2395
  119. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-csv-product-event.md +2295 -2295
  120. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-json-product-event.md +2602 -2602
  121. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-parquet-product-event.md +2589 -2589
  122. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-xml-product-event.md +3578 -3578
  123. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/graphql-mutations-guide.md +93 -93
  124. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-json-order-update-graphql.md +1260 -1260
  125. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-xml-order-update-graphql.md +1472 -1472
  126. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-control-graphql.md +2417 -2417
  127. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-location-graphql.md +2811 -2811
  128. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-price-graphql.md +2619 -2619
  129. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-json-location-graphql.md +2807 -2807
  130. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-xml-location-graphql.md +2373 -2373
  131. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-control-graphql.md +2740 -2740
  132. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-location-graphql.md +2760 -2760
  133. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-json-location-graphql.md +1710 -1710
  134. package/docs/01-TEMPLATES/versori/workflows/ingestion/ingestion-workflows-guide.md +136 -136
  135. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/rubix-webhooks-guide.md +520 -520
  136. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-inline.md +1418 -1418
  137. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-universal-mapper.md +1785 -1785
  138. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-order-attribute-update.md +824 -824
  139. package/docs/01-TEMPLATES/versori/workflows/workflows-overview-guide.md +646 -646
  140. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-batch-archival.md +724 -724
  141. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-job-tracker.md +627 -627
  142. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-partial-batch-recovery.md +561 -561
  143. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-quick-reference.md +367 -367
  144. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-readme.md +407 -407
  145. package/docs/02-CORE-GUIDES/advanced-services/readme.md +49 -49
  146. package/docs/02-CORE-GUIDES/api-reference/api-reference-quick-reference.md +548 -548
  147. package/docs/02-CORE-GUIDES/api-reference/event-api-input-output-reference.md +702 -1171
  148. package/docs/02-CORE-GUIDES/api-reference/examples/client-initialization.ts +286 -286
  149. package/docs/02-CORE-GUIDES/api-reference/graphql-error-classification.md +337 -337
  150. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-01-client-api.md +399 -482
  151. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-03-authentication.md +199 -199
  152. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-04-graphql-mapping.md +925 -925
  153. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-05-services.md +1198 -1198
  154. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-06-data-sources.md +1083 -1083
  155. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-07-parsers.md +1097 -1097
  156. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-pagination.md +513 -513
  157. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-types.md +545 -597
  158. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-error-handling.md +527 -527
  159. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-webhook-validation.md +514 -514
  160. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-extraction.md +557 -557
  161. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-utilities.md +412 -412
  162. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-cli-tools.md +423 -423
  163. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-error-handling.md +716 -716
  164. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-analyze-source-structure.md +518 -518
  165. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-partial-responses.md +212 -212
  166. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-testing.md +300 -300
  167. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-13-resolver-builder.md +322 -322
  168. package/docs/02-CORE-GUIDES/api-reference/readme.md +279 -279
  169. package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-quick-reference.md +351 -351
  170. package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-readme.md +277 -277
  171. package/docs/02-CORE-GUIDES/auto-pagination/examples/auto-pagination-readme.md +178 -178
  172. package/docs/02-CORE-GUIDES/auto-pagination/examples/common-patterns.ts +351 -351
  173. package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-products.ts +384 -384
  174. package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-virtual-positions.ts +308 -308
  175. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-01-foundations.md +470 -470
  176. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-02-quick-start.md +713 -713
  177. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-03-configuration.md +754 -754
  178. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-04-advanced-patterns.md +732 -732
  179. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-05-sdk-integration.md +847 -847
  180. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-06-troubleshooting.md +359 -359
  181. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-07-api-reference.md +462 -462
  182. package/docs/02-CORE-GUIDES/auto-pagination/readme.md +54 -54
  183. package/docs/02-CORE-GUIDES/data-sources/data-sources-file-operations-error-handling.md +1487 -1487
  184. package/docs/02-CORE-GUIDES/data-sources/data-sources-quick-reference.md +836 -836
  185. package/docs/02-CORE-GUIDES/data-sources/data-sources-readme.md +276 -276
  186. package/docs/02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md +553 -553
  187. package/docs/02-CORE-GUIDES/data-sources/examples/common-patterns.ts +409 -409
  188. package/docs/02-CORE-GUIDES/data-sources/examples/data-sources-readme.md +178 -178
  189. package/docs/02-CORE-GUIDES/data-sources/examples/s3-operations.ts +308 -308
  190. package/docs/02-CORE-GUIDES/data-sources/examples/sftp-operations.ts +371 -371
  191. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-01-foundations.md +735 -735
  192. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-02-s3-operations.md +1302 -1302
  193. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-03-sftp-operations.md +1379 -1379
  194. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-04-file-patterns.md +941 -941
  195. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-05-advanced-topics.md +813 -813
  196. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-06-integration-patterns.md +486 -486
  197. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-07-troubleshooting.md +387 -387
  198. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-08-api-reference.md +417 -417
  199. package/docs/02-CORE-GUIDES/data-sources/readme.md +77 -77
  200. package/docs/02-CORE-GUIDES/error-handling-guide.md +936 -936
  201. package/docs/02-CORE-GUIDES/extraction/examples/02-core-guides-extraction-readme.md +116 -116
  202. package/docs/02-CORE-GUIDES/extraction/examples/common-patterns.ts +428 -428
  203. package/docs/02-CORE-GUIDES/extraction/examples/extract-inventory-basic.ts +187 -187
  204. package/docs/02-CORE-GUIDES/extraction/extraction-quick-reference.md +596 -596
  205. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-01-foundations.md +514 -514
  206. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-02-basic-extraction.md +823 -823
  207. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-03-parquet-processing.md +507 -507
  208. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-04-data-enrichment.md +546 -546
  209. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-05-transformation.md +494 -494
  210. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-export-formats.md +458 -458
  211. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-performance.md +138 -138
  212. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-api-reference.md +148 -148
  213. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-optimization.md +692 -692
  214. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-08-extraction-orchestrator.md +1008 -1008
  215. package/docs/02-CORE-GUIDES/extraction/readme.md +151 -151
  216. package/docs/02-CORE-GUIDES/ingestion/examples/_simple-kv-store.ts +40 -40
  217. package/docs/02-CORE-GUIDES/ingestion/examples/error-recovery.ts +728 -728
  218. package/docs/02-CORE-GUIDES/ingestion/examples/event-driven.ts +501 -501
  219. package/docs/02-CORE-GUIDES/ingestion/examples/local-file-ingestion.ts +88 -88
  220. package/docs/02-CORE-GUIDES/ingestion/examples/parquet-ingestion.ts +117 -117
  221. package/docs/02-CORE-GUIDES/ingestion/examples/performance-optimized.ts +647 -647
  222. package/docs/02-CORE-GUIDES/ingestion/examples/s3-csv-ingestion.ts +169 -169
  223. package/docs/02-CORE-GUIDES/ingestion/examples/sftp-csv-ingestion.ts +134 -134
  224. package/docs/02-CORE-GUIDES/ingestion/ingestion-quick-reference.md +546 -546
  225. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-01-introduction.md +626 -626
  226. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-02-quick-start.md +658 -658
  227. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-03-data-sources.md +1052 -1052
  228. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-04-field-mapping.md +763 -763
  229. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-05-advanced-parsers.md +676 -676
  230. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-06-batch-api.md +1295 -1295
  231. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-api-reference.md +138 -138
  232. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-state-management.md +1037 -1037
  233. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-08-performance-optimization.md +1349 -1349
  234. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-09-best-practices.md +1893 -1893
  235. package/docs/02-CORE-GUIDES/ingestion/readme.md +160 -160
  236. package/docs/02-CORE-GUIDES/logging-guide.md +585 -585
  237. package/docs/02-CORE-GUIDES/mapping/error-handling-patterns.md +401 -401
  238. package/docs/02-CORE-GUIDES/mapping/examples/02-core-guides-mapping-readme.md +128 -128
  239. package/docs/02-CORE-GUIDES/mapping/examples/common-patterns.ts +273 -273
  240. package/docs/02-CORE-GUIDES/mapping/examples/csv-location-ingestion.json +36 -36
  241. package/docs/02-CORE-GUIDES/mapping/examples/csv-mapping.ts +242 -242
  242. package/docs/02-CORE-GUIDES/mapping/examples/graphql-to-parquet-extraction.json +36 -36
  243. package/docs/02-CORE-GUIDES/mapping/examples/json-mapping.ts +213 -213
  244. package/docs/02-CORE-GUIDES/mapping/examples/json-product-to-mutation.json +48 -48
  245. package/docs/02-CORE-GUIDES/mapping/examples/xml-mapping.ts +291 -291
  246. package/docs/02-CORE-GUIDES/mapping/examples/xml-order-to-mutation.json +45 -45
  247. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-quick-reference.md +463 -463
  248. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-readme.md +227 -227
  249. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-01-introduction.md +222 -222
  250. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-02-quick-start.md +351 -351
  251. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-03-schema-validation.md +569 -569
  252. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-04-mapping-patterns.md +471 -471
  253. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-05-configuration-reference.md +611 -611
  254. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-advanced-xpath.md +148 -148
  255. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-path-syntax.md +464 -464
  256. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-api-reference.md +94 -94
  257. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-array-handling.md +307 -307
  258. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-08-custom-resolvers.md +544 -544
  259. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-09-advanced-patterns.md +427 -427
  260. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-10-hooks-and-variables.md +336 -336
  261. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-11-error-handling.md +488 -488
  262. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-12-arguments-vs-nodes.md +383 -383
  263. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-13-best-practices.md +477 -477
  264. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/readme.md +62 -62
  265. package/docs/02-CORE-GUIDES/mapping/mapping-format-decision-tree.md +480 -480
  266. package/docs/02-CORE-GUIDES/mapping/mapping-graphql-alias-batching-guide.md +820 -820
  267. package/docs/02-CORE-GUIDES/mapping/mapping-javascript-objects.md +2369 -2369
  268. package/docs/02-CORE-GUIDES/mapping/mapping-mapper-comparison-guide.md +682 -682
  269. package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-07-api-reference.md +1327 -1327
  270. package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-08-error-handling.md +1142 -1142
  271. package/docs/02-CORE-GUIDES/mapping/modules/mapping-04-use-cases.md +891 -891
  272. package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-helpers-resolvers.md +1126 -1126
  273. package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-sdk-resolvers.md +199 -199
  274. package/docs/02-CORE-GUIDES/mapping/modules/mapping-07-api-reference.md +1319 -1319
  275. package/docs/02-CORE-GUIDES/mapping/readme.md +178 -178
  276. package/docs/02-CORE-GUIDES/mapping/resolver-registration.md +410 -410
  277. package/docs/02-CORE-GUIDES/mapping/resolvers/examples/common-patterns.ts +226 -226
  278. package/docs/02-CORE-GUIDES/mapping/resolvers/examples/custom-resolvers.ts +227 -227
  279. package/docs/02-CORE-GUIDES/mapping/resolvers/examples/sdk-resolvers-usage.ts +203 -203
  280. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-readme.md +274 -274
  281. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-api-reference.md +679 -679
  282. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-cookbook.md +826 -826
  283. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-guide.md +1330 -1330
  284. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-helpers-reference.md +1437 -1437
  285. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-parameters-reference.md +553 -553
  286. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-troubleshooting.md +854 -854
  287. package/docs/02-CORE-GUIDES/mapping/resolvers/readme.md +75 -75
  288. package/docs/02-CORE-GUIDES/parsers/examples/02-core-guides-parsers-readme.md +161 -161
  289. package/docs/02-CORE-GUIDES/parsers/examples/csv-parser-examples.ts +110 -110
  290. package/docs/02-CORE-GUIDES/parsers/examples/json-parser-examples.ts +33 -33
  291. package/docs/02-CORE-GUIDES/parsers/examples/parquet-parser-examples.ts +47 -47
  292. package/docs/02-CORE-GUIDES/parsers/examples/xml-parser-examples.ts +38 -38
  293. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-01-foundations.md +355 -355
  294. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-02-csv-parser.md +772 -772
  295. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-03-json-parser.md +789 -789
  296. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-04-xml-parser.md +857 -857
  297. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-05-parquet-parser.md +603 -603
  298. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-integration-patterns.md +702 -702
  299. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-streaming.md +121 -121
  300. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-api-reference.md +89 -89
  301. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-troubleshooting.md +727 -727
  302. package/docs/02-CORE-GUIDES/parsers/parsers-quick-reference.md +482 -482
  303. package/docs/02-CORE-GUIDES/parsers/parsers-readme.md +258 -258
  304. package/docs/02-CORE-GUIDES/parsers/readme.md +65 -65
  305. package/docs/02-CORE-GUIDES/readme.md +194 -194
  306. package/docs/02-CORE-GUIDES/webhook-validation/examples/basic-validation.ts +108 -108
  307. package/docs/02-CORE-GUIDES/webhook-validation/examples/common-patterns.ts +316 -316
  308. package/docs/02-CORE-GUIDES/webhook-validation/examples/webhook-validation-readme.md +61 -61
  309. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-01-foundations.md +440 -440
  310. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-02-quick-start.md +525 -525
  311. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-03-versori-integration.md +741 -741
  312. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-04-platform-integration.md +629 -629
  313. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-05-configuration.md +535 -535
  314. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-error-handling.md +611 -611
  315. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-troubleshooting.md +124 -124
  316. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-07-api-reference.md +511 -511
  317. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-08-rubix-webhooks.md +590 -590
  318. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-09-rubix-event-vs-http-call.md +432 -432
  319. package/docs/02-CORE-GUIDES/webhook-validation/readme.md +239 -239
  320. package/docs/02-CORE-GUIDES/webhook-validation/webhook-validation-quick-reference.md +392 -392
  321. package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-quick-reference.md +498 -498
  322. package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-readme.md +313 -313
  323. package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/common-patterns.ts +612 -612
  324. package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/connector-scenarios-readme.md +253 -253
  325. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-01-foundations.md +452 -452
  326. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-02-simple-scenarios.md +681 -681
  327. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-03-intermediate-scenarios.md +637 -637
  328. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-04-advanced-scenarios.md +650 -650
  329. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-05-bidirectional-sync.md +233 -233
  330. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-06-production-patterns.md +442 -442
  331. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-07-reference.md +445 -445
  332. package/docs/03-PATTERN-GUIDES/connector-scenarios/readme.md +31 -31
  333. package/docs/03-PATTERN-GUIDES/enterprise-integration-patterns.md +1528 -1528
  334. package/docs/03-PATTERN-GUIDES/error-handling/comprehensive-error-handling-guide.md +1437 -1437
  335. package/docs/03-PATTERN-GUIDES/error-handling/error-handling-quick-reference.md +390 -390
  336. package/docs/03-PATTERN-GUIDES/error-handling/examples/common-patterns.ts +438 -438
  337. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-01-foundations.md +362 -362
  338. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-02-error-types.md +850 -850
  339. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-03-utf8-handling.md +456 -456
  340. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-04-error-scenarios.md +658 -658
  341. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-05-calling-patterns.md +671 -671
  342. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-06-retry-strategies.md +1034 -1034
  343. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-07-monitoring.md +653 -653
  344. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-08-api-reference.md +847 -847
  345. package/docs/03-PATTERN-GUIDES/error-handling/readme.md +36 -36
  346. package/docs/03-PATTERN-GUIDES/examples/__tests__/readme.md +40 -40
  347. package/docs/03-PATTERN-GUIDES/examples/__tests__/resolver-examples.test.js +282 -282
  348. package/docs/03-PATTERN-GUIDES/examples/test-data/03-pattern-guides-readme.md +110 -110
  349. package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-inventory.json +123 -123
  350. package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-order.json +171 -171
  351. package/docs/03-PATTERN-GUIDES/examples/test-data/readme.md +28 -28
  352. package/docs/03-PATTERN-GUIDES/extraction/extraction-readme.md +15 -15
  353. package/docs/03-PATTERN-GUIDES/extraction/readme.md +25 -25
  354. package/docs/03-PATTERN-GUIDES/file-operations/examples/common-patterns.ts +407 -407
  355. package/docs/03-PATTERN-GUIDES/file-operations/examples/file-operations-readme.md +142 -142
  356. package/docs/03-PATTERN-GUIDES/file-operations/file-operations-quick-reference.md +462 -462
  357. package/docs/03-PATTERN-GUIDES/file-operations/file-operations-readme.md +379 -379
  358. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-01-foundations.md +430 -430
  359. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-02-quick-start.md +484 -484
  360. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-03-s3-operations.md +507 -507
  361. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-04-sftp-operations.md +963 -963
  362. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-05-streaming-performance.md +503 -503
  363. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-archive-patterns.md +386 -386
  364. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-error-handling.md +117 -117
  365. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-api-reference.md +78 -78
  366. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-testing-troubleshooting.md +567 -567
  367. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-08-api-reference.md +1055 -1055
  368. package/docs/03-PATTERN-GUIDES/file-operations/readme.md +32 -32
  369. package/docs/03-PATTERN-GUIDES/ingestion/ingestion-readme.md +15 -15
  370. package/docs/03-PATTERN-GUIDES/ingestion/readme.md +25 -25
  371. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/batch-processing.ts +130 -130
  372. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/common-patterns.ts +360 -360
  373. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/delta-sync.ts +130 -130
  374. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/integration-patterns-readme.md +100 -100
  375. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/real-time-webhook.ts +398 -398
  376. package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-quick-reference.md +962 -962
  377. package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-readme.md +134 -134
  378. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-01-real-time-processing.md +991 -991
  379. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-02-batch-processing.md +1547 -1547
  380. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-03-delta-sync.md +1108 -1108
  381. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-04-webhook-patterns.md +1181 -1181
  382. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-05-error-handling.md +1061 -1061
  383. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-advanced-integration-services.md +1547 -1547
  384. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-performance.md +109 -109
  385. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-07-api-reference.md +34 -34
  386. package/docs/03-PATTERN-GUIDES/integration-patterns/readme.md +30 -30
  387. package/docs/03-PATTERN-GUIDES/logging-minimal-mode.md +128 -128
  388. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/common-patterns.ts +380 -380
  389. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/multiple-connections-readme.md +139 -139
  390. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/parallel-root-connections.ts +149 -149
  391. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/real-world-scenarios.ts +405 -405
  392. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-01-foundations.md +378 -378
  393. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-02-quick-start.md +566 -566
  394. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-03-targeting-connections.md +659 -659
  395. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-04-parallel-queries.md +656 -656
  396. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-05-best-practices.md +624 -624
  397. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-api-reference.md +824 -824
  398. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-versori.md +119 -119
  399. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-07-api-reference.md +87 -87
  400. package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-quick-reference.md +353 -353
  401. package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-readme.md +270 -270
  402. package/docs/03-PATTERN-GUIDES/multiple-connections/readme.md +30 -30
  403. package/docs/03-PATTERN-GUIDES/pagination/pagination-readme.md +14 -14
  404. package/docs/03-PATTERN-GUIDES/pagination/readme.md +24 -24
  405. package/docs/03-PATTERN-GUIDES/parquet/examples/common-patterns.ts +180 -180
  406. package/docs/03-PATTERN-GUIDES/parquet/examples/read-parquet.ts +48 -48
  407. package/docs/03-PATTERN-GUIDES/parquet/examples/write-parquet.ts +65 -65
  408. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-01-introduction.md +393 -393
  409. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-02-quick-start.md +572 -572
  410. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-03-reading-parquet.md +525 -525
  411. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-04-writing-parquet.md +554 -554
  412. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-05-graphql-extraction.md +405 -405
  413. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-performance.md +104 -104
  414. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-s3-integration.md +511 -511
  415. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-api-reference.md +90 -90
  416. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-performance-optimization.md +525 -525
  417. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-08-best-practices.md +712 -712
  418. package/docs/03-PATTERN-GUIDES/parquet/parquet-quick-reference.md +683 -683
  419. package/docs/03-PATTERN-GUIDES/parquet/parquet-readme.md +248 -248
  420. package/docs/03-PATTERN-GUIDES/parquet/readme.md +32 -32
  421. package/docs/03-PATTERN-GUIDES/parsers/parsers-readme.md +12 -12
  422. package/docs/03-PATTERN-GUIDES/parsers/readme.md +24 -24
  423. package/docs/03-PATTERN-GUIDES/readme.md +159 -159
  424. package/docs/03-PATTERN-GUIDES/webhooks/readme.md +24 -24
  425. package/docs/03-PATTERN-GUIDES/webhooks/webhooks-readme.md +8 -8
  426. package/docs/04-REFERENCE/architecture/architecture-01-overview.md +427 -427
  427. package/docs/04-REFERENCE/architecture/architecture-02-client-architecture.md +424 -424
  428. package/docs/04-REFERENCE/architecture/architecture-03-data-flow.md +690 -690
  429. package/docs/04-REFERENCE/architecture/architecture-04-service-layer.md +834 -834
  430. package/docs/04-REFERENCE/architecture/architecture-05-integration-architecture.md +655 -655
  431. package/docs/04-REFERENCE/architecture/architecture-06-state-management.md +653 -653
  432. package/docs/04-REFERENCE/architecture/architecture-adding-new-data-sources.md +686 -686
  433. package/docs/04-REFERENCE/architecture/readme.md +279 -279
  434. package/docs/04-REFERENCE/platforms/deno/readme.md +117 -117
  435. package/docs/04-REFERENCE/platforms/nodejs/readme.md +146 -146
  436. package/docs/04-REFERENCE/platforms/readme.md +135 -135
  437. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-01-introduction.md +398 -398
  438. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-02-quick-start.md +560 -560
  439. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-03-authentication.md +757 -757
  440. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-04-workflows.md +2476 -2476
  441. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-05-connections.md +1167 -1167
  442. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-kv-storage.md +990 -990
  443. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-state-management.md +121 -121
  444. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-api-reference.md +68 -68
  445. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-deployment.md +731 -731
  446. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-08-best-practices.md +1111 -1111
  447. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-09-signature-reference.md +766 -766
  448. package/docs/04-REFERENCE/platforms/versori/platforms-versori-readme.md +299 -299
  449. package/docs/04-REFERENCE/platforms/versori/platforms-versori-s3-sftp-configuration-guide.md +1425 -1425
  450. package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-api-key-security.md +816 -816
  451. package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-connection-security.md +681 -681
  452. package/docs/04-REFERENCE/platforms/versori/platforms-versori-workflow-task-types.md +708 -708
  453. package/docs/04-REFERENCE/platforms/versori/readme.md +108 -108
  454. package/docs/04-REFERENCE/readme.md +148 -148
  455. package/docs/04-REFERENCE/resolver-signature/examples/advanced-resolvers.ts +482 -482
  456. package/docs/04-REFERENCE/resolver-signature/examples/async-resolvers.ts +496 -496
  457. package/docs/04-REFERENCE/resolver-signature/examples/basic-resolvers.ts +343 -343
  458. package/docs/04-REFERENCE/resolver-signature/examples/resolver-signature-readme.md +188 -188
  459. package/docs/04-REFERENCE/resolver-signature/examples/testing-resolvers.ts +463 -463
  460. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-01-foundations.md +286 -286
  461. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-02-parameter-reference.md +643 -643
  462. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-03-basic-examples.md +521 -521
  463. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-04-advanced-patterns.md +739 -739
  464. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-05-sdk-resolvers.md +531 -531
  465. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-migration-guide.md +650 -650
  466. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-testing.md +125 -125
  467. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-07-api-reference.md +794 -794
  468. package/docs/04-REFERENCE/resolver-signature/readme.md +64 -64
  469. package/docs/04-REFERENCE/resolver-signature/resolver-signature-quick-reference.md +270 -270
  470. package/docs/04-REFERENCE/resolver-signature/resolver-signature-readme.md +351 -351
  471. package/docs/04-REFERENCE/schema/fluent-commerce-schema.json +764 -764
  472. package/docs/04-REFERENCE/schema/readme.md +141 -141
  473. package/docs/04-REFERENCE/testing/examples/04-reference-testing-readme.md +158 -158
  474. package/docs/04-REFERENCE/testing/examples/fluent-testing.ts +62 -62
  475. package/docs/04-REFERENCE/testing/examples/health-check.ts +155 -155
  476. package/docs/04-REFERENCE/testing/examples/integration-test.ts +119 -119
  477. package/docs/04-REFERENCE/testing/examples/performance-test.ts +183 -183
  478. package/docs/04-REFERENCE/testing/examples/s3-testing.ts +127 -127
  479. package/docs/04-REFERENCE/testing/modules/04-reference-testing-01-foundations.md +267 -267
  480. package/docs/04-REFERENCE/testing/modules/04-reference-testing-02-s3-testing.md +599 -599
  481. package/docs/04-REFERENCE/testing/modules/04-reference-testing-03-fluent-testing.md +589 -589
  482. package/docs/04-REFERENCE/testing/modules/04-reference-testing-04-integration-testing.md +699 -699
  483. package/docs/04-REFERENCE/testing/modules/04-reference-testing-05-debugging.md +478 -478
  484. package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-cicd-integration.md +463 -463
  485. package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-preflight-validation.md +131 -131
  486. package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-best-practices.md +499 -499
  487. package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-coverage-ci.md +165 -165
  488. package/docs/04-REFERENCE/testing/modules/04-reference-testing-08-api-reference.md +634 -634
  489. package/docs/04-REFERENCE/testing/readme.md +86 -86
  490. package/docs/04-REFERENCE/testing/testing-quick-reference.md +667 -667
  491. package/docs/04-REFERENCE/testing/testing-readme.md +286 -286
  492. package/docs/04-REFERENCE/troubleshooting/readme.md +144 -144
  493. package/docs/04-REFERENCE/troubleshooting/troubleshooting-deno-sftp-compatibility.md +392 -392
  494. package/docs/template-loading-matrix.md +242 -242
  495. package/package.json +5 -3
@@ -1,2373 +1,2373 @@
1
- ---
2
- template_id: tpl-ingest-s3-xml-to-location-graphql
3
- canonical_filename: template-ingestion-s3-xml-location-graphql.md
4
- version: 2.0.0
5
- sdk_version: ^0.1.39
6
- runtime: versori
7
- direction: ingestion
8
- source: s3-xml
9
- destination: fluent-graphql
10
- entity: location
11
- format: xml
12
- logging: versori
13
- status: stable
14
- compliance: gold-standard
15
- features:
16
- - graphql-mutation-mapper
17
- - memory-management
18
- - enhanced-logging
19
- - attribute-transformation
20
- ---
21
-
22
- Template: Ingestion - S3 XML to Location GraphQL
23
-
24
- **Template Version:** 2.0.0
25
- **SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
26
- **Last Updated:** 2025-01-24
27
-
28
- **🆕 Version 2.0.0 Enhancements:**
29
- - ✅ **GraphQL Mutation Mapper** - Direct field mapping to mutation variables
30
- - ✅ **Memory Management** - Clear large arrays after processing batches
31
- - ✅ **Enhanced Logging** - Track mutation execution with emoji indicators
32
- - ✅ **Attribute Transformation** - Handle complex nested data structures
33
-
34
- ---
35
-
36
- ## Implementation Prompt
37
-
38
- "Create a Versori scheduled workflow that reads location XML files from S3, transforms the data using GraphQLMutationMapper with nested object mapping, and creates/updates Fluent Commerce locations via direct GraphQL mutations with alias batching support.
39
-
40
- **Requirements:**
41
-
42
- 1. **S3 Source**:
43
- - List and download XML files with configurable prefix and pattern
44
- - Archive processed files to `processed/` directory (deduplication via archiving)
45
- - Move failed files to `errors/` directory
46
- - Use S3DataSource with retry logic and built-in archival
47
- - Support configurable bucket, region, credentials
48
-
49
- 2. **XML Parsing**:
50
- - Use XMLParserService with @ prefix for attribute access
51
- - Handle both single and multiple `<location>` elements (array normalization)
52
- - Support nested XML paths (`location.address.street1`, `location.coordinates.@lat`)
53
- - Parse complex structures (addresses, coordinates, opening schedules)
54
-
55
- 3. **Field Mapping**:
56
- - Map XML fields to Location GraphQL input type with nested objects:
57
- - `ref`, `name`, `type` (root fields)
58
- - `primaryAddress.*` (nested address object with coordinates)
59
- - `openingSchedule.*` (nested schedule object with 7-day hours)
60
- - Use SDK resolvers (trim, uppercase, parseFloat, parseInt, boolean)
61
- - Support custom resolvers for complex transformations
62
- - Validate required fields
63
- - Note: Check your GraphQL schema to determine if `retailer.id` field exists and is mandatory/optional
64
-
65
- 4. **GraphQL Mutations** (Direct - NO Batch API):
66
- - Execute `createLocation` mutation directly for each location
67
- - **NO BPP (Batch Pre-Processing)** - Not applicable for direct GraphQL mutations
68
- - **NO Batch API** - Use direct GraphQL mutations with rate limiting instead
69
- - Use rate limiting (configurable mutations per second)
70
- - Add delay between mutations to avoid API throttling
71
- - Retry failed mutations with exponential backoff
72
- - Track successful vs failed mutations per file
73
-
74
- 5. **Job Tracking & State Management**:
75
- - **Use JobTracker** - Track job lifecycle (start, complete, fail) in KV store
76
- - Use StateService + VersoriKVAdapter for duplicate file prevention
77
- - Track processed files in KV store with metadata
78
- - Store error state with exponential backoff tracking
79
- - Support distributed state across workflow runs
80
- - Provide job status endpoint for monitoring
81
-
82
- 6. **Error Handling**:
83
- - File-level errors archived to `/errors/` subdirectory
84
- - Record-level errors tracked but don't stop file processing
85
- - Mapping errors logged with specific location context
86
- - Mutation errors retried with exponential backoff
87
- - Error state tracking with next retry timestamp
88
-
89
- 7. **Advanced Features**:
90
- - Configurable rate limiting (mutations per second)
91
- - Empty file detection and archival
92
- - Timestamp-based error tracking
93
- - Comprehensive monitoring and logging
94
- - Manual webhook trigger with job tracking
95
- - Job status query endpoint
96
-
97
- **Use SDK Components:**
98
-
99
- - `createClient()` - Universal client factory for Versori
100
- - `S3DataSource` - S3 operations with retry logic
101
- - `XMLParserService` - XML parsing with @ prefix attribute support
102
- - `GraphQLMutationMapper` - Field transformation with schema validation and nested object support
103
- - `StateService` + `VersoriKVAdapter` - Duplicate prevention with KV storage
104
- - Native Versori `log` - Structured logging
105
-
106
- **Configuration Variables** (from Versori activation):
107
-
108
- ```typescript
109
- {
110
- s3: {
111
- bucketName: string;
112
- region: string;
113
- accessKeyId: string;
114
- secretAccessKey: string;
115
- prefix: string; // e.g., 'locations/'
116
- archivePrefix: string; // e.g., 'processed/'
117
- errorPrefix: string; // e.g., 'errors/'
118
- filePattern: string; // e.g., '.xml'
119
- maxFilesToProcess: number;
120
- enableArchival: boolean;
121
- },
122
- fluent: {
123
- retailerId: string; // Optional: Only if mutation schema requires retailerId in input
124
- mutationRateLimit: number; // mutations per second (e.g., 5)
125
- }
126
- }
127
- ```
128
-
129
- **Architecture Pattern:**
130
-
131
- ```
132
- S3 Bucket → List Files → Download XML → Parse → Map → GraphQL Mutation → Archive/Move
133
- ↓ ↓ ↓ ↓ ↓ ↓ ↓
134
- Configure Filter by S3DataSource XML Universal Direct S3 moveFile()
135
- Connection Pattern + Retry Parser Mapper createLocation to processed/
136
- (@) (nested) + Rate Limit or errors/
137
- (deduplication)
138
- ```
139
-
140
- **Deliverables:**
141
-
142
- 1. Complete Versori workflow with package.json
143
- 2. Main workflow logic with S3 + XML + GraphQL patterns
144
- 3. Helper functions for rate limiting and retry
145
- 4. XML path resolution examples
146
- 5. Sample XML files with nested structures
147
- 6. Schema validation CLI commands
148
- 7. Testing and deployment instructions
149
- 8. Monitoring and troubleshooting guidance"
150
-
151
- ---
152
-
153
- # STEP 3: Complete Implementation
154
-
155
- ## Versori Scheduled: S3 XML → Location GraphQL
156
-
157
- **FC Connect SDK Use Case Guide**
158
-
159
- > **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
160
- > **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
161
-
162
- **Context**: Versori scheduled workflow that reads location XML files from S3 and creates/updates Fluent Commerce locations via GraphQL mutations with XML path resolution and rate limiting
163
-
164
- **Complexity**: Medium
165
-
166
- **Runtime**: Versori Platform
167
-
168
- **Estimated Lines**: ~850 lines (with comprehensive documentation)
169
-
170
- ---
171
-
172
- ## What You'll Build
173
-
174
- - Scheduled Versori workflow (daily location sync)
175
- - S3 file listing, download, and archival/move
176
- - XML parsing with @ prefix for attributes (XPath-style)
177
- - Array normalization (single element → array conversion)
178
- - GraphQLMutationMapper-based field transformations with nested objects
179
- - GraphQL mutations for location upserts with alias batching support
180
- - Retry logic with exponential backoff
181
- - StateService duplicate prevention (KV-backed)
182
- - Error state tracking and file error archival
183
- - Manual webhook trigger and job status endpoint
184
-
185
- ---
186
-
187
- ## When to Use GraphQL Mutations vs Batch API vs Event API
188
-
189
- ### ✅ Use GraphQL Mutations For:
190
-
191
- | Entity Type | Use Case | Why GraphQL |
192
- | -------------- | ---------------------------------------- | ------------------------------------- |
193
- | **Locations** | Store/warehouse master data (low volume) | Direct control, immediate validation |
194
- | **Controls** | System configuration, settings | Single operations, complex queries |
195
- | **Prices** | Price updates (moderate volume) | Immediate feedback, custom logic |
196
- | **Single Ops** | One-off creates/updates | Testing, debugging, direct API access |
197
-
198
- ### ❌ Use Event API Instead For:
199
-
200
- | Entity Type | Use Case | Why Event API |
201
- | ------------------- | -------------------------------------- | -------------------------------------------- |
202
- | **Products** | Product catalog sync, variant updates | Triggers workflows, validates business rules |
203
- | **Customers** | Customer registration, profile updates | Needs workflow for downstream systems |
204
- | **Orders** | Order creation, status updates | Event-driven fulfillment workflows |
205
- | **Custom Entities** | Any entity needing workflow triggers | Full Rubix workflow support |
206
-
207
- ### 🔄 Use Batch API For:
208
-
209
- | Entity Type | Use Case | Why Batch API |
210
- | ------------------ | ---------------------------------- | ----------------------------------------------- |
211
- | **Inventory ONLY** | Bulk inventory updates, daily sync | Optimized for high-volume, BPP change detection |
212
-
213
- ---
214
-
215
- ## XML File Format
216
-
217
- ### Sample: locations.xml
218
-
219
- ```xml
220
- <?xml version="1.0" encoding="UTF-8"?>
221
- <locations>
222
- <location ref="LOC-001" type="WAREHOUSE">
223
- <name>Downtown Warehouse</name>
224
- <address country="USA">
225
- <street1>123 Main St</street1>
226
- <street2>Building A</street2>
227
- <city>New York</city>
228
- <state>NY</state>
229
- <postalCode>10001</postalCode>
230
- </address>
231
- <coordinates lat="40.7128" lon="-74.0060"/>
232
- <timeZone>America/New_York</timeZone>
233
- <openingSchedule>
234
- <allHours>false</allHours>
235
- <monStart>800</monStart>
236
- <monEnd>1800</monEnd>
237
- <tueStart>800</tueStart>
238
- <tueEnd>1800</tueEnd>
239
- <wedStart>800</wedStart>
240
- <wedEnd>1800</wedEnd>
241
- <thuStart>800</thuStart>
242
- <thuEnd>1800</thuEnd>
243
- <friStart>800</friStart>
244
- <friEnd>1800</friEnd>
245
- <satStart>0</satStart>
246
- <satEnd>0</satEnd>
247
- <sunStart>0</sunStart>
248
- <sunEnd>0</sunEnd>
249
- </openingSchedule>
250
- </location>
251
-
252
- <location ref="LOC-002" type="DC">
253
- <name>Regional DC</name>
254
- <address country="USA">
255
- <street1>456 Industrial Pkwy</street1>
256
- <city>Los Angeles</city>
257
- <state>CA</state>
258
- <postalCode>90001</postalCode>
259
- </address>
260
- <coordinates lat="34.0522" lon="-118.2437"/>
261
- <timeZone>America/Los_Angeles</timeZone>
262
- <openingSchedule>
263
- <allHours>true</allHours>
264
- <monStart>0</monStart>
265
- <monEnd>0</monEnd>
266
- <tueStart>0</tueStart>
267
- <tueEnd>0</tueEnd>
268
- <wedStart>0</wedStart>
269
- <wedEnd>0</wedEnd>
270
- <thuStart>0</thuStart>
271
- <thuEnd>0</thuEnd>
272
- <friStart>0</friStart>
273
- <friEnd>0</friEnd>
274
- <satStart>0</satStart>
275
- <satEnd>0</satEnd>
276
- <sunStart>0</sunStart>
277
- <sunEnd>0</sunEnd>
278
- </openingSchedule>
279
- </location>
280
- </locations>
281
- ```
282
-
283
- **XML Path Syntax (with @ prefix for attributes):**
284
-
285
- - `location.@ref` → XML attribute `ref` on `<location>` element
286
- - `location.name` → Text content of `<name>` element
287
- - `location.address.street1` → Nested element path
288
- - `location.address.@country` → XML attribute on nested `<address>` element
289
- - `location.coordinates.@lat` → XML attribute for latitude
290
- - `location.openingSchedule.monStart` → Deeply nested element
291
-
292
- **Note**: The SDK's `XMLParserService` automatically handles XML attributes using `@` prefix notation.
293
-
294
- ---
295
-
296
- ## Versori Workflows Structure
297
-
298
- **Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
299
-
300
- **Trigger Types:**
301
- - **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
302
- - **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
303
- - **`workflow()`** → Durable workflows (advanced, rarely used)
304
-
305
- **Execution Steps (chained to triggers):**
306
- - **`http()`** → External API calls (chained from schedule/webhook)
307
- - **`fn()`** → Internal processing (chained from schedule/webhook)
308
-
309
- ### Recommended Project Structure
310
-
311
- ```
312
- s3-xml-location-graphql/
313
- ├── index.ts # Entry point - exports all workflows
314
- └── src/
315
- ├── workflows/
316
- │ ├── scheduled/
317
- │ │ └── daily-location-sync.ts # Scheduled: Daily location sync
318
- │ │
319
- │ └── webhook/
320
- │ ├── adhoc-location-sync.ts # Webhook: Manual trigger
321
- │ └── job-status-check.ts # Webhook: Status query
322
-
323
- ├── services/
324
- │ └── location-sync.service.ts # Shared orchestration logic (reusable)
325
-
326
- └── config/
327
- └── location-mapping.json # GraphQL mapping config
328
- ```
329
-
330
- ---
331
-
332
- ## Complete Versori Workflow
333
-
334
- ### Step 1: Package Configuration
335
-
336
- **File: package.json**
337
-
338
- ```json
339
- {
340
- "name": "versori-s3-xml-location-sync",
341
- "version": "1.0.0",
342
- "description": "Versori workflow: S3 XML location sync to Fluent GraphQL",
343
- "versori": {
344
- "workflows": "./index.ts"
345
- },
346
- "type": "module",
347
- "scripts": {
348
- "deploy": "versori deploy",
349
- "logs": "versori logs"
350
- },
351
- "dependencies": {
352
- "@fluentcommerce/fc-connect-sdk": "^0.1.39",
353
- "@versori/run": "latest"
354
- },
355
- "devDependencies": {
356
- "typescript": "^5.0.0",
357
- "@types/node": "^20.0.0"
358
- }
359
- }
360
- ```
361
-
362
- ### Step 2: Workflow Entry Point (`index.ts`)
363
-
364
- **Purpose**: Register all workflows with Versori platform
365
-
366
- ```typescript
367
- /**
368
- * Entry Point - Registers all workflows with Versori platform
369
- *
370
- * Versori automatically discovers and registers exported workflows
371
- *
372
- * File Structure:
373
- * - src/workflows/scheduled/ → Time-based triggers (cron)
374
- * - src/workflows/webhook/ → HTTP-based triggers (webhooks)
375
- */
376
-
377
- // Scheduled workflows
378
- export { dailyLocationSync } from './src/workflows/scheduled/daily-location-sync';
379
-
380
- // Webhook workflows
381
- export { adhocLocationSync } from './src/workflows/webhook/adhoc-location-sync';
382
- export { locationSyncJobStatus } from './src/workflows/webhook/job-status-check';
383
- ```
384
-
385
- **What Gets Exposed:**
386
- - ✅ `adhocLocationSync` → `https://{workspace}.versori.run/location-sync-adhoc`
387
- - ✅ `locationSyncJobStatus` → `https://{workspace}.versori.run/location-sync-job-status`
388
- - ❌ `dailyLocationSync` → NOT exposed (runs automatically on cron)
389
-
390
- ---
391
-
392
- ### Step 3: Workflow Files
393
-
394
- #### `src/workflows/scheduled/daily-location-sync.ts`
395
-
396
- **Purpose**: Automatic daily location sync
397
- **Trigger**: Cron schedule (`0 2 * * *`)
398
- **Exposed as Endpoint**: ❌ NO - Runs automatically
399
-
400
- ```typescript
401
- import { schedule, http } from '@versori/run';
402
- import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
403
- import { executeLocationSync } from '../../services/location-sync.service';
404
-
405
- /**
406
- * Scheduled Workflow: Daily Location Sync
407
- *
408
- * Runs automatically daily at 2 AM UTC
409
- * NOT exposed as HTTP endpoint - Versori executes on schedule
410
- *
411
- * Uses shared service: location-sync.service.ts
412
- */
413
- export const dailyLocationSync = schedule(
414
- 'location-sync-scheduled',
415
- '0 2 * * *' // Daily at 2 AM UTC
416
- ).then(
417
- http('run-location-sync', { connection: 'fluent_commerce' }, async ctx => {
418
- const startTime = Date.now();
419
- const { log, openKv } = ctx;
420
- const jobId = `location-sync-${Date.now()}`;
421
- const tracker = new JobTracker(openKv(':project:'), log);
422
-
423
- log.info('🚀 [START] Daily location sync initiated', { jobId, trigger: 'schedule' });
424
-
425
- await tracker.createJob(jobId, { triggeredBy: 'schedule', stage: 'initialization' });
426
- await tracker.updateJob(jobId, { status: 'processing' });
427
-
428
- try {
429
- log.info('⚙️ [PROCESSING] Starting location synchronization workflow', { jobId });
430
- const result = await executeLocationSync(ctx, jobId, tracker);
431
- await tracker.markCompleted(jobId, result);
432
-
433
- const duration = Date.now() - startTime;
434
- log.info('✅ [SUCCESS] Daily location sync completed', {
435
- jobId,
436
- duration: `${duration}ms`,
437
- processed: result.processed,
438
- totalRecords: result.totalRecords
439
- });
440
-
441
- return { success: true, jobId, duration, ...result };
442
- } catch (e: any) {
443
- await tracker.markFailed(jobId, e);
444
- const duration = Date.now() - startTime;
445
-
446
- log.error('❌ [FAILED] Daily location sync failed', {
447
- jobId,
448
- duration: `${duration}ms`,
449
- error: e?.message,
450
- errorType: e?.name
451
- });
452
-
453
- return { success: false, jobId, duration, error: e?.message };
454
- }
455
- })
456
- );
457
- ```
458
-
459
- ---
460
-
461
- #### `src/workflows/webhook/adhoc-location-sync.ts`
462
-
463
- **Purpose**: Manual location sync trigger (on-demand)
464
- **Trigger**: HTTP POST
465
- **Endpoint**: `POST https://{workspace}.versori.run/location-sync-adhoc`
466
- **Use Cases**: Testing, priority processing, ad-hoc runs
467
-
468
- ```typescript
469
- import { webhook, http } from '@versori/run';
470
- import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
471
- import { executeLocationSync } from '../../services/location-sync.service';
472
-
473
- /**
474
- * Webhook: Manual Location Sync Trigger
475
- *
476
- * Endpoint: POST https://{workspace}.versori.run/location-sync-adhoc
477
- * Request body (optional): { filePattern: "urgent_*.xml", maxFiles: 5 }
478
- *
479
- * Pattern: webhook().then(http()) - needs Fluent API access
480
- * Uses shared service: location-sync.service.ts
481
- *
482
- * SECURITY: Authentication handled via connection parameter
483
- * No manual API key validation needed - Versori manages this via connection auth
484
- */
485
- export const adhocLocationSync = webhook('location-sync-adhoc', {
486
- response: { mode: 'sync' },
487
- connection: 'location-sync-adhoc', // Versori validates API key
488
- }).then(
489
- http('run-location-sync-adhoc', { connection: 'fluent_commerce' }, async ctx => {
490
- const startTime = Date.now();
491
- const { log, openKv, data } = ctx;
492
- const jobId = `location-sync-adhoc-${Date.now()}`;
493
- const tracker = new JobTracker(openKv(':project:'), log);
494
-
495
- log.info('🚀 [START] Manual location sync triggered', { jobId, trigger: 'webhook', options: data });
496
-
497
- await tracker.createJob(jobId, {
498
- triggeredBy: 'manual',
499
- stage: 'initialization',
500
- options: data // Optional: filePattern, maxFiles, etc.
501
- });
502
- await tracker.updateJob(jobId, { status: 'processing' });
503
-
504
- try {
505
- log.info('⚙️ [PROCESSING] Starting manual location synchronization', { jobId });
506
- const result = await executeLocationSync(ctx, jobId, tracker);
507
- await tracker.markCompleted(jobId, result);
508
-
509
- const duration = Date.now() - startTime;
510
- log.info('✅ [SUCCESS] Manual location sync completed', {
511
- jobId,
512
- duration: `${duration}ms`,
513
- processed: result.processed,
514
- totalRecords: result.totalRecords
515
- });
516
-
517
- return { success: true, jobId, duration, ...result };
518
- } catch (e: any) {
519
- await tracker.markFailed(jobId, e);
520
- const duration = Date.now() - startTime;
521
-
522
- log.error('❌ [FAILED] Manual location sync failed', {
523
- jobId,
524
- duration: `${duration}ms`,
525
- error: e?.message,
526
- errorType: e?.name
527
- });
528
-
529
- return { success: false, jobId, duration, error: e?.message };
530
- }
531
- })
532
- );
533
- ```
534
-
535
- ---
536
-
537
- #### `src/workflows/webhook/job-status-check.ts`
538
-
539
- **Purpose**: Query job status
540
- **Trigger**: HTTP POST
541
- **Endpoint**: `POST https://{workspace}.versori.run/location-sync-job-status`
542
- **Request body**: `{ "jobId": "location-sync-1234567890" }`
543
-
544
- ```typescript
545
- import { webhook, fn } from '@versori/run';
546
- import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
547
-
548
- /**
549
- * Webhook: Job Status Check
550
- *
551
- * Endpoint: POST https://{workspace}.versori.run/location-sync-job-status
552
- * Request body: { "jobId": "location-sync-1234567890" }
553
- *
554
- * Pattern: webhook().then(fn()) - no external API needed, only KV storage
555
- * Lightweight: Only queries KV store, no Fluent API calls
556
- *
557
- * SECURITY: Authentication handled via connection parameter
558
- * No manual API key validation needed - Versori manages this via connection auth
559
- */
560
- export const locationSyncJobStatus = webhook('location-sync-job-status', {
561
- response: { mode: 'sync' },
562
- connection: 'location-sync-job-status',
563
- }).then(
564
- fn('status', async ctx => {
565
- const { data, log, openKv } = ctx;
566
- const jobId = data?.jobId as string;
567
-
568
- if (!jobId) {
569
- return { success: false, error: 'jobId required' };
570
- }
571
-
572
- const tracker = new JobTracker(openKv(':project:'), log);
573
- const status = await tracker.getJob(jobId);
574
-
575
- return status
576
- ? { success: true, jobId, ...status }
577
- : { success: false, error: 'Job not found', jobId };
578
- })
579
- );
580
- ```
581
-
582
- ---
583
-
584
- ### Step 4: Main Orchestration Service (`src/services/location-sync.service.ts`)
585
-
586
- **Note:** This service file should contain the `executeLocationSync` function (renamed from `runLocationXmlWorkflow`). The main workflow logic should be moved here.
587
-
588
- ```typescript
589
- /**
590
- * Main Orchestration Service: Location Sync
591
- *
592
- * This service contains the core business logic for location synchronization.
593
- *
594
- * Features:
595
- * - S3 file operations with archival-based deduplication (moveFile to processed/)
596
- * - XML parsing with @ prefix for attributes
597
- * - Single element → array normalization
598
- * - GraphQLMutationMapper for nested field transformations
599
- * - GraphQL mutations with alias batching support
600
- * - StateService for metadata tracking (secondary to archival)
601
- * - Error state tracking with exponential backoff
602
- *
603
- * Deduplication Strategy:
604
- * - PRIMARY: S3 archival via moveFile() - Files in processed/ won't be re-listed
605
- * - SECONDARY: StateService KV tracking - Provides metadata and processing history
606
- */
607
- import { Buffer } from 'node:buffer'; // Required for Deno/Versori runtime
608
- import {
609
- createClient,
610
- S3DataSource,
611
- XMLParserService,
612
- GraphQLMutationMapper,
613
- StateService,
614
- VersoriKVAdapter,
615
- JobTracker,
616
- FluentClient,
617
- } from '@fluentcommerce/fc-connect-sdk';
618
-
619
- // ============================================================================
620
- // Type Definitions
621
- // ============================================================================
622
-
623
- interface FileProcessingResult {
624
- success: boolean;
625
- locations: any[];
626
- errors: string[];
627
- }
628
-
629
- interface MutationResult {
630
- successful: number;
631
- failed: number;
632
- errors: string[];
633
- }
634
-
635
- interface MutationLogEntry {
636
- timestamp: string;
637
- fileName: string;
638
- locationRef: string;
639
- status: 'success' | 'failure';
640
- error?: string;
641
- }
642
-
643
- // ============================================================================
644
- // Utility Functions
645
- // ============================================================================
646
-
647
- /**
648
- * Retry utility with exponential backoff
649
- * (User-defined - not part of SDK public API)
650
- */
651
- async function retryWithBackoff<T>(
652
- operation: () => Promise<T>,
653
- maxRetries = 3,
654
- baseDelayMs = 1000
655
- ): Promise<T> {
656
- let lastError: any;
657
- for (let attempt = 0; attempt < maxRetries; attempt++) {
658
- try {
659
- return await operation();
660
- } catch (error) {
661
- lastError = error;
662
- if (attempt < maxRetries - 1) {
663
- const delay = baseDelayMs * Math.pow(2, attempt) + Math.floor(Math.random() * 200);
664
- await new Promise(resolve => setTimeout(resolve, delay));
665
- }
666
- }
667
- }
668
- throw lastError;
669
- }
670
-
671
- /**
672
- * Rate limiter for GraphQL mutations
673
- */
674
- async function rateLimitedMutation(operation: () => Promise<any>, delayMs: number): Promise<any> {
675
- const result = await operation();
676
- await new Promise(resolve => setTimeout(resolve, delayMs));
677
- return result;
678
- }
679
-
680
- // ============================================================================
681
- // Service Functions
682
- // ============================================================================
683
-
684
- /**
685
- * Service Function 1: Process File
686
- *
687
- * Downloads XML from S3, parses with XMLParserService, normalizes arrays,
688
- * and maps fields with GraphQLMutationMapper.
689
- *
690
- * @param s3 - S3DataSource instance
691
- * @param parser - XMLParserService instance
692
- * @param mapper - GraphQLMutationMapper instance
693
- * @param filePath - Full S3 path to file
694
- * @param fileName - File name only (for logging)
695
- * @param log - Logger instance
696
- * @returns FileProcessingResult with locations array and errors
697
- */
698
- async function processFile(
699
- s3: S3DataSource,
700
- parser: XMLParserService,
701
- mapper: GraphQLMutationMapper,
702
- filePath: string,
703
- fileName: string,
704
- log: any
705
- ): Promise<FileProcessingResult> {
706
- try {
707
- log.info('Processing file', { fileName });
708
-
709
- // Download with retry
710
- const content = await retryWithBackoff(
711
- () => s3.downloadFile(filePath, { encoding: 'utf8' }) as Promise<string>
712
- );
713
-
714
- // Parse XML
715
- const xmlData = await parser.parse(content);
716
-
717
- // Extract location array (handle both single and multiple locations)
718
- // CRITICAL: XML array normalization - single <location> becomes object, not array
719
- const locationsData = xmlData.locations?.location;
720
- const locationsArray = Array.isArray(locationsData) ? locationsData : [locationsData];
721
-
722
- if (!locationsArray || locationsArray.length === 0) {
723
- log.warn('Empty file (no locations)', { fileName });
724
- return {
725
- success: true,
726
- locations: [],
727
- errors: [],
728
- };
729
- }
730
-
731
- // Map each location using GraphQLMutationMapper
732
- const mappedLocations: Array<{ query: string; variables: any; input: any }> = [];
733
- const mappingErrors: string[] = [];
734
-
735
- // ✅ PRODUCTION ENHANCEMENT: Log transformation start
736
- log.info('Transforming locations to GraphQL mutations', {
737
- fileName,
738
- totalLocations: locationsArray.length,
739
- });
740
-
741
- for (let i = 0; i < locationsArray.length; i++) {
742
- const locationNumber = i + 1;
743
-
744
- // ✅ PRODUCTION ENHANCEMENT: Log progress every 50 locations
745
- if (locationNumber % 50 === 0) {
746
- log.info(`📤 Transforming location ${locationNumber}/${locationsArray.length}`, {
747
- fileName,
748
- locationNumber,
749
- totalLocations: locationsArray.length,
750
- validSoFar: mappedLocations.length,
751
- errorsSoFar: mappingErrors.length,
752
- progress: `${((locationNumber / locationsArray.length) * 100).toFixed(1)}%`,
753
- });
754
- }
755
-
756
- // Wrap location in context for mapping
757
- const record = { location: locationsArray[i] };
758
- try {
759
- // GraphQLMutationMapper returns { query, variables } directly
760
- const mappingResult = await mapper.map(record);
761
-
762
- mappedLocations.push({
763
- query: mappingResult.query,
764
- variables: mappingResult.variables,
765
- input: mappingResult.variables.input || mappingResult.variables,
766
- });
767
- } catch (error: unknown) {
768
- const errorMsg = error instanceof Error ? error.message : String(error);
769
- mappingErrors.push(`Location ${locationNumber}: ${errorMsg}`);
770
- log.warn('Mapping failed for location', { fileName, index: i, error: errorMsg });
771
- }
772
- }
773
-
774
- log.info('File processed', {
775
- fileName,
776
- total: locationsArray.length,
777
- mapped: mappedLocations.length,
778
- errors: mappingErrors.length,
779
- successRate: `${((mappedLocations.length / locationsArray.length) * 100).toFixed(1)}%`,
780
- });
781
-
782
- return {
783
- success: true,
784
- locations: mappedLocations,
785
- errors: mappingErrors,
786
- };
787
- } catch (error: any) {
788
- // ✅ Enhanced error logging: Extract all error details for visibility
789
- const errorDetails = {
790
- message: error?.message || 'Unknown error',
791
- stack: error?.stack,
792
- fileName: error?.fileName,
793
- lineNumber: error?.lineNumber,
794
- originalError: error?.context?.originalError?.message,
795
- errorType: error?.name || 'Error',
796
- };
797
- log.error('File processing failed', errorDetails, { fileName });
798
- return {
799
- success: false,
800
- locations: [],
801
- errors: [error.message || 'Unknown error'],
802
- };
803
- }
804
- }
805
-
806
- /**
807
- * Service Function 2: Execute Mutations
808
- *
809
- * Executes GraphQL createLocation mutations with alias batching support.
810
- *
811
- * @param client - FluentClient instance
812
- * @param mapper - GraphQLMutationMapper instance
813
- * @param locations - Array of mapped location objects with query and variables
814
- * @param log - Logger instance
815
- * @param retailerId - Fluent retailer ID
816
- * @param batchSize - Number of concurrent requests (default: 1)
817
- * @param mutationsPerAliasBatch - Optional: Number of mutations per aliased request (default: undefined)
818
- * @returns MutationResult with success/failure counts
819
- */
820
- async function executeMutations(
821
- client: FluentClient,
822
- mapper: GraphQLMutationMapper,
823
- locations: Array<{ query: string; variables: any; input: any }>,
824
- log: any,
825
- retailerId: string,
826
- batchSize: number = 1, // ✅ Default: 1 (sequential)
827
- mutationsPerAliasBatch?: number // ✅ NEW: Alias batching parameter (default: undefined = disabled)
828
- ): Promise<MutationResult> {
829
- // Determine mode: use aliases if mutationsPerAliasBatch is set and > 1
830
- const useAliases = mutationsPerAliasBatch && mutationsPerAliasBatch > 1;
831
-
832
- if (useAliases) {
833
- return await executeMutationsWithAliases(
834
- client,
835
- mapper,
836
- locations,
837
- log,
838
- retailerId,
839
- batchSize,
840
- mutationsPerAliasBatch!
841
- );
842
- } else {
843
- return await executeMutationsSeparate(
844
- client,
845
- locations,
846
- log,
847
- batchSize
848
- );
849
- }
850
- }
851
-
852
- /**
853
- * Execute mutations using separate concurrent requests (current mode)
854
- */
855
- async function executeMutationsSeparate(
856
- client: FluentClient,
857
- locations: Array<{ query: string; variables: any; input: any }>,
858
- log: any,
859
- batchSize: number
860
- ): Promise<MutationResult> {
861
- const result: MutationResult = {
862
- successful: 0,
863
- failed: 0,
864
- errors: [],
865
- };
866
-
867
- const safeConc = Math.max(1, Math.floor(batchSize));
868
-
869
- // Sequential mode
870
- if (safeConc === 1) {
871
- for (const location of locations) {
872
- try {
873
- await retryWithBackoff(() =>
874
- client.graphql({
875
- query: location.query,
876
- variables: location.variables,
877
- })
878
- );
879
- result.successful++;
880
- } catch (error: unknown) {
881
- result.failed++;
882
- const errorMsg = error instanceof Error ? error.message : String(error);
883
- result.errors.push(errorMsg);
884
- }
885
- }
886
- return result;
887
- }
888
-
889
- // Parallel mode
890
- for (let i = 0; i < locations.length; i += safeConc) {
891
- const chunk = locations.slice(i, i + safeConc);
892
- const results = await Promise.allSettled(
893
- chunk.map(loc =>
894
- retryWithBackoff(() =>
895
- client.graphql({
896
- query: loc.query,
897
- variables: loc.variables,
898
- })
899
- )
900
- )
901
- );
902
-
903
- results.forEach((settledResult, idx) => {
904
- if (settledResult.status === 'fulfilled') {
905
- result.successful++;
906
- } else {
907
- result.failed++;
908
- result.errors.push(
909
- settledResult.reason instanceof Error ? settledResult.reason.message : String(settledResult.reason)
910
- );
911
- }
912
- });
913
- }
914
-
915
- return result;
916
- }
917
-
918
- /**
919
- * ✅ NEW: Execute mutations using GraphQL aliases (batched requests)
920
- */
921
- async function executeMutationsWithAliases(
922
- client: FluentClient,
923
- mapper: GraphQLMutationMapper,
924
- locations: Array<{ query: string; variables: any; input: any }>,
925
- log: any,
926
- retailerId: string,
927
- maxParallel: number,
928
- mutationsPerAliasBatch: number
929
- ): Promise<MutationResult> {
930
- const result: MutationResult = { successful: 0, failed: 0, errors: [] };
931
-
932
- const mutationName = (mapper as any).config.mutation || 'createLocation';
933
- const aliasBatches: Array<Array<typeof locations[0]>> = [];
934
-
935
- for (let i = 0; i < locations.length; i += mutationsPerAliasBatch) {
936
- aliasBatches.push(locations.slice(i, i + mutationsPerAliasBatch));
937
- }
938
-
939
- // Process batches with concurrency control
940
- for (let i = 0; i < aliasBatches.length; i += maxParallel) {
941
- const concurrentBatches = aliasBatches.slice(i, i + maxParallel);
942
-
943
- const batchResults = await Promise.allSettled(
944
- concurrentBatches.map(async (batch) => {
945
- const { query, variables } = buildAliasedBatch(batch, mutationName, retailerId);
946
- const response = await retryWithBackoff(() => client.graphql({ query, variables }));
947
- return parseAliasResponse(response, batch, mutationName);
948
- })
949
- );
950
-
951
- batchResults.forEach((batchResult, idx) => {
952
- if (batchResult.status === 'fulfilled') {
953
- const batchRes = batchResult.value;
954
- result.successful += batchRes.executed;
955
- result.failed += batchRes.failed;
956
- result.errors.push(...batchRes.errors);
957
- } else {
958
- const batch = concurrentBatches[idx];
959
- const errorMsg = batchResult.reason instanceof Error ? batchResult.reason.message : String(batchResult.reason);
960
- batch.forEach(loc => {
961
- result.failed++;
962
- result.errors.push(`Batch execution failed: ${errorMsg}`);
963
- });
964
- }
965
- });
966
-
967
- if (i + maxParallel < aliasBatches.length) {
968
- await new Promise(resolve => setTimeout(resolve, 500));
969
- }
970
- }
971
-
972
- return result;
973
- }
974
-
975
- /**
976
- * ✅ NEW: Build aliased batch query and variables
977
- */
978
- function buildAliasedBatch(
979
- batch: Array<{ query: string; variables: any; input: any }>,
980
- mutationName: string,
981
- retailerId: string
982
- ): { query: string; variables: Record<string, any> } {
983
- const batchSize = batch.length;
984
- const inputTypeName = `${mutationName.charAt(0).toUpperCase() + mutationName.slice(1)}Input`;
985
-
986
- const variables = Array.from({ length: batchSize }, (_, i) =>
987
- `$input${i + 1}: ${inputTypeName}!`
988
- ).join(', ');
989
-
990
- const aliasedMutations = Array.from({ length: batchSize }, (_, i) => {
991
- const alias = `${mutationName}${i + 1}`;
992
- return ` ${alias}: ${mutationName}(input: $input${i + 1}) { id ref name }`;
993
- }).join('\n');
994
-
995
- const operationName = `Batch${mutationName.charAt(0).toUpperCase() + mutationName.slice(1)}s`;
996
- const query = `mutation ${operationName}(${variables}) {\n${aliasedMutations}\n}`;
997
-
998
- const variablesObj: Record<string, any> = {};
999
- batch.forEach((loc, index) => {
1000
- const input = loc.variables.input || loc.variables;
1001
- if (input && !input.retailer) {
1002
- input.retailer = { id: parseInt(retailerId) };
1003
- }
1004
- variablesObj[`input${index + 1}`] = input;
1005
- });
1006
-
1007
- return { query, variables: variablesObj };
1008
- }
1009
-
1010
- /**
1011
- * ✅ NEW: Parse aliased GraphQL response
1012
- */
1013
- function parseAliasResponse(
1014
- response: any,
1015
- batch: Array<{ query: string; variables: any; input: any }>,
1016
- mutationName: string
1017
- ): { executed: number; failed: number; errors: string[] } {
1018
- const result = { executed: 0, failed: 0, errors: [] as string[] };
1019
-
1020
- const data = response.data || {};
1021
- const errors = response.errors || [];
1022
-
1023
- batch.forEach((loc, index) => {
1024
- const alias = `${mutationName}${index + 1}`;
1025
- const aliasData = data[alias];
1026
- const aliasErrors = errors.filter((e: any) =>
1027
- e.path && Array.isArray(e.path) && e.path.includes(alias)
1028
- );
1029
-
1030
- if (aliasData && !aliasErrors.length) {
1031
- result.executed++;
1032
- } else {
1033
- result.failed++;
1034
- const errorMsg = aliasErrors[0]?.message || 'Mutation failed';
1035
- result.errors.push(`${loc.input?.ref || 'unknown'}: ${errorMsg}`);
1036
- }
1037
- });
1038
-
1039
- return result;
1040
- }
1041
-
1042
- /**
1043
- * Service Function 3: Write Mutation Log
1044
- *
1045
- * Writes mutation results to S3 as a JSON log file.
1046
- *
1047
- * @param s3 - S3DataSource instance
1048
- * @param logEntries - Array of mutation log entries
1049
- * @param fileName - Original file name (used to generate log path)
1050
- * @param logPrefix - S3 prefix for logs (e.g., 'logs/')
1051
- * @param log - Logger instance
1052
- */
1053
- async function writeMutationLog(
1054
- s3: S3DataSource,
1055
- logEntries: MutationLogEntry[],
1056
- fileName: string,
1057
- logPrefix: string,
1058
- log: any
1059
- ): Promise<void> {
1060
- try {
1061
- const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
1062
- const logFileName = `${logPrefix}${fileName.replace('.xml', '')}_${timestamp}.json`;
1063
-
1064
- const logContent = JSON.stringify(
1065
- {
1066
- fileName,
1067
- timestamp: new Date().toISOString(),
1068
- totalMutations: logEntries.length,
1069
- successful: logEntries.filter(e => e.status === 'success').length,
1070
- failed: logEntries.filter(e => e.status === 'failure').length,
1071
- entries: logEntries,
1072
- },
1073
- null,
1074
- 2
1075
- );
1076
-
1077
- // Write to S3 (uploadFile accepts string or Buffer)
1078
- await s3.uploadFile(logFileName, logContent);
1079
-
1080
- log.info('Mutation log written', { logFileName, entries: logEntries.length });
1081
- } catch (error: any) {
1082
- // ✅ Enhanced error logging: Extract all error details for visibility
1083
- const errorDetails = {
1084
- message: error?.message || 'Unknown error',
1085
- stack: error?.stack,
1086
- errorType: error?.name || 'Error',
1087
- };
1088
- log.error('Failed to write mutation log', errorDetails, { fileName });
1089
- // Don't throw - logging failure shouldn't stop workflow
1090
- }
1091
- }
1092
-
1093
- // ============================================================================
1094
- // Main Workflow Function
1095
- // ============================================================================
1096
-
1097
- /**
1098
- * Main Orchestration Function: Execute Location Sync
1099
- *
1100
- * This function orchestrates the location synchronization workflow.
1101
- *
1102
- * Architecture:
1103
- * 1. List files from S3
1104
- * 2. For each file:
1105
- * a. processFile() - Download, parse, map
1106
- * b. executeMutations() - Send GraphQL mutations with rate limiting
1107
- * c. writeMutationLog() - Log results to S3
1108
- * d. Archive file (primary deduplication)
1109
- * e. Mark processed in KV (metadata tracking)
1110
- *
1111
- * @param ctx - Versori context
1112
- * @param jobId - Job identifier
1113
- * @param tracker - JobTracker instance
1114
- * @returns Processing result
1115
- */
1116
- export async function executeLocationSync(ctx: any, jobId: string, tracker: JobTracker) {
1117
- const { log, activation } = ctx;
1118
-
1119
- log.info('📋 [INIT] Reading activation variables', { jobId });
1120
-
1121
- // Read activation variables
1122
- const s3Bucket = activation?.getVariable('s3BucketName');
1123
- const s3Region = activation?.getVariable('awsRegion') || 'us-east-1';
1124
- const s3AccessKeyId = activation?.getVariable('awsAccessKeyId');
1125
- const s3SecretAccessKey = activation?.getVariable('awsSecretAccessKey');
1126
- const s3Prefix = activation?.getVariable('s3Prefix') || 'locations/';
1127
- const archivePrefix = activation?.getVariable('archivePrefix') || 'processed/';
1128
- const errorPrefix = activation?.getVariable('errorPrefix') || 'errors/';
1129
- const logPrefix = activation?.getVariable('logPrefix') || 'logs/';
1130
- const filePattern = (activation?.getVariable('filePattern') || '.xml').toLowerCase();
1131
- const maxFiles = parseInt(activation?.getVariable('maxFilesToProcess') || '10', 10);
1132
- const retailerId = activation?.getVariable('retailerId'); // Optional: Only if mutation schema requires it
1133
- const enableArchival = activation?.getVariable('enableArchival') !== 'false';
1134
- const enableMutationLogs = activation?.getVariable('enableMutationLogs') !== 'false';
1135
- const enableFileTracking = activation?.getVariable('enableFileTracking') !== 'false';
1136
-
1137
- // ✅ Configuration with defaults
1138
- const mutationBatchSize = parseInt(
1139
- activation?.getVariable('mutationBatchSize') || '1', // ✅ Default: 1 (sequential)
1140
- 10
1141
- );
1142
-
1143
- const mutationsPerAliasBatch = activation?.getVariable('mutationsPerAliasBatch')
1144
- ? parseInt(activation?.getVariable('mutationsPerAliasBatch') || '1', 10)
1145
- : undefined; // ✅ Default: undefined (disabled, use separate requests)
1146
-
1147
- // Validate required variables
1148
- const missingVars: string[] = [];
1149
- if (!s3Bucket) missingVars.push('s3BucketName');
1150
- if (!s3AccessKeyId) missingVars.push('awsAccessKeyId');
1151
- if (!s3SecretAccessKey) missingVars.push('awsSecretAccessKey');
1152
- // Note: retailerId is optional - only needed if mutation schema requires it
1153
-
1154
- if (missingVars.length > 0) {
1155
- const errorMsg = `Missing required variables: ${missingVars.join(', ')}`;
1156
- log.error('❌ [VALIDATION] Missing required activation variables', {
1157
- missingVars,
1158
- recommendation: 'Add missing variables in Versori activation settings'
1159
- });
1160
- return { success: false, error: errorMsg, processed: 0 };
1161
- }
1162
-
1163
- log.info('✅ [VALIDATION] All required variables present', {
1164
- s3Bucket,
1165
- s3Region,
1166
- s3Prefix,
1167
- enableFileTracking,
1168
- mutationBatchSize
1169
- });
1170
-
1171
- try {
1172
- log.info('🔧 [INIT] Initializing Fluent Commerce client', { jobId });
1173
-
1174
- // Initialize services with validateConnection
1175
- const client = await createClient(ctx, { validateConnection: true });
1176
- if (!client) {
1177
- throw new Error('Failed to create Fluent Commerce client');
1178
- }
1179
-
1180
- log.info('✅ [INIT] Fluent Commerce client validated and ready', { jobId });
1181
-
1182
- // ✅ CORRECT: GraphQL mutations do NOT need client.setRetailerId()
1183
- // setRetailerId() is only for Job/Event API, NOT GraphQL
1184
- // Check your GraphQL schema to determine retailerId handling:
1185
- // - Mandatory retailerId → Must pass it in mutation input
1186
- // - Optional retailerId → Can pass it if needed
1187
- // - No retailerId field → Don't pass it
1188
- // See: fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md
1189
-
1190
- log.info('🗄️ [INIT] Initializing S3 data source', { s3Bucket, s3Region, s3Prefix });
1191
-
1192
- const s3 = new S3DataSource(
1193
- {
1194
- type: 'S3_XML',
1195
- connectionId: 's3-location-sync',
1196
- name: 'Source S3',
1197
- s3Config: {
1198
- bucket: s3Bucket,
1199
- region: s3Region,
1200
- accessKeyId: s3AccessKeyId,
1201
- secretAccessKey: s3SecretAccessKey,
1202
- },
1203
- },
1204
- log
1205
- );
1206
-
1207
- const parser = new XMLParserService();
1208
-
1209
- // Initialize state tracking (only if enabled)
1210
- let stateService: StateService | null = null;
1211
- if (enableFileTracking) {
1212
- log.info('🔄 [INIT] Enabling file tracking with StateService', { jobId });
1213
- const stateKV = new VersoriKVAdapter(ctx);
1214
- stateService = new StateService(stateKV);
1215
- } else {
1216
- log.info('⏭️ [INIT] File tracking disabled - relying on S3 archival only', { jobId });
1217
- }
1218
-
1219
- log.info('📝 [INIT] Loading mapping configuration', { jobId });
1220
-
1221
- // ✅ CRITICAL: Load mapping config from external JSON file
1222
- // Mapping config uses GraphQLMutationMapper structure (nested objects, not dot notation)
1223
- // File: src/config/location-mapping.json
1224
- const mappingConfigJson = await import('../config/location-mapping.json', { assert: { type: 'json' } });
1225
- const mappingConfig = mappingConfigJson.default;
1226
-
1227
- // Initialize GraphQLMutationMapper with client for schema introspection
1228
- const mapper = new GraphQLMutationMapper(mappingConfig, log, { fluentClient: client });
1229
-
1230
- log.info('📂 [S3] Listing files from S3', { s3Bucket, s3Prefix, filePattern });
1231
-
1232
- // List files (pattern filtering handled by listFiles)
1233
- const files = await s3.listFiles({
1234
- prefix: s3Prefix,
1235
- pattern: filePattern,
1236
- maxKeys: 1000
1237
- });
1238
-
1239
- // Newest-first ordering
1240
- const xmlFiles = files
1241
- .sort((a: any, b: any) => {
1242
- const aTime = a.lastModified ? new Date(a.lastModified).getTime() : 0;
1243
- const bTime = b.lastModified ? new Date(b.lastModified).getTime() : 0;
1244
- return bTime - aTime;
1245
- })
1246
- .slice(0, maxFiles);
1247
-
1248
- log.info('📊 [S3] File discovery complete', {
1249
- totalFiles: files.length,
1250
- xmlFiles: xmlFiles.length,
1251
- maxFiles,
1252
- selectedFiles: xmlFiles.map(f => f.name)
1253
- });
1254
-
1255
- const results = {
1256
- processed: 0,
1257
- skipped: 0,
1258
- failed: 0,
1259
- totalRecords: 0,
1260
- errors: [] as string[],
1261
- };
1262
-
1263
- log.info('🔄 [PROCESSING] Starting file processing loop', {
1264
- fileCount: xmlFiles.length,
1265
- jobId
1266
- });
1267
-
1268
- // Per-file processing loop
1269
- for (const file of xmlFiles) {
1270
- const filePath = file.path;
1271
- const fileName = file.name;
1272
-
1273
- log.info('📄 [FILE] Processing file', { fileName, filePath });
1274
-
1275
- // Duplicate prevention (secondary check - files in processed/ won't be listed)
1276
- // Primary deduplication: S3 archival (files moved to processed/ subdirectory)
1277
- if (enableFileTracking && stateService) {
1278
- const wasProcessed = await stateService.isFileProcessed(fileName);
1279
- if (wasProcessed) {
1280
- log.info('⏭️ [SKIP] File already processed (KV check)', { fileName });
1281
- results.skipped++;
1282
- continue;
1283
- }
1284
- }
1285
-
1286
- try {
1287
- // Step 1: Process file (download, parse, map)
1288
- log.info('📥 [DOWNLOAD] Downloading and parsing file', { fileName });
1289
- const processingResult = await processFile(s3, parser, mapper, filePath, fileName, log);
1290
-
1291
- if (!processingResult.success) {
1292
- throw new Error(`File processing failed: ${processingResult.errors.join(', ')}`);
1293
- }
1294
-
1295
- if (processingResult.locations.length === 0) {
1296
- log.warn('⚠️ [SKIP] Empty file detected, archiving', { fileName });
1297
- if (enableArchival) {
1298
- await s3.moveFile(filePath, `${archivePrefix}${fileName}`);
1299
- log.info('📦 [ARCHIVE] Empty file archived', { fileName, destination: `${archivePrefix}${fileName}` });
1300
- }
1301
- results.skipped++;
1302
- continue;
1303
- }
1304
-
1305
- log.info('✅ [PARSE] File parsed successfully', {
1306
- fileName,
1307
- locationCount: processingResult.locations.length,
1308
- mappingErrors: processingResult.errors.length
1309
- });
1310
-
1311
- // Step 2: Execute mutations
1312
- // Step 2: Execute mutations with alias batching support
1313
- // ? Enhanced: Extract context for progress logging
1314
- const sampleLocationRefs = processingResult.locations.slice(0, 5).map((loc: any) => loc.input?.ref || loc.ref || 'unknown');
1315
- const mutationType = mapper?.mutationName || 'createLocation';
1316
-
1317
- // ? Enhanced: Start logging with context
1318
- log.info(`[GraphQLMutations] Sending mutations for file "${fileName}"`, {
1319
- totalMutations: processingResult.locations.length,
1320
- mutationType,
1321
- batchSize: mutationBatchSize,
1322
- batchMode: mutationBatchSize === 1 ? 'sequential' : `parallel (${mutationBatchSize})`,
1323
- sampleLocationRefs: sampleLocationRefs.join(', '),
1324
- aliasBatching: mutationsPerAliasBatch ? `enabled (${mutationsPerAliasBatch} per alias)` : 'disabled'
1325
- });
1326
-
1327
- const mutationResult = await executeMutations(
1328
- client,
1329
- mapper,
1330
- processingResult.locations,
1331
- log,
1332
- retailerId, // Pass retailerId for mutations that require it in input
1333
- mutationBatchSize, // Concurrency control (default: 1)
1334
- mutationsPerAliasBatch // ✅ NEW: Alias batching (default: undefined)
1335
- );
1336
-
1337
- // ? Enhanced: Completion logging with summary
1338
- log.info(`[GraphQLMutations] Mutation submission completed for file "${fileName}"`, {
1339
- totalMutations: processingResult.locations.length,
1340
- successful: mutationResult.successful,
1341
- failed: mutationResult.failed,
1342
- successRate: processingResult.locations.length > 0 ? `${Math.round((mutationResult.successful / processingResult.locations.length) * 100)}%` : '0%',
1343
- mutationType
1344
- });
1345
-
1346
- // Step 3: Write mutation log (if enabled)
1347
- if (enableMutationLogs) {
1348
- const logEntries: MutationLogEntry[] = processingResult.locations.map(loc => {
1349
- const failed = mutationResult.errors.find(e => e.startsWith(loc.ref));
1350
- return {
1351
- timestamp: new Date().toISOString(),
1352
- fileName,
1353
- locationRef: loc.ref,
1354
- status: failed ? 'failure' : 'success',
1355
- error: failed,
1356
- };
1357
- });
1358
-
1359
- await writeMutationLog(s3, logEntries, fileName, logPrefix, log);
1360
- }
1361
-
1362
- // Step 4: Archive file (PRIMARY deduplication - file won't be listed again)
1363
- if (enableArchival) {
1364
- log.info('📦 [ARCHIVE] Moving file to processed directory', { fileName });
1365
- await s3.moveFile(filePath, `${archivePrefix}${fileName}`);
1366
- log.info('✅ [ARCHIVE] File archived successfully', {
1367
- fileName,
1368
- destination: `${archivePrefix}${fileName}`
1369
- });
1370
- }
1371
-
1372
- // Step 5: Mark processed in KV (SECONDARY - provides metadata/history)
1373
- if (enableFileTracking && stateService) {
1374
- log.info('💾 [STATE] Recording file processing metadata', { fileName });
1375
- await stateService.markFileProcessed(fileName, {
1376
- recordCount: processingResult.locations.length,
1377
- successful: mutationResult.successful,
1378
- failed: mutationResult.failed,
1379
- mappingErrors: processingResult.errors.length,
1380
- timestamp: new Date().toISOString(),
1381
- });
1382
- }
1383
-
1384
- results.processed++;
1385
- results.totalRecords += mutationResult.successful;
1386
-
1387
- log.info('✅ [COMPLETE] File processing complete', {
1388
- fileName,
1389
- locations: processingResult.locations.length,
1390
- successful: mutationResult.successful,
1391
- failed: mutationResult.failed,
1392
- mappingErrors: processingResult.errors.length,
1393
- });
1394
-
1395
- if (processingResult.errors.length > 0 || mutationResult.errors.length > 0) {
1396
- results.errors.push(
1397
- `${fileName}: ${processingResult.errors.length} mapping errors, ${mutationResult.errors.length} mutation errors`
1398
- );
1399
- }
1400
- } catch (error: any) {
1401
- // ✅ Enhanced error logging: Extract all error details for visibility
1402
- const errorDetails = {
1403
- message: error?.message || 'Unknown error',
1404
- stack: error?.stack,
1405
- fileName: error?.fileName,
1406
- lineNumber: error?.lineNumber,
1407
- originalError: error?.context?.originalError?.message,
1408
- errorType: error?.name || 'Error',
1409
- };
1410
-
1411
- log.error('❌ [ERROR] File processing failed', errorDetails, { fileName });
1412
-
1413
- // Provide error recommendations based on error type
1414
- const recommendation = getErrorRecommendation(error);
1415
- if (recommendation) {
1416
- log.warn('💡 [RECOMMENDATION]', { fileName, recommendation });
1417
- }
1418
-
1419
- results.failed++;
1420
- results.errors.push(`${fileName}: ${error.message}`);
1421
-
1422
- // Attempt to move to error directory, ignore failures
1423
- try {
1424
- log.info('🗂️ [ERROR] Moving failed file to error directory', { fileName });
1425
- await s3.moveFile(filePath, `${errorPrefix}${fileName}`);
1426
- log.info('✅ [ERROR] Failed file moved to error directory', {
1427
- fileName,
1428
- destination: `${errorPrefix}${fileName}`
1429
- });
1430
- } catch (moveError) {
1431
- log.warn('⚠️ [ERROR] Failed to move file to error directory', {
1432
- fileName,
1433
- moveError: moveError instanceof Error ? moveError.message : String(moveError)
1434
- });
1435
- }
1436
-
1437
- // Track error state with exponential backoff (only if file tracking enabled)
1438
- if (enableFileTracking && stateService) {
1439
- try {
1440
- const stateKV = new VersoriKVAdapter(ctx);
1441
- const key = ['error-state', fileName];
1442
- const prev = (await stateKV.get(key))?.value as any;
1443
- const attempts = (prev?.attemptCount || 0) + 1;
1444
- const backoffMinutes = Math.min(Math.pow(2, attempts) * 5, 24 * 60);
1445
- const nextRetryAt = new Date(Date.now() + backoffMinutes * 60000).toISOString();
1446
-
1447
- await stateKV.set(key, {
1448
- fileName,
1449
- attemptCount: attempts,
1450
- lastError: error?.message || 'unknown',
1451
- lastAttemptAt: new Date().toISOString(),
1452
- firstFailedAt: prev?.firstFailedAt || new Date().toISOString(),
1453
- nextRetryAt,
1454
- });
1455
-
1456
- log.info('💾 [ERROR] Error state tracked with exponential backoff', {
1457
- fileName,
1458
- attempts,
1459
- nextRetryAt
1460
- });
1461
- } catch (stateError) {
1462
- log.warn('⚠️ [ERROR] Failed to track error state', {
1463
- fileName,
1464
- stateError: stateError instanceof Error ? stateError.message : String(stateError)
1465
- });
1466
- }
1467
- }
1468
- }
1469
- }
1470
-
1471
- log.info('🏁 [COMPLETE] File processing loop finished', {
1472
- processed: results.processed,
1473
- skipped: results.skipped,
1474
- failed: results.failed,
1475
- totalRecords: results.totalRecords
1476
- });
1477
-
1478
- return results;
1479
- } catch (error: any) {
1480
- // ✅ Enhanced error logging: Extract all error details for visibility
1481
- const errorDetails = {
1482
- message: error?.message || 'Unknown error',
1483
- stack: error?.stack,
1484
- errorType: error?.name || 'Error',
1485
- };
1486
-
1487
- log.error('❌ [FATAL] Location sync failed', errorDetails);
1488
-
1489
- // Provide fatal error recommendations
1490
- const recommendation = getErrorRecommendation(error);
1491
- if (recommendation) {
1492
- log.warn('💡 [RECOMMENDATION]', { recommendation });
1493
- }
1494
-
1495
- return {
1496
- success: false,
1497
- error: error.message,
1498
- processed: 0,
1499
- timestamp: new Date().toISOString(),
1500
- };
1501
- }
1502
- }
1503
-
1504
- /**
1505
- * Get error recommendation based on error type
1506
- */
1507
- function getErrorRecommendation(error: any): string | null {
1508
- const message = error?.message?.toLowerCase() || '';
1509
-
1510
- if (message.includes('s3') || message.includes('access denied')) {
1511
- return 'Check S3 credentials and bucket permissions. Verify IAM policy includes s3:ListBucket, s3:GetObject, s3:PutObject, s3:DeleteObject.';
1512
- }
1513
-
1514
- if (message.includes('xml') || message.includes('parse')) {
1515
- return 'Verify XML file structure matches expected schema. Check for malformed XML or encoding issues.';
1516
- }
1517
-
1518
- if (message.includes('mapping') || message.includes('field')) {
1519
- return 'Review field mapping configuration. Ensure all required fields are present and source paths are correct.';
1520
- }
1521
-
1522
- if (message.includes('graphql') || message.includes('mutation')) {
1523
- return 'Check GraphQL schema and mutation input. Verify all required fields are provided and types match schema.';
1524
- }
1525
-
1526
- if (message.includes('auth') || message.includes('401') || message.includes('403')) {
1527
- return 'Verify Fluent Commerce credentials. Check OAuth2 client ID/secret and ensure connection is active.';
1528
- }
1529
-
1530
- if (message.includes('timeout') || message.includes('econnrefused')) {
1531
- return 'Check network connectivity. Verify API endpoints are accessible and not rate-limited.';
1532
- }
1533
-
1534
- return null;
1535
- }
1536
- ```
1537
-
1538
- **Note:** The `executeLocationSync` function should contain the full implementation of `runLocationXmlWorkflow` (renamed to `executeLocationSync`), including all the logic for processing files, executing mutations, and logging. The implementation details are shown in the service function code above.
1539
-
1540
- ---
1541
-
1542
- ### Step 5: TypeScript Configuration
1543
-
1544
- **File: tsconfig.json**
1545
-
1546
- ```json
1547
- {
1548
- "compilerOptions": {
1549
- "module": "ES2022",
1550
- "target": "ES2024",
1551
- "moduleResolution": "node"
1552
- }
1553
- }
1554
- ```
1555
-
1556
- ---
1557
-
1558
- ## Code Flow Explanation
1559
-
1560
- ### Initialization Phase
1561
-
1562
- 1. **Read activation variables** - S3 config, Fluent config, rate limiting, logging options
1563
- 2. **Validate required variables** - Fail fast if missing credentials
1564
- 3. **Initialize SDK services** - FluentClient, S3DataSource, XMLParserService, StateService
1565
- 4. **Create mapping configuration** - Define XML → GraphQL field mappings with nested objects
1566
- 5. **Calculate rate limit delay** - Convert mutations/second to delay in milliseconds
1567
-
1568
- ### File Discovery Phase
1569
-
1570
- 1. **List S3 files** - Use `s3.listFiles()` with prefix filter (excludes processed/ subdirectory)
1571
- 2. **Filter by pattern** - Match file extension (e.g., `.xml`)
1572
- 3. **Sort newest-first** - Process most recent files first
1573
- 4. **Apply max files limit** - Prevent overwhelming the workflow
1574
- 5. **Note**: Files in `processed/` subdirectory won't be listed (primary deduplication)
1575
-
1576
- ### Per-File Processing (Service Functions)
1577
-
1578
- **Step 1: processFile()** - Download, parse, map
1579
-
1580
- 1. **Download file** - Use S3DataSource with retry logic
1581
- 2. **Parse XML** - XMLParserService converts to JavaScript object
1582
- 3. **Normalize array** - Handle single vs multiple `<location>` elements (CRITICAL for XML)
1583
- 4. **Map locations** - Use UniversalMapper with nested field mapping
1584
- 5. **Collect errors** - Track mapping errors without stopping
1585
- 6. **Return result** - FileProcessingResult with locations array and errors
1586
-
1587
- **Step 2: executeMutations()** - GraphQL mutations with rate limiting
1588
-
1589
- 1. **Loop through locations** - Process each location individually
1590
- 2. **Build mutation input** - Extract nested fields (primaryAddress, openingSchedule)
1591
- 3. **Execute mutation** - Direct GraphQL `createLocation` with retry logic
1592
- 4. **Apply rate limiting** - Add configurable delay between mutations
1593
- 5. **Track results** - Count successful vs failed, collect error messages
1594
- 6. **Return result** - MutationResult with counts and errors
1595
-
1596
- **Step 3: writeMutationLog()** - S3 log file (optional)
1597
-
1598
- 1. **Create log entries** - Map locations to log entries with status
1599
- 2. **Build JSON log** - Include timestamp, summary, detailed entries
1600
- 3. **Write to S3** - Use Buffer.from() for Deno/Versori runtime compatibility
1601
- 4. **Non-blocking** - Logging failure doesn't stop workflow
1602
- 5. **Timestamped naming** - Unique log file per processed file
1603
-
1604
- ### Cleanup Phase
1605
-
1606
- 1. **Archive file FIRST** - Move to `processed/` or `errors/` (PRIMARY deduplication)
1607
- 2. **Mark processed in KV** - StateService tracks metadata and processing history
1608
- 3. **Error state tracking** - Store error info with exponential backoff timestamp
1609
- 4. **Return results** - Summary of processed, skipped, failed files
1610
-
1611
- **Note**: The order matters! Archive first (primary deduplication), then KV tracking (metadata/history).
1612
-
1613
- ### Service Function Benefits
1614
-
1615
- - **Composability**: Each function has single responsibility and can be reused
1616
- - **Testability**: Service functions can be unit tested independently
1617
- - **Clarity**: Main workflow shows high-level orchestration
1618
- - **Error Handling**: Isolated error handling per service function
1619
- - **Logging**: Detailed logging at each step with structured context
1620
-
1621
- ---
1622
-
1623
- ## S3 Archival Deduplication Pattern
1624
-
1625
- This template uses **S3 archival** as the primary deduplication mechanism, NOT VersoriFileTracker.
1626
-
1627
- ### How It Works
1628
-
1629
- **S3 Directory Structure:**
1630
-
1631
- ```
1632
- s3://my-bucket/
1633
- ├── locations/ ← listFiles() reads from here
1634
- │ ├── new-file-1.xml
1635
- │ └── new-file-2.xml
1636
- ├── processed/ ← Successfully processed files
1637
- │ ├── old-file-1.xml
1638
- │ └── old-file-2.xml
1639
- └── errors/ ← Failed files
1640
- └── bad-file.xml
1641
- ```
1642
-
1643
- **Deduplication Flow:**
1644
-
1645
- 1. **List files**: `s3.listFiles({ prefix: 'locations/' })` - Only lists `locations/` subdirectory
1646
- 2. **Process file**: Download, parse, transform, send mutations
1647
- 3. **Archive**: `s3.moveFile(filePath, 'processed/new-file-1.xml')` - Moves file out of `locations/`
1648
- 4. **Next run**: File is now in `processed/`, won't be listed again
1649
-
1650
- **Why This Works:**
1651
-
1652
- - Files in `processed/` subdirectory are **never listed** when prefix is `locations/`
1653
- - No need to track file state in KV store for deduplication
1654
- - S3 is the single source of truth for file status
1655
- - Simple, reliable, scales to millions of files
1656
-
1657
- **StateService Role (Secondary):**
1658
-
1659
- - Provides metadata and processing history
1660
- - Backup check in case archival fails mid-process
1661
- - Useful for monitoring and debugging
1662
- - NOT the primary deduplication mechanism
1663
-
1664
- **When to Use VersoriFileTracker:**
1665
-
1666
- - **NEVER for S3 sources** - Use archival pattern instead
1667
- - **Only for SFTP sources** - Where archival might not be possible
1668
- - See SFTP templates for VersoriFileTracker usage
1669
-
1670
- ---
1671
-
1672
- ## XML Path Resolution Patterns
1673
-
1674
- ### Pattern 1: XML Attribute Access with @ Prefix
1675
-
1676
- ```typescript
1677
- const mappingConfig = {
1678
- fields: {
1679
- // XML attribute access
1680
- ref: { source: 'location.@ref' }, // <location ref="LOC-001">
1681
- type: { source: 'location.@type' }, // <location type="WAREHOUSE">
1682
- country: { source: 'location.address.@country' }, // <address country="USA">
1683
-
1684
- // XML element text content
1685
- name: { source: 'location.name' }, // <name>Downtown</name>
1686
- city: { source: 'location.address.city' }, // <address><city>NYC</city></address>
1687
- },
1688
- };
1689
- ```
1690
-
1691
- ### Pattern 2: Handling Single vs Multiple Elements
1692
-
1693
- ```typescript
1694
- // XML can have single or multiple <location> elements
1695
- const locationsData = xmlData.locations?.location;
1696
-
1697
- // Normalize to array
1698
- const locations = Array.isArray(locationsData) ? locationsData : [locationsData];
1699
-
1700
- // Process each location
1701
- for (const loc of locations) {
1702
- const record = { location: loc }; // Wrap for mapping
1703
- const result = await mapper.map(record);
1704
- }
1705
- ```
1706
-
1707
- ### Pattern 3: Nested Object Mapping
1708
-
1709
- ```typescript
1710
- const mappingConfig = {
1711
- fields: {
1712
- // Root fields
1713
- ref: { source: 'location.@ref', required: true },
1714
- name: { source: 'location.name', required: true },
1715
- type: { source: 'location.@type', required: true },
1716
-
1717
- // Nested primaryAddress object
1718
- 'primaryAddress.ref': { source: 'location.@ref' },
1719
- 'primaryAddress.street': { source: 'location.address.street1' },
1720
- 'primaryAddress.city': { source: 'location.address.city' },
1721
- 'primaryAddress.latitude': { source: 'location.coordinates.@lat', resolver: 'sdk.parseFloat' },
1722
-
1723
- // Nested openingSchedule object
1724
- 'openingSchedule.allHours': {
1725
- source: 'location.openingSchedule.allHours',
1726
- resolver: 'sdk.boolean',
1727
- },
1728
- 'openingSchedule.monStart': {
1729
- source: 'location.openingSchedule.monStart',
1730
- resolver: 'sdk.parseInt',
1731
- },
1732
-
1733
- // Note: retailer.id not shown - standard createLocation does not have this field
1734
- // If your schema requires it, add: 'retailer.id': { value: parseInt(retailerId) }
1735
- },
1736
- };
1737
- ```
1738
-
1739
- ---
1740
-
1741
- ## Sample XML Files
1742
-
1743
- ### Minimal Test File
1744
-
1745
- **File: test-location.xml**
1746
-
1747
- ```xml
1748
- <?xml version="1.0" encoding="UTF-8"?>
1749
- <locations>
1750
- <location ref="TEST-001" type="WAREHOUSE">
1751
- <name>Test Warehouse</name>
1752
- <address country="USA">
1753
- <street1>123 Test St</street1>
1754
- <city>TestCity</city>
1755
- <state>TC</state>
1756
- <postalCode>12345</postalCode>
1757
- </address>
1758
- <coordinates lat="40.7128" lon="-74.0060"/>
1759
- <timeZone>America/New_York</timeZone>
1760
- <openingSchedule>
1761
- <allHours>false</allHours>
1762
- <monStart>800</monStart>
1763
- <monEnd>1800</monEnd>
1764
- <tueStart>800</tueStart>
1765
- <tueEnd>1800</tueEnd>
1766
- <wedStart>800</wedStart>
1767
- <wedEnd>1800</wedEnd>
1768
- <thuStart>800</thuStart>
1769
- <thuEnd>1800</thuEnd>
1770
- <friStart>800</friStart>
1771
- <friEnd>1800</friEnd>
1772
- <satStart>0</satStart>
1773
- <satEnd>0</satEnd>
1774
- <sunStart>0</sunStart>
1775
- <sunEnd>0</sunEnd>
1776
- </openingSchedule>
1777
- </location>
1778
- </locations>
1779
- ```
1780
-
1781
- ---
1782
-
1783
- ## Service Functions Deep Dive
1784
-
1785
- This template demonstrates service function composition for maintainable workflows.
1786
-
1787
- ### Function 1: processFile()
1788
-
1789
- **Purpose**: Download, parse, normalize, and map XML data
1790
-
1791
- **Inputs**:
1792
- - `s3: S3DataSource` - For file download
1793
- - `parser: XMLParserService` - For XML parsing
1794
- - `mapper: UniversalMapper` - For field mapping
1795
- - `filePath: string` - S3 path to file
1796
- - `fileName: string` - File name for logging
1797
- - `log: any` - Logger instance
1798
-
1799
- **Outputs**: `FileProcessingResult`
1800
- ```typescript
1801
- {
1802
- success: boolean; // Overall success
1803
- locations: any[]; // Mapped location objects
1804
- errors: string[]; // Mapping error messages
1805
- }
1806
- ```
1807
-
1808
- **Key Operations**:
1809
- 1. Downloads file with retry logic
1810
- 2. Parses XML using XMLParserService
1811
- 3. **Normalizes arrays** - Handles single vs multiple `<location>` elements
1812
- 4. Maps each location using UniversalMapper
1813
- 5. Collects errors without stopping processing
1814
- 6. Returns all mapped locations + errors
1815
-
1816
- **Why separate function?**
1817
- - Testable independently with mock data
1818
- - Reusable across workflows (scheduled, webhook, adhoc)
1819
- - Clear error boundaries - file-level errors vs record-level errors
1820
- - Easy to add XML validation or schema checks
1821
-
1822
- ### Function 2: executeMutations()
1823
-
1824
- **Purpose**: Execute GraphQL createLocation mutations with rate limiting
1825
-
1826
- **Inputs**:
1827
- - `client: FluentClient` - For GraphQL mutations
1828
- - `locations: any[]` - Mapped location data
1829
- - `mutationDelayMs: number` - Rate limit delay
1830
- - `fileName: string` - For logging context
1831
- - `log: any` - Logger instance
1832
-
1833
- **Outputs**: `MutationResult`
1834
- ```typescript
1835
- {
1836
- successful: number; // Count of successful mutations
1837
- failed: number; // Count of failed mutations
1838
- errors: string[]; // Error messages with location refs
1839
- }
1840
- ```
1841
-
1842
- **Key Operations**:
1843
- 1. Loops through locations
1844
- 2. Builds GraphQL mutation input
1845
- 3. Executes with retry logic + rate limiting
1846
- 4. Tracks success/failure per location
1847
- 5. Returns summary counts + errors
1848
-
1849
- **Why separate function?**
1850
- - Clear separation: mapping vs mutation execution
1851
- - Rate limiting logic isolated and configurable
1852
- - Easy to swap mutation types (create vs update)
1853
- - Testable with mock FluentClient
1854
- - Can parallelize in future (batch mutations)
1855
-
1856
- ### Function 3: writeMutationLog()
1857
-
1858
- **Purpose**: Write detailed mutation results to S3 as JSON log
1859
-
1860
- **Inputs**:
1861
- - `s3: S3DataSource` - For log upload
1862
- - `logEntries: MutationLogEntry[]` - Mutation status per location
1863
- - `fileName: string` - Original file name
1864
- - `logPrefix: string` - S3 prefix for logs (e.g., `logs/`)
1865
- - `log: any` - Logger instance
1866
-
1867
- **Outputs**: `void` (non-blocking - errors logged but not thrown)
1868
-
1869
- **Key Operations**:
1870
- 1. Creates timestamped log file name
1871
- 2. Builds JSON log with summary + entries
1872
- 3. **Uses Buffer.from()** - Required for Deno/Versori runtime
1873
- 4. Writes to S3 with `uploadFile()`
1874
- 5. Errors don't stop workflow (logging is non-critical)
1875
-
1876
- **Why separate function?**
1877
- - Optional feature - can be disabled via config
1878
- - Non-blocking - logging failure doesn't fail workflow
1879
- - Structured logging for audit trails
1880
- - Easy to change log format (JSON, CSV, XML)
1881
- - Can add log rotation/cleanup logic later
1882
-
1883
- ### Service Function Composition Benefits
1884
-
1885
- **1. Maintainability**
1886
- ```typescript
1887
- // Clear workflow orchestration
1888
- const processingResult = await processFile(...);
1889
- const mutationResult = await executeMutations(...);
1890
- await writeMutationLog(...);
1891
- ```
1892
-
1893
- **2. Testability**
1894
- ```typescript
1895
- // Unit test processFile() with mock XML
1896
- const mockS3 = { downloadFile: jest.fn() };
1897
- const result = await processFile(mockS3, parser, mapper, ...);
1898
- expect(result.locations).toHaveLength(5);
1899
- ```
1900
-
1901
- **3. Reusability**
1902
- ```typescript
1903
- // Use processFile() in different workflows
1904
- export const webhook = webhook('location-webhook').then(async ctx => {
1905
- const result = await processFile(s3, parser, mapper, filePath, fileName, log);
1906
- return { locations: result.locations };
1907
- });
1908
- ```
1909
-
1910
- **4. Error Isolation**
1911
- ```typescript
1912
- // Each function has clear error boundaries
1913
- try {
1914
- const processingResult = await processFile(...);
1915
- // File-level errors caught here
1916
- } catch (error) {
1917
- // Handle file processing failure
1918
- }
1919
-
1920
- // Mutation errors don't stop file processing
1921
- const mutationResult = await executeMutations(...);
1922
- // mutationResult.errors contains per-location failures
1923
- ```
1924
-
1925
- **5. Progressive Enhancement**
1926
- ```typescript
1927
- // Easy to add features without touching core logic
1928
- async function validateLocations(locations: any[]) {
1929
- // Add validation step
1930
- }
1931
-
1932
- const processingResult = await processFile(...);
1933
- await validateLocations(processingResult.locations); // New step
1934
- const mutationResult = await executeMutations(...);
1935
- ```
1936
-
1937
- ---
1938
-
1939
- ## Versori Environment Variables
1940
-
1941
- **Activation Variables:**
1942
-
1943
- ```bash
1944
- # ============================================================================
1945
- # Required Variables
1946
- # ============================================================================
1947
- s3BucketName=my-location-bucket
1948
- awsAccessKeyId=AKIAXXXXXXXXXXXX
1949
- awsSecretAccessKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
1950
-
1951
- # ============================================================================
1952
- # S3 Configuration (Optional - with defaults)
1953
- # ============================================================================
1954
- awsRegion=us-east-1
1955
- s3Prefix=locations/
1956
- archivePrefix=processed/
1957
- errorPrefix=errors/
1958
- logPrefix=logs/
1959
- filePattern=.xml
1960
- maxFilesToProcess=10
1961
-
1962
- # ============================================================================
1963
- # Feature Toggles (Optional - with defaults)
1964
- # ============================================================================
1965
- # Enable S3 archival (move files to processed/errors directories)
1966
- enableArchival=true
1967
-
1968
- # Enable mutation logs (write detailed mutation results to S3)
1969
- enableMutationLogs=true
1970
-
1971
- # Enable file tracking via StateService + KV store
1972
- # When disabled, relies on S3 archival only for deduplication
1973
- enableFileTracking=true
1974
-
1975
- # ============================================================================
1976
- # Mutation Configuration (Optional - with defaults)
1977
- # ============================================================================
1978
- # Mutation batch size (concurrent requests)
1979
- # - 1 = Sequential (default, safest)
1980
- # - 5 = Process 5 mutations in parallel
1981
- # - 10 = Process 10 mutations in parallel
1982
- mutationBatchSize=1
1983
-
1984
- # Alias batching (combine multiple mutations into single request)
1985
- # - undefined = Disabled (default, use separate requests)
1986
- # - 5 = Combine 5 mutations per aliased request
1987
- # - 10 = Combine 10 mutations per aliased request
1988
- mutationsPerAliasBatch=
1989
-
1990
- # ============================================================================
1991
- # Fluent Commerce Configuration (Optional)
1992
- # ============================================================================
1993
- # Retailer ID - Only if mutation schema requires retailerId in input
1994
- # Standard createLocation does NOT require this field
1995
- # Check your GraphQL schema to determine if needed
1996
- retailerId=my-retailer-id
1997
- ```
1998
-
1999
- **Notes:**
2000
- - Webhook security is handled by Versori's native connection authentication. No manual API key configuration needed.
2001
- - `retailerId` - Standard createLocation does not have this field. Only use if YOUR custom schema requires it.
2002
- - See `fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md` for details.
2003
-
2004
- ---
2005
-
2006
- ## Schema Validation CLI Commands
2007
-
2008
- Before deploying, validate your field mappings against the Fluent GraphQL schema:
2009
-
2010
- ### 1. Introspect Schema
2011
-
2012
- ```bash
2013
- # Generate schema.json from live Fluent API
2014
- npx fc-connect introspect-schema \
2015
- --url https://api.fluentcommerce.com/graphql \
2016
- --client-id your-client-id \
2017
- --client-secret your-client-secret \
2018
- --output schema.json
2019
- ```
2020
-
2021
- ### 2. Create Mapping Config File
2022
-
2023
- **File: location-mapping.json**
2024
-
2025
- ```json
2026
- {
2027
- "version": "1.0.0",
2028
- "mutation": "createLocation",
2029
- "sourceFormat": "xml",
2030
- "returnFields": ["id", "ref", "name", "type", "status"],
2031
- "description": "XML location to Fluent Commerce GraphQL mapping",
2032
- "fields": {
2033
- "ref": { "source": "location.@ref", "required": true, "resolver": "sdk.trim" },
2034
- "name": { "source": "location.name", "required": true, "resolver": "sdk.trim" },
2035
- "type": { "source": "location.@type", "required": true, "resolver": "sdk.uppercase" },
2036
- "primaryAddress.ref": { "source": "location.@ref", "required": true },
2037
- "primaryAddress.street": { "source": "location.address.street1" },
2038
- "primaryAddress.latitude": {
2039
- "source": "location.coordinates.@lat",
2040
- "resolver": "sdk.parseFloat"
2041
- },
2042
- "primaryAddress.longitude": {
2043
- "source": "location.coordinates.@lon",
2044
- "resolver": "sdk.parseFloat"
2045
- }
2046
- }
2047
- }
2048
- ```
2049
-
2050
- ### 3. Validate Mapping
2051
-
2052
- ```bash
2053
- # Validate that all target fields exist in schema
2054
- npx fc-connect validate-schema \
2055
- --mapping location-mapping.json \
2056
- --schema schema.json
2057
- ```
2058
-
2059
- ### 4. Analyze Coverage
2060
-
2061
- ```bash
2062
- # Check which Location fields are mapped vs available
2063
- npx fc-connect analyze-coverage \
2064
- --mapping location-mapping.json \
2065
- --schema schema.json \
2066
- --type CreateLocationInput
2067
- ```
2068
-
2069
- **Output:**
2070
-
2071
- ```
2072
- ✅ Mapped: 15/42 fields (35%)
2073
- ❌ Missing required: timezone (String!)
2074
- ⚠️ Optional unmapped: supportPhoneNumber, networkId, attributes
2075
- ```
2076
-
2077
- ---
2078
-
2079
- ## Testing Locally
2080
-
2081
- ### 1. Upload Test XML to S3
2082
-
2083
- ```bash
2084
- aws s3 cp test-location.xml s3://my-location-bucket/locations/test-location.xml
2085
- ```
2086
-
2087
- ### 2. Deploy to Versori
2088
-
2089
- ```bash
2090
- npm run deploy
2091
- ```
2092
-
2093
- ### 3. Manual Testing
2094
-
2095
- ```bash
2096
- # Trigger manual sync (auth handled by Versori connection)
2097
- curl -X POST https://your-workspace.versori.run/location-xml-adhoc
2098
-
2099
- # Check job status
2100
- curl -X POST https://your-workspace.versori.run/location-xml-job-status \
2101
- -H "Content-Type: application/json" \
2102
- -d '{"jobId": "location-xml-adhoc-1737525600000"}'
2103
- ```
2104
-
2105
- ### 4. Verify Processing
2106
-
2107
- - Upload a small XML to S3 (2-3 locations) and trigger `adhoc` webhook
2108
- - Verify GraphQL mutations are executed for each location
2109
- - Confirm file moved from `locations/` to `processed/` in S3
2110
- - Check KV state: errors and last processed metadata
2111
- - Monitor rate limiting: verify mutations respect configured rate
2112
-
2113
- ---
2114
-
2115
- ## Deployment
2116
-
2117
- ```bash
2118
- # Deploy to Versori
2119
- npm run deploy
2120
-
2121
- # View logs
2122
- npm run logs
2123
-
2124
- # Monitor execution
2125
- versori logs --follow
2126
- ```
2127
-
2128
- ---
2129
-
2130
- ## Monitoring
2131
-
2132
- ### Success Response
2133
-
2134
- ```json
2135
- {
2136
- "success": true,
2137
- "jobId": "location-xml-scheduled-1737525600000",
2138
- "processed": 3,
2139
- "skipped": 0,
2140
- "failed": 0,
2141
- "totalRecords": 12,
2142
- "errors": []
2143
- }
2144
- ```
2145
-
2146
- ### Partial Success Response
2147
-
2148
- ```json
2149
- {
2150
- "success": true,
2151
- "jobId": "location-xml-scheduled-1737525600000",
2152
- "processed": 3,
2153
- "skipped": 1,
2154
- "failed": 0,
2155
- "totalRecords": 10,
2156
- "errors": ["locations-003.xml: 2 mapping errors"]
2157
- }
2158
- ```
2159
-
2160
- ### Error Response
2161
-
2162
- ```json
2163
- {
2164
- "success": false,
2165
- "jobId": "location-xml-scheduled-1737525600000",
2166
- "processed": 0,
2167
- "skipped": 0,
2168
- "failed": 1,
2169
- "totalRecords": 0,
2170
- "errors": ["locations-001.xml: Invalid XML structure"]
2171
- }
2172
- ```
2173
-
2174
- ---
2175
-
2176
- ## Common Pitfalls and Solutions
2177
-
2178
- ### 1. XML Attribute Not Found
2179
-
2180
- **Symptoms**: Mapping errors like "field not found"
2181
-
2182
- **Solution**:
2183
-
2184
- ```typescript
2185
- // ❌ WRONG - Missing @ prefix for attribute
2186
- ref: {
2187
- source: 'location.ref';
2188
- }
2189
-
2190
- // ✅ CORRECT - Use @ prefix for XML attributes
2191
- ref: {
2192
- source: 'location.@ref';
2193
- }
2194
- ```
2195
-
2196
- ### 2. Single Element Not Array
2197
-
2198
- **Symptoms**: "Cannot read property forEach of undefined"
2199
-
2200
- **Solution**:
2201
-
2202
- ```typescript
2203
- // Always normalize to array
2204
- const locationsData = xmlData.locations?.location;
2205
- const locations = Array.isArray(locationsData) ? locationsData : [locationsData];
2206
- ```
2207
-
2208
- ### 3. Rate Limiting Too Aggressive
2209
-
2210
- **Symptoms**: Slow processing, mutations taking too long
2211
-
2212
- **Solution**:
2213
-
2214
- ```bash
2215
- # Increase rate limit (mutations per second)
2216
- mutationRateLimit=10 # Default is 5
2217
- ```
2218
-
2219
- ### 4. Empty Element vs Missing Element
2220
-
2221
- **Solution**:
2222
-
2223
- ```typescript
2224
- // Use required: false and defaultValue for optional fields
2225
- 'primaryAddress.street2': {
2226
- source: 'location.address.street2',
2227
- required: false,
2228
- defaultValue: ''
2229
- }
2230
- ```
2231
-
2232
- ### 5. S3 Access Denied
2233
-
2234
- **Symptoms**: S3 operations fail with 403 errors
2235
-
2236
- **Solution**: Validate IAM permissions
2237
-
2238
- **Required IAM Permissions:**
2239
-
2240
- ```json
2241
- {
2242
- "Version": "2012-10-17",
2243
- "Statement": [
2244
- {
2245
- "Effect": "Allow",
2246
- "Action": ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
2247
- "Resource": ["arn:aws:s3:::my-location-bucket", "arn:aws:s3:::my-location-bucket/*"]
2248
- }
2249
- ]
2250
- }
2251
- ```
2252
-
2253
- ### 6. GraphQL Schema Mismatch
2254
-
2255
- **Symptoms**: Mutation errors like "Unknown field", "Invalid input type"
2256
-
2257
- **Solution**: Use CLI tools to validate mappings
2258
-
2259
- ```bash
2260
- npx fc-connect validate-schema --mapping location-mapping.json --schema schema.json
2261
- ```
2262
-
2263
- ### 7. Nested Object Mapping Errors
2264
-
2265
- **Symptoms**: Flat structure instead of nested objects in mutation input
2266
-
2267
- **Solution**: Use dot notation in field mapping
2268
-
2269
- ```typescript
2270
- // ✅ CORRECT - Creates nested structure
2271
- 'primaryAddress.city': { source: 'location.address.city' }
2272
-
2273
- // ❌ WRONG - Creates flat structure
2274
- primaryAddress_city: { source: 'location.address.city' }
2275
- ```
2276
-
2277
- ### 8. retailerId Configuration Errors
2278
-
2279
- **Symptoms**: "retailerId is required" errors or confusion about when to use `setRetailerId()`
2280
-
2281
- **Solution**: Understand the correct pattern
2282
-
2283
- ```typescript
2284
- // ✅ CORRECT - GraphQL mutations don't need setRetailerId()
2285
- // Check your GraphQL schema to determine retailerId handling:
2286
- // - Mandatory retailerId → Must pass it in mutation input
2287
- // - Optional retailerId → Can pass it if needed
2288
- // - No retailerId field → Don't pass it
2289
- // Standard createLocation does not have retailerId field in schema
2290
-
2291
- // ✅ IF mutation schema requires retailerId (mandatory):
2292
- const { query, variables } = await mapper.map(location);
2293
- if (retailerId && variables.input) {
2294
- variables.input.retailer = { id: parseInt(retailerId) };
2295
- }
2296
- await client.graphql({ query, variables });
2297
- ```
2298
-
2299
- **Reference:** `fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md`
2300
-
2301
- ---
2302
-
2303
- ## Key Takeaways
2304
-
2305
- ### Architecture Patterns
2306
-
2307
- - **Service Function Composition**: Workflow broken into 3 service functions - `processFile()`, `executeMutations()`, `writeMutationLog()`
2308
- - **Per-File Processing**: Main workflow orchestrates service functions for each file
2309
- - **Clear Separation of Concerns**: Parse/map, execute mutations, logging are independent functions
2310
-
2311
- ### SDK Usage
2312
-
2313
- - **Buffer Import (CRITICAL)**: Always `import { Buffer } from 'node:buffer'` for Deno/Versori runtime
2314
- - **S3 Archival Deduplication**: Use `s3.moveFile()` to `processed/` subdirectory - PRIMARY deduplication mechanism
2315
- - **NO VersoriFileTracker for S3**: S3 archival is simpler and more reliable
2316
- - **StateService Role**: SECONDARY - provides metadata/history, not primary deduplication
2317
-
2318
- ### XML Processing
2319
-
2320
- - **XML @ Prefix**: Always use `@` prefix for XML attributes (`location.@ref`)
2321
- - **Array Normalization (CRITICAL)**: Handle single vs multiple elements with `Array.isArray()` check - single `<location>` becomes object, not array
2322
- - **Nested Mapping**: Use dot notation for nested objects (`primaryAddress.street`, `openingSchedule.monStart`)
2323
-
2324
- ### GraphQL Mutations
2325
-
2326
- - **NO setRetailerId() Required**: GraphQL mutations do NOT need `client.setRetailerId()` - only Job/Event API needs it
2327
- - **retailerId in Input**: Check your GraphQL schema to determine retailerId handling:
2328
- - **Mandatory retailerId** - Field exists and is required (`!`) → Must pass it
2329
- - **Optional retailerId** - Field exists and is optional → Can pass it if needed
2330
- - **No retailerId field** - Field doesn't exist → Don't pass it
2331
- - **Rate Limiting**: Implement configurable delays between mutations to avoid API throttling
2332
- - **Retry Logic**: Exponential backoff for failed mutations with `retryWithBackoff()`
2333
- - **Direct Mutations**: Use `client.graphql()` for location upserts (NOT Batch API)
2334
- - **GraphQL vs Batch API**: Use GraphQL for low-volume master data, Batch API for high-volume inventory
2335
-
2336
- ### Error Handling & Monitoring
2337
-
2338
- - **Error Recovery**: Exponential backoff for error state tracking with retry timestamps
2339
- - **Mutation Logging**: Optional S3 JSON logs with detailed per-location mutation status
2340
- - **Schema Validation**: Use CLI tools before deployment to catch mapping errors
2341
- - **Archival Order**: Archive FIRST (deduplication), then KV tracking (metadata)
2342
-
2343
- ---
2344
-
2345
- ## Related Documentation
2346
-
2347
- ### Core Guides
2348
-
2349
- - **GraphQL Mutation Mapping**: `fc-connect-sdk/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/readme.md`
2350
- - **Universal Mapping**: `fc-connect-sdk/docs/02-CORE-GUIDES/mapping/modules/readme.md`
2351
- - **XML Parser**: `fc-connect-sdk/docs/02-CORE-GUIDES/parsers/modules/05-xml-parser.md`
2352
- - **Data Sources**: `fc-connect-sdk/docs/02-CORE-GUIDES/data-sources/readme.md`
2353
- - **State Management**: `fc-connect-sdk/docs/03-PATTERN-GUIDES/file-operations/state-duplicate-prevention.md`
2354
-
2355
- ### Related Templates
2356
-
2357
- - **CSV Version**: `template-ingestion-s3-csv-location-graphql.md`
2358
- - **JSON Version**: `template-ingestion-s3-json-location-graphql.md`
2359
- - **SFTP XML Version**: `template-ingestion-sftp-xml-location-graphql.md`
2360
- - **Event API Pattern**: `../event-api/template-ingestion-s3-xml-product-event.md`
2361
- - **Batch API Pattern**: `../batch-api/template-ingestion-s3-xml-inventory-batch.md`
2362
-
2363
- ### CLI Tools
2364
-
2365
- - **Schema Introspection**: `fc-connect-sdk/bin/readme.md#introspect-schema`
2366
- - **Mapping Validation**: `fc-connect-sdk/bin/readme.md#validate-schema`
2367
- - **Coverage Analysis**: `fc-connect-sdk/bin/readme.md#analyze-coverage`
2368
-
2369
- ### Patterns
2370
-
2371
- - **Error Handling**: `fc-connect-sdk/docs/01-TEMPLATES/patterns/error-handling-retry.md`
2372
- - **Rate Limiting**: `fc-connect-sdk/docs/03-PATTERN-GUIDES/integration-patterns/rate-limiting.md`
2373
- - **XML Patterns**: `fc-connect-sdk/docs/01-TEMPLATES/versori/patterns/xml-response-patterns.md`
1
+ ---
2
+ template_id: tpl-ingest-s3-xml-to-location-graphql
3
+ canonical_filename: template-ingestion-s3-xml-location-graphql.md
4
+ version: 2.0.0
5
+ sdk_version: ^0.1.39
6
+ runtime: versori
7
+ direction: ingestion
8
+ source: s3-xml
9
+ destination: fluent-graphql
10
+ entity: location
11
+ format: xml
12
+ logging: versori
13
+ status: stable
14
+ compliance: gold-standard
15
+ features:
16
+ - graphql-mutation-mapper
17
+ - memory-management
18
+ - enhanced-logging
19
+ - attribute-transformation
20
+ ---
21
+
22
+ Template: Ingestion - S3 XML to Location GraphQL
23
+
24
+ **Template Version:** 2.0.0
25
+ **SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
26
+ **Last Updated:** 2025-01-24
27
+
28
+ **🆕 Version 2.0.0 Enhancements:**
29
+ - ✅ **GraphQL Mutation Mapper** - Direct field mapping to mutation variables
30
+ - ✅ **Memory Management** - Clear large arrays after processing batches
31
+ - ✅ **Enhanced Logging** - Track mutation execution with emoji indicators
32
+ - ✅ **Attribute Transformation** - Handle complex nested data structures
33
+
34
+ ---
35
+
36
+ ## Implementation Prompt
37
+
38
+ "Create a Versori scheduled workflow that reads location XML files from S3, transforms the data using GraphQLMutationMapper with nested object mapping, and creates/updates Fluent Commerce locations via direct GraphQL mutations with alias batching support.
39
+
40
+ **Requirements:**
41
+
42
+ 1. **S3 Source**:
43
+ - List and download XML files with configurable prefix and pattern
44
+ - Archive processed files to `processed/` directory (deduplication via archiving)
45
+ - Move failed files to `errors/` directory
46
+ - Use S3DataSource with retry logic and built-in archival
47
+ - Support configurable bucket, region, credentials
48
+
49
+ 2. **XML Parsing**:
50
+ - Use XMLParserService with @ prefix for attribute access
51
+ - Handle both single and multiple `<location>` elements (array normalization)
52
+ - Support nested XML paths (`location.address.street1`, `location.coordinates.@lat`)
53
+ - Parse complex structures (addresses, coordinates, opening schedules)
54
+
55
+ 3. **Field Mapping**:
56
+ - Map XML fields to Location GraphQL input type with nested objects:
57
+ - `ref`, `name`, `type` (root fields)
58
+ - `primaryAddress.*` (nested address object with coordinates)
59
+ - `openingSchedule.*` (nested schedule object with 7-day hours)
60
+ - Use SDK resolvers (trim, uppercase, parseFloat, parseInt, boolean)
61
+ - Support custom resolvers for complex transformations
62
+ - Validate required fields
63
+ - Note: Check your GraphQL schema to determine if `retailer.id` field exists and is mandatory/optional
64
+
65
+ 4. **GraphQL Mutations** (Direct - NO Batch API):
66
+ - Execute `createLocation` mutation directly for each location
67
+ - **NO BPP (Batch Pre-Processing)** - Not applicable for direct GraphQL mutations
68
+ - **NO Batch API** - Use direct GraphQL mutations with rate limiting instead
69
+ - Use rate limiting (configurable mutations per second)
70
+ - Add delay between mutations to avoid API throttling
71
+ - Retry failed mutations with exponential backoff
72
+ - Track successful vs failed mutations per file
73
+
74
+ 5. **Job Tracking & State Management**:
75
+ - **Use JobTracker** - Track job lifecycle (start, complete, fail) in KV store
76
+ - Use StateService + VersoriKVAdapter for duplicate file prevention
77
+ - Track processed files in KV store with metadata
78
+ - Store error state with exponential backoff tracking
79
+ - Support distributed state across workflow runs
80
+ - Provide job status endpoint for monitoring
81
+
82
+ 6. **Error Handling**:
83
+ - File-level errors archived to `/errors/` subdirectory
84
+ - Record-level errors tracked but don't stop file processing
85
+ - Mapping errors logged with specific location context
86
+ - Mutation errors retried with exponential backoff
87
+ - Error state tracking with next retry timestamp
88
+
89
+ 7. **Advanced Features**:
90
+ - Configurable rate limiting (mutations per second)
91
+ - Empty file detection and archival
92
+ - Timestamp-based error tracking
93
+ - Comprehensive monitoring and logging
94
+ - Manual webhook trigger with job tracking
95
+ - Job status query endpoint
96
+
97
+ **Use SDK Components:**
98
+
99
+ - `createClient()` - Universal client factory for Versori
100
+ - `S3DataSource` - S3 operations with retry logic
101
+ - `XMLParserService` - XML parsing with @ prefix attribute support
102
+ - `GraphQLMutationMapper` - Field transformation with schema validation and nested object support
103
+ - `StateService` + `VersoriKVAdapter` - Duplicate prevention with KV storage
104
+ - Native Versori `log` - Structured logging
105
+
106
+ **Configuration Variables** (from Versori activation):
107
+
108
+ ```typescript
109
+ {
110
+ s3: {
111
+ bucketName: string;
112
+ region: string;
113
+ accessKeyId: string;
114
+ secretAccessKey: string;
115
+ prefix: string; // e.g., 'locations/'
116
+ archivePrefix: string; // e.g., 'processed/'
117
+ errorPrefix: string; // e.g., 'errors/'
118
+ filePattern: string; // e.g., '.xml'
119
+ maxFilesToProcess: number;
120
+ enableArchival: boolean;
121
+ },
122
+ fluent: {
123
+ retailerId: string; // Optional: Only if mutation schema requires retailerId in input
124
+ mutationRateLimit: number; // mutations per second (e.g., 5)
125
+ }
126
+ }
127
+ ```
128
+
129
+ **Architecture Pattern:**
130
+
131
+ ```
132
+ S3 Bucket → List Files → Download XML → Parse → Map → GraphQL Mutation → Archive/Move
133
+ ↓ ↓ ↓ ↓ ↓ ↓ ↓
134
+ Configure Filter by S3DataSource XML Universal Direct S3 moveFile()
135
+ Connection Pattern + Retry Parser Mapper createLocation to processed/
136
+ (@) (nested) + Rate Limit or errors/
137
+ (deduplication)
138
+ ```
139
+
140
+ **Deliverables:**
141
+
142
+ 1. Complete Versori workflow with package.json
143
+ 2. Main workflow logic with S3 + XML + GraphQL patterns
144
+ 3. Helper functions for rate limiting and retry
145
+ 4. XML path resolution examples
146
+ 5. Sample XML files with nested structures
147
+ 6. Schema validation CLI commands
148
+ 7. Testing and deployment instructions
149
+ 8. Monitoring and troubleshooting guidance"
150
+
151
+ ---
152
+
153
+ # STEP 3: Complete Implementation
154
+
155
+ ## Versori Scheduled: S3 XML → Location GraphQL
156
+
157
+ **FC Connect SDK Use Case Guide**
158
+
159
+ > **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
160
+ > **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
161
+
162
+ **Context**: Versori scheduled workflow that reads location XML files from S3 and creates/updates Fluent Commerce locations via GraphQL mutations with XML path resolution and rate limiting
163
+
164
+ **Complexity**: Medium
165
+
166
+ **Runtime**: Versori Platform
167
+
168
+ **Estimated Lines**: ~850 lines (with comprehensive documentation)
169
+
170
+ ---
171
+
172
+ ## What You'll Build
173
+
174
+ - Scheduled Versori workflow (daily location sync)
175
+ - S3 file listing, download, and archival/move
176
+ - XML parsing with @ prefix for attributes (XPath-style)
177
+ - Array normalization (single element → array conversion)
178
+ - GraphQLMutationMapper-based field transformations with nested objects
179
+ - GraphQL mutations for location upserts with alias batching support
180
+ - Retry logic with exponential backoff
181
+ - StateService duplicate prevention (KV-backed)
182
+ - Error state tracking and file error archival
183
+ - Manual webhook trigger and job status endpoint
184
+
185
+ ---
186
+
187
+ ## When to Use GraphQL Mutations vs Batch API vs Event API
188
+
189
+ ### ✅ Use GraphQL Mutations For:
190
+
191
+ | Entity Type | Use Case | Why GraphQL |
192
+ | -------------- | ---------------------------------------- | ------------------------------------- |
193
+ | **Locations** | Store/warehouse master data (low volume) | Direct control, immediate validation |
194
+ | **Controls** | System configuration, settings | Single operations, complex queries |
195
+ | **Prices** | Price updates (moderate volume) | Immediate feedback, custom logic |
196
+ | **Single Ops** | One-off creates/updates | Testing, debugging, direct API access |
197
+
198
+ ### ❌ Use Event API Instead For:
199
+
200
+ | Entity Type | Use Case | Why Event API |
201
+ | ------------------- | -------------------------------------- | -------------------------------------------- |
202
+ | **Products** | Product catalog sync, variant updates | Triggers workflows, validates business rules |
203
+ | **Customers** | Customer registration, profile updates | Needs workflow for downstream systems |
204
+ | **Orders** | Order creation, status updates | Event-driven fulfillment workflows |
205
+ | **Custom Entities** | Any entity needing workflow triggers | Full Rubix workflow support |
206
+
207
+ ### 🔄 Use Batch API For:
208
+
209
+ | Entity Type | Use Case | Why Batch API |
210
+ | ------------------ | ---------------------------------- | ----------------------------------------------- |
211
+ | **Inventory ONLY** | Bulk inventory updates, daily sync | Optimized for high-volume, BPP change detection |
212
+
213
+ ---
214
+
215
+ ## XML File Format
216
+
217
+ ### Sample: locations.xml
218
+
219
+ ```xml
220
+ <?xml version="1.0" encoding="UTF-8"?>
221
+ <locations>
222
+ <location ref="LOC-001" type="WAREHOUSE">
223
+ <name>Downtown Warehouse</name>
224
+ <address country="USA">
225
+ <street1>123 Main St</street1>
226
+ <street2>Building A</street2>
227
+ <city>New York</city>
228
+ <state>NY</state>
229
+ <postalCode>10001</postalCode>
230
+ </address>
231
+ <coordinates lat="40.7128" lon="-74.0060"/>
232
+ <timeZone>America/New_York</timeZone>
233
+ <openingSchedule>
234
+ <allHours>false</allHours>
235
+ <monStart>800</monStart>
236
+ <monEnd>1800</monEnd>
237
+ <tueStart>800</tueStart>
238
+ <tueEnd>1800</tueEnd>
239
+ <wedStart>800</wedStart>
240
+ <wedEnd>1800</wedEnd>
241
+ <thuStart>800</thuStart>
242
+ <thuEnd>1800</thuEnd>
243
+ <friStart>800</friStart>
244
+ <friEnd>1800</friEnd>
245
+ <satStart>0</satStart>
246
+ <satEnd>0</satEnd>
247
+ <sunStart>0</sunStart>
248
+ <sunEnd>0</sunEnd>
249
+ </openingSchedule>
250
+ </location>
251
+
252
+ <location ref="LOC-002" type="DC">
253
+ <name>Regional DC</name>
254
+ <address country="USA">
255
+ <street1>456 Industrial Pkwy</street1>
256
+ <city>Los Angeles</city>
257
+ <state>CA</state>
258
+ <postalCode>90001</postalCode>
259
+ </address>
260
+ <coordinates lat="34.0522" lon="-118.2437"/>
261
+ <timeZone>America/Los_Angeles</timeZone>
262
+ <openingSchedule>
263
+ <allHours>true</allHours>
264
+ <monStart>0</monStart>
265
+ <monEnd>0</monEnd>
266
+ <tueStart>0</tueStart>
267
+ <tueEnd>0</tueEnd>
268
+ <wedStart>0</wedStart>
269
+ <wedEnd>0</wedEnd>
270
+ <thuStart>0</thuStart>
271
+ <thuEnd>0</thuEnd>
272
+ <friStart>0</friStart>
273
+ <friEnd>0</friEnd>
274
+ <satStart>0</satStart>
275
+ <satEnd>0</satEnd>
276
+ <sunStart>0</sunStart>
277
+ <sunEnd>0</sunEnd>
278
+ </openingSchedule>
279
+ </location>
280
+ </locations>
281
+ ```
282
+
283
+ **XML Path Syntax (with @ prefix for attributes):**
284
+
285
+ - `location.@ref` → XML attribute `ref` on `<location>` element
286
+ - `location.name` → Text content of `<name>` element
287
+ - `location.address.street1` → Nested element path
288
+ - `location.address.@country` → XML attribute on nested `<address>` element
289
+ - `location.coordinates.@lat` → XML attribute for latitude
290
+ - `location.openingSchedule.monStart` → Deeply nested element
291
+
292
+ **Note**: The SDK's `XMLParserService` automatically handles XML attributes using `@` prefix notation.
293
+
294
+ ---
295
+
296
+ ## Versori Workflows Structure
297
+
298
+ **Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
299
+
300
+ **Trigger Types:**
301
+ - **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
302
+ - **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
303
+ - **`workflow()`** → Durable workflows (advanced, rarely used)
304
+
305
+ **Execution Steps (chained to triggers):**
306
+ - **`http()`** → External API calls (chained from schedule/webhook)
307
+ - **`fn()`** → Internal processing (chained from schedule/webhook)
308
+
309
+ ### Recommended Project Structure
310
+
311
+ ```
312
+ s3-xml-location-graphql/
313
+ ├── index.ts # Entry point - exports all workflows
314
+ └── src/
315
+ ├── workflows/
316
+ │ ├── scheduled/
317
+ │ │ └── daily-location-sync.ts # Scheduled: Daily location sync
318
+ │ │
319
+ │ └── webhook/
320
+ │ ├── adhoc-location-sync.ts # Webhook: Manual trigger
321
+ │ └── job-status-check.ts # Webhook: Status query
322
+
323
+ ├── services/
324
+ │ └── location-sync.service.ts # Shared orchestration logic (reusable)
325
+
326
+ └── config/
327
+ └── location-mapping.json # GraphQL mapping config
328
+ ```
329
+
330
+ ---
331
+
332
+ ## Complete Versori Workflow
333
+
334
+ ### Step 1: Package Configuration
335
+
336
+ **File: package.json**
337
+
338
+ ```json
339
+ {
340
+ "name": "versori-s3-xml-location-sync",
341
+ "version": "1.0.0",
342
+ "description": "Versori workflow: S3 XML location sync to Fluent GraphQL",
343
+ "versori": {
344
+ "workflows": "./index.ts"
345
+ },
346
+ "type": "module",
347
+ "scripts": {
348
+ "deploy": "versori deploy",
349
+ "logs": "versori logs"
350
+ },
351
+ "dependencies": {
352
+ "@fluentcommerce/fc-connect-sdk": "^0.1.39",
353
+ "@versori/run": "latest"
354
+ },
355
+ "devDependencies": {
356
+ "typescript": "^5.0.0",
357
+ "@types/node": "^20.0.0"
358
+ }
359
+ }
360
+ ```
361
+
362
+ ### Step 2: Workflow Entry Point (`index.ts`)
363
+
364
+ **Purpose**: Register all workflows with Versori platform
365
+
366
+ ```typescript
367
+ /**
368
+ * Entry Point - Registers all workflows with Versori platform
369
+ *
370
+ * Versori automatically discovers and registers exported workflows
371
+ *
372
+ * File Structure:
373
+ * - src/workflows/scheduled/ → Time-based triggers (cron)
374
+ * - src/workflows/webhook/ → HTTP-based triggers (webhooks)
375
+ */
376
+
377
+ // Scheduled workflows
378
+ export { dailyLocationSync } from './src/workflows/scheduled/daily-location-sync';
379
+
380
+ // Webhook workflows
381
+ export { adhocLocationSync } from './src/workflows/webhook/adhoc-location-sync';
382
+ export { locationSyncJobStatus } from './src/workflows/webhook/job-status-check';
383
+ ```
384
+
385
+ **What Gets Exposed:**
386
+ - ✅ `adhocLocationSync` → `https://{workspace}.versori.run/location-sync-adhoc`
387
+ - ✅ `locationSyncJobStatus` → `https://{workspace}.versori.run/location-sync-job-status`
388
+ - ❌ `dailyLocationSync` → NOT exposed (runs automatically on cron)
389
+
390
+ ---
391
+
392
+ ### Step 3: Workflow Files
393
+
394
+ #### `src/workflows/scheduled/daily-location-sync.ts`
395
+
396
+ **Purpose**: Automatic daily location sync
397
+ **Trigger**: Cron schedule (`0 2 * * *`)
398
+ **Exposed as Endpoint**: ❌ NO - Runs automatically
399
+
400
+ ```typescript
401
+ import { schedule, http } from '@versori/run';
402
+ import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
403
+ import { executeLocationSync } from '../../services/location-sync.service';
404
+
405
+ /**
406
+ * Scheduled Workflow: Daily Location Sync
407
+ *
408
+ * Runs automatically daily at 2 AM UTC
409
+ * NOT exposed as HTTP endpoint - Versori executes on schedule
410
+ *
411
+ * Uses shared service: location-sync.service.ts
412
+ */
413
+ export const dailyLocationSync = schedule(
414
+ 'location-sync-scheduled',
415
+ '0 2 * * *' // Daily at 2 AM UTC
416
+ ).then(
417
+ http('run-location-sync', { connection: 'fluent_commerce' }, async ctx => {
418
+ const startTime = Date.now();
419
+ const { log, openKv } = ctx;
420
+ const jobId = `location-sync-${Date.now()}`;
421
+ const tracker = new JobTracker(openKv(':project:'), log);
422
+
423
+ log.info('🚀 [START] Daily location sync initiated', { jobId, trigger: 'schedule' });
424
+
425
+ await tracker.createJob(jobId, { triggeredBy: 'schedule', stage: 'initialization' });
426
+ await tracker.updateJob(jobId, { status: 'processing' });
427
+
428
+ try {
429
+ log.info('⚙️ [PROCESSING] Starting location synchronization workflow', { jobId });
430
+ const result = await executeLocationSync(ctx, jobId, tracker);
431
+ await tracker.markCompleted(jobId, result);
432
+
433
+ const duration = Date.now() - startTime;
434
+ log.info('✅ [SUCCESS] Daily location sync completed', {
435
+ jobId,
436
+ duration: `${duration}ms`,
437
+ processed: result.processed,
438
+ totalRecords: result.totalRecords
439
+ });
440
+
441
+ return { success: true, jobId, duration, ...result };
442
+ } catch (e: any) {
443
+ await tracker.markFailed(jobId, e);
444
+ const duration = Date.now() - startTime;
445
+
446
+ log.error('❌ [FAILED] Daily location sync failed', {
447
+ jobId,
448
+ duration: `${duration}ms`,
449
+ error: e?.message,
450
+ errorType: e?.name
451
+ });
452
+
453
+ return { success: false, jobId, duration, error: e?.message };
454
+ }
455
+ })
456
+ );
457
+ ```
458
+
459
+ ---
460
+
461
+ #### `src/workflows/webhook/adhoc-location-sync.ts`
462
+
463
+ **Purpose**: Manual location sync trigger (on-demand)
464
+ **Trigger**: HTTP POST
465
+ **Endpoint**: `POST https://{workspace}.versori.run/location-sync-adhoc`
466
+ **Use Cases**: Testing, priority processing, ad-hoc runs
467
+
468
+ ```typescript
469
+ import { webhook, http } from '@versori/run';
470
+ import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
471
+ import { executeLocationSync } from '../../services/location-sync.service';
472
+
473
+ /**
474
+ * Webhook: Manual Location Sync Trigger
475
+ *
476
+ * Endpoint: POST https://{workspace}.versori.run/location-sync-adhoc
477
+ * Request body (optional): { filePattern: "urgent_*.xml", maxFiles: 5 }
478
+ *
479
+ * Pattern: webhook().then(http()) - needs Fluent API access
480
+ * Uses shared service: location-sync.service.ts
481
+ *
482
+ * SECURITY: Authentication handled via connection parameter
483
+ * No manual API key validation needed - Versori manages this via connection auth
484
+ */
485
+ export const adhocLocationSync = webhook('location-sync-adhoc', {
486
+ response: { mode: 'sync' },
487
+ connection: 'location-sync-adhoc', // Versori validates API key
488
+ }).then(
489
+ http('run-location-sync-adhoc', { connection: 'fluent_commerce' }, async ctx => {
490
+ const startTime = Date.now();
491
+ const { log, openKv, data } = ctx;
492
+ const jobId = `location-sync-adhoc-${Date.now()}`;
493
+ const tracker = new JobTracker(openKv(':project:'), log);
494
+
495
+ log.info('🚀 [START] Manual location sync triggered', { jobId, trigger: 'webhook', options: data });
496
+
497
+ await tracker.createJob(jobId, {
498
+ triggeredBy: 'manual',
499
+ stage: 'initialization',
500
+ options: data // Optional: filePattern, maxFiles, etc.
501
+ });
502
+ await tracker.updateJob(jobId, { status: 'processing' });
503
+
504
+ try {
505
+ log.info('⚙️ [PROCESSING] Starting manual location synchronization', { jobId });
506
+ const result = await executeLocationSync(ctx, jobId, tracker);
507
+ await tracker.markCompleted(jobId, result);
508
+
509
+ const duration = Date.now() - startTime;
510
+ log.info('✅ [SUCCESS] Manual location sync completed', {
511
+ jobId,
512
+ duration: `${duration}ms`,
513
+ processed: result.processed,
514
+ totalRecords: result.totalRecords
515
+ });
516
+
517
+ return { success: true, jobId, duration, ...result };
518
+ } catch (e: any) {
519
+ await tracker.markFailed(jobId, e);
520
+ const duration = Date.now() - startTime;
521
+
522
+ log.error('❌ [FAILED] Manual location sync failed', {
523
+ jobId,
524
+ duration: `${duration}ms`,
525
+ error: e?.message,
526
+ errorType: e?.name
527
+ });
528
+
529
+ return { success: false, jobId, duration, error: e?.message };
530
+ }
531
+ })
532
+ );
533
+ ```
534
+
535
+ ---
536
+
537
+ #### `src/workflows/webhook/job-status-check.ts`
538
+
539
+ **Purpose**: Query job status
540
+ **Trigger**: HTTP POST
541
+ **Endpoint**: `POST https://{workspace}.versori.run/location-sync-job-status`
542
+ **Request body**: `{ "jobId": "location-sync-1234567890" }`
543
+
544
+ ```typescript
545
+ import { webhook, fn } from '@versori/run';
546
+ import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
547
+
548
+ /**
549
+ * Webhook: Job Status Check
550
+ *
551
+ * Endpoint: POST https://{workspace}.versori.run/location-sync-job-status
552
+ * Request body: { "jobId": "location-sync-1234567890" }
553
+ *
554
+ * Pattern: webhook().then(fn()) - no external API needed, only KV storage
555
+ * Lightweight: Only queries KV store, no Fluent API calls
556
+ *
557
+ * SECURITY: Authentication handled via connection parameter
558
+ * No manual API key validation needed - Versori manages this via connection auth
559
+ */
560
+ export const locationSyncJobStatus = webhook('location-sync-job-status', {
561
+ response: { mode: 'sync' },
562
+ connection: 'location-sync-job-status',
563
+ }).then(
564
+ fn('status', async ctx => {
565
+ const { data, log, openKv } = ctx;
566
+ const jobId = data?.jobId as string;
567
+
568
+ if (!jobId) {
569
+ return { success: false, error: 'jobId required' };
570
+ }
571
+
572
+ const tracker = new JobTracker(openKv(':project:'), log);
573
+ const status = await tracker.getJob(jobId);
574
+
575
+ return status
576
+ ? { success: true, jobId, ...status }
577
+ : { success: false, error: 'Job not found', jobId };
578
+ })
579
+ );
580
+ ```
581
+
582
+ ---
583
+
584
+ ### Step 4: Main Orchestration Service (`src/services/location-sync.service.ts`)
585
+
586
+ **Note:** This service file should contain the `executeLocationSync` function (renamed from `runLocationXmlWorkflow`). The main workflow logic should be moved here.
587
+
588
+ ```typescript
589
+ /**
590
+ * Main Orchestration Service: Location Sync
591
+ *
592
+ * This service contains the core business logic for location synchronization.
593
+ *
594
+ * Features:
595
+ * - S3 file operations with archival-based deduplication (moveFile to processed/)
596
+ * - XML parsing with @ prefix for attributes
597
+ * - Single element → array normalization
598
+ * - GraphQLMutationMapper for nested field transformations
599
+ * - GraphQL mutations with alias batching support
600
+ * - StateService for metadata tracking (secondary to archival)
601
+ * - Error state tracking with exponential backoff
602
+ *
603
+ * Deduplication Strategy:
604
+ * - PRIMARY: S3 archival via moveFile() - Files in processed/ won't be re-listed
605
+ * - SECONDARY: StateService KV tracking - Provides metadata and processing history
606
+ */
607
+ import { Buffer } from 'node:buffer'; // Required for Deno/Versori runtime
608
+ import {
609
+ createClient,
610
+ S3DataSource,
611
+ XMLParserService,
612
+ GraphQLMutationMapper,
613
+ StateService,
614
+ VersoriKVAdapter,
615
+ JobTracker,
616
+ FluentClient,
617
+ } from '@fluentcommerce/fc-connect-sdk';
618
+
619
+ // ============================================================================
620
+ // Type Definitions
621
+ // ============================================================================
622
+
623
+ interface FileProcessingResult {
624
+ success: boolean;
625
+ locations: any[];
626
+ errors: string[];
627
+ }
628
+
629
+ interface MutationResult {
630
+ successful: number;
631
+ failed: number;
632
+ errors: string[];
633
+ }
634
+
635
+ interface MutationLogEntry {
636
+ timestamp: string;
637
+ fileName: string;
638
+ locationRef: string;
639
+ status: 'success' | 'failure';
640
+ error?: string;
641
+ }
642
+
643
+ // ============================================================================
644
+ // Utility Functions
645
+ // ============================================================================
646
+
647
+ /**
648
+ * Retry utility with exponential backoff
649
+ * (User-defined - not part of SDK public API)
650
+ */
651
+ async function retryWithBackoff<T>(
652
+ operation: () => Promise<T>,
653
+ maxRetries = 3,
654
+ baseDelayMs = 1000
655
+ ): Promise<T> {
656
+ let lastError: any;
657
+ for (let attempt = 0; attempt < maxRetries; attempt++) {
658
+ try {
659
+ return await operation();
660
+ } catch (error) {
661
+ lastError = error;
662
+ if (attempt < maxRetries - 1) {
663
+ const delay = baseDelayMs * Math.pow(2, attempt) + Math.floor(Math.random() * 200);
664
+ await new Promise(resolve => setTimeout(resolve, delay));
665
+ }
666
+ }
667
+ }
668
+ throw lastError;
669
+ }
670
+
671
+ /**
672
+ * Rate limiter for GraphQL mutations
673
+ */
674
+ async function rateLimitedMutation(operation: () => Promise<any>, delayMs: number): Promise<any> {
675
+ const result = await operation();
676
+ await new Promise(resolve => setTimeout(resolve, delayMs));
677
+ return result;
678
+ }
679
+
680
+ // ============================================================================
681
+ // Service Functions
682
+ // ============================================================================
683
+
684
+ /**
685
+ * Service Function 1: Process File
686
+ *
687
+ * Downloads XML from S3, parses with XMLParserService, normalizes arrays,
688
+ * and maps fields with GraphQLMutationMapper.
689
+ *
690
+ * @param s3 - S3DataSource instance
691
+ * @param parser - XMLParserService instance
692
+ * @param mapper - GraphQLMutationMapper instance
693
+ * @param filePath - Full S3 path to file
694
+ * @param fileName - File name only (for logging)
695
+ * @param log - Logger instance
696
+ * @returns FileProcessingResult with locations array and errors
697
+ */
698
+ async function processFile(
699
+ s3: S3DataSource,
700
+ parser: XMLParserService,
701
+ mapper: GraphQLMutationMapper,
702
+ filePath: string,
703
+ fileName: string,
704
+ log: any
705
+ ): Promise<FileProcessingResult> {
706
+ try {
707
+ log.info('Processing file', { fileName });
708
+
709
+ // Download with retry
710
+ const content = await retryWithBackoff(
711
+ () => s3.downloadFile(filePath, { encoding: 'utf8' }) as Promise<string>
712
+ );
713
+
714
+ // Parse XML
715
+ const xmlData = await parser.parse(content);
716
+
717
+ // Extract location array (handle both single and multiple locations)
718
+ // CRITICAL: XML array normalization - single <location> becomes object, not array
719
+ const locationsData = xmlData.locations?.location;
720
+ const locationsArray = Array.isArray(locationsData) ? locationsData : [locationsData];
721
+
722
+ if (!locationsArray || locationsArray.length === 0) {
723
+ log.warn('Empty file (no locations)', { fileName });
724
+ return {
725
+ success: true,
726
+ locations: [],
727
+ errors: [],
728
+ };
729
+ }
730
+
731
+ // Map each location using GraphQLMutationMapper
732
+ const mappedLocations: Array<{ query: string; variables: any; input: any }> = [];
733
+ const mappingErrors: string[] = [];
734
+
735
+ // ✅ PRODUCTION ENHANCEMENT: Log transformation start
736
+ log.info('Transforming locations to GraphQL mutations', {
737
+ fileName,
738
+ totalLocations: locationsArray.length,
739
+ });
740
+
741
+ for (let i = 0; i < locationsArray.length; i++) {
742
+ const locationNumber = i + 1;
743
+
744
+ // ✅ PRODUCTION ENHANCEMENT: Log progress every 50 locations
745
+ if (locationNumber % 50 === 0) {
746
+ log.info(`📤 Transforming location ${locationNumber}/${locationsArray.length}`, {
747
+ fileName,
748
+ locationNumber,
749
+ totalLocations: locationsArray.length,
750
+ validSoFar: mappedLocations.length,
751
+ errorsSoFar: mappingErrors.length,
752
+ progress: `${((locationNumber / locationsArray.length) * 100).toFixed(1)}%`,
753
+ });
754
+ }
755
+
756
+ // Wrap location in context for mapping
757
+ const record = { location: locationsArray[i] };
758
+ try {
759
+ // GraphQLMutationMapper returns { query, variables } directly
760
+ const mappingResult = await mapper.map(record);
761
+
762
+ mappedLocations.push({
763
+ query: mappingResult.query,
764
+ variables: mappingResult.variables,
765
+ input: mappingResult.variables.input || mappingResult.variables,
766
+ });
767
+ } catch (error: unknown) {
768
+ const errorMsg = error instanceof Error ? error.message : String(error);
769
+ mappingErrors.push(`Location ${locationNumber}: ${errorMsg}`);
770
+ log.warn('Mapping failed for location', { fileName, index: i, error: errorMsg });
771
+ }
772
+ }
773
+
774
+ log.info('File processed', {
775
+ fileName,
776
+ total: locationsArray.length,
777
+ mapped: mappedLocations.length,
778
+ errors: mappingErrors.length,
779
+ successRate: `${((mappedLocations.length / locationsArray.length) * 100).toFixed(1)}%`,
780
+ });
781
+
782
+ return {
783
+ success: true,
784
+ locations: mappedLocations,
785
+ errors: mappingErrors,
786
+ };
787
+ } catch (error: any) {
788
+ // ✅ Enhanced error logging: Extract all error details for visibility
789
+ const errorDetails = {
790
+ message: error?.message || 'Unknown error',
791
+ stack: error?.stack,
792
+ fileName: error?.fileName,
793
+ lineNumber: error?.lineNumber,
794
+ originalError: error?.context?.originalError?.message,
795
+ errorType: error?.name || 'Error',
796
+ };
797
+ log.error('File processing failed', errorDetails, { fileName });
798
+ return {
799
+ success: false,
800
+ locations: [],
801
+ errors: [error.message || 'Unknown error'],
802
+ };
803
+ }
804
+ }
805
+
806
+ /**
807
+ * Service Function 2: Execute Mutations
808
+ *
809
+ * Executes GraphQL createLocation mutations with alias batching support.
810
+ *
811
+ * @param client - FluentClient instance
812
+ * @param mapper - GraphQLMutationMapper instance
813
+ * @param locations - Array of mapped location objects with query and variables
814
+ * @param log - Logger instance
815
+ * @param retailerId - Fluent retailer ID
816
+ * @param batchSize - Number of concurrent requests (default: 1)
817
+ * @param mutationsPerAliasBatch - Optional: Number of mutations per aliased request (default: undefined)
818
+ * @returns MutationResult with success/failure counts
819
+ */
820
+ async function executeMutations(
821
+ client: FluentClient,
822
+ mapper: GraphQLMutationMapper,
823
+ locations: Array<{ query: string; variables: any; input: any }>,
824
+ log: any,
825
+ retailerId: string,
826
+ batchSize: number = 1, // ✅ Default: 1 (sequential)
827
+ mutationsPerAliasBatch?: number // ✅ NEW: Alias batching parameter (default: undefined = disabled)
828
+ ): Promise<MutationResult> {
829
+ // Determine mode: use aliases if mutationsPerAliasBatch is set and > 1
830
+ const useAliases = mutationsPerAliasBatch && mutationsPerAliasBatch > 1;
831
+
832
+ if (useAliases) {
833
+ return await executeMutationsWithAliases(
834
+ client,
835
+ mapper,
836
+ locations,
837
+ log,
838
+ retailerId,
839
+ batchSize,
840
+ mutationsPerAliasBatch!
841
+ );
842
+ } else {
843
+ return await executeMutationsSeparate(
844
+ client,
845
+ locations,
846
+ log,
847
+ batchSize
848
+ );
849
+ }
850
+ }
851
+
852
+ /**
853
+ * Execute mutations using separate concurrent requests (current mode)
854
+ */
855
+ async function executeMutationsSeparate(
856
+ client: FluentClient,
857
+ locations: Array<{ query: string; variables: any; input: any }>,
858
+ log: any,
859
+ batchSize: number
860
+ ): Promise<MutationResult> {
861
+ const result: MutationResult = {
862
+ successful: 0,
863
+ failed: 0,
864
+ errors: [],
865
+ };
866
+
867
+ const safeConc = Math.max(1, Math.floor(batchSize));
868
+
869
+ // Sequential mode
870
+ if (safeConc === 1) {
871
+ for (const location of locations) {
872
+ try {
873
+ await retryWithBackoff(() =>
874
+ client.graphql({
875
+ query: location.query,
876
+ variables: location.variables,
877
+ })
878
+ );
879
+ result.successful++;
880
+ } catch (error: unknown) {
881
+ result.failed++;
882
+ const errorMsg = error instanceof Error ? error.message : String(error);
883
+ result.errors.push(errorMsg);
884
+ }
885
+ }
886
+ return result;
887
+ }
888
+
889
+ // Parallel mode
890
+ for (let i = 0; i < locations.length; i += safeConc) {
891
+ const chunk = locations.slice(i, i + safeConc);
892
+ const results = await Promise.allSettled(
893
+ chunk.map(loc =>
894
+ retryWithBackoff(() =>
895
+ client.graphql({
896
+ query: loc.query,
897
+ variables: loc.variables,
898
+ })
899
+ )
900
+ )
901
+ );
902
+
903
+ results.forEach((settledResult, idx) => {
904
+ if (settledResult.status === 'fulfilled') {
905
+ result.successful++;
906
+ } else {
907
+ result.failed++;
908
+ result.errors.push(
909
+ settledResult.reason instanceof Error ? settledResult.reason.message : String(settledResult.reason)
910
+ );
911
+ }
912
+ });
913
+ }
914
+
915
+ return result;
916
+ }
917
+
918
+ /**
919
+ * ✅ NEW: Execute mutations using GraphQL aliases (batched requests)
920
+ */
921
+ async function executeMutationsWithAliases(
922
+ client: FluentClient,
923
+ mapper: GraphQLMutationMapper,
924
+ locations: Array<{ query: string; variables: any; input: any }>,
925
+ log: any,
926
+ retailerId: string,
927
+ maxParallel: number,
928
+ mutationsPerAliasBatch: number
929
+ ): Promise<MutationResult> {
930
+ const result: MutationResult = { successful: 0, failed: 0, errors: [] };
931
+
932
+ const mutationName = (mapper as any).config.mutation || 'createLocation';
933
+ const aliasBatches: Array<Array<typeof locations[0]>> = [];
934
+
935
+ for (let i = 0; i < locations.length; i += mutationsPerAliasBatch) {
936
+ aliasBatches.push(locations.slice(i, i + mutationsPerAliasBatch));
937
+ }
938
+
939
+ // Process batches with concurrency control
940
+ for (let i = 0; i < aliasBatches.length; i += maxParallel) {
941
+ const concurrentBatches = aliasBatches.slice(i, i + maxParallel);
942
+
943
+ const batchResults = await Promise.allSettled(
944
+ concurrentBatches.map(async (batch) => {
945
+ const { query, variables } = buildAliasedBatch(batch, mutationName, retailerId);
946
+ const response = await retryWithBackoff(() => client.graphql({ query, variables }));
947
+ return parseAliasResponse(response, batch, mutationName);
948
+ })
949
+ );
950
+
951
+ batchResults.forEach((batchResult, idx) => {
952
+ if (batchResult.status === 'fulfilled') {
953
+ const batchRes = batchResult.value;
954
+ result.successful += batchRes.executed;
955
+ result.failed += batchRes.failed;
956
+ result.errors.push(...batchRes.errors);
957
+ } else {
958
+ const batch = concurrentBatches[idx];
959
+ const errorMsg = batchResult.reason instanceof Error ? batchResult.reason.message : String(batchResult.reason);
960
+ batch.forEach(loc => {
961
+ result.failed++;
962
+ result.errors.push(`Batch execution failed: ${errorMsg}`);
963
+ });
964
+ }
965
+ });
966
+
967
+ if (i + maxParallel < aliasBatches.length) {
968
+ await new Promise(resolve => setTimeout(resolve, 500));
969
+ }
970
+ }
971
+
972
+ return result;
973
+ }
974
+
975
+ /**
976
+ * ✅ NEW: Build aliased batch query and variables
977
+ */
978
+ function buildAliasedBatch(
979
+ batch: Array<{ query: string; variables: any; input: any }>,
980
+ mutationName: string,
981
+ retailerId: string
982
+ ): { query: string; variables: Record<string, any> } {
983
+ const batchSize = batch.length;
984
+ const inputTypeName = `${mutationName.charAt(0).toUpperCase() + mutationName.slice(1)}Input`;
985
+
986
+ const variables = Array.from({ length: batchSize }, (_, i) =>
987
+ `$input${i + 1}: ${inputTypeName}!`
988
+ ).join(', ');
989
+
990
+ const aliasedMutations = Array.from({ length: batchSize }, (_, i) => {
991
+ const alias = `${mutationName}${i + 1}`;
992
+ return ` ${alias}: ${mutationName}(input: $input${i + 1}) { id ref name }`;
993
+ }).join('\n');
994
+
995
+ const operationName = `Batch${mutationName.charAt(0).toUpperCase() + mutationName.slice(1)}s`;
996
+ const query = `mutation ${operationName}(${variables}) {\n${aliasedMutations}\n}`;
997
+
998
+ const variablesObj: Record<string, any> = {};
999
+ batch.forEach((loc, index) => {
1000
+ const input = loc.variables.input || loc.variables;
1001
+ if (input && !input.retailer) {
1002
+ input.retailer = { id: parseInt(retailerId) };
1003
+ }
1004
+ variablesObj[`input${index + 1}`] = input;
1005
+ });
1006
+
1007
+ return { query, variables: variablesObj };
1008
+ }
1009
+
1010
+ /**
1011
+ * ✅ NEW: Parse aliased GraphQL response
1012
+ */
1013
+ function parseAliasResponse(
1014
+ response: any,
1015
+ batch: Array<{ query: string; variables: any; input: any }>,
1016
+ mutationName: string
1017
+ ): { executed: number; failed: number; errors: string[] } {
1018
+ const result = { executed: 0, failed: 0, errors: [] as string[] };
1019
+
1020
+ const data = response.data || {};
1021
+ const errors = response.errors || [];
1022
+
1023
+ batch.forEach((loc, index) => {
1024
+ const alias = `${mutationName}${index + 1}`;
1025
+ const aliasData = data[alias];
1026
+ const aliasErrors = errors.filter((e: any) =>
1027
+ e.path && Array.isArray(e.path) && e.path.includes(alias)
1028
+ );
1029
+
1030
+ if (aliasData && !aliasErrors.length) {
1031
+ result.executed++;
1032
+ } else {
1033
+ result.failed++;
1034
+ const errorMsg = aliasErrors[0]?.message || 'Mutation failed';
1035
+ result.errors.push(`${loc.input?.ref || 'unknown'}: ${errorMsg}`);
1036
+ }
1037
+ });
1038
+
1039
+ return result;
1040
+ }
1041
+
1042
+ /**
1043
+ * Service Function 3: Write Mutation Log
1044
+ *
1045
+ * Writes mutation results to S3 as a JSON log file.
1046
+ *
1047
+ * @param s3 - S3DataSource instance
1048
+ * @param logEntries - Array of mutation log entries
1049
+ * @param fileName - Original file name (used to generate log path)
1050
+ * @param logPrefix - S3 prefix for logs (e.g., 'logs/')
1051
+ * @param log - Logger instance
1052
+ */
1053
+ async function writeMutationLog(
1054
+ s3: S3DataSource,
1055
+ logEntries: MutationLogEntry[],
1056
+ fileName: string,
1057
+ logPrefix: string,
1058
+ log: any
1059
+ ): Promise<void> {
1060
+ try {
1061
+ const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
1062
+ const logFileName = `${logPrefix}${fileName.replace('.xml', '')}_${timestamp}.json`;
1063
+
1064
+ const logContent = JSON.stringify(
1065
+ {
1066
+ fileName,
1067
+ timestamp: new Date().toISOString(),
1068
+ totalMutations: logEntries.length,
1069
+ successful: logEntries.filter(e => e.status === 'success').length,
1070
+ failed: logEntries.filter(e => e.status === 'failure').length,
1071
+ entries: logEntries,
1072
+ },
1073
+ null,
1074
+ 2
1075
+ );
1076
+
1077
+ // Write to S3 (uploadFile accepts string or Buffer)
1078
+ await s3.uploadFile(logFileName, logContent);
1079
+
1080
+ log.info('Mutation log written', { logFileName, entries: logEntries.length });
1081
+ } catch (error: any) {
1082
+ // ✅ Enhanced error logging: Extract all error details for visibility
1083
+ const errorDetails = {
1084
+ message: error?.message || 'Unknown error',
1085
+ stack: error?.stack,
1086
+ errorType: error?.name || 'Error',
1087
+ };
1088
+ log.error('Failed to write mutation log', errorDetails, { fileName });
1089
+ // Don't throw - logging failure shouldn't stop workflow
1090
+ }
1091
+ }
1092
+
1093
+ // ============================================================================
1094
+ // Main Workflow Function
1095
+ // ============================================================================
1096
+
1097
+ /**
1098
+ * Main Orchestration Function: Execute Location Sync
1099
+ *
1100
+ * This function orchestrates the location synchronization workflow.
1101
+ *
1102
+ * Architecture:
1103
+ * 1. List files from S3
1104
+ * 2. For each file:
1105
+ * a. processFile() - Download, parse, map
1106
+ * b. executeMutations() - Send GraphQL mutations with rate limiting
1107
+ * c. writeMutationLog() - Log results to S3
1108
+ * d. Archive file (primary deduplication)
1109
+ * e. Mark processed in KV (metadata tracking)
1110
+ *
1111
+ * @param ctx - Versori context
1112
+ * @param jobId - Job identifier
1113
+ * @param tracker - JobTracker instance
1114
+ * @returns Processing result
1115
+ */
1116
+ export async function executeLocationSync(ctx: any, jobId: string, tracker: JobTracker) {
1117
+ const { log, activation } = ctx;
1118
+
1119
+ log.info('📋 [INIT] Reading activation variables', { jobId });
1120
+
1121
+ // Read activation variables
1122
+ const s3Bucket = activation?.getVariable('s3BucketName');
1123
+ const s3Region = activation?.getVariable('awsRegion') || 'us-east-1';
1124
+ const s3AccessKeyId = activation?.getVariable('awsAccessKeyId');
1125
+ const s3SecretAccessKey = activation?.getVariable('awsSecretAccessKey');
1126
+ const s3Prefix = activation?.getVariable('s3Prefix') || 'locations/';
1127
+ const archivePrefix = activation?.getVariable('archivePrefix') || 'processed/';
1128
+ const errorPrefix = activation?.getVariable('errorPrefix') || 'errors/';
1129
+ const logPrefix = activation?.getVariable('logPrefix') || 'logs/';
1130
+ const filePattern = (activation?.getVariable('filePattern') || '.xml').toLowerCase();
1131
+ const maxFiles = parseInt(activation?.getVariable('maxFilesToProcess') || '10', 10);
1132
+ const retailerId = activation?.getVariable('retailerId'); // Optional: Only if mutation schema requires it
1133
+ const enableArchival = activation?.getVariable('enableArchival') !== 'false';
1134
+ const enableMutationLogs = activation?.getVariable('enableMutationLogs') !== 'false';
1135
+ const enableFileTracking = activation?.getVariable('enableFileTracking') !== 'false';
1136
+
1137
+ // ✅ Configuration with defaults
1138
+ const mutationBatchSize = parseInt(
1139
+ activation?.getVariable('mutationBatchSize') || '1', // ✅ Default: 1 (sequential)
1140
+ 10
1141
+ );
1142
+
1143
+ const mutationsPerAliasBatch = activation?.getVariable('mutationsPerAliasBatch')
1144
+ ? parseInt(activation?.getVariable('mutationsPerAliasBatch') || '1', 10)
1145
+ : undefined; // ✅ Default: undefined (disabled, use separate requests)
1146
+
1147
+ // Validate required variables
1148
+ const missingVars: string[] = [];
1149
+ if (!s3Bucket) missingVars.push('s3BucketName');
1150
+ if (!s3AccessKeyId) missingVars.push('awsAccessKeyId');
1151
+ if (!s3SecretAccessKey) missingVars.push('awsSecretAccessKey');
1152
+ // Note: retailerId is optional - only needed if mutation schema requires it
1153
+
1154
+ if (missingVars.length > 0) {
1155
+ const errorMsg = `Missing required variables: ${missingVars.join(', ')}`;
1156
+ log.error('❌ [VALIDATION] Missing required activation variables', {
1157
+ missingVars,
1158
+ recommendation: 'Add missing variables in Versori activation settings'
1159
+ });
1160
+ return { success: false, error: errorMsg, processed: 0 };
1161
+ }
1162
+
1163
+ log.info('✅ [VALIDATION] All required variables present', {
1164
+ s3Bucket,
1165
+ s3Region,
1166
+ s3Prefix,
1167
+ enableFileTracking,
1168
+ mutationBatchSize
1169
+ });
1170
+
1171
+ try {
1172
+ log.info('🔧 [INIT] Initializing Fluent Commerce client', { jobId });
1173
+
1174
+ // Initialize services with validateConnection
1175
+ const client = await createClient(ctx, { validateConnection: true });
1176
+ if (!client) {
1177
+ throw new Error('Failed to create Fluent Commerce client');
1178
+ }
1179
+
1180
+ log.info('✅ [INIT] Fluent Commerce client validated and ready', { jobId });
1181
+
1182
+ // ✅ CORRECT: GraphQL mutations do NOT need client.setRetailerId()
1183
+ // setRetailerId() is only for Job/Event API, NOT GraphQL
1184
+ // Check your GraphQL schema to determine retailerId handling:
1185
+ // - Mandatory retailerId → Must pass it in mutation input
1186
+ // - Optional retailerId → Can pass it if needed
1187
+ // - No retailerId field → Don't pass it
1188
+ // See: fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md
1189
+
1190
+ log.info('🗄️ [INIT] Initializing S3 data source', { s3Bucket, s3Region, s3Prefix });
1191
+
1192
+ const s3 = new S3DataSource(
1193
+ {
1194
+ type: 'S3_XML',
1195
+ connectionId: 's3-location-sync',
1196
+ name: 'Source S3',
1197
+ s3Config: {
1198
+ bucket: s3Bucket,
1199
+ region: s3Region,
1200
+ accessKeyId: s3AccessKeyId,
1201
+ secretAccessKey: s3SecretAccessKey,
1202
+ },
1203
+ },
1204
+ log
1205
+ );
1206
+
1207
+ const parser = new XMLParserService();
1208
+
1209
+ // Initialize state tracking (only if enabled)
1210
+ let stateService: StateService | null = null;
1211
+ if (enableFileTracking) {
1212
+ log.info('🔄 [INIT] Enabling file tracking with StateService', { jobId });
1213
+ const stateKV = new VersoriKVAdapter(ctx);
1214
+ stateService = new StateService(stateKV);
1215
+ } else {
1216
+ log.info('⏭️ [INIT] File tracking disabled - relying on S3 archival only', { jobId });
1217
+ }
1218
+
1219
+ log.info('📝 [INIT] Loading mapping configuration', { jobId });
1220
+
1221
+ // ✅ CRITICAL: Load mapping config from external JSON file
1222
+ // Mapping config uses GraphQLMutationMapper structure (nested objects, not dot notation)
1223
+ // File: src/config/location-mapping.json
1224
+ const mappingConfigJson = await import('../config/location-mapping.json', { assert: { type: 'json' } });
1225
+ const mappingConfig = mappingConfigJson.default;
1226
+
1227
+ // Initialize GraphQLMutationMapper with client for schema introspection
1228
+ const mapper = new GraphQLMutationMapper(mappingConfig, log, { fluentClient: client });
1229
+
1230
+ log.info('📂 [S3] Listing files from S3', { s3Bucket, s3Prefix, filePattern });
1231
+
1232
+ // List files (pattern filtering handled by listFiles)
1233
+ const files = await s3.listFiles({
1234
+ prefix: s3Prefix,
1235
+ pattern: filePattern,
1236
+ maxKeys: 1000
1237
+ });
1238
+
1239
+ // Newest-first ordering
1240
+ const xmlFiles = files
1241
+ .sort((a: any, b: any) => {
1242
+ const aTime = a.lastModified ? new Date(a.lastModified).getTime() : 0;
1243
+ const bTime = b.lastModified ? new Date(b.lastModified).getTime() : 0;
1244
+ return bTime - aTime;
1245
+ })
1246
+ .slice(0, maxFiles);
1247
+
1248
+ log.info('📊 [S3] File discovery complete', {
1249
+ totalFiles: files.length,
1250
+ xmlFiles: xmlFiles.length,
1251
+ maxFiles,
1252
+ selectedFiles: xmlFiles.map(f => f.name)
1253
+ });
1254
+
1255
+ const results = {
1256
+ processed: 0,
1257
+ skipped: 0,
1258
+ failed: 0,
1259
+ totalRecords: 0,
1260
+ errors: [] as string[],
1261
+ };
1262
+
1263
+ log.info('🔄 [PROCESSING] Starting file processing loop', {
1264
+ fileCount: xmlFiles.length,
1265
+ jobId
1266
+ });
1267
+
1268
+ // Per-file processing loop
1269
+ for (const file of xmlFiles) {
1270
+ const filePath = file.path;
1271
+ const fileName = file.name;
1272
+
1273
+ log.info('📄 [FILE] Processing file', { fileName, filePath });
1274
+
1275
+ // Duplicate prevention (secondary check - files in processed/ won't be listed)
1276
+ // Primary deduplication: S3 archival (files moved to processed/ subdirectory)
1277
+ if (enableFileTracking && stateService) {
1278
+ const wasProcessed = await stateService.isFileProcessed(fileName);
1279
+ if (wasProcessed) {
1280
+ log.info('⏭️ [SKIP] File already processed (KV check)', { fileName });
1281
+ results.skipped++;
1282
+ continue;
1283
+ }
1284
+ }
1285
+
1286
+ try {
1287
+ // Step 1: Process file (download, parse, map)
1288
+ log.info('📥 [DOWNLOAD] Downloading and parsing file', { fileName });
1289
+ const processingResult = await processFile(s3, parser, mapper, filePath, fileName, log);
1290
+
1291
+ if (!processingResult.success) {
1292
+ throw new Error(`File processing failed: ${processingResult.errors.join(', ')}`);
1293
+ }
1294
+
1295
+ if (processingResult.locations.length === 0) {
1296
+ log.warn('⚠️ [SKIP] Empty file detected, archiving', { fileName });
1297
+ if (enableArchival) {
1298
+ await s3.moveFile(filePath, `${archivePrefix}${fileName}`);
1299
+ log.info('📦 [ARCHIVE] Empty file archived', { fileName, destination: `${archivePrefix}${fileName}` });
1300
+ }
1301
+ results.skipped++;
1302
+ continue;
1303
+ }
1304
+
1305
+ log.info('✅ [PARSE] File parsed successfully', {
1306
+ fileName,
1307
+ locationCount: processingResult.locations.length,
1308
+ mappingErrors: processingResult.errors.length
1309
+ });
1310
+
1311
+ // Step 2: Execute mutations
1312
+ // Step 2: Execute mutations with alias batching support
1313
+ // ? Enhanced: Extract context for progress logging
1314
+ const sampleLocationRefs = processingResult.locations.slice(0, 5).map((loc: any) => loc.input?.ref || loc.ref || 'unknown');
1315
+ const mutationType = mapper?.mutationName || 'createLocation';
1316
+
1317
+ // ? Enhanced: Start logging with context
1318
+ log.info(`[GraphQLMutations] Sending mutations for file "${fileName}"`, {
1319
+ totalMutations: processingResult.locations.length,
1320
+ mutationType,
1321
+ batchSize: mutationBatchSize,
1322
+ batchMode: mutationBatchSize === 1 ? 'sequential' : `parallel (${mutationBatchSize})`,
1323
+ sampleLocationRefs: sampleLocationRefs.join(', '),
1324
+ aliasBatching: mutationsPerAliasBatch ? `enabled (${mutationsPerAliasBatch} per alias)` : 'disabled'
1325
+ });
1326
+
1327
+ const mutationResult = await executeMutations(
1328
+ client,
1329
+ mapper,
1330
+ processingResult.locations,
1331
+ log,
1332
+ retailerId, // Pass retailerId for mutations that require it in input
1333
+ mutationBatchSize, // Concurrency control (default: 1)
1334
+ mutationsPerAliasBatch // ✅ NEW: Alias batching (default: undefined)
1335
+ );
1336
+
1337
+ // ? Enhanced: Completion logging with summary
1338
+ log.info(`[GraphQLMutations] Mutation submission completed for file "${fileName}"`, {
1339
+ totalMutations: processingResult.locations.length,
1340
+ successful: mutationResult.successful,
1341
+ failed: mutationResult.failed,
1342
+ successRate: processingResult.locations.length > 0 ? `${Math.round((mutationResult.successful / processingResult.locations.length) * 100)}%` : '0%',
1343
+ mutationType
1344
+ });
1345
+
1346
+ // Step 3: Write mutation log (if enabled)
1347
+ if (enableMutationLogs) {
1348
+ const logEntries: MutationLogEntry[] = processingResult.locations.map(loc => {
1349
+ const failed = mutationResult.errors.find(e => e.startsWith(loc.ref));
1350
+ return {
1351
+ timestamp: new Date().toISOString(),
1352
+ fileName,
1353
+ locationRef: loc.ref,
1354
+ status: failed ? 'failure' : 'success',
1355
+ error: failed,
1356
+ };
1357
+ });
1358
+
1359
+ await writeMutationLog(s3, logEntries, fileName, logPrefix, log);
1360
+ }
1361
+
1362
+ // Step 4: Archive file (PRIMARY deduplication - file won't be listed again)
1363
+ if (enableArchival) {
1364
+ log.info('📦 [ARCHIVE] Moving file to processed directory', { fileName });
1365
+ await s3.moveFile(filePath, `${archivePrefix}${fileName}`);
1366
+ log.info('✅ [ARCHIVE] File archived successfully', {
1367
+ fileName,
1368
+ destination: `${archivePrefix}${fileName}`
1369
+ });
1370
+ }
1371
+
1372
+ // Step 5: Mark processed in KV (SECONDARY - provides metadata/history)
1373
+ if (enableFileTracking && stateService) {
1374
+ log.info('💾 [STATE] Recording file processing metadata', { fileName });
1375
+ await stateService.markFileProcessed(fileName, {
1376
+ recordCount: processingResult.locations.length,
1377
+ successful: mutationResult.successful,
1378
+ failed: mutationResult.failed,
1379
+ mappingErrors: processingResult.errors.length,
1380
+ timestamp: new Date().toISOString(),
1381
+ });
1382
+ }
1383
+
1384
+ results.processed++;
1385
+ results.totalRecords += mutationResult.successful;
1386
+
1387
+ log.info('✅ [COMPLETE] File processing complete', {
1388
+ fileName,
1389
+ locations: processingResult.locations.length,
1390
+ successful: mutationResult.successful,
1391
+ failed: mutationResult.failed,
1392
+ mappingErrors: processingResult.errors.length,
1393
+ });
1394
+
1395
+ if (processingResult.errors.length > 0 || mutationResult.errors.length > 0) {
1396
+ results.errors.push(
1397
+ `${fileName}: ${processingResult.errors.length} mapping errors, ${mutationResult.errors.length} mutation errors`
1398
+ );
1399
+ }
1400
+ } catch (error: any) {
1401
+ // ✅ Enhanced error logging: Extract all error details for visibility
1402
+ const errorDetails = {
1403
+ message: error?.message || 'Unknown error',
1404
+ stack: error?.stack,
1405
+ fileName: error?.fileName,
1406
+ lineNumber: error?.lineNumber,
1407
+ originalError: error?.context?.originalError?.message,
1408
+ errorType: error?.name || 'Error',
1409
+ };
1410
+
1411
+ log.error('❌ [ERROR] File processing failed', errorDetails, { fileName });
1412
+
1413
+ // Provide error recommendations based on error type
1414
+ const recommendation = getErrorRecommendation(error);
1415
+ if (recommendation) {
1416
+ log.warn('💡 [RECOMMENDATION]', { fileName, recommendation });
1417
+ }
1418
+
1419
+ results.failed++;
1420
+ results.errors.push(`${fileName}: ${error.message}`);
1421
+
1422
+ // Attempt to move to error directory, ignore failures
1423
+ try {
1424
+ log.info('🗂️ [ERROR] Moving failed file to error directory', { fileName });
1425
+ await s3.moveFile(filePath, `${errorPrefix}${fileName}`);
1426
+ log.info('✅ [ERROR] Failed file moved to error directory', {
1427
+ fileName,
1428
+ destination: `${errorPrefix}${fileName}`
1429
+ });
1430
+ } catch (moveError) {
1431
+ log.warn('⚠️ [ERROR] Failed to move file to error directory', {
1432
+ fileName,
1433
+ moveError: moveError instanceof Error ? moveError.message : String(moveError)
1434
+ });
1435
+ }
1436
+
1437
+ // Track error state with exponential backoff (only if file tracking enabled)
1438
+ if (enableFileTracking && stateService) {
1439
+ try {
1440
+ const stateKV = new VersoriKVAdapter(ctx);
1441
+ const key = ['error-state', fileName];
1442
+ const prev = (await stateKV.get(key))?.value as any;
1443
+ const attempts = (prev?.attemptCount || 0) + 1;
1444
+ const backoffMinutes = Math.min(Math.pow(2, attempts) * 5, 24 * 60);
1445
+ const nextRetryAt = new Date(Date.now() + backoffMinutes * 60000).toISOString();
1446
+
1447
+ await stateKV.set(key, {
1448
+ fileName,
1449
+ attemptCount: attempts,
1450
+ lastError: error?.message || 'unknown',
1451
+ lastAttemptAt: new Date().toISOString(),
1452
+ firstFailedAt: prev?.firstFailedAt || new Date().toISOString(),
1453
+ nextRetryAt,
1454
+ });
1455
+
1456
+ log.info('💾 [ERROR] Error state tracked with exponential backoff', {
1457
+ fileName,
1458
+ attempts,
1459
+ nextRetryAt
1460
+ });
1461
+ } catch (stateError) {
1462
+ log.warn('⚠️ [ERROR] Failed to track error state', {
1463
+ fileName,
1464
+ stateError: stateError instanceof Error ? stateError.message : String(stateError)
1465
+ });
1466
+ }
1467
+ }
1468
+ }
1469
+ }
1470
+
1471
+ log.info('🏁 [COMPLETE] File processing loop finished', {
1472
+ processed: results.processed,
1473
+ skipped: results.skipped,
1474
+ failed: results.failed,
1475
+ totalRecords: results.totalRecords
1476
+ });
1477
+
1478
+ return results;
1479
+ } catch (error: any) {
1480
+ // ✅ Enhanced error logging: Extract all error details for visibility
1481
+ const errorDetails = {
1482
+ message: error?.message || 'Unknown error',
1483
+ stack: error?.stack,
1484
+ errorType: error?.name || 'Error',
1485
+ };
1486
+
1487
+ log.error('❌ [FATAL] Location sync failed', errorDetails);
1488
+
1489
+ // Provide fatal error recommendations
1490
+ const recommendation = getErrorRecommendation(error);
1491
+ if (recommendation) {
1492
+ log.warn('💡 [RECOMMENDATION]', { recommendation });
1493
+ }
1494
+
1495
+ return {
1496
+ success: false,
1497
+ error: error.message,
1498
+ processed: 0,
1499
+ timestamp: new Date().toISOString(),
1500
+ };
1501
+ }
1502
+ }
1503
+
1504
+ /**
1505
+ * Get error recommendation based on error type
1506
+ */
1507
+ function getErrorRecommendation(error: any): string | null {
1508
+ const message = error?.message?.toLowerCase() || '';
1509
+
1510
+ if (message.includes('s3') || message.includes('access denied')) {
1511
+ return 'Check S3 credentials and bucket permissions. Verify IAM policy includes s3:ListBucket, s3:GetObject, s3:PutObject, s3:DeleteObject.';
1512
+ }
1513
+
1514
+ if (message.includes('xml') || message.includes('parse')) {
1515
+ return 'Verify XML file structure matches expected schema. Check for malformed XML or encoding issues.';
1516
+ }
1517
+
1518
+ if (message.includes('mapping') || message.includes('field')) {
1519
+ return 'Review field mapping configuration. Ensure all required fields are present and source paths are correct.';
1520
+ }
1521
+
1522
+ if (message.includes('graphql') || message.includes('mutation')) {
1523
+ return 'Check GraphQL schema and mutation input. Verify all required fields are provided and types match schema.';
1524
+ }
1525
+
1526
+ if (message.includes('auth') || message.includes('401') || message.includes('403')) {
1527
+ return 'Verify Fluent Commerce credentials. Check OAuth2 client ID/secret and ensure connection is active.';
1528
+ }
1529
+
1530
+ if (message.includes('timeout') || message.includes('econnrefused')) {
1531
+ return 'Check network connectivity. Verify API endpoints are accessible and not rate-limited.';
1532
+ }
1533
+
1534
+ return null;
1535
+ }
1536
+ ```
1537
+
1538
+ **Note:** The `executeLocationSync` function should contain the full implementation of `runLocationXmlWorkflow` (renamed to `executeLocationSync`), including all the logic for processing files, executing mutations, and logging. The implementation details are shown in the service function code above.
1539
+
1540
+ ---
1541
+
1542
+ ### Step 5: TypeScript Configuration
1543
+
1544
+ **File: tsconfig.json**
1545
+
1546
+ ```json
1547
+ {
1548
+ "compilerOptions": {
1549
+ "module": "ES2022",
1550
+ "target": "ES2024",
1551
+ "moduleResolution": "node"
1552
+ }
1553
+ }
1554
+ ```
1555
+
1556
+ ---
1557
+
1558
+ ## Code Flow Explanation
1559
+
1560
+ ### Initialization Phase
1561
+
1562
+ 1. **Read activation variables** - S3 config, Fluent config, rate limiting, logging options
1563
+ 2. **Validate required variables** - Fail fast if missing credentials
1564
+ 3. **Initialize SDK services** - FluentClient, S3DataSource, XMLParserService, StateService
1565
+ 4. **Create mapping configuration** - Define XML → GraphQL field mappings with nested objects
1566
+ 5. **Calculate rate limit delay** - Convert mutations/second to delay in milliseconds
1567
+
1568
+ ### File Discovery Phase
1569
+
1570
+ 1. **List S3 files** - Use `s3.listFiles()` with prefix filter (excludes processed/ subdirectory)
1571
+ 2. **Filter by pattern** - Match file extension (e.g., `.xml`)
1572
+ 3. **Sort newest-first** - Process most recent files first
1573
+ 4. **Apply max files limit** - Prevent overwhelming the workflow
1574
+ 5. **Note**: Files in `processed/` subdirectory won't be listed (primary deduplication)
1575
+
1576
+ ### Per-File Processing (Service Functions)
1577
+
1578
+ **Step 1: processFile()** - Download, parse, map
1579
+
1580
+ 1. **Download file** - Use S3DataSource with retry logic
1581
+ 2. **Parse XML** - XMLParserService converts to JavaScript object
1582
+ 3. **Normalize array** - Handle single vs multiple `<location>` elements (CRITICAL for XML)
1583
+ 4. **Map locations** - Use UniversalMapper with nested field mapping
1584
+ 5. **Collect errors** - Track mapping errors without stopping
1585
+ 6. **Return result** - FileProcessingResult with locations array and errors
1586
+
1587
+ **Step 2: executeMutations()** - GraphQL mutations with rate limiting
1588
+
1589
+ 1. **Loop through locations** - Process each location individually
1590
+ 2. **Build mutation input** - Extract nested fields (primaryAddress, openingSchedule)
1591
+ 3. **Execute mutation** - Direct GraphQL `createLocation` with retry logic
1592
+ 4. **Apply rate limiting** - Add configurable delay between mutations
1593
+ 5. **Track results** - Count successful vs failed, collect error messages
1594
+ 6. **Return result** - MutationResult with counts and errors
1595
+
1596
+ **Step 3: writeMutationLog()** - S3 log file (optional)
1597
+
1598
+ 1. **Create log entries** - Map locations to log entries with status
1599
+ 2. **Build JSON log** - Include timestamp, summary, detailed entries
1600
+ 3. **Write to S3** - Use Buffer.from() for Deno/Versori runtime compatibility
1601
+ 4. **Non-blocking** - Logging failure doesn't stop workflow
1602
+ 5. **Timestamped naming** - Unique log file per processed file
1603
+
1604
+ ### Cleanup Phase
1605
+
1606
+ 1. **Archive file FIRST** - Move to `processed/` or `errors/` (PRIMARY deduplication)
1607
+ 2. **Mark processed in KV** - StateService tracks metadata and processing history
1608
+ 3. **Error state tracking** - Store error info with exponential backoff timestamp
1609
+ 4. **Return results** - Summary of processed, skipped, failed files
1610
+
1611
+ **Note**: The order matters! Archive first (primary deduplication), then KV tracking (metadata/history).
1612
+
1613
+ ### Service Function Benefits
1614
+
1615
+ - **Composability**: Each function has single responsibility and can be reused
1616
+ - **Testability**: Service functions can be unit tested independently
1617
+ - **Clarity**: Main workflow shows high-level orchestration
1618
+ - **Error Handling**: Isolated error handling per service function
1619
+ - **Logging**: Detailed logging at each step with structured context
1620
+
1621
+ ---
1622
+
1623
+ ## S3 Archival Deduplication Pattern
1624
+
1625
+ This template uses **S3 archival** as the primary deduplication mechanism, NOT VersoriFileTracker.
1626
+
1627
+ ### How It Works
1628
+
1629
+ **S3 Directory Structure:**
1630
+
1631
+ ```
1632
+ s3://my-bucket/
1633
+ ├── locations/ ← listFiles() reads from here
1634
+ │ ├── new-file-1.xml
1635
+ │ └── new-file-2.xml
1636
+ ├── processed/ ← Successfully processed files
1637
+ │ ├── old-file-1.xml
1638
+ │ └── old-file-2.xml
1639
+ └── errors/ ← Failed files
1640
+ └── bad-file.xml
1641
+ ```
1642
+
1643
+ **Deduplication Flow:**
1644
+
1645
+ 1. **List files**: `s3.listFiles({ prefix: 'locations/' })` - Only lists `locations/` subdirectory
1646
+ 2. **Process file**: Download, parse, transform, send mutations
1647
+ 3. **Archive**: `s3.moveFile(filePath, 'processed/new-file-1.xml')` - Moves file out of `locations/`
1648
+ 4. **Next run**: File is now in `processed/`, won't be listed again
1649
+
1650
+ **Why This Works:**
1651
+
1652
+ - Files in `processed/` subdirectory are **never listed** when prefix is `locations/`
1653
+ - No need to track file state in KV store for deduplication
1654
+ - S3 is the single source of truth for file status
1655
+ - Simple, reliable, scales to millions of files
1656
+
1657
+ **StateService Role (Secondary):**
1658
+
1659
+ - Provides metadata and processing history
1660
+ - Backup check in case archival fails mid-process
1661
+ - Useful for monitoring and debugging
1662
+ - NOT the primary deduplication mechanism
1663
+
1664
+ **When to Use VersoriFileTracker:**
1665
+
1666
+ - **NEVER for S3 sources** - Use archival pattern instead
1667
+ - **Only for SFTP sources** - Where archival might not be possible
1668
+ - See SFTP templates for VersoriFileTracker usage
1669
+
1670
+ ---
1671
+
1672
+ ## XML Path Resolution Patterns
1673
+
1674
+ ### Pattern 1: XML Attribute Access with @ Prefix
1675
+
1676
+ ```typescript
1677
+ const mappingConfig = {
1678
+ fields: {
1679
+ // XML attribute access
1680
+ ref: { source: 'location.@ref' }, // <location ref="LOC-001">
1681
+ type: { source: 'location.@type' }, // <location type="WAREHOUSE">
1682
+ country: { source: 'location.address.@country' }, // <address country="USA">
1683
+
1684
+ // XML element text content
1685
+ name: { source: 'location.name' }, // <name>Downtown</name>
1686
+ city: { source: 'location.address.city' }, // <address><city>NYC</city></address>
1687
+ },
1688
+ };
1689
+ ```
1690
+
1691
+ ### Pattern 2: Handling Single vs Multiple Elements
1692
+
1693
+ ```typescript
1694
+ // XML can have single or multiple <location> elements
1695
+ const locationsData = xmlData.locations?.location;
1696
+
1697
+ // Normalize to array
1698
+ const locations = Array.isArray(locationsData) ? locationsData : [locationsData];
1699
+
1700
+ // Process each location
1701
+ for (const loc of locations) {
1702
+ const record = { location: loc }; // Wrap for mapping
1703
+ const result = await mapper.map(record);
1704
+ }
1705
+ ```
1706
+
1707
+ ### Pattern 3: Nested Object Mapping
1708
+
1709
+ ```typescript
1710
+ const mappingConfig = {
1711
+ fields: {
1712
+ // Root fields
1713
+ ref: { source: 'location.@ref', required: true },
1714
+ name: { source: 'location.name', required: true },
1715
+ type: { source: 'location.@type', required: true },
1716
+
1717
+ // Nested primaryAddress object
1718
+ 'primaryAddress.ref': { source: 'location.@ref' },
1719
+ 'primaryAddress.street': { source: 'location.address.street1' },
1720
+ 'primaryAddress.city': { source: 'location.address.city' },
1721
+ 'primaryAddress.latitude': { source: 'location.coordinates.@lat', resolver: 'sdk.parseFloat' },
1722
+
1723
+ // Nested openingSchedule object
1724
+ 'openingSchedule.allHours': {
1725
+ source: 'location.openingSchedule.allHours',
1726
+ resolver: 'sdk.boolean',
1727
+ },
1728
+ 'openingSchedule.monStart': {
1729
+ source: 'location.openingSchedule.monStart',
1730
+ resolver: 'sdk.parseInt',
1731
+ },
1732
+
1733
+ // Note: retailer.id not shown - standard createLocation does not have this field
1734
+ // If your schema requires it, add: 'retailer.id': { value: parseInt(retailerId) }
1735
+ },
1736
+ };
1737
+ ```
1738
+
1739
+ ---
1740
+
1741
+ ## Sample XML Files
1742
+
1743
+ ### Minimal Test File
1744
+
1745
+ **File: test-location.xml**
1746
+
1747
+ ```xml
1748
+ <?xml version="1.0" encoding="UTF-8"?>
1749
+ <locations>
1750
+ <location ref="TEST-001" type="WAREHOUSE">
1751
+ <name>Test Warehouse</name>
1752
+ <address country="USA">
1753
+ <street1>123 Test St</street1>
1754
+ <city>TestCity</city>
1755
+ <state>TC</state>
1756
+ <postalCode>12345</postalCode>
1757
+ </address>
1758
+ <coordinates lat="40.7128" lon="-74.0060"/>
1759
+ <timeZone>America/New_York</timeZone>
1760
+ <openingSchedule>
1761
+ <allHours>false</allHours>
1762
+ <monStart>800</monStart>
1763
+ <monEnd>1800</monEnd>
1764
+ <tueStart>800</tueStart>
1765
+ <tueEnd>1800</tueEnd>
1766
+ <wedStart>800</wedStart>
1767
+ <wedEnd>1800</wedEnd>
1768
+ <thuStart>800</thuStart>
1769
+ <thuEnd>1800</thuEnd>
1770
+ <friStart>800</friStart>
1771
+ <friEnd>1800</friEnd>
1772
+ <satStart>0</satStart>
1773
+ <satEnd>0</satEnd>
1774
+ <sunStart>0</sunStart>
1775
+ <sunEnd>0</sunEnd>
1776
+ </openingSchedule>
1777
+ </location>
1778
+ </locations>
1779
+ ```
1780
+
1781
+ ---
1782
+
1783
+ ## Service Functions Deep Dive
1784
+
1785
+ This template demonstrates service function composition for maintainable workflows.
1786
+
1787
+ ### Function 1: processFile()
1788
+
1789
+ **Purpose**: Download, parse, normalize, and map XML data
1790
+
1791
+ **Inputs**:
1792
+ - `s3: S3DataSource` - For file download
1793
+ - `parser: XMLParserService` - For XML parsing
1794
+ - `mapper: UniversalMapper` - For field mapping
1795
+ - `filePath: string` - S3 path to file
1796
+ - `fileName: string` - File name for logging
1797
+ - `log: any` - Logger instance
1798
+
1799
+ **Outputs**: `FileProcessingResult`
1800
+ ```typescript
1801
+ {
1802
+ success: boolean; // Overall success
1803
+ locations: any[]; // Mapped location objects
1804
+ errors: string[]; // Mapping error messages
1805
+ }
1806
+ ```
1807
+
1808
+ **Key Operations**:
1809
+ 1. Downloads file with retry logic
1810
+ 2. Parses XML using XMLParserService
1811
+ 3. **Normalizes arrays** - Handles single vs multiple `<location>` elements
1812
+ 4. Maps each location using UniversalMapper
1813
+ 5. Collects errors without stopping processing
1814
+ 6. Returns all mapped locations + errors
1815
+
1816
+ **Why separate function?**
1817
+ - Testable independently with mock data
1818
+ - Reusable across workflows (scheduled, webhook, adhoc)
1819
+ - Clear error boundaries - file-level errors vs record-level errors
1820
+ - Easy to add XML validation or schema checks
1821
+
1822
+ ### Function 2: executeMutations()
1823
+
1824
+ **Purpose**: Execute GraphQL createLocation mutations with rate limiting
1825
+
1826
+ **Inputs**:
1827
+ - `client: FluentClient` - For GraphQL mutations
1828
+ - `locations: any[]` - Mapped location data
1829
+ - `mutationDelayMs: number` - Rate limit delay
1830
+ - `fileName: string` - For logging context
1831
+ - `log: any` - Logger instance
1832
+
1833
+ **Outputs**: `MutationResult`
1834
+ ```typescript
1835
+ {
1836
+ successful: number; // Count of successful mutations
1837
+ failed: number; // Count of failed mutations
1838
+ errors: string[]; // Error messages with location refs
1839
+ }
1840
+ ```
1841
+
1842
+ **Key Operations**:
1843
+ 1. Loops through locations
1844
+ 2. Builds GraphQL mutation input
1845
+ 3. Executes with retry logic + rate limiting
1846
+ 4. Tracks success/failure per location
1847
+ 5. Returns summary counts + errors
1848
+
1849
+ **Why separate function?**
1850
+ - Clear separation: mapping vs mutation execution
1851
+ - Rate limiting logic isolated and configurable
1852
+ - Easy to swap mutation types (create vs update)
1853
+ - Testable with mock FluentClient
1854
+ - Can parallelize in future (batch mutations)
1855
+
1856
+ ### Function 3: writeMutationLog()
1857
+
1858
+ **Purpose**: Write detailed mutation results to S3 as JSON log
1859
+
1860
+ **Inputs**:
1861
+ - `s3: S3DataSource` - For log upload
1862
+ - `logEntries: MutationLogEntry[]` - Mutation status per location
1863
+ - `fileName: string` - Original file name
1864
+ - `logPrefix: string` - S3 prefix for logs (e.g., `logs/`)
1865
+ - `log: any` - Logger instance
1866
+
1867
+ **Outputs**: `void` (non-blocking - errors logged but not thrown)
1868
+
1869
+ **Key Operations**:
1870
+ 1. Creates timestamped log file name
1871
+ 2. Builds JSON log with summary + entries
1872
+ 3. **Uses Buffer.from()** - Required for Deno/Versori runtime
1873
+ 4. Writes to S3 with `uploadFile()`
1874
+ 5. Errors don't stop workflow (logging is non-critical)
1875
+
1876
+ **Why separate function?**
1877
+ - Optional feature - can be disabled via config
1878
+ - Non-blocking - logging failure doesn't fail workflow
1879
+ - Structured logging for audit trails
1880
+ - Easy to change log format (JSON, CSV, XML)
1881
+ - Can add log rotation/cleanup logic later
1882
+
1883
+ ### Service Function Composition Benefits
1884
+
1885
+ **1. Maintainability**
1886
+ ```typescript
1887
+ // Clear workflow orchestration
1888
+ const processingResult = await processFile(...);
1889
+ const mutationResult = await executeMutations(...);
1890
+ await writeMutationLog(...);
1891
+ ```
1892
+
1893
+ **2. Testability**
1894
+ ```typescript
1895
+ // Unit test processFile() with mock XML
1896
+ const mockS3 = { downloadFile: jest.fn() };
1897
+ const result = await processFile(mockS3, parser, mapper, ...);
1898
+ expect(result.locations).toHaveLength(5);
1899
+ ```
1900
+
1901
+ **3. Reusability**
1902
+ ```typescript
1903
+ // Use processFile() in different workflows
1904
+ export const webhook = webhook('location-webhook').then(async ctx => {
1905
+ const result = await processFile(s3, parser, mapper, filePath, fileName, log);
1906
+ return { locations: result.locations };
1907
+ });
1908
+ ```
1909
+
1910
+ **4. Error Isolation**
1911
+ ```typescript
1912
+ // Each function has clear error boundaries
1913
+ try {
1914
+ const processingResult = await processFile(...);
1915
+ // File-level errors caught here
1916
+ } catch (error) {
1917
+ // Handle file processing failure
1918
+ }
1919
+
1920
+ // Mutation errors don't stop file processing
1921
+ const mutationResult = await executeMutations(...);
1922
+ // mutationResult.errors contains per-location failures
1923
+ ```
1924
+
1925
+ **5. Progressive Enhancement**
1926
+ ```typescript
1927
+ // Easy to add features without touching core logic
1928
+ async function validateLocations(locations: any[]) {
1929
+ // Add validation step
1930
+ }
1931
+
1932
+ const processingResult = await processFile(...);
1933
+ await validateLocations(processingResult.locations); // New step
1934
+ const mutationResult = await executeMutations(...);
1935
+ ```
1936
+
1937
+ ---
1938
+
1939
+ ## Versori Environment Variables
1940
+
1941
+ **Activation Variables:**
1942
+
1943
+ ```bash
1944
+ # ============================================================================
1945
+ # Required Variables
1946
+ # ============================================================================
1947
+ s3BucketName=my-location-bucket
1948
+ awsAccessKeyId=AKIAXXXXXXXXXXXX
1949
+ awsSecretAccessKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
1950
+
1951
+ # ============================================================================
1952
+ # S3 Configuration (Optional - with defaults)
1953
+ # ============================================================================
1954
+ awsRegion=us-east-1
1955
+ s3Prefix=locations/
1956
+ archivePrefix=processed/
1957
+ errorPrefix=errors/
1958
+ logPrefix=logs/
1959
+ filePattern=.xml
1960
+ maxFilesToProcess=10
1961
+
1962
+ # ============================================================================
1963
+ # Feature Toggles (Optional - with defaults)
1964
+ # ============================================================================
1965
+ # Enable S3 archival (move files to processed/errors directories)
1966
+ enableArchival=true
1967
+
1968
+ # Enable mutation logs (write detailed mutation results to S3)
1969
+ enableMutationLogs=true
1970
+
1971
+ # Enable file tracking via StateService + KV store
1972
+ # When disabled, relies on S3 archival only for deduplication
1973
+ enableFileTracking=true
1974
+
1975
+ # ============================================================================
1976
+ # Mutation Configuration (Optional - with defaults)
1977
+ # ============================================================================
1978
+ # Mutation batch size (concurrent requests)
1979
+ # - 1 = Sequential (default, safest)
1980
+ # - 5 = Process 5 mutations in parallel
1981
+ # - 10 = Process 10 mutations in parallel
1982
+ mutationBatchSize=1
1983
+
1984
+ # Alias batching (combine multiple mutations into single request)
1985
+ # - undefined = Disabled (default, use separate requests)
1986
+ # - 5 = Combine 5 mutations per aliased request
1987
+ # - 10 = Combine 10 mutations per aliased request
1988
+ mutationsPerAliasBatch=
1989
+
1990
+ # ============================================================================
1991
+ # Fluent Commerce Configuration (Optional)
1992
+ # ============================================================================
1993
+ # Retailer ID - Only if mutation schema requires retailerId in input
1994
+ # Standard createLocation does NOT require this field
1995
+ # Check your GraphQL schema to determine if needed
1996
+ retailerId=my-retailer-id
1997
+ ```
1998
+
1999
+ **Notes:**
2000
+ - Webhook security is handled by Versori's native connection authentication. No manual API key configuration needed.
2001
+ - `retailerId` - Standard createLocation does not have this field. Only use if YOUR custom schema requires it.
2002
+ - See `fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md` for details.
2003
+
2004
+ ---
2005
+
2006
+ ## Schema Validation CLI Commands
2007
+
2008
+ Before deploying, validate your field mappings against the Fluent GraphQL schema:
2009
+
2010
+ ### 1. Introspect Schema
2011
+
2012
+ ```bash
2013
+ # Generate schema.json from live Fluent API
2014
+ npx fc-connect introspect-schema \
2015
+ --url https://api.fluentcommerce.com/graphql \
2016
+ --client-id your-client-id \
2017
+ --client-secret your-client-secret \
2018
+ --output schema.json
2019
+ ```
2020
+
2021
+ ### 2. Create Mapping Config File
2022
+
2023
+ **File: location-mapping.json**
2024
+
2025
+ ```json
2026
+ {
2027
+ "version": "1.0.0",
2028
+ "mutation": "createLocation",
2029
+ "sourceFormat": "xml",
2030
+ "returnFields": ["id", "ref", "name", "type", "status"],
2031
+ "description": "XML location to Fluent Commerce GraphQL mapping",
2032
+ "fields": {
2033
+ "ref": { "source": "location.@ref", "required": true, "resolver": "sdk.trim" },
2034
+ "name": { "source": "location.name", "required": true, "resolver": "sdk.trim" },
2035
+ "type": { "source": "location.@type", "required": true, "resolver": "sdk.uppercase" },
2036
+ "primaryAddress.ref": { "source": "location.@ref", "required": true },
2037
+ "primaryAddress.street": { "source": "location.address.street1" },
2038
+ "primaryAddress.latitude": {
2039
+ "source": "location.coordinates.@lat",
2040
+ "resolver": "sdk.parseFloat"
2041
+ },
2042
+ "primaryAddress.longitude": {
2043
+ "source": "location.coordinates.@lon",
2044
+ "resolver": "sdk.parseFloat"
2045
+ }
2046
+ }
2047
+ }
2048
+ ```
2049
+
2050
+ ### 3. Validate Mapping
2051
+
2052
+ ```bash
2053
+ # Validate that all target fields exist in schema
2054
+ npx fc-connect validate-schema \
2055
+ --mapping location-mapping.json \
2056
+ --schema schema.json
2057
+ ```
2058
+
2059
+ ### 4. Analyze Coverage
2060
+
2061
+ ```bash
2062
+ # Check which Location fields are mapped vs available
2063
+ npx fc-connect analyze-coverage \
2064
+ --mapping location-mapping.json \
2065
+ --schema schema.json \
2066
+ --type CreateLocationInput
2067
+ ```
2068
+
2069
+ **Output:**
2070
+
2071
+ ```
2072
+ ✅ Mapped: 15/42 fields (35%)
2073
+ ❌ Missing required: timezone (String!)
2074
+ ⚠️ Optional unmapped: supportPhoneNumber, networkId, attributes
2075
+ ```
2076
+
2077
+ ---
2078
+
2079
+ ## Testing Locally
2080
+
2081
+ ### 1. Upload Test XML to S3
2082
+
2083
+ ```bash
2084
+ aws s3 cp test-location.xml s3://my-location-bucket/locations/test-location.xml
2085
+ ```
2086
+
2087
+ ### 2. Deploy to Versori
2088
+
2089
+ ```bash
2090
+ npm run deploy
2091
+ ```
2092
+
2093
+ ### 3. Manual Testing
2094
+
2095
+ ```bash
2096
+ # Trigger manual sync (auth handled by Versori connection)
2097
+ curl -X POST https://your-workspace.versori.run/location-xml-adhoc
2098
+
2099
+ # Check job status
2100
+ curl -X POST https://your-workspace.versori.run/location-xml-job-status \
2101
+ -H "Content-Type: application/json" \
2102
+ -d '{"jobId": "location-xml-adhoc-1737525600000"}'
2103
+ ```
2104
+
2105
+ ### 4. Verify Processing
2106
+
2107
+ - Upload a small XML to S3 (2-3 locations) and trigger `adhoc` webhook
2108
+ - Verify GraphQL mutations are executed for each location
2109
+ - Confirm file moved from `locations/` to `processed/` in S3
2110
+ - Check KV state: errors and last processed metadata
2111
+ - Monitor rate limiting: verify mutations respect configured rate
2112
+
2113
+ ---
2114
+
2115
+ ## Deployment
2116
+
2117
+ ```bash
2118
+ # Deploy to Versori
2119
+ npm run deploy
2120
+
2121
+ # View logs
2122
+ npm run logs
2123
+
2124
+ # Monitor execution
2125
+ versori logs --follow
2126
+ ```
2127
+
2128
+ ---
2129
+
2130
+ ## Monitoring
2131
+
2132
+ ### Success Response
2133
+
2134
+ ```json
2135
+ {
2136
+ "success": true,
2137
+ "jobId": "location-xml-scheduled-1737525600000",
2138
+ "processed": 3,
2139
+ "skipped": 0,
2140
+ "failed": 0,
2141
+ "totalRecords": 12,
2142
+ "errors": []
2143
+ }
2144
+ ```
2145
+
2146
+ ### Partial Success Response
2147
+
2148
+ ```json
2149
+ {
2150
+ "success": true,
2151
+ "jobId": "location-xml-scheduled-1737525600000",
2152
+ "processed": 3,
2153
+ "skipped": 1,
2154
+ "failed": 0,
2155
+ "totalRecords": 10,
2156
+ "errors": ["locations-003.xml: 2 mapping errors"]
2157
+ }
2158
+ ```
2159
+
2160
+ ### Error Response
2161
+
2162
+ ```json
2163
+ {
2164
+ "success": false,
2165
+ "jobId": "location-xml-scheduled-1737525600000",
2166
+ "processed": 0,
2167
+ "skipped": 0,
2168
+ "failed": 1,
2169
+ "totalRecords": 0,
2170
+ "errors": ["locations-001.xml: Invalid XML structure"]
2171
+ }
2172
+ ```
2173
+
2174
+ ---
2175
+
2176
+ ## Common Pitfalls and Solutions
2177
+
2178
+ ### 1. XML Attribute Not Found
2179
+
2180
+ **Symptoms**: Mapping errors like "field not found"
2181
+
2182
+ **Solution**:
2183
+
2184
+ ```typescript
2185
+ // ❌ WRONG - Missing @ prefix for attribute
2186
+ ref: {
2187
+ source: 'location.ref';
2188
+ }
2189
+
2190
+ // ✅ CORRECT - Use @ prefix for XML attributes
2191
+ ref: {
2192
+ source: 'location.@ref';
2193
+ }
2194
+ ```
2195
+
2196
+ ### 2. Single Element Not Array
2197
+
2198
+ **Symptoms**: "Cannot read property forEach of undefined"
2199
+
2200
+ **Solution**:
2201
+
2202
+ ```typescript
2203
+ // Always normalize to array
2204
+ const locationsData = xmlData.locations?.location;
2205
+ const locations = Array.isArray(locationsData) ? locationsData : [locationsData];
2206
+ ```
2207
+
2208
+ ### 3. Rate Limiting Too Aggressive
2209
+
2210
+ **Symptoms**: Slow processing, mutations taking too long
2211
+
2212
+ **Solution**:
2213
+
2214
+ ```bash
2215
+ # Increase rate limit (mutations per second)
2216
+ mutationRateLimit=10 # Default is 5
2217
+ ```
2218
+
2219
+ ### 4. Empty Element vs Missing Element
2220
+
2221
+ **Solution**:
2222
+
2223
+ ```typescript
2224
+ // Use required: false and defaultValue for optional fields
2225
+ 'primaryAddress.street2': {
2226
+ source: 'location.address.street2',
2227
+ required: false,
2228
+ defaultValue: ''
2229
+ }
2230
+ ```
2231
+
2232
+ ### 5. S3 Access Denied
2233
+
2234
+ **Symptoms**: S3 operations fail with 403 errors
2235
+
2236
+ **Solution**: Validate IAM permissions
2237
+
2238
+ **Required IAM Permissions:**
2239
+
2240
+ ```json
2241
+ {
2242
+ "Version": "2012-10-17",
2243
+ "Statement": [
2244
+ {
2245
+ "Effect": "Allow",
2246
+ "Action": ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
2247
+ "Resource": ["arn:aws:s3:::my-location-bucket", "arn:aws:s3:::my-location-bucket/*"]
2248
+ }
2249
+ ]
2250
+ }
2251
+ ```
2252
+
2253
+ ### 6. GraphQL Schema Mismatch
2254
+
2255
+ **Symptoms**: Mutation errors like "Unknown field", "Invalid input type"
2256
+
2257
+ **Solution**: Use CLI tools to validate mappings
2258
+
2259
+ ```bash
2260
+ npx fc-connect validate-schema --mapping location-mapping.json --schema schema.json
2261
+ ```
2262
+
2263
+ ### 7. Nested Object Mapping Errors
2264
+
2265
+ **Symptoms**: Flat structure instead of nested objects in mutation input
2266
+
2267
+ **Solution**: Use dot notation in field mapping
2268
+
2269
+ ```typescript
2270
+ // ✅ CORRECT - Creates nested structure
2271
+ 'primaryAddress.city': { source: 'location.address.city' }
2272
+
2273
+ // ❌ WRONG - Creates flat structure
2274
+ primaryAddress_city: { source: 'location.address.city' }
2275
+ ```
2276
+
2277
+ ### 8. retailerId Configuration Errors
2278
+
2279
+ **Symptoms**: "retailerId is required" errors or confusion about when to use `setRetailerId()`
2280
+
2281
+ **Solution**: Understand the correct pattern
2282
+
2283
+ ```typescript
2284
+ // ✅ CORRECT - GraphQL mutations don't need setRetailerId()
2285
+ // Check your GraphQL schema to determine retailerId handling:
2286
+ // - Mandatory retailerId → Must pass it in mutation input
2287
+ // - Optional retailerId → Can pass it if needed
2288
+ // - No retailerId field → Don't pass it
2289
+ // Standard createLocation does not have retailerId field in schema
2290
+
2291
+ // ✅ IF mutation schema requires retailerId (mandatory):
2292
+ const { query, variables } = await mapper.map(location);
2293
+ if (retailerId && variables.input) {
2294
+ variables.input.retailer = { id: parseInt(retailerId) };
2295
+ }
2296
+ await client.graphql({ query, variables });
2297
+ ```
2298
+
2299
+ **Reference:** `fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md`
2300
+
2301
+ ---
2302
+
2303
+ ## Key Takeaways
2304
+
2305
+ ### Architecture Patterns
2306
+
2307
+ - **Service Function Composition**: Workflow broken into 3 service functions - `processFile()`, `executeMutations()`, `writeMutationLog()`
2308
+ - **Per-File Processing**: Main workflow orchestrates service functions for each file
2309
+ - **Clear Separation of Concerns**: Parse/map, execute mutations, logging are independent functions
2310
+
2311
+ ### SDK Usage
2312
+
2313
+ - **Buffer Import (CRITICAL)**: Always `import { Buffer } from 'node:buffer'` for Deno/Versori runtime
2314
+ - **S3 Archival Deduplication**: Use `s3.moveFile()` to `processed/` subdirectory - PRIMARY deduplication mechanism
2315
+ - **NO VersoriFileTracker for S3**: S3 archival is simpler and more reliable
2316
+ - **StateService Role**: SECONDARY - provides metadata/history, not primary deduplication
2317
+
2318
+ ### XML Processing
2319
+
2320
+ - **XML @ Prefix**: Always use `@` prefix for XML attributes (`location.@ref`)
2321
+ - **Array Normalization (CRITICAL)**: Handle single vs multiple elements with `Array.isArray()` check - single `<location>` becomes object, not array
2322
+ - **Nested Mapping**: Use dot notation for nested objects (`primaryAddress.street`, `openingSchedule.monStart`)
2323
+
2324
+ ### GraphQL Mutations
2325
+
2326
+ - **NO setRetailerId() Required**: GraphQL mutations do NOT need `client.setRetailerId()` - only Job/Event API needs it
2327
+ - **retailerId in Input**: Check your GraphQL schema to determine retailerId handling:
2328
+ - **Mandatory retailerId** - Field exists and is required (`!`) → Must pass it
2329
+ - **Optional retailerId** - Field exists and is optional → Can pass it if needed
2330
+ - **No retailerId field** - Field doesn't exist → Don't pass it
2331
+ - **Rate Limiting**: Implement configurable delays between mutations to avoid API throttling
2332
+ - **Retry Logic**: Exponential backoff for failed mutations with `retryWithBackoff()`
2333
+ - **Direct Mutations**: Use `client.graphql()` for location upserts (NOT Batch API)
2334
+ - **GraphQL vs Batch API**: Use GraphQL for low-volume master data, Batch API for high-volume inventory
2335
+
2336
+ ### Error Handling & Monitoring
2337
+
2338
+ - **Error Recovery**: Exponential backoff for error state tracking with retry timestamps
2339
+ - **Mutation Logging**: Optional S3 JSON logs with detailed per-location mutation status
2340
+ - **Schema Validation**: Use CLI tools before deployment to catch mapping errors
2341
+ - **Archival Order**: Archive FIRST (deduplication), then KV tracking (metadata)
2342
+
2343
+ ---
2344
+
2345
+ ## Related Documentation
2346
+
2347
+ ### Core Guides
2348
+
2349
+ - **GraphQL Mutation Mapping**: `fc-connect-sdk/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/readme.md`
2350
+ - **Universal Mapping**: `fc-connect-sdk/docs/02-CORE-GUIDES/mapping/modules/readme.md`
2351
+ - **XML Parser**: `fc-connect-sdk/docs/02-CORE-GUIDES/parsers/modules/05-xml-parser.md`
2352
+ - **Data Sources**: `fc-connect-sdk/docs/02-CORE-GUIDES/data-sources/readme.md`
2353
+ - **State Management**: `fc-connect-sdk/docs/03-PATTERN-GUIDES/file-operations/state-duplicate-prevention.md`
2354
+
2355
+ ### Related Templates
2356
+
2357
+ - **CSV Version**: `template-ingestion-s3-csv-location-graphql.md`
2358
+ - **JSON Version**: `template-ingestion-s3-json-location-graphql.md`
2359
+ - **SFTP XML Version**: `template-ingestion-sftp-xml-location-graphql.md`
2360
+ - **Event API Pattern**: `../event-api/template-ingestion-s3-xml-product-event.md`
2361
+ - **Batch API Pattern**: `../batch-api/template-ingestion-s3-xml-inventory-batch.md`
2362
+
2363
+ ### CLI Tools
2364
+
2365
+ - **Schema Introspection**: `fc-connect-sdk/bin/readme.md#introspect-schema`
2366
+ - **Mapping Validation**: `fc-connect-sdk/bin/readme.md#validate-schema`
2367
+ - **Coverage Analysis**: `fc-connect-sdk/bin/readme.md#analyze-coverage`
2368
+
2369
+ ### Patterns
2370
+
2371
+ - **Error Handling**: `fc-connect-sdk/docs/01-TEMPLATES/patterns/error-handling-retry.md`
2372
+ - **Rate Limiting**: `fc-connect-sdk/docs/03-PATTERN-GUIDES/integration-patterns/rate-limiting.md`
2373
+ - **XML Patterns**: `fc-connect-sdk/docs/01-TEMPLATES/versori/patterns/xml-response-patterns.md`