@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.55

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (475) hide show
  1. package/CHANGELOG.md +12 -0
  2. package/dist/cjs/clients/fluent-client.js +13 -6
  3. package/dist/cjs/utils/pagination-helpers.js +38 -2
  4. package/dist/cjs/versori/fluent-versori-client.js +11 -5
  5. package/dist/esm/clients/fluent-client.js +13 -6
  6. package/dist/esm/utils/pagination-helpers.js +38 -2
  7. package/dist/esm/versori/fluent-versori-client.js +11 -5
  8. package/dist/tsconfig.esm.tsbuildinfo +1 -1
  9. package/dist/tsconfig.tsbuildinfo +1 -1
  10. package/dist/tsconfig.types.tsbuildinfo +1 -1
  11. package/docs/00-START-HERE/EXPORT-VALIDATION.md +158 -158
  12. package/docs/00-START-HERE/cli-analyze-source-structure-guide.md +655 -655
  13. package/docs/00-START-HERE/cli-documentation-index.md +202 -202
  14. package/docs/00-START-HERE/cli-quick-reference.md +252 -252
  15. package/docs/00-START-HERE/decision-tree.md +552 -552
  16. package/docs/00-START-HERE/getting-started.md +1070 -1070
  17. package/docs/00-START-HERE/mapper-quick-decision-guide.md +235 -235
  18. package/docs/00-START-HERE/readme.md +237 -237
  19. package/docs/00-START-HERE/retailerid-configuration.md +404 -404
  20. package/docs/00-START-HERE/sdk-philosophy.md +794 -794
  21. package/docs/00-START-HERE/troubleshooting-quick-reference.md +1086 -1086
  22. package/docs/01-TEMPLATES/faq.md +686 -686
  23. package/docs/01-TEMPLATES/patterns/pattern-templates-guide.md +68 -68
  24. package/docs/01-TEMPLATES/patterns/patterns-csv-schema-validation-and-rejection-report.md +233 -233
  25. package/docs/01-TEMPLATES/patterns/patterns-custom-resolvers.md +407 -407
  26. package/docs/01-TEMPLATES/patterns/patterns-error-handling-retry.md +511 -511
  27. package/docs/01-TEMPLATES/patterns/patterns-field-mapping-universal.md +701 -701
  28. package/docs/01-TEMPLATES/patterns/patterns-large-file-splitting.md +1430 -1430
  29. package/docs/01-TEMPLATES/patterns/patterns-master-data-etl.md +2399 -2399
  30. package/docs/01-TEMPLATES/patterns/patterns-pagination-streaming.md +447 -447
  31. package/docs/01-TEMPLATES/patterns/patterns-state-duplicate-prevention.md +385 -385
  32. package/docs/01-TEMPLATES/readme.md +957 -957
  33. package/docs/01-TEMPLATES/standalone/standalone-asn-inbound-processing.md +1209 -1209
  34. package/docs/01-TEMPLATES/standalone/standalone-graphql-query-export.md +1140 -1140
  35. package/docs/01-TEMPLATES/standalone/standalone-graphql-to-parquet-partitioned-s3.md +432 -432
  36. package/docs/01-TEMPLATES/standalone/standalone-multi-channel-inventory-sync.md +1185 -1185
  37. package/docs/01-TEMPLATES/standalone/standalone-multi-source-aggregation.md +1462 -1462
  38. package/docs/01-TEMPLATES/standalone/standalone-s3-csv-batch-api.md +1390 -1390
  39. package/docs/01-TEMPLATES/standalone/standalone-s3-csv-inventory-to-batch.md +330 -330
  40. package/docs/01-TEMPLATES/standalone/standalone-scripts-guide.md +87 -87
  41. package/docs/01-TEMPLATES/standalone/standalone-sftp-xml-graphql.md +1444 -1444
  42. package/docs/01-TEMPLATES/standalone/standalone-webhook-payload-processing.md +688 -688
  43. package/docs/01-TEMPLATES/versori/business-examples/business-examples-dropship-order-routing.md +193 -193
  44. package/docs/01-TEMPLATES/versori/business-examples/business-examples-graphql-parquet-extraction.md +518 -518
  45. package/docs/01-TEMPLATES/versori/business-examples/business-examples-inter-location-transfers.md +2162 -2162
  46. package/docs/01-TEMPLATES/versori/business-examples/business-examples-pre-order-allocation.md +2226 -2226
  47. package/docs/01-TEMPLATES/versori/business-examples/business-scenarios-guide.md +87 -87
  48. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-connection-validation-pattern.md +656 -656
  49. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-dual-workflow-connector.md +835 -835
  50. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-guide.md +108 -108
  51. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-kv-state-management.md +1533 -1533
  52. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-xml-response-patterns.md +1160 -1160
  53. package/docs/01-TEMPLATES/versori/versori-platform-guide.md +201 -201
  54. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-asn-purchase-order.md +1906 -1906
  55. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-dropship-routing.md +1074 -1074
  56. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-flash-sale-reserve.md +1395 -1395
  57. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-generic-xml-order.md +888 -888
  58. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-payment-gateway-integration.md +2478 -2478
  59. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-rma-returns-comprehensive.md +2240 -2240
  60. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-xml-order-ingestion.md +2029 -2029
  61. package/docs/01-TEMPLATES/versori/webhooks/webhook-templates-guide.md +140 -140
  62. package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/inventory-mapping.json +20 -20
  63. package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/products_2025-01-22.csv +11 -11
  64. package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/sample-data-guide.md +34 -34
  65. package/docs/01-TEMPLATES/versori/workflows/_examples/workflow-examples-guide.md +36 -36
  66. package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-modes-guide.md +1038 -1038
  67. package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-workflows-guide.md +138 -138
  68. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/graphql-extraction-guide.md +63 -63
  69. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-csv.md +2062 -2062
  70. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-xml.md +2294 -2294
  71. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-s3-csv.md +2461 -2461
  72. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-sftp-xml.md +2529 -2529
  73. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-csv.md +2464 -2464
  74. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-json.md +1959 -1959
  75. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-s3-csv.md +1953 -1953
  76. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-sftp-xml.md +2541 -2541
  77. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-s3-json.md +2384 -2384
  78. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-sftp-xml.md +2445 -2445
  79. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-csv.md +2355 -2355
  80. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-json.md +2042 -2042
  81. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-sftp-xml.md +2726 -2726
  82. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/batch-api-guide.md +206 -206
  83. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-cycle-count-reconciliation.md +2030 -2030
  84. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-multi-channel-inventory-sync.md +1882 -1882
  85. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-csv-inventory-batch.md +2827 -2827
  86. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-json-inventory-batch.md +1952 -1952
  87. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-xml-inventory-batch.md +3289 -3289
  88. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-csv-inventory-batch.md +3064 -3064
  89. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-json-inventory-batch.md +3238 -3238
  90. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-xml-inventory-batch.md +2977 -2977
  91. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/event-api-guide.md +321 -321
  92. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-json-order-cancel-event.md +959 -959
  93. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-xml-order-cancel-event.md +1170 -1170
  94. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-csv-product-event.md +2312 -2312
  95. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-json-product-event.md +2999 -2999
  96. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-parquet-product-event.md +2836 -2836
  97. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-xml-product-event.md +2395 -2395
  98. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-csv-product-event.md +2295 -2295
  99. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-json-product-event.md +2602 -2602
  100. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-parquet-product-event.md +2589 -2589
  101. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-xml-product-event.md +3578 -3578
  102. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/graphql-mutations-guide.md +93 -93
  103. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-json-order-update-graphql.md +1260 -1260
  104. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-xml-order-update-graphql.md +1472 -1472
  105. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-control-graphql.md +2417 -2417
  106. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-location-graphql.md +2811 -2811
  107. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-price-graphql.md +2619 -2619
  108. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-json-location-graphql.md +2807 -2807
  109. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-xml-location-graphql.md +2373 -2373
  110. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-control-graphql.md +2740 -2740
  111. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-location-graphql.md +2760 -2760
  112. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-json-location-graphql.md +1710 -1710
  113. package/docs/01-TEMPLATES/versori/workflows/ingestion/ingestion-workflows-guide.md +136 -136
  114. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/rubix-webhooks-guide.md +520 -520
  115. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-inline.md +1418 -1418
  116. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-universal-mapper.md +1785 -1785
  117. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-order-attribute-update.md +824 -824
  118. package/docs/01-TEMPLATES/versori/workflows/workflows-overview-guide.md +646 -646
  119. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-batch-archival.md +724 -724
  120. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-job-tracker.md +627 -627
  121. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-partial-batch-recovery.md +561 -561
  122. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-quick-reference.md +367 -367
  123. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-readme.md +407 -407
  124. package/docs/02-CORE-GUIDES/advanced-services/readme.md +49 -49
  125. package/docs/02-CORE-GUIDES/api-reference/api-reference-quick-reference.md +548 -548
  126. package/docs/02-CORE-GUIDES/api-reference/event-api-input-output-reference.md +702 -1171
  127. package/docs/02-CORE-GUIDES/api-reference/examples/client-initialization.ts +286 -286
  128. package/docs/02-CORE-GUIDES/api-reference/graphql-error-classification.md +337 -337
  129. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-01-client-api.md +399 -520
  130. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-03-authentication.md +199 -199
  131. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-04-graphql-mapping.md +925 -925
  132. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-05-services.md +1198 -1198
  133. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-06-data-sources.md +1083 -1083
  134. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-07-parsers.md +1097 -1097
  135. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-pagination.md +513 -513
  136. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-types.md +545 -597
  137. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-error-handling.md +527 -527
  138. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-webhook-validation.md +514 -514
  139. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-extraction.md +557 -557
  140. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-utilities.md +412 -412
  141. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-cli-tools.md +423 -423
  142. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-error-handling.md +716 -716
  143. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-analyze-source-structure.md +518 -518
  144. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-partial-responses.md +212 -212
  145. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-testing.md +300 -300
  146. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-13-resolver-builder.md +322 -322
  147. package/docs/02-CORE-GUIDES/api-reference/readme.md +279 -279
  148. package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-quick-reference.md +351 -351
  149. package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-readme.md +277 -277
  150. package/docs/02-CORE-GUIDES/auto-pagination/examples/auto-pagination-readme.md +178 -178
  151. package/docs/02-CORE-GUIDES/auto-pagination/examples/common-patterns.ts +351 -351
  152. package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-products.ts +384 -384
  153. package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-virtual-positions.ts +308 -308
  154. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-01-foundations.md +470 -470
  155. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-02-quick-start.md +713 -713
  156. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-03-configuration.md +754 -754
  157. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-04-advanced-patterns.md +732 -732
  158. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-05-sdk-integration.md +847 -847
  159. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-06-troubleshooting.md +359 -359
  160. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-07-api-reference.md +462 -462
  161. package/docs/02-CORE-GUIDES/auto-pagination/readme.md +54 -54
  162. package/docs/02-CORE-GUIDES/data-sources/data-sources-file-operations-error-handling.md +1487 -1487
  163. package/docs/02-CORE-GUIDES/data-sources/data-sources-quick-reference.md +836 -836
  164. package/docs/02-CORE-GUIDES/data-sources/data-sources-readme.md +276 -276
  165. package/docs/02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md +553 -553
  166. package/docs/02-CORE-GUIDES/data-sources/examples/common-patterns.ts +409 -409
  167. package/docs/02-CORE-GUIDES/data-sources/examples/data-sources-readme.md +178 -178
  168. package/docs/02-CORE-GUIDES/data-sources/examples/s3-operations.ts +308 -308
  169. package/docs/02-CORE-GUIDES/data-sources/examples/sftp-operations.ts +371 -371
  170. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-01-foundations.md +735 -735
  171. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-02-s3-operations.md +1302 -1302
  172. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-03-sftp-operations.md +1379 -1379
  173. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-04-file-patterns.md +941 -941
  174. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-05-advanced-topics.md +813 -813
  175. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-06-integration-patterns.md +486 -486
  176. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-07-troubleshooting.md +387 -387
  177. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-08-api-reference.md +417 -417
  178. package/docs/02-CORE-GUIDES/data-sources/readme.md +77 -77
  179. package/docs/02-CORE-GUIDES/error-handling-guide.md +936 -936
  180. package/docs/02-CORE-GUIDES/extraction/examples/02-core-guides-extraction-readme.md +116 -116
  181. package/docs/02-CORE-GUIDES/extraction/examples/common-patterns.ts +428 -428
  182. package/docs/02-CORE-GUIDES/extraction/examples/extract-inventory-basic.ts +187 -187
  183. package/docs/02-CORE-GUIDES/extraction/extraction-quick-reference.md +596 -596
  184. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-01-foundations.md +514 -514
  185. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-02-basic-extraction.md +823 -823
  186. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-03-parquet-processing.md +507 -507
  187. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-04-data-enrichment.md +546 -546
  188. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-05-transformation.md +494 -494
  189. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-export-formats.md +458 -458
  190. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-performance.md +138 -138
  191. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-api-reference.md +148 -148
  192. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-optimization.md +692 -692
  193. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-08-extraction-orchestrator.md +1008 -1008
  194. package/docs/02-CORE-GUIDES/extraction/readme.md +151 -151
  195. package/docs/02-CORE-GUIDES/ingestion/examples/_simple-kv-store.ts +40 -40
  196. package/docs/02-CORE-GUIDES/ingestion/examples/error-recovery.ts +728 -728
  197. package/docs/02-CORE-GUIDES/ingestion/examples/event-driven.ts +501 -501
  198. package/docs/02-CORE-GUIDES/ingestion/examples/local-file-ingestion.ts +88 -88
  199. package/docs/02-CORE-GUIDES/ingestion/examples/parquet-ingestion.ts +117 -117
  200. package/docs/02-CORE-GUIDES/ingestion/examples/performance-optimized.ts +647 -647
  201. package/docs/02-CORE-GUIDES/ingestion/examples/s3-csv-ingestion.ts +169 -169
  202. package/docs/02-CORE-GUIDES/ingestion/examples/sftp-csv-ingestion.ts +134 -134
  203. package/docs/02-CORE-GUIDES/ingestion/ingestion-quick-reference.md +546 -546
  204. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-01-introduction.md +626 -626
  205. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-02-quick-start.md +658 -658
  206. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-03-data-sources.md +1052 -1052
  207. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-04-field-mapping.md +763 -763
  208. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-05-advanced-parsers.md +676 -676
  209. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-06-batch-api.md +1295 -1295
  210. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-api-reference.md +138 -138
  211. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-state-management.md +1037 -1037
  212. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-08-performance-optimization.md +1349 -1349
  213. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-09-best-practices.md +1893 -1893
  214. package/docs/02-CORE-GUIDES/ingestion/readme.md +160 -160
  215. package/docs/02-CORE-GUIDES/logging-guide.md +585 -585
  216. package/docs/02-CORE-GUIDES/mapping/error-handling-patterns.md +401 -401
  217. package/docs/02-CORE-GUIDES/mapping/examples/02-core-guides-mapping-readme.md +128 -128
  218. package/docs/02-CORE-GUIDES/mapping/examples/common-patterns.ts +273 -273
  219. package/docs/02-CORE-GUIDES/mapping/examples/csv-location-ingestion.json +36 -36
  220. package/docs/02-CORE-GUIDES/mapping/examples/csv-mapping.ts +242 -242
  221. package/docs/02-CORE-GUIDES/mapping/examples/graphql-to-parquet-extraction.json +36 -36
  222. package/docs/02-CORE-GUIDES/mapping/examples/json-mapping.ts +213 -213
  223. package/docs/02-CORE-GUIDES/mapping/examples/json-product-to-mutation.json +48 -48
  224. package/docs/02-CORE-GUIDES/mapping/examples/xml-mapping.ts +291 -291
  225. package/docs/02-CORE-GUIDES/mapping/examples/xml-order-to-mutation.json +45 -45
  226. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-quick-reference.md +463 -463
  227. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-readme.md +227 -227
  228. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-01-introduction.md +222 -222
  229. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-02-quick-start.md +351 -351
  230. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-03-schema-validation.md +569 -569
  231. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-04-mapping-patterns.md +471 -471
  232. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-05-configuration-reference.md +611 -611
  233. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-advanced-xpath.md +148 -148
  234. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-path-syntax.md +464 -464
  235. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-api-reference.md +94 -94
  236. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-array-handling.md +307 -307
  237. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-08-custom-resolvers.md +544 -544
  238. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-09-advanced-patterns.md +427 -427
  239. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-10-hooks-and-variables.md +336 -336
  240. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-11-error-handling.md +488 -488
  241. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-12-arguments-vs-nodes.md +383 -383
  242. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-13-best-practices.md +477 -477
  243. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/readme.md +62 -62
  244. package/docs/02-CORE-GUIDES/mapping/mapping-format-decision-tree.md +480 -480
  245. package/docs/02-CORE-GUIDES/mapping/mapping-graphql-alias-batching-guide.md +820 -820
  246. package/docs/02-CORE-GUIDES/mapping/mapping-javascript-objects.md +2369 -2369
  247. package/docs/02-CORE-GUIDES/mapping/mapping-mapper-comparison-guide.md +682 -682
  248. package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-07-api-reference.md +1327 -1327
  249. package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-08-error-handling.md +1142 -1142
  250. package/docs/02-CORE-GUIDES/mapping/modules/mapping-04-use-cases.md +891 -891
  251. package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-helpers-resolvers.md +1126 -1126
  252. package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-sdk-resolvers.md +199 -199
  253. package/docs/02-CORE-GUIDES/mapping/modules/mapping-07-api-reference.md +1319 -1319
  254. package/docs/02-CORE-GUIDES/mapping/readme.md +178 -178
  255. package/docs/02-CORE-GUIDES/mapping/resolver-registration.md +410 -410
  256. package/docs/02-CORE-GUIDES/mapping/resolvers/examples/common-patterns.ts +226 -226
  257. package/docs/02-CORE-GUIDES/mapping/resolvers/examples/custom-resolvers.ts +227 -227
  258. package/docs/02-CORE-GUIDES/mapping/resolvers/examples/sdk-resolvers-usage.ts +203 -203
  259. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-readme.md +274 -274
  260. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-api-reference.md +679 -679
  261. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-cookbook.md +826 -826
  262. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-guide.md +1330 -1330
  263. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-helpers-reference.md +1437 -1437
  264. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-parameters-reference.md +553 -553
  265. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-troubleshooting.md +854 -854
  266. package/docs/02-CORE-GUIDES/mapping/resolvers/readme.md +75 -75
  267. package/docs/02-CORE-GUIDES/parsers/examples/02-core-guides-parsers-readme.md +161 -161
  268. package/docs/02-CORE-GUIDES/parsers/examples/csv-parser-examples.ts +110 -110
  269. package/docs/02-CORE-GUIDES/parsers/examples/json-parser-examples.ts +33 -33
  270. package/docs/02-CORE-GUIDES/parsers/examples/parquet-parser-examples.ts +47 -47
  271. package/docs/02-CORE-GUIDES/parsers/examples/xml-parser-examples.ts +38 -38
  272. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-01-foundations.md +355 -355
  273. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-02-csv-parser.md +772 -772
  274. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-03-json-parser.md +789 -789
  275. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-04-xml-parser.md +857 -857
  276. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-05-parquet-parser.md +603 -603
  277. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-integration-patterns.md +702 -702
  278. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-streaming.md +121 -121
  279. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-api-reference.md +89 -89
  280. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-troubleshooting.md +727 -727
  281. package/docs/02-CORE-GUIDES/parsers/parsers-quick-reference.md +482 -482
  282. package/docs/02-CORE-GUIDES/parsers/parsers-readme.md +258 -258
  283. package/docs/02-CORE-GUIDES/parsers/readme.md +65 -65
  284. package/docs/02-CORE-GUIDES/readme.md +194 -194
  285. package/docs/02-CORE-GUIDES/webhook-validation/examples/basic-validation.ts +108 -108
  286. package/docs/02-CORE-GUIDES/webhook-validation/examples/common-patterns.ts +316 -316
  287. package/docs/02-CORE-GUIDES/webhook-validation/examples/webhook-validation-readme.md +61 -61
  288. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-01-foundations.md +440 -440
  289. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-02-quick-start.md +525 -525
  290. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-03-versori-integration.md +741 -741
  291. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-04-platform-integration.md +629 -629
  292. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-05-configuration.md +535 -535
  293. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-error-handling.md +611 -611
  294. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-troubleshooting.md +124 -124
  295. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-07-api-reference.md +511 -511
  296. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-08-rubix-webhooks.md +590 -590
  297. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-09-rubix-event-vs-http-call.md +432 -432
  298. package/docs/02-CORE-GUIDES/webhook-validation/readme.md +239 -239
  299. package/docs/02-CORE-GUIDES/webhook-validation/webhook-validation-quick-reference.md +392 -392
  300. package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-quick-reference.md +498 -498
  301. package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-readme.md +313 -313
  302. package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/common-patterns.ts +612 -612
  303. package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/connector-scenarios-readme.md +253 -253
  304. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-01-foundations.md +452 -452
  305. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-02-simple-scenarios.md +681 -681
  306. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-03-intermediate-scenarios.md +637 -637
  307. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-04-advanced-scenarios.md +650 -650
  308. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-05-bidirectional-sync.md +233 -233
  309. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-06-production-patterns.md +442 -442
  310. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-07-reference.md +445 -445
  311. package/docs/03-PATTERN-GUIDES/connector-scenarios/readme.md +31 -31
  312. package/docs/03-PATTERN-GUIDES/enterprise-integration-patterns.md +1528 -1528
  313. package/docs/03-PATTERN-GUIDES/error-handling/comprehensive-error-handling-guide.md +1437 -1437
  314. package/docs/03-PATTERN-GUIDES/error-handling/error-handling-quick-reference.md +390 -390
  315. package/docs/03-PATTERN-GUIDES/error-handling/examples/common-patterns.ts +438 -438
  316. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-01-foundations.md +362 -362
  317. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-02-error-types.md +850 -850
  318. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-03-utf8-handling.md +456 -456
  319. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-04-error-scenarios.md +658 -658
  320. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-05-calling-patterns.md +671 -671
  321. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-06-retry-strategies.md +1034 -1034
  322. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-07-monitoring.md +653 -653
  323. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-08-api-reference.md +847 -847
  324. package/docs/03-PATTERN-GUIDES/error-handling/readme.md +36 -36
  325. package/docs/03-PATTERN-GUIDES/examples/__tests__/readme.md +40 -40
  326. package/docs/03-PATTERN-GUIDES/examples/__tests__/resolver-examples.test.js +282 -282
  327. package/docs/03-PATTERN-GUIDES/examples/test-data/03-pattern-guides-readme.md +110 -110
  328. package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-inventory.json +123 -123
  329. package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-order.json +171 -171
  330. package/docs/03-PATTERN-GUIDES/examples/test-data/readme.md +28 -28
  331. package/docs/03-PATTERN-GUIDES/extraction/extraction-readme.md +15 -15
  332. package/docs/03-PATTERN-GUIDES/extraction/readme.md +25 -25
  333. package/docs/03-PATTERN-GUIDES/file-operations/examples/common-patterns.ts +407 -407
  334. package/docs/03-PATTERN-GUIDES/file-operations/examples/file-operations-readme.md +142 -142
  335. package/docs/03-PATTERN-GUIDES/file-operations/file-operations-quick-reference.md +462 -462
  336. package/docs/03-PATTERN-GUIDES/file-operations/file-operations-readme.md +379 -379
  337. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-01-foundations.md +430 -430
  338. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-02-quick-start.md +484 -484
  339. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-03-s3-operations.md +507 -507
  340. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-04-sftp-operations.md +963 -963
  341. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-05-streaming-performance.md +503 -503
  342. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-archive-patterns.md +386 -386
  343. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-error-handling.md +117 -117
  344. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-api-reference.md +78 -78
  345. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-testing-troubleshooting.md +567 -567
  346. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-08-api-reference.md +1055 -1055
  347. package/docs/03-PATTERN-GUIDES/file-operations/readme.md +32 -32
  348. package/docs/03-PATTERN-GUIDES/ingestion/ingestion-readme.md +15 -15
  349. package/docs/03-PATTERN-GUIDES/ingestion/readme.md +25 -25
  350. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/batch-processing.ts +130 -130
  351. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/common-patterns.ts +360 -360
  352. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/delta-sync.ts +130 -130
  353. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/integration-patterns-readme.md +100 -100
  354. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/real-time-webhook.ts +398 -398
  355. package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-quick-reference.md +962 -962
  356. package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-readme.md +134 -134
  357. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-01-real-time-processing.md +991 -991
  358. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-02-batch-processing.md +1547 -1547
  359. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-03-delta-sync.md +1108 -1108
  360. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-04-webhook-patterns.md +1181 -1181
  361. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-05-error-handling.md +1061 -1061
  362. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-advanced-integration-services.md +1547 -1547
  363. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-performance.md +109 -109
  364. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-07-api-reference.md +34 -34
  365. package/docs/03-PATTERN-GUIDES/integration-patterns/readme.md +30 -30
  366. package/docs/03-PATTERN-GUIDES/logging-minimal-mode.md +128 -128
  367. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/common-patterns.ts +380 -380
  368. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/multiple-connections-readme.md +139 -139
  369. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/parallel-root-connections.ts +149 -149
  370. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/real-world-scenarios.ts +405 -405
  371. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-01-foundations.md +378 -378
  372. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-02-quick-start.md +566 -566
  373. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-03-targeting-connections.md +659 -659
  374. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-04-parallel-queries.md +656 -656
  375. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-05-best-practices.md +624 -624
  376. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-api-reference.md +824 -824
  377. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-versori.md +119 -119
  378. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-07-api-reference.md +87 -87
  379. package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-quick-reference.md +353 -353
  380. package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-readme.md +270 -270
  381. package/docs/03-PATTERN-GUIDES/multiple-connections/readme.md +30 -30
  382. package/docs/03-PATTERN-GUIDES/pagination/pagination-readme.md +14 -14
  383. package/docs/03-PATTERN-GUIDES/pagination/readme.md +24 -24
  384. package/docs/03-PATTERN-GUIDES/parquet/examples/common-patterns.ts +180 -180
  385. package/docs/03-PATTERN-GUIDES/parquet/examples/read-parquet.ts +48 -48
  386. package/docs/03-PATTERN-GUIDES/parquet/examples/write-parquet.ts +65 -65
  387. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-01-introduction.md +393 -393
  388. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-02-quick-start.md +572 -572
  389. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-03-reading-parquet.md +525 -525
  390. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-04-writing-parquet.md +554 -554
  391. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-05-graphql-extraction.md +405 -405
  392. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-performance.md +104 -104
  393. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-s3-integration.md +511 -511
  394. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-api-reference.md +90 -90
  395. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-performance-optimization.md +525 -525
  396. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-08-best-practices.md +712 -712
  397. package/docs/03-PATTERN-GUIDES/parquet/parquet-quick-reference.md +683 -683
  398. package/docs/03-PATTERN-GUIDES/parquet/parquet-readme.md +248 -248
  399. package/docs/03-PATTERN-GUIDES/parquet/readme.md +32 -32
  400. package/docs/03-PATTERN-GUIDES/parsers/parsers-readme.md +12 -12
  401. package/docs/03-PATTERN-GUIDES/parsers/readme.md +24 -24
  402. package/docs/03-PATTERN-GUIDES/readme.md +159 -159
  403. package/docs/03-PATTERN-GUIDES/webhooks/readme.md +24 -24
  404. package/docs/03-PATTERN-GUIDES/webhooks/webhooks-readme.md +8 -8
  405. package/docs/04-REFERENCE/architecture/architecture-01-overview.md +427 -427
  406. package/docs/04-REFERENCE/architecture/architecture-02-client-architecture.md +424 -424
  407. package/docs/04-REFERENCE/architecture/architecture-03-data-flow.md +690 -690
  408. package/docs/04-REFERENCE/architecture/architecture-04-service-layer.md +834 -834
  409. package/docs/04-REFERENCE/architecture/architecture-05-integration-architecture.md +655 -655
  410. package/docs/04-REFERENCE/architecture/architecture-06-state-management.md +653 -653
  411. package/docs/04-REFERENCE/architecture/architecture-adding-new-data-sources.md +686 -686
  412. package/docs/04-REFERENCE/architecture/readme.md +279 -279
  413. package/docs/04-REFERENCE/platforms/deno/readme.md +117 -117
  414. package/docs/04-REFERENCE/platforms/nodejs/readme.md +146 -146
  415. package/docs/04-REFERENCE/platforms/readme.md +135 -135
  416. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-01-introduction.md +398 -398
  417. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-02-quick-start.md +560 -560
  418. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-03-authentication.md +757 -757
  419. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-04-workflows.md +2476 -2476
  420. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-05-connections.md +1167 -1167
  421. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-kv-storage.md +990 -990
  422. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-state-management.md +121 -121
  423. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-api-reference.md +68 -68
  424. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-deployment.md +731 -731
  425. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-08-best-practices.md +1111 -1111
  426. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-09-signature-reference.md +766 -766
  427. package/docs/04-REFERENCE/platforms/versori/platforms-versori-readme.md +299 -299
  428. package/docs/04-REFERENCE/platforms/versori/platforms-versori-s3-sftp-configuration-guide.md +1425 -1425
  429. package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-api-key-security.md +816 -816
  430. package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-connection-security.md +681 -681
  431. package/docs/04-REFERENCE/platforms/versori/platforms-versori-workflow-task-types.md +708 -708
  432. package/docs/04-REFERENCE/platforms/versori/readme.md +108 -108
  433. package/docs/04-REFERENCE/readme.md +148 -148
  434. package/docs/04-REFERENCE/resolver-signature/examples/advanced-resolvers.ts +482 -482
  435. package/docs/04-REFERENCE/resolver-signature/examples/async-resolvers.ts +496 -496
  436. package/docs/04-REFERENCE/resolver-signature/examples/basic-resolvers.ts +343 -343
  437. package/docs/04-REFERENCE/resolver-signature/examples/resolver-signature-readme.md +188 -188
  438. package/docs/04-REFERENCE/resolver-signature/examples/testing-resolvers.ts +463 -463
  439. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-01-foundations.md +286 -286
  440. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-02-parameter-reference.md +643 -643
  441. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-03-basic-examples.md +521 -521
  442. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-04-advanced-patterns.md +739 -739
  443. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-05-sdk-resolvers.md +531 -531
  444. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-migration-guide.md +650 -650
  445. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-testing.md +125 -125
  446. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-07-api-reference.md +794 -794
  447. package/docs/04-REFERENCE/resolver-signature/readme.md +64 -64
  448. package/docs/04-REFERENCE/resolver-signature/resolver-signature-quick-reference.md +270 -270
  449. package/docs/04-REFERENCE/resolver-signature/resolver-signature-readme.md +351 -351
  450. package/docs/04-REFERENCE/schema/fluent-commerce-schema.json +764 -764
  451. package/docs/04-REFERENCE/schema/readme.md +141 -141
  452. package/docs/04-REFERENCE/testing/examples/04-reference-testing-readme.md +158 -158
  453. package/docs/04-REFERENCE/testing/examples/fluent-testing.ts +62 -62
  454. package/docs/04-REFERENCE/testing/examples/health-check.ts +155 -155
  455. package/docs/04-REFERENCE/testing/examples/integration-test.ts +119 -119
  456. package/docs/04-REFERENCE/testing/examples/performance-test.ts +183 -183
  457. package/docs/04-REFERENCE/testing/examples/s3-testing.ts +127 -127
  458. package/docs/04-REFERENCE/testing/modules/04-reference-testing-01-foundations.md +267 -267
  459. package/docs/04-REFERENCE/testing/modules/04-reference-testing-02-s3-testing.md +599 -599
  460. package/docs/04-REFERENCE/testing/modules/04-reference-testing-03-fluent-testing.md +589 -589
  461. package/docs/04-REFERENCE/testing/modules/04-reference-testing-04-integration-testing.md +699 -699
  462. package/docs/04-REFERENCE/testing/modules/04-reference-testing-05-debugging.md +478 -478
  463. package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-cicd-integration.md +463 -463
  464. package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-preflight-validation.md +131 -131
  465. package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-best-practices.md +499 -499
  466. package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-coverage-ci.md +165 -165
  467. package/docs/04-REFERENCE/testing/modules/04-reference-testing-08-api-reference.md +634 -634
  468. package/docs/04-REFERENCE/testing/readme.md +86 -86
  469. package/docs/04-REFERENCE/testing/testing-quick-reference.md +667 -667
  470. package/docs/04-REFERENCE/testing/testing-readme.md +286 -286
  471. package/docs/04-REFERENCE/troubleshooting/readme.md +144 -144
  472. package/docs/04-REFERENCE/troubleshooting/troubleshooting-deno-sftp-compatibility.md +392 -392
  473. package/docs/template-loading-matrix.md +242 -242
  474. package/package.json +5 -3
  475. package/docs/02-CORE-GUIDES/api-reference/cli-profile-integration.md +0 -377
@@ -1,2373 +1,2373 @@
1
- ---
2
- template_id: tpl-ingest-s3-xml-to-location-graphql
3
- canonical_filename: template-ingestion-s3-xml-location-graphql.md
4
- version: 2.0.0
5
- sdk_version: ^0.1.39
6
- runtime: versori
7
- direction: ingestion
8
- source: s3-xml
9
- destination: fluent-graphql
10
- entity: location
11
- format: xml
12
- logging: versori
13
- status: stable
14
- compliance: gold-standard
15
- features:
16
- - graphql-mutation-mapper
17
- - memory-management
18
- - enhanced-logging
19
- - attribute-transformation
20
- ---
21
-
22
- Template: Ingestion - S3 XML to Location GraphQL
23
-
24
- **Template Version:** 2.0.0
25
- **SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
26
- **Last Updated:** 2025-01-24
27
-
28
- **🆕 Version 2.0.0 Enhancements:**
29
- - ✅ **GraphQL Mutation Mapper** - Direct field mapping to mutation variables
30
- - ✅ **Memory Management** - Clear large arrays after processing batches
31
- - ✅ **Enhanced Logging** - Track mutation execution with emoji indicators
32
- - ✅ **Attribute Transformation** - Handle complex nested data structures
33
-
34
- ---
35
-
36
- ## Implementation Prompt
37
-
38
- "Create a Versori scheduled workflow that reads location XML files from S3, transforms the data using GraphQLMutationMapper with nested object mapping, and creates/updates Fluent Commerce locations via direct GraphQL mutations with alias batching support.
39
-
40
- **Requirements:**
41
-
42
- 1. **S3 Source**:
43
- - List and download XML files with configurable prefix and pattern
44
- - Archive processed files to `processed/` directory (deduplication via archiving)
45
- - Move failed files to `errors/` directory
46
- - Use S3DataSource with retry logic and built-in archival
47
- - Support configurable bucket, region, credentials
48
-
49
- 2. **XML Parsing**:
50
- - Use XMLParserService with @ prefix for attribute access
51
- - Handle both single and multiple `<location>` elements (array normalization)
52
- - Support nested XML paths (`location.address.street1`, `location.coordinates.@lat`)
53
- - Parse complex structures (addresses, coordinates, opening schedules)
54
-
55
- 3. **Field Mapping**:
56
- - Map XML fields to Location GraphQL input type with nested objects:
57
- - `ref`, `name`, `type` (root fields)
58
- - `primaryAddress.*` (nested address object with coordinates)
59
- - `openingSchedule.*` (nested schedule object with 7-day hours)
60
- - Use SDK resolvers (trim, uppercase, parseFloat, parseInt, boolean)
61
- - Support custom resolvers for complex transformations
62
- - Validate required fields
63
- - Note: Check your GraphQL schema to determine if `retailer.id` field exists and is mandatory/optional
64
-
65
- 4. **GraphQL Mutations** (Direct - NO Batch API):
66
- - Execute `createLocation` mutation directly for each location
67
- - **NO BPP (Batch Pre-Processing)** - Not applicable for direct GraphQL mutations
68
- - **NO Batch API** - Use direct GraphQL mutations with rate limiting instead
69
- - Use rate limiting (configurable mutations per second)
70
- - Add delay between mutations to avoid API throttling
71
- - Retry failed mutations with exponential backoff
72
- - Track successful vs failed mutations per file
73
-
74
- 5. **Job Tracking & State Management**:
75
- - **Use JobTracker** - Track job lifecycle (start, complete, fail) in KV store
76
- - Use StateService + VersoriKVAdapter for duplicate file prevention
77
- - Track processed files in KV store with metadata
78
- - Store error state with exponential backoff tracking
79
- - Support distributed state across workflow runs
80
- - Provide job status endpoint for monitoring
81
-
82
- 6. **Error Handling**:
83
- - File-level errors archived to `/errors/` subdirectory
84
- - Record-level errors tracked but don't stop file processing
85
- - Mapping errors logged with specific location context
86
- - Mutation errors retried with exponential backoff
87
- - Error state tracking with next retry timestamp
88
-
89
- 7. **Advanced Features**:
90
- - Configurable rate limiting (mutations per second)
91
- - Empty file detection and archival
92
- - Timestamp-based error tracking
93
- - Comprehensive monitoring and logging
94
- - Manual webhook trigger with job tracking
95
- - Job status query endpoint
96
-
97
- **Use SDK Components:**
98
-
99
- - `createClient()` - Universal client factory for Versori
100
- - `S3DataSource` - S3 operations with retry logic
101
- - `XMLParserService` - XML parsing with @ prefix attribute support
102
- - `GraphQLMutationMapper` - Field transformation with schema validation and nested object support
103
- - `StateService` + `VersoriKVAdapter` - Duplicate prevention with KV storage
104
- - Native Versori `log` - Structured logging
105
-
106
- **Configuration Variables** (from Versori activation):
107
-
108
- ```typescript
109
- {
110
- s3: {
111
- bucketName: string;
112
- region: string;
113
- accessKeyId: string;
114
- secretAccessKey: string;
115
- prefix: string; // e.g., 'locations/'
116
- archivePrefix: string; // e.g., 'processed/'
117
- errorPrefix: string; // e.g., 'errors/'
118
- filePattern: string; // e.g., '.xml'
119
- maxFilesToProcess: number;
120
- enableArchival: boolean;
121
- },
122
- fluent: {
123
- retailerId: string; // Optional: Only if mutation schema requires retailerId in input
124
- mutationRateLimit: number; // mutations per second (e.g., 5)
125
- }
126
- }
127
- ```
128
-
129
- **Architecture Pattern:**
130
-
131
- ```
132
- S3 Bucket → List Files → Download XML → Parse → Map → GraphQL Mutation → Archive/Move
133
- ↓ ↓ ↓ ↓ ↓ ↓ ↓
134
- Configure Filter by S3DataSource XML Universal Direct S3 moveFile()
135
- Connection Pattern + Retry Parser Mapper createLocation to processed/
136
- (@) (nested) + Rate Limit or errors/
137
- (deduplication)
138
- ```
139
-
140
- **Deliverables:**
141
-
142
- 1. Complete Versori workflow with package.json
143
- 2. Main workflow logic with S3 + XML + GraphQL patterns
144
- 3. Helper functions for rate limiting and retry
145
- 4. XML path resolution examples
146
- 5. Sample XML files with nested structures
147
- 6. Schema validation CLI commands
148
- 7. Testing and deployment instructions
149
- 8. Monitoring and troubleshooting guidance"
150
-
151
- ---
152
-
153
- # STEP 3: Complete Implementation
154
-
155
- ## Versori Scheduled: S3 XML → Location GraphQL
156
-
157
- **FC Connect SDK Use Case Guide**
158
-
159
- > **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
160
- > **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
161
-
162
- **Context**: Versori scheduled workflow that reads location XML files from S3 and creates/updates Fluent Commerce locations via GraphQL mutations with XML path resolution and rate limiting
163
-
164
- **Complexity**: Medium
165
-
166
- **Runtime**: Versori Platform
167
-
168
- **Estimated Lines**: ~850 lines (with comprehensive documentation)
169
-
170
- ---
171
-
172
- ## What You'll Build
173
-
174
- - Scheduled Versori workflow (daily location sync)
175
- - S3 file listing, download, and archival/move
176
- - XML parsing with @ prefix for attributes (XPath-style)
177
- - Array normalization (single element → array conversion)
178
- - GraphQLMutationMapper-based field transformations with nested objects
179
- - GraphQL mutations for location upserts with alias batching support
180
- - Retry logic with exponential backoff
181
- - StateService duplicate prevention (KV-backed)
182
- - Error state tracking and file error archival
183
- - Manual webhook trigger and job status endpoint
184
-
185
- ---
186
-
187
- ## When to Use GraphQL Mutations vs Batch API vs Event API
188
-
189
- ### ✅ Use GraphQL Mutations For:
190
-
191
- | Entity Type | Use Case | Why GraphQL |
192
- | -------------- | ---------------------------------------- | ------------------------------------- |
193
- | **Locations** | Store/warehouse master data (low volume) | Direct control, immediate validation |
194
- | **Controls** | System configuration, settings | Single operations, complex queries |
195
- | **Prices** | Price updates (moderate volume) | Immediate feedback, custom logic |
196
- | **Single Ops** | One-off creates/updates | Testing, debugging, direct API access |
197
-
198
- ### ❌ Use Event API Instead For:
199
-
200
- | Entity Type | Use Case | Why Event API |
201
- | ------------------- | -------------------------------------- | -------------------------------------------- |
202
- | **Products** | Product catalog sync, variant updates | Triggers workflows, validates business rules |
203
- | **Customers** | Customer registration, profile updates | Needs workflow for downstream systems |
204
- | **Orders** | Order creation, status updates | Event-driven fulfillment workflows |
205
- | **Custom Entities** | Any entity needing workflow triggers | Full Rubix workflow support |
206
-
207
- ### 🔄 Use Batch API For:
208
-
209
- | Entity Type | Use Case | Why Batch API |
210
- | ------------------ | ---------------------------------- | ----------------------------------------------- |
211
- | **Inventory ONLY** | Bulk inventory updates, daily sync | Optimized for high-volume, BPP change detection |
212
-
213
- ---
214
-
215
- ## XML File Format
216
-
217
- ### Sample: locations.xml
218
-
219
- ```xml
220
- <?xml version="1.0" encoding="UTF-8"?>
221
- <locations>
222
- <location ref="LOC-001" type="WAREHOUSE">
223
- <name>Downtown Warehouse</name>
224
- <address country="USA">
225
- <street1>123 Main St</street1>
226
- <street2>Building A</street2>
227
- <city>New York</city>
228
- <state>NY</state>
229
- <postalCode>10001</postalCode>
230
- </address>
231
- <coordinates lat="40.7128" lon="-74.0060"/>
232
- <timeZone>America/New_York</timeZone>
233
- <openingSchedule>
234
- <allHours>false</allHours>
235
- <monStart>800</monStart>
236
- <monEnd>1800</monEnd>
237
- <tueStart>800</tueStart>
238
- <tueEnd>1800</tueEnd>
239
- <wedStart>800</wedStart>
240
- <wedEnd>1800</wedEnd>
241
- <thuStart>800</thuStart>
242
- <thuEnd>1800</thuEnd>
243
- <friStart>800</friStart>
244
- <friEnd>1800</friEnd>
245
- <satStart>0</satStart>
246
- <satEnd>0</satEnd>
247
- <sunStart>0</sunStart>
248
- <sunEnd>0</sunEnd>
249
- </openingSchedule>
250
- </location>
251
-
252
- <location ref="LOC-002" type="DC">
253
- <name>Regional DC</name>
254
- <address country="USA">
255
- <street1>456 Industrial Pkwy</street1>
256
- <city>Los Angeles</city>
257
- <state>CA</state>
258
- <postalCode>90001</postalCode>
259
- </address>
260
- <coordinates lat="34.0522" lon="-118.2437"/>
261
- <timeZone>America/Los_Angeles</timeZone>
262
- <openingSchedule>
263
- <allHours>true</allHours>
264
- <monStart>0</monStart>
265
- <monEnd>0</monEnd>
266
- <tueStart>0</tueStart>
267
- <tueEnd>0</tueEnd>
268
- <wedStart>0</wedStart>
269
- <wedEnd>0</wedEnd>
270
- <thuStart>0</thuStart>
271
- <thuEnd>0</thuEnd>
272
- <friStart>0</friStart>
273
- <friEnd>0</friEnd>
274
- <satStart>0</satStart>
275
- <satEnd>0</satEnd>
276
- <sunStart>0</sunStart>
277
- <sunEnd>0</sunEnd>
278
- </openingSchedule>
279
- </location>
280
- </locations>
281
- ```
282
-
283
- **XML Path Syntax (with @ prefix for attributes):**
284
-
285
- - `location.@ref` → XML attribute `ref` on `<location>` element
286
- - `location.name` → Text content of `<name>` element
287
- - `location.address.street1` → Nested element path
288
- - `location.address.@country` → XML attribute on nested `<address>` element
289
- - `location.coordinates.@lat` → XML attribute for latitude
290
- - `location.openingSchedule.monStart` → Deeply nested element
291
-
292
- **Note**: The SDK's `XMLParserService` automatically handles XML attributes using `@` prefix notation.
293
-
294
- ---
295
-
296
- ## Versori Workflows Structure
297
-
298
- **Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
299
-
300
- **Trigger Types:**
301
- - **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
302
- - **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
303
- - **`workflow()`** → Durable workflows (advanced, rarely used)
304
-
305
- **Execution Steps (chained to triggers):**
306
- - **`http()`** → External API calls (chained from schedule/webhook)
307
- - **`fn()`** → Internal processing (chained from schedule/webhook)
308
-
309
- ### Recommended Project Structure
310
-
311
- ```
312
- s3-xml-location-graphql/
313
- ├── index.ts # Entry point - exports all workflows
314
- └── src/
315
- ├── workflows/
316
- │ ├── scheduled/
317
- │ │ └── daily-location-sync.ts # Scheduled: Daily location sync
318
- │ │
319
- │ └── webhook/
320
- │ ├── adhoc-location-sync.ts # Webhook: Manual trigger
321
- │ └── job-status-check.ts # Webhook: Status query
322
-
323
- ├── services/
324
- │ └── location-sync.service.ts # Shared orchestration logic (reusable)
325
-
326
- └── config/
327
- └── location-mapping.json # GraphQL mapping config
328
- ```
329
-
330
- ---
331
-
332
- ## Complete Versori Workflow
333
-
334
- ### Step 1: Package Configuration
335
-
336
- **File: package.json**
337
-
338
- ```json
339
- {
340
- "name": "versori-s3-xml-location-sync",
341
- "version": "1.0.0",
342
- "description": "Versori workflow: S3 XML location sync to Fluent GraphQL",
343
- "versori": {
344
- "workflows": "./index.ts"
345
- },
346
- "type": "module",
347
- "scripts": {
348
- "deploy": "versori deploy",
349
- "logs": "versori logs"
350
- },
351
- "dependencies": {
352
- "@fluentcommerce/fc-connect-sdk": "^0.1.39",
353
- "@versori/run": "latest"
354
- },
355
- "devDependencies": {
356
- "typescript": "^5.0.0",
357
- "@types/node": "^20.0.0"
358
- }
359
- }
360
- ```
361
-
362
- ### Step 2: Workflow Entry Point (`index.ts`)
363
-
364
- **Purpose**: Register all workflows with Versori platform
365
-
366
- ```typescript
367
- /**
368
- * Entry Point - Registers all workflows with Versori platform
369
- *
370
- * Versori automatically discovers and registers exported workflows
371
- *
372
- * File Structure:
373
- * - src/workflows/scheduled/ → Time-based triggers (cron)
374
- * - src/workflows/webhook/ → HTTP-based triggers (webhooks)
375
- */
376
-
377
- // Scheduled workflows
378
- export { dailyLocationSync } from './src/workflows/scheduled/daily-location-sync';
379
-
380
- // Webhook workflows
381
- export { adhocLocationSync } from './src/workflows/webhook/adhoc-location-sync';
382
- export { locationSyncJobStatus } from './src/workflows/webhook/job-status-check';
383
- ```
384
-
385
- **What Gets Exposed:**
386
- - ✅ `adhocLocationSync` → `https://{workspace}.versori.run/location-sync-adhoc`
387
- - ✅ `locationSyncJobStatus` → `https://{workspace}.versori.run/location-sync-job-status`
388
- - ❌ `dailyLocationSync` → NOT exposed (runs automatically on cron)
389
-
390
- ---
391
-
392
- ### Step 3: Workflow Files
393
-
394
- #### `src/workflows/scheduled/daily-location-sync.ts`
395
-
396
- **Purpose**: Automatic daily location sync
397
- **Trigger**: Cron schedule (`0 2 * * *`)
398
- **Exposed as Endpoint**: ❌ NO - Runs automatically
399
-
400
- ```typescript
401
- import { schedule, http } from '@versori/run';
402
- import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
403
- import { executeLocationSync } from '../../services/location-sync.service';
404
-
405
- /**
406
- * Scheduled Workflow: Daily Location Sync
407
- *
408
- * Runs automatically daily at 2 AM UTC
409
- * NOT exposed as HTTP endpoint - Versori executes on schedule
410
- *
411
- * Uses shared service: location-sync.service.ts
412
- */
413
- export const dailyLocationSync = schedule(
414
- 'location-sync-scheduled',
415
- '0 2 * * *' // Daily at 2 AM UTC
416
- ).then(
417
- http('run-location-sync', { connection: 'fluent_commerce' }, async ctx => {
418
- const startTime = Date.now();
419
- const { log, openKv } = ctx;
420
- const jobId = `location-sync-${Date.now()}`;
421
- const tracker = new JobTracker(openKv(':project:'), log);
422
-
423
- log.info('🚀 [START] Daily location sync initiated', { jobId, trigger: 'schedule' });
424
-
425
- await tracker.createJob(jobId, { triggeredBy: 'schedule', stage: 'initialization' });
426
- await tracker.updateJob(jobId, { status: 'processing' });
427
-
428
- try {
429
- log.info('⚙️ [PROCESSING] Starting location synchronization workflow', { jobId });
430
- const result = await executeLocationSync(ctx, jobId, tracker);
431
- await tracker.markCompleted(jobId, result);
432
-
433
- const duration = Date.now() - startTime;
434
- log.info('✅ [SUCCESS] Daily location sync completed', {
435
- jobId,
436
- duration: `${duration}ms`,
437
- processed: result.processed,
438
- totalRecords: result.totalRecords
439
- });
440
-
441
- return { success: true, jobId, duration, ...result };
442
- } catch (e: any) {
443
- await tracker.markFailed(jobId, e);
444
- const duration = Date.now() - startTime;
445
-
446
- log.error('❌ [FAILED] Daily location sync failed', {
447
- jobId,
448
- duration: `${duration}ms`,
449
- error: e?.message,
450
- errorType: e?.name
451
- });
452
-
453
- return { success: false, jobId, duration, error: e?.message };
454
- }
455
- })
456
- );
457
- ```
458
-
459
- ---
460
-
461
- #### `src/workflows/webhook/adhoc-location-sync.ts`
462
-
463
- **Purpose**: Manual location sync trigger (on-demand)
464
- **Trigger**: HTTP POST
465
- **Endpoint**: `POST https://{workspace}.versori.run/location-sync-adhoc`
466
- **Use Cases**: Testing, priority processing, ad-hoc runs
467
-
468
- ```typescript
469
- import { webhook, http } from '@versori/run';
470
- import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
471
- import { executeLocationSync } from '../../services/location-sync.service';
472
-
473
- /**
474
- * Webhook: Manual Location Sync Trigger
475
- *
476
- * Endpoint: POST https://{workspace}.versori.run/location-sync-adhoc
477
- * Request body (optional): { filePattern: "urgent_*.xml", maxFiles: 5 }
478
- *
479
- * Pattern: webhook().then(http()) - needs Fluent API access
480
- * Uses shared service: location-sync.service.ts
481
- *
482
- * SECURITY: Authentication handled via connection parameter
483
- * No manual API key validation needed - Versori manages this via connection auth
484
- */
485
- export const adhocLocationSync = webhook('location-sync-adhoc', {
486
- response: { mode: 'sync' },
487
- connection: 'location-sync-adhoc', // Versori validates API key
488
- }).then(
489
- http('run-location-sync-adhoc', { connection: 'fluent_commerce' }, async ctx => {
490
- const startTime = Date.now();
491
- const { log, openKv, data } = ctx;
492
- const jobId = `location-sync-adhoc-${Date.now()}`;
493
- const tracker = new JobTracker(openKv(':project:'), log);
494
-
495
- log.info('🚀 [START] Manual location sync triggered', { jobId, trigger: 'webhook', options: data });
496
-
497
- await tracker.createJob(jobId, {
498
- triggeredBy: 'manual',
499
- stage: 'initialization',
500
- options: data // Optional: filePattern, maxFiles, etc.
501
- });
502
- await tracker.updateJob(jobId, { status: 'processing' });
503
-
504
- try {
505
- log.info('⚙️ [PROCESSING] Starting manual location synchronization', { jobId });
506
- const result = await executeLocationSync(ctx, jobId, tracker);
507
- await tracker.markCompleted(jobId, result);
508
-
509
- const duration = Date.now() - startTime;
510
- log.info('✅ [SUCCESS] Manual location sync completed', {
511
- jobId,
512
- duration: `${duration}ms`,
513
- processed: result.processed,
514
- totalRecords: result.totalRecords
515
- });
516
-
517
- return { success: true, jobId, duration, ...result };
518
- } catch (e: any) {
519
- await tracker.markFailed(jobId, e);
520
- const duration = Date.now() - startTime;
521
-
522
- log.error('❌ [FAILED] Manual location sync failed', {
523
- jobId,
524
- duration: `${duration}ms`,
525
- error: e?.message,
526
- errorType: e?.name
527
- });
528
-
529
- return { success: false, jobId, duration, error: e?.message };
530
- }
531
- })
532
- );
533
- ```
534
-
535
- ---
536
-
537
- #### `src/workflows/webhook/job-status-check.ts`
538
-
539
- **Purpose**: Query job status
540
- **Trigger**: HTTP POST
541
- **Endpoint**: `POST https://{workspace}.versori.run/location-sync-job-status`
542
- **Request body**: `{ "jobId": "location-sync-1234567890" }`
543
-
544
- ```typescript
545
- import { webhook, fn } from '@versori/run';
546
- import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
547
-
548
- /**
549
- * Webhook: Job Status Check
550
- *
551
- * Endpoint: POST https://{workspace}.versori.run/location-sync-job-status
552
- * Request body: { "jobId": "location-sync-1234567890" }
553
- *
554
- * Pattern: webhook().then(fn()) - no external API needed, only KV storage
555
- * Lightweight: Only queries KV store, no Fluent API calls
556
- *
557
- * SECURITY: Authentication handled via connection parameter
558
- * No manual API key validation needed - Versori manages this via connection auth
559
- */
560
- export const locationSyncJobStatus = webhook('location-sync-job-status', {
561
- response: { mode: 'sync' },
562
- connection: 'location-sync-job-status',
563
- }).then(
564
- fn('status', async ctx => {
565
- const { data, log, openKv } = ctx;
566
- const jobId = data?.jobId as string;
567
-
568
- if (!jobId) {
569
- return { success: false, error: 'jobId required' };
570
- }
571
-
572
- const tracker = new JobTracker(openKv(':project:'), log);
573
- const status = await tracker.getJob(jobId);
574
-
575
- return status
576
- ? { success: true, jobId, ...status }
577
- : { success: false, error: 'Job not found', jobId };
578
- })
579
- );
580
- ```
581
-
582
- ---
583
-
584
- ### Step 4: Main Orchestration Service (`src/services/location-sync.service.ts`)
585
-
586
- **Note:** This service file should contain the `executeLocationSync` function (renamed from `runLocationXmlWorkflow`). The main workflow logic should be moved here.
587
-
588
- ```typescript
589
- /**
590
- * Main Orchestration Service: Location Sync
591
- *
592
- * This service contains the core business logic for location synchronization.
593
- *
594
- * Features:
595
- * - S3 file operations with archival-based deduplication (moveFile to processed/)
596
- * - XML parsing with @ prefix for attributes
597
- * - Single element → array normalization
598
- * - GraphQLMutationMapper for nested field transformations
599
- * - GraphQL mutations with alias batching support
600
- * - StateService for metadata tracking (secondary to archival)
601
- * - Error state tracking with exponential backoff
602
- *
603
- * Deduplication Strategy:
604
- * - PRIMARY: S3 archival via moveFile() - Files in processed/ won't be re-listed
605
- * - SECONDARY: StateService KV tracking - Provides metadata and processing history
606
- */
607
- import { Buffer } from 'node:buffer'; // Required for Deno/Versori runtime
608
- import {
609
- createClient,
610
- S3DataSource,
611
- XMLParserService,
612
- GraphQLMutationMapper,
613
- StateService,
614
- VersoriKVAdapter,
615
- JobTracker,
616
- FluentClient,
617
- } from '@fluentcommerce/fc-connect-sdk';
618
-
619
- // ============================================================================
620
- // Type Definitions
621
- // ============================================================================
622
-
623
- interface FileProcessingResult {
624
- success: boolean;
625
- locations: any[];
626
- errors: string[];
627
- }
628
-
629
- interface MutationResult {
630
- successful: number;
631
- failed: number;
632
- errors: string[];
633
- }
634
-
635
- interface MutationLogEntry {
636
- timestamp: string;
637
- fileName: string;
638
- locationRef: string;
639
- status: 'success' | 'failure';
640
- error?: string;
641
- }
642
-
643
- // ============================================================================
644
- // Utility Functions
645
- // ============================================================================
646
-
647
- /**
648
- * Retry utility with exponential backoff
649
- * (User-defined - not part of SDK public API)
650
- */
651
- async function retryWithBackoff<T>(
652
- operation: () => Promise<T>,
653
- maxRetries = 3,
654
- baseDelayMs = 1000
655
- ): Promise<T> {
656
- let lastError: any;
657
- for (let attempt = 0; attempt < maxRetries; attempt++) {
658
- try {
659
- return await operation();
660
- } catch (error) {
661
- lastError = error;
662
- if (attempt < maxRetries - 1) {
663
- const delay = baseDelayMs * Math.pow(2, attempt) + Math.floor(Math.random() * 200);
664
- await new Promise(resolve => setTimeout(resolve, delay));
665
- }
666
- }
667
- }
668
- throw lastError;
669
- }
670
-
671
- /**
672
- * Rate limiter for GraphQL mutations
673
- */
674
- async function rateLimitedMutation(operation: () => Promise<any>, delayMs: number): Promise<any> {
675
- const result = await operation();
676
- await new Promise(resolve => setTimeout(resolve, delayMs));
677
- return result;
678
- }
679
-
680
- // ============================================================================
681
- // Service Functions
682
- // ============================================================================
683
-
684
- /**
685
- * Service Function 1: Process File
686
- *
687
- * Downloads XML from S3, parses with XMLParserService, normalizes arrays,
688
- * and maps fields with GraphQLMutationMapper.
689
- *
690
- * @param s3 - S3DataSource instance
691
- * @param parser - XMLParserService instance
692
- * @param mapper - GraphQLMutationMapper instance
693
- * @param filePath - Full S3 path to file
694
- * @param fileName - File name only (for logging)
695
- * @param log - Logger instance
696
- * @returns FileProcessingResult with locations array and errors
697
- */
698
- async function processFile(
699
- s3: S3DataSource,
700
- parser: XMLParserService,
701
- mapper: GraphQLMutationMapper,
702
- filePath: string,
703
- fileName: string,
704
- log: any
705
- ): Promise<FileProcessingResult> {
706
- try {
707
- log.info('Processing file', { fileName });
708
-
709
- // Download with retry
710
- const content = await retryWithBackoff(
711
- () => s3.downloadFile(filePath, { encoding: 'utf8' }) as Promise<string>
712
- );
713
-
714
- // Parse XML
715
- const xmlData = await parser.parse(content);
716
-
717
- // Extract location array (handle both single and multiple locations)
718
- // CRITICAL: XML array normalization - single <location> becomes object, not array
719
- const locationsData = xmlData.locations?.location;
720
- const locationsArray = Array.isArray(locationsData) ? locationsData : [locationsData];
721
-
722
- if (!locationsArray || locationsArray.length === 0) {
723
- log.warn('Empty file (no locations)', { fileName });
724
- return {
725
- success: true,
726
- locations: [],
727
- errors: [],
728
- };
729
- }
730
-
731
- // Map each location using GraphQLMutationMapper
732
- const mappedLocations: Array<{ query: string; variables: any; input: any }> = [];
733
- const mappingErrors: string[] = [];
734
-
735
- // ✅ PRODUCTION ENHANCEMENT: Log transformation start
736
- log.info('Transforming locations to GraphQL mutations', {
737
- fileName,
738
- totalLocations: locationsArray.length,
739
- });
740
-
741
- for (let i = 0; i < locationsArray.length; i++) {
742
- const locationNumber = i + 1;
743
-
744
- // ✅ PRODUCTION ENHANCEMENT: Log progress every 50 locations
745
- if (locationNumber % 50 === 0) {
746
- log.info(`📤 Transforming location ${locationNumber}/${locationsArray.length}`, {
747
- fileName,
748
- locationNumber,
749
- totalLocations: locationsArray.length,
750
- validSoFar: mappedLocations.length,
751
- errorsSoFar: mappingErrors.length,
752
- progress: `${((locationNumber / locationsArray.length) * 100).toFixed(1)}%`,
753
- });
754
- }
755
-
756
- // Wrap location in context for mapping
757
- const record = { location: locationsArray[i] };
758
- try {
759
- // GraphQLMutationMapper returns { query, variables } directly
760
- const mappingResult = await mapper.map(record);
761
-
762
- mappedLocations.push({
763
- query: mappingResult.query,
764
- variables: mappingResult.variables,
765
- input: mappingResult.variables.input || mappingResult.variables,
766
- });
767
- } catch (error: unknown) {
768
- const errorMsg = error instanceof Error ? error.message : String(error);
769
- mappingErrors.push(`Location ${locationNumber}: ${errorMsg}`);
770
- log.warn('Mapping failed for location', { fileName, index: i, error: errorMsg });
771
- }
772
- }
773
-
774
- log.info('File processed', {
775
- fileName,
776
- total: locationsArray.length,
777
- mapped: mappedLocations.length,
778
- errors: mappingErrors.length,
779
- successRate: `${((mappedLocations.length / locationsArray.length) * 100).toFixed(1)}%`,
780
- });
781
-
782
- return {
783
- success: true,
784
- locations: mappedLocations,
785
- errors: mappingErrors,
786
- };
787
- } catch (error: any) {
788
- // ✅ Enhanced error logging: Extract all error details for visibility
789
- const errorDetails = {
790
- message: error?.message || 'Unknown error',
791
- stack: error?.stack,
792
- fileName: error?.fileName,
793
- lineNumber: error?.lineNumber,
794
- originalError: error?.context?.originalError?.message,
795
- errorType: error?.name || 'Error',
796
- };
797
- log.error('File processing failed', errorDetails, { fileName });
798
- return {
799
- success: false,
800
- locations: [],
801
- errors: [error.message || 'Unknown error'],
802
- };
803
- }
804
- }
805
-
806
- /**
807
- * Service Function 2: Execute Mutations
808
- *
809
- * Executes GraphQL createLocation mutations with alias batching support.
810
- *
811
- * @param client - FluentClient instance
812
- * @param mapper - GraphQLMutationMapper instance
813
- * @param locations - Array of mapped location objects with query and variables
814
- * @param log - Logger instance
815
- * @param retailerId - Fluent retailer ID
816
- * @param batchSize - Number of concurrent requests (default: 1)
817
- * @param mutationsPerAliasBatch - Optional: Number of mutations per aliased request (default: undefined)
818
- * @returns MutationResult with success/failure counts
819
- */
820
- async function executeMutations(
821
- client: FluentClient,
822
- mapper: GraphQLMutationMapper,
823
- locations: Array<{ query: string; variables: any; input: any }>,
824
- log: any,
825
- retailerId: string,
826
- batchSize: number = 1, // ✅ Default: 1 (sequential)
827
- mutationsPerAliasBatch?: number // ✅ NEW: Alias batching parameter (default: undefined = disabled)
828
- ): Promise<MutationResult> {
829
- // Determine mode: use aliases if mutationsPerAliasBatch is set and > 1
830
- const useAliases = mutationsPerAliasBatch && mutationsPerAliasBatch > 1;
831
-
832
- if (useAliases) {
833
- return await executeMutationsWithAliases(
834
- client,
835
- mapper,
836
- locations,
837
- log,
838
- retailerId,
839
- batchSize,
840
- mutationsPerAliasBatch!
841
- );
842
- } else {
843
- return await executeMutationsSeparate(
844
- client,
845
- locations,
846
- log,
847
- batchSize
848
- );
849
- }
850
- }
851
-
852
- /**
853
- * Execute mutations using separate concurrent requests (current mode)
854
- */
855
- async function executeMutationsSeparate(
856
- client: FluentClient,
857
- locations: Array<{ query: string; variables: any; input: any }>,
858
- log: any,
859
- batchSize: number
860
- ): Promise<MutationResult> {
861
- const result: MutationResult = {
862
- successful: 0,
863
- failed: 0,
864
- errors: [],
865
- };
866
-
867
- const safeConc = Math.max(1, Math.floor(batchSize));
868
-
869
- // Sequential mode
870
- if (safeConc === 1) {
871
- for (const location of locations) {
872
- try {
873
- await retryWithBackoff(() =>
874
- client.graphql({
875
- query: location.query,
876
- variables: location.variables,
877
- })
878
- );
879
- result.successful++;
880
- } catch (error: unknown) {
881
- result.failed++;
882
- const errorMsg = error instanceof Error ? error.message : String(error);
883
- result.errors.push(errorMsg);
884
- }
885
- }
886
- return result;
887
- }
888
-
889
- // Parallel mode
890
- for (let i = 0; i < locations.length; i += safeConc) {
891
- const chunk = locations.slice(i, i + safeConc);
892
- const results = await Promise.allSettled(
893
- chunk.map(loc =>
894
- retryWithBackoff(() =>
895
- client.graphql({
896
- query: loc.query,
897
- variables: loc.variables,
898
- })
899
- )
900
- )
901
- );
902
-
903
- results.forEach((settledResult, idx) => {
904
- if (settledResult.status === 'fulfilled') {
905
- result.successful++;
906
- } else {
907
- result.failed++;
908
- result.errors.push(
909
- settledResult.reason instanceof Error ? settledResult.reason.message : String(settledResult.reason)
910
- );
911
- }
912
- });
913
- }
914
-
915
- return result;
916
- }
917
-
918
- /**
919
- * ✅ NEW: Execute mutations using GraphQL aliases (batched requests)
920
- */
921
- async function executeMutationsWithAliases(
922
- client: FluentClient,
923
- mapper: GraphQLMutationMapper,
924
- locations: Array<{ query: string; variables: any; input: any }>,
925
- log: any,
926
- retailerId: string,
927
- maxParallel: number,
928
- mutationsPerAliasBatch: number
929
- ): Promise<MutationResult> {
930
- const result: MutationResult = { successful: 0, failed: 0, errors: [] };
931
-
932
- const mutationName = (mapper as any).config.mutation || 'createLocation';
933
- const aliasBatches: Array<Array<typeof locations[0]>> = [];
934
-
935
- for (let i = 0; i < locations.length; i += mutationsPerAliasBatch) {
936
- aliasBatches.push(locations.slice(i, i + mutationsPerAliasBatch));
937
- }
938
-
939
- // Process batches with concurrency control
940
- for (let i = 0; i < aliasBatches.length; i += maxParallel) {
941
- const concurrentBatches = aliasBatches.slice(i, i + maxParallel);
942
-
943
- const batchResults = await Promise.allSettled(
944
- concurrentBatches.map(async (batch) => {
945
- const { query, variables } = buildAliasedBatch(batch, mutationName, retailerId);
946
- const response = await retryWithBackoff(() => client.graphql({ query, variables }));
947
- return parseAliasResponse(response, batch, mutationName);
948
- })
949
- );
950
-
951
- batchResults.forEach((batchResult, idx) => {
952
- if (batchResult.status === 'fulfilled') {
953
- const batchRes = batchResult.value;
954
- result.successful += batchRes.executed;
955
- result.failed += batchRes.failed;
956
- result.errors.push(...batchRes.errors);
957
- } else {
958
- const batch = concurrentBatches[idx];
959
- const errorMsg = batchResult.reason instanceof Error ? batchResult.reason.message : String(batchResult.reason);
960
- batch.forEach(loc => {
961
- result.failed++;
962
- result.errors.push(`Batch execution failed: ${errorMsg}`);
963
- });
964
- }
965
- });
966
-
967
- if (i + maxParallel < aliasBatches.length) {
968
- await new Promise(resolve => setTimeout(resolve, 500));
969
- }
970
- }
971
-
972
- return result;
973
- }
974
-
975
- /**
976
- * ✅ NEW: Build aliased batch query and variables
977
- */
978
- function buildAliasedBatch(
979
- batch: Array<{ query: string; variables: any; input: any }>,
980
- mutationName: string,
981
- retailerId: string
982
- ): { query: string; variables: Record<string, any> } {
983
- const batchSize = batch.length;
984
- const inputTypeName = `${mutationName.charAt(0).toUpperCase() + mutationName.slice(1)}Input`;
985
-
986
- const variables = Array.from({ length: batchSize }, (_, i) =>
987
- `$input${i + 1}: ${inputTypeName}!`
988
- ).join(', ');
989
-
990
- const aliasedMutations = Array.from({ length: batchSize }, (_, i) => {
991
- const alias = `${mutationName}${i + 1}`;
992
- return ` ${alias}: ${mutationName}(input: $input${i + 1}) { id ref name }`;
993
- }).join('\n');
994
-
995
- const operationName = `Batch${mutationName.charAt(0).toUpperCase() + mutationName.slice(1)}s`;
996
- const query = `mutation ${operationName}(${variables}) {\n${aliasedMutations}\n}`;
997
-
998
- const variablesObj: Record<string, any> = {};
999
- batch.forEach((loc, index) => {
1000
- const input = loc.variables.input || loc.variables;
1001
- if (input && !input.retailer) {
1002
- input.retailer = { id: parseInt(retailerId) };
1003
- }
1004
- variablesObj[`input${index + 1}`] = input;
1005
- });
1006
-
1007
- return { query, variables: variablesObj };
1008
- }
1009
-
1010
- /**
1011
- * ✅ NEW: Parse aliased GraphQL response
1012
- */
1013
- function parseAliasResponse(
1014
- response: any,
1015
- batch: Array<{ query: string; variables: any; input: any }>,
1016
- mutationName: string
1017
- ): { executed: number; failed: number; errors: string[] } {
1018
- const result = { executed: 0, failed: 0, errors: [] as string[] };
1019
-
1020
- const data = response.data || {};
1021
- const errors = response.errors || [];
1022
-
1023
- batch.forEach((loc, index) => {
1024
- const alias = `${mutationName}${index + 1}`;
1025
- const aliasData = data[alias];
1026
- const aliasErrors = errors.filter((e: any) =>
1027
- e.path && Array.isArray(e.path) && e.path.includes(alias)
1028
- );
1029
-
1030
- if (aliasData && !aliasErrors.length) {
1031
- result.executed++;
1032
- } else {
1033
- result.failed++;
1034
- const errorMsg = aliasErrors[0]?.message || 'Mutation failed';
1035
- result.errors.push(`${loc.input?.ref || 'unknown'}: ${errorMsg}`);
1036
- }
1037
- });
1038
-
1039
- return result;
1040
- }
1041
-
1042
- /**
1043
- * Service Function 3: Write Mutation Log
1044
- *
1045
- * Writes mutation results to S3 as a JSON log file.
1046
- *
1047
- * @param s3 - S3DataSource instance
1048
- * @param logEntries - Array of mutation log entries
1049
- * @param fileName - Original file name (used to generate log path)
1050
- * @param logPrefix - S3 prefix for logs (e.g., 'logs/')
1051
- * @param log - Logger instance
1052
- */
1053
- async function writeMutationLog(
1054
- s3: S3DataSource,
1055
- logEntries: MutationLogEntry[],
1056
- fileName: string,
1057
- logPrefix: string,
1058
- log: any
1059
- ): Promise<void> {
1060
- try {
1061
- const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
1062
- const logFileName = `${logPrefix}${fileName.replace('.xml', '')}_${timestamp}.json`;
1063
-
1064
- const logContent = JSON.stringify(
1065
- {
1066
- fileName,
1067
- timestamp: new Date().toISOString(),
1068
- totalMutations: logEntries.length,
1069
- successful: logEntries.filter(e => e.status === 'success').length,
1070
- failed: logEntries.filter(e => e.status === 'failure').length,
1071
- entries: logEntries,
1072
- },
1073
- null,
1074
- 2
1075
- );
1076
-
1077
- // Write to S3 (uploadFile accepts string or Buffer)
1078
- await s3.uploadFile(logFileName, logContent);
1079
-
1080
- log.info('Mutation log written', { logFileName, entries: logEntries.length });
1081
- } catch (error: any) {
1082
- // ✅ Enhanced error logging: Extract all error details for visibility
1083
- const errorDetails = {
1084
- message: error?.message || 'Unknown error',
1085
- stack: error?.stack,
1086
- errorType: error?.name || 'Error',
1087
- };
1088
- log.error('Failed to write mutation log', errorDetails, { fileName });
1089
- // Don't throw - logging failure shouldn't stop workflow
1090
- }
1091
- }
1092
-
1093
- // ============================================================================
1094
- // Main Workflow Function
1095
- // ============================================================================
1096
-
1097
- /**
1098
- * Main Orchestration Function: Execute Location Sync
1099
- *
1100
- * This function orchestrates the location synchronization workflow.
1101
- *
1102
- * Architecture:
1103
- * 1. List files from S3
1104
- * 2. For each file:
1105
- * a. processFile() - Download, parse, map
1106
- * b. executeMutations() - Send GraphQL mutations with rate limiting
1107
- * c. writeMutationLog() - Log results to S3
1108
- * d. Archive file (primary deduplication)
1109
- * e. Mark processed in KV (metadata tracking)
1110
- *
1111
- * @param ctx - Versori context
1112
- * @param jobId - Job identifier
1113
- * @param tracker - JobTracker instance
1114
- * @returns Processing result
1115
- */
1116
- export async function executeLocationSync(ctx: any, jobId: string, tracker: JobTracker) {
1117
- const { log, activation } = ctx;
1118
-
1119
- log.info('📋 [INIT] Reading activation variables', { jobId });
1120
-
1121
- // Read activation variables
1122
- const s3Bucket = activation?.getVariable('s3BucketName');
1123
- const s3Region = activation?.getVariable('awsRegion') || 'us-east-1';
1124
- const s3AccessKeyId = activation?.getVariable('awsAccessKeyId');
1125
- const s3SecretAccessKey = activation?.getVariable('awsSecretAccessKey');
1126
- const s3Prefix = activation?.getVariable('s3Prefix') || 'locations/';
1127
- const archivePrefix = activation?.getVariable('archivePrefix') || 'processed/';
1128
- const errorPrefix = activation?.getVariable('errorPrefix') || 'errors/';
1129
- const logPrefix = activation?.getVariable('logPrefix') || 'logs/';
1130
- const filePattern = (activation?.getVariable('filePattern') || '.xml').toLowerCase();
1131
- const maxFiles = parseInt(activation?.getVariable('maxFilesToProcess') || '10', 10);
1132
- const retailerId = activation?.getVariable('retailerId'); // Optional: Only if mutation schema requires it
1133
- const enableArchival = activation?.getVariable('enableArchival') !== 'false';
1134
- const enableMutationLogs = activation?.getVariable('enableMutationLogs') !== 'false';
1135
- const enableFileTracking = activation?.getVariable('enableFileTracking') !== 'false';
1136
-
1137
- // ✅ Configuration with defaults
1138
- const mutationBatchSize = parseInt(
1139
- activation?.getVariable('mutationBatchSize') || '1', // ✅ Default: 1 (sequential)
1140
- 10
1141
- );
1142
-
1143
- const mutationsPerAliasBatch = activation?.getVariable('mutationsPerAliasBatch')
1144
- ? parseInt(activation?.getVariable('mutationsPerAliasBatch') || '1', 10)
1145
- : undefined; // ✅ Default: undefined (disabled, use separate requests)
1146
-
1147
- // Validate required variables
1148
- const missingVars: string[] = [];
1149
- if (!s3Bucket) missingVars.push('s3BucketName');
1150
- if (!s3AccessKeyId) missingVars.push('awsAccessKeyId');
1151
- if (!s3SecretAccessKey) missingVars.push('awsSecretAccessKey');
1152
- // Note: retailerId is optional - only needed if mutation schema requires it
1153
-
1154
- if (missingVars.length > 0) {
1155
- const errorMsg = `Missing required variables: ${missingVars.join(', ')}`;
1156
- log.error('❌ [VALIDATION] Missing required activation variables', {
1157
- missingVars,
1158
- recommendation: 'Add missing variables in Versori activation settings'
1159
- });
1160
- return { success: false, error: errorMsg, processed: 0 };
1161
- }
1162
-
1163
- log.info('✅ [VALIDATION] All required variables present', {
1164
- s3Bucket,
1165
- s3Region,
1166
- s3Prefix,
1167
- enableFileTracking,
1168
- mutationBatchSize
1169
- });
1170
-
1171
- try {
1172
- log.info('🔧 [INIT] Initializing Fluent Commerce client', { jobId });
1173
-
1174
- // Initialize services with validateConnection
1175
- const client = await createClient(ctx, { validateConnection: true });
1176
- if (!client) {
1177
- throw new Error('Failed to create Fluent Commerce client');
1178
- }
1179
-
1180
- log.info('✅ [INIT] Fluent Commerce client validated and ready', { jobId });
1181
-
1182
- // ✅ CORRECT: GraphQL mutations do NOT need client.setRetailerId()
1183
- // setRetailerId() is only for Job/Event API, NOT GraphQL
1184
- // Check your GraphQL schema to determine retailerId handling:
1185
- // - Mandatory retailerId → Must pass it in mutation input
1186
- // - Optional retailerId → Can pass it if needed
1187
- // - No retailerId field → Don't pass it
1188
- // See: fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md
1189
-
1190
- log.info('🗄️ [INIT] Initializing S3 data source', { s3Bucket, s3Region, s3Prefix });
1191
-
1192
- const s3 = new S3DataSource(
1193
- {
1194
- type: 'S3_XML',
1195
- connectionId: 's3-location-sync',
1196
- name: 'Source S3',
1197
- s3Config: {
1198
- bucket: s3Bucket,
1199
- region: s3Region,
1200
- accessKeyId: s3AccessKeyId,
1201
- secretAccessKey: s3SecretAccessKey,
1202
- },
1203
- },
1204
- log
1205
- );
1206
-
1207
- const parser = new XMLParserService();
1208
-
1209
- // Initialize state tracking (only if enabled)
1210
- let stateService: StateService | null = null;
1211
- if (enableFileTracking) {
1212
- log.info('🔄 [INIT] Enabling file tracking with StateService', { jobId });
1213
- const stateKV = new VersoriKVAdapter(ctx);
1214
- stateService = new StateService(stateKV);
1215
- } else {
1216
- log.info('⏭️ [INIT] File tracking disabled - relying on S3 archival only', { jobId });
1217
- }
1218
-
1219
- log.info('📝 [INIT] Loading mapping configuration', { jobId });
1220
-
1221
- // ✅ CRITICAL: Load mapping config from external JSON file
1222
- // Mapping config uses GraphQLMutationMapper structure (nested objects, not dot notation)
1223
- // File: src/config/location-mapping.json
1224
- const mappingConfigJson = await import('../config/location-mapping.json', { assert: { type: 'json' } });
1225
- const mappingConfig = mappingConfigJson.default;
1226
-
1227
- // Initialize GraphQLMutationMapper with client for schema introspection
1228
- const mapper = new GraphQLMutationMapper(mappingConfig, log, { fluentClient: client });
1229
-
1230
- log.info('📂 [S3] Listing files from S3', { s3Bucket, s3Prefix, filePattern });
1231
-
1232
- // List files (pattern filtering handled by listFiles)
1233
- const files = await s3.listFiles({
1234
- prefix: s3Prefix,
1235
- pattern: filePattern,
1236
- maxKeys: 1000
1237
- });
1238
-
1239
- // Newest-first ordering
1240
- const xmlFiles = files
1241
- .sort((a: any, b: any) => {
1242
- const aTime = a.lastModified ? new Date(a.lastModified).getTime() : 0;
1243
- const bTime = b.lastModified ? new Date(b.lastModified).getTime() : 0;
1244
- return bTime - aTime;
1245
- })
1246
- .slice(0, maxFiles);
1247
-
1248
- log.info('📊 [S3] File discovery complete', {
1249
- totalFiles: files.length,
1250
- xmlFiles: xmlFiles.length,
1251
- maxFiles,
1252
- selectedFiles: xmlFiles.map(f => f.name)
1253
- });
1254
-
1255
- const results = {
1256
- processed: 0,
1257
- skipped: 0,
1258
- failed: 0,
1259
- totalRecords: 0,
1260
- errors: [] as string[],
1261
- };
1262
-
1263
- log.info('🔄 [PROCESSING] Starting file processing loop', {
1264
- fileCount: xmlFiles.length,
1265
- jobId
1266
- });
1267
-
1268
- // Per-file processing loop
1269
- for (const file of xmlFiles) {
1270
- const filePath = file.path;
1271
- const fileName = file.name;
1272
-
1273
- log.info('📄 [FILE] Processing file', { fileName, filePath });
1274
-
1275
- // Duplicate prevention (secondary check - files in processed/ won't be listed)
1276
- // Primary deduplication: S3 archival (files moved to processed/ subdirectory)
1277
- if (enableFileTracking && stateService) {
1278
- const wasProcessed = await stateService.isFileProcessed(fileName);
1279
- if (wasProcessed) {
1280
- log.info('⏭️ [SKIP] File already processed (KV check)', { fileName });
1281
- results.skipped++;
1282
- continue;
1283
- }
1284
- }
1285
-
1286
- try {
1287
- // Step 1: Process file (download, parse, map)
1288
- log.info('📥 [DOWNLOAD] Downloading and parsing file', { fileName });
1289
- const processingResult = await processFile(s3, parser, mapper, filePath, fileName, log);
1290
-
1291
- if (!processingResult.success) {
1292
- throw new Error(`File processing failed: ${processingResult.errors.join(', ')}`);
1293
- }
1294
-
1295
- if (processingResult.locations.length === 0) {
1296
- log.warn('⚠️ [SKIP] Empty file detected, archiving', { fileName });
1297
- if (enableArchival) {
1298
- await s3.moveFile(filePath, `${archivePrefix}${fileName}`);
1299
- log.info('📦 [ARCHIVE] Empty file archived', { fileName, destination: `${archivePrefix}${fileName}` });
1300
- }
1301
- results.skipped++;
1302
- continue;
1303
- }
1304
-
1305
- log.info('✅ [PARSE] File parsed successfully', {
1306
- fileName,
1307
- locationCount: processingResult.locations.length,
1308
- mappingErrors: processingResult.errors.length
1309
- });
1310
-
1311
- // Step 2: Execute mutations
1312
- // Step 2: Execute mutations with alias batching support
1313
- // ? Enhanced: Extract context for progress logging
1314
- const sampleLocationRefs = processingResult.locations.slice(0, 5).map((loc: any) => loc.input?.ref || loc.ref || 'unknown');
1315
- const mutationType = mapper?.mutationName || 'createLocation';
1316
-
1317
- // ? Enhanced: Start logging with context
1318
- log.info(`[GraphQLMutations] Sending mutations for file "${fileName}"`, {
1319
- totalMutations: processingResult.locations.length,
1320
- mutationType,
1321
- batchSize: mutationBatchSize,
1322
- batchMode: mutationBatchSize === 1 ? 'sequential' : `parallel (${mutationBatchSize})`,
1323
- sampleLocationRefs: sampleLocationRefs.join(', '),
1324
- aliasBatching: mutationsPerAliasBatch ? `enabled (${mutationsPerAliasBatch} per alias)` : 'disabled'
1325
- });
1326
-
1327
- const mutationResult = await executeMutations(
1328
- client,
1329
- mapper,
1330
- processingResult.locations,
1331
- log,
1332
- retailerId, // Pass retailerId for mutations that require it in input
1333
- mutationBatchSize, // Concurrency control (default: 1)
1334
- mutationsPerAliasBatch // ✅ NEW: Alias batching (default: undefined)
1335
- );
1336
-
1337
- // ? Enhanced: Completion logging with summary
1338
- log.info(`[GraphQLMutations] Mutation submission completed for file "${fileName}"`, {
1339
- totalMutations: processingResult.locations.length,
1340
- successful: mutationResult.successful,
1341
- failed: mutationResult.failed,
1342
- successRate: processingResult.locations.length > 0 ? `${Math.round((mutationResult.successful / processingResult.locations.length) * 100)}%` : '0%',
1343
- mutationType
1344
- });
1345
-
1346
- // Step 3: Write mutation log (if enabled)
1347
- if (enableMutationLogs) {
1348
- const logEntries: MutationLogEntry[] = processingResult.locations.map(loc => {
1349
- const failed = mutationResult.errors.find(e => e.startsWith(loc.ref));
1350
- return {
1351
- timestamp: new Date().toISOString(),
1352
- fileName,
1353
- locationRef: loc.ref,
1354
- status: failed ? 'failure' : 'success',
1355
- error: failed,
1356
- };
1357
- });
1358
-
1359
- await writeMutationLog(s3, logEntries, fileName, logPrefix, log);
1360
- }
1361
-
1362
- // Step 4: Archive file (PRIMARY deduplication - file won't be listed again)
1363
- if (enableArchival) {
1364
- log.info('📦 [ARCHIVE] Moving file to processed directory', { fileName });
1365
- await s3.moveFile(filePath, `${archivePrefix}${fileName}`);
1366
- log.info('✅ [ARCHIVE] File archived successfully', {
1367
- fileName,
1368
- destination: `${archivePrefix}${fileName}`
1369
- });
1370
- }
1371
-
1372
- // Step 5: Mark processed in KV (SECONDARY - provides metadata/history)
1373
- if (enableFileTracking && stateService) {
1374
- log.info('💾 [STATE] Recording file processing metadata', { fileName });
1375
- await stateService.markFileProcessed(fileName, {
1376
- recordCount: processingResult.locations.length,
1377
- successful: mutationResult.successful,
1378
- failed: mutationResult.failed,
1379
- mappingErrors: processingResult.errors.length,
1380
- timestamp: new Date().toISOString(),
1381
- });
1382
- }
1383
-
1384
- results.processed++;
1385
- results.totalRecords += mutationResult.successful;
1386
-
1387
- log.info('✅ [COMPLETE] File processing complete', {
1388
- fileName,
1389
- locations: processingResult.locations.length,
1390
- successful: mutationResult.successful,
1391
- failed: mutationResult.failed,
1392
- mappingErrors: processingResult.errors.length,
1393
- });
1394
-
1395
- if (processingResult.errors.length > 0 || mutationResult.errors.length > 0) {
1396
- results.errors.push(
1397
- `${fileName}: ${processingResult.errors.length} mapping errors, ${mutationResult.errors.length} mutation errors`
1398
- );
1399
- }
1400
- } catch (error: any) {
1401
- // ✅ Enhanced error logging: Extract all error details for visibility
1402
- const errorDetails = {
1403
- message: error?.message || 'Unknown error',
1404
- stack: error?.stack,
1405
- fileName: error?.fileName,
1406
- lineNumber: error?.lineNumber,
1407
- originalError: error?.context?.originalError?.message,
1408
- errorType: error?.name || 'Error',
1409
- };
1410
-
1411
- log.error('❌ [ERROR] File processing failed', errorDetails, { fileName });
1412
-
1413
- // Provide error recommendations based on error type
1414
- const recommendation = getErrorRecommendation(error);
1415
- if (recommendation) {
1416
- log.warn('💡 [RECOMMENDATION]', { fileName, recommendation });
1417
- }
1418
-
1419
- results.failed++;
1420
- results.errors.push(`${fileName}: ${error.message}`);
1421
-
1422
- // Attempt to move to error directory, ignore failures
1423
- try {
1424
- log.info('🗂️ [ERROR] Moving failed file to error directory', { fileName });
1425
- await s3.moveFile(filePath, `${errorPrefix}${fileName}`);
1426
- log.info('✅ [ERROR] Failed file moved to error directory', {
1427
- fileName,
1428
- destination: `${errorPrefix}${fileName}`
1429
- });
1430
- } catch (moveError) {
1431
- log.warn('⚠️ [ERROR] Failed to move file to error directory', {
1432
- fileName,
1433
- moveError: moveError instanceof Error ? moveError.message : String(moveError)
1434
- });
1435
- }
1436
-
1437
- // Track error state with exponential backoff (only if file tracking enabled)
1438
- if (enableFileTracking && stateService) {
1439
- try {
1440
- const stateKV = new VersoriKVAdapter(ctx);
1441
- const key = ['error-state', fileName];
1442
- const prev = (await stateKV.get(key))?.value as any;
1443
- const attempts = (prev?.attemptCount || 0) + 1;
1444
- const backoffMinutes = Math.min(Math.pow(2, attempts) * 5, 24 * 60);
1445
- const nextRetryAt = new Date(Date.now() + backoffMinutes * 60000).toISOString();
1446
-
1447
- await stateKV.set(key, {
1448
- fileName,
1449
- attemptCount: attempts,
1450
- lastError: error?.message || 'unknown',
1451
- lastAttemptAt: new Date().toISOString(),
1452
- firstFailedAt: prev?.firstFailedAt || new Date().toISOString(),
1453
- nextRetryAt,
1454
- });
1455
-
1456
- log.info('💾 [ERROR] Error state tracked with exponential backoff', {
1457
- fileName,
1458
- attempts,
1459
- nextRetryAt
1460
- });
1461
- } catch (stateError) {
1462
- log.warn('⚠️ [ERROR] Failed to track error state', {
1463
- fileName,
1464
- stateError: stateError instanceof Error ? stateError.message : String(stateError)
1465
- });
1466
- }
1467
- }
1468
- }
1469
- }
1470
-
1471
- log.info('🏁 [COMPLETE] File processing loop finished', {
1472
- processed: results.processed,
1473
- skipped: results.skipped,
1474
- failed: results.failed,
1475
- totalRecords: results.totalRecords
1476
- });
1477
-
1478
- return results;
1479
- } catch (error: any) {
1480
- // ✅ Enhanced error logging: Extract all error details for visibility
1481
- const errorDetails = {
1482
- message: error?.message || 'Unknown error',
1483
- stack: error?.stack,
1484
- errorType: error?.name || 'Error',
1485
- };
1486
-
1487
- log.error('❌ [FATAL] Location sync failed', errorDetails);
1488
-
1489
- // Provide fatal error recommendations
1490
- const recommendation = getErrorRecommendation(error);
1491
- if (recommendation) {
1492
- log.warn('💡 [RECOMMENDATION]', { recommendation });
1493
- }
1494
-
1495
- return {
1496
- success: false,
1497
- error: error.message,
1498
- processed: 0,
1499
- timestamp: new Date().toISOString(),
1500
- };
1501
- }
1502
- }
1503
-
1504
- /**
1505
- * Get error recommendation based on error type
1506
- */
1507
- function getErrorRecommendation(error: any): string | null {
1508
- const message = error?.message?.toLowerCase() || '';
1509
-
1510
- if (message.includes('s3') || message.includes('access denied')) {
1511
- return 'Check S3 credentials and bucket permissions. Verify IAM policy includes s3:ListBucket, s3:GetObject, s3:PutObject, s3:DeleteObject.';
1512
- }
1513
-
1514
- if (message.includes('xml') || message.includes('parse')) {
1515
- return 'Verify XML file structure matches expected schema. Check for malformed XML or encoding issues.';
1516
- }
1517
-
1518
- if (message.includes('mapping') || message.includes('field')) {
1519
- return 'Review field mapping configuration. Ensure all required fields are present and source paths are correct.';
1520
- }
1521
-
1522
- if (message.includes('graphql') || message.includes('mutation')) {
1523
- return 'Check GraphQL schema and mutation input. Verify all required fields are provided and types match schema.';
1524
- }
1525
-
1526
- if (message.includes('auth') || message.includes('401') || message.includes('403')) {
1527
- return 'Verify Fluent Commerce credentials. Check OAuth2 client ID/secret and ensure connection is active.';
1528
- }
1529
-
1530
- if (message.includes('timeout') || message.includes('econnrefused')) {
1531
- return 'Check network connectivity. Verify API endpoints are accessible and not rate-limited.';
1532
- }
1533
-
1534
- return null;
1535
- }
1536
- ```
1537
-
1538
- **Note:** The `executeLocationSync` function should contain the full implementation of `runLocationXmlWorkflow` (renamed to `executeLocationSync`), including all the logic for processing files, executing mutations, and logging. The implementation details are shown in the service function code above.
1539
-
1540
- ---
1541
-
1542
- ### Step 5: TypeScript Configuration
1543
-
1544
- **File: tsconfig.json**
1545
-
1546
- ```json
1547
- {
1548
- "compilerOptions": {
1549
- "module": "ES2022",
1550
- "target": "ES2024",
1551
- "moduleResolution": "node"
1552
- }
1553
- }
1554
- ```
1555
-
1556
- ---
1557
-
1558
- ## Code Flow Explanation
1559
-
1560
- ### Initialization Phase
1561
-
1562
- 1. **Read activation variables** - S3 config, Fluent config, rate limiting, logging options
1563
- 2. **Validate required variables** - Fail fast if missing credentials
1564
- 3. **Initialize SDK services** - FluentClient, S3DataSource, XMLParserService, StateService
1565
- 4. **Create mapping configuration** - Define XML → GraphQL field mappings with nested objects
1566
- 5. **Calculate rate limit delay** - Convert mutations/second to delay in milliseconds
1567
-
1568
- ### File Discovery Phase
1569
-
1570
- 1. **List S3 files** - Use `s3.listFiles()` with prefix filter (excludes processed/ subdirectory)
1571
- 2. **Filter by pattern** - Match file extension (e.g., `.xml`)
1572
- 3. **Sort newest-first** - Process most recent files first
1573
- 4. **Apply max files limit** - Prevent overwhelming the workflow
1574
- 5. **Note**: Files in `processed/` subdirectory won't be listed (primary deduplication)
1575
-
1576
- ### Per-File Processing (Service Functions)
1577
-
1578
- **Step 1: processFile()** - Download, parse, map
1579
-
1580
- 1. **Download file** - Use S3DataSource with retry logic
1581
- 2. **Parse XML** - XMLParserService converts to JavaScript object
1582
- 3. **Normalize array** - Handle single vs multiple `<location>` elements (CRITICAL for XML)
1583
- 4. **Map locations** - Use UniversalMapper with nested field mapping
1584
- 5. **Collect errors** - Track mapping errors without stopping
1585
- 6. **Return result** - FileProcessingResult with locations array and errors
1586
-
1587
- **Step 2: executeMutations()** - GraphQL mutations with rate limiting
1588
-
1589
- 1. **Loop through locations** - Process each location individually
1590
- 2. **Build mutation input** - Extract nested fields (primaryAddress, openingSchedule)
1591
- 3. **Execute mutation** - Direct GraphQL `createLocation` with retry logic
1592
- 4. **Apply rate limiting** - Add configurable delay between mutations
1593
- 5. **Track results** - Count successful vs failed, collect error messages
1594
- 6. **Return result** - MutationResult with counts and errors
1595
-
1596
- **Step 3: writeMutationLog()** - S3 log file (optional)
1597
-
1598
- 1. **Create log entries** - Map locations to log entries with status
1599
- 2. **Build JSON log** - Include timestamp, summary, detailed entries
1600
- 3. **Write to S3** - Use Buffer.from() for Deno/Versori runtime compatibility
1601
- 4. **Non-blocking** - Logging failure doesn't stop workflow
1602
- 5. **Timestamped naming** - Unique log file per processed file
1603
-
1604
- ### Cleanup Phase
1605
-
1606
- 1. **Archive file FIRST** - Move to `processed/` or `errors/` (PRIMARY deduplication)
1607
- 2. **Mark processed in KV** - StateService tracks metadata and processing history
1608
- 3. **Error state tracking** - Store error info with exponential backoff timestamp
1609
- 4. **Return results** - Summary of processed, skipped, failed files
1610
-
1611
- **Note**: The order matters! Archive first (primary deduplication), then KV tracking (metadata/history).
1612
-
1613
- ### Service Function Benefits
1614
-
1615
- - **Composability**: Each function has single responsibility and can be reused
1616
- - **Testability**: Service functions can be unit tested independently
1617
- - **Clarity**: Main workflow shows high-level orchestration
1618
- - **Error Handling**: Isolated error handling per service function
1619
- - **Logging**: Detailed logging at each step with structured context
1620
-
1621
- ---
1622
-
1623
- ## S3 Archival Deduplication Pattern
1624
-
1625
- This template uses **S3 archival** as the primary deduplication mechanism, NOT VersoriFileTracker.
1626
-
1627
- ### How It Works
1628
-
1629
- **S3 Directory Structure:**
1630
-
1631
- ```
1632
- s3://my-bucket/
1633
- ├── locations/ ← listFiles() reads from here
1634
- │ ├── new-file-1.xml
1635
- │ └── new-file-2.xml
1636
- ├── processed/ ← Successfully processed files
1637
- │ ├── old-file-1.xml
1638
- │ └── old-file-2.xml
1639
- └── errors/ ← Failed files
1640
- └── bad-file.xml
1641
- ```
1642
-
1643
- **Deduplication Flow:**
1644
-
1645
- 1. **List files**: `s3.listFiles({ prefix: 'locations/' })` - Only lists `locations/` subdirectory
1646
- 2. **Process file**: Download, parse, transform, send mutations
1647
- 3. **Archive**: `s3.moveFile(filePath, 'processed/new-file-1.xml')` - Moves file out of `locations/`
1648
- 4. **Next run**: File is now in `processed/`, won't be listed again
1649
-
1650
- **Why This Works:**
1651
-
1652
- - Files in `processed/` subdirectory are **never listed** when prefix is `locations/`
1653
- - No need to track file state in KV store for deduplication
1654
- - S3 is the single source of truth for file status
1655
- - Simple, reliable, scales to millions of files
1656
-
1657
- **StateService Role (Secondary):**
1658
-
1659
- - Provides metadata and processing history
1660
- - Backup check in case archival fails mid-process
1661
- - Useful for monitoring and debugging
1662
- - NOT the primary deduplication mechanism
1663
-
1664
- **When to Use VersoriFileTracker:**
1665
-
1666
- - **NEVER for S3 sources** - Use archival pattern instead
1667
- - **Only for SFTP sources** - Where archival might not be possible
1668
- - See SFTP templates for VersoriFileTracker usage
1669
-
1670
- ---
1671
-
1672
- ## XML Path Resolution Patterns
1673
-
1674
- ### Pattern 1: XML Attribute Access with @ Prefix
1675
-
1676
- ```typescript
1677
- const mappingConfig = {
1678
- fields: {
1679
- // XML attribute access
1680
- ref: { source: 'location.@ref' }, // <location ref="LOC-001">
1681
- type: { source: 'location.@type' }, // <location type="WAREHOUSE">
1682
- country: { source: 'location.address.@country' }, // <address country="USA">
1683
-
1684
- // XML element text content
1685
- name: { source: 'location.name' }, // <name>Downtown</name>
1686
- city: { source: 'location.address.city' }, // <address><city>NYC</city></address>
1687
- },
1688
- };
1689
- ```
1690
-
1691
- ### Pattern 2: Handling Single vs Multiple Elements
1692
-
1693
- ```typescript
1694
- // XML can have single or multiple <location> elements
1695
- const locationsData = xmlData.locations?.location;
1696
-
1697
- // Normalize to array
1698
- const locations = Array.isArray(locationsData) ? locationsData : [locationsData];
1699
-
1700
- // Process each location
1701
- for (const loc of locations) {
1702
- const record = { location: loc }; // Wrap for mapping
1703
- const result = await mapper.map(record);
1704
- }
1705
- ```
1706
-
1707
- ### Pattern 3: Nested Object Mapping
1708
-
1709
- ```typescript
1710
- const mappingConfig = {
1711
- fields: {
1712
- // Root fields
1713
- ref: { source: 'location.@ref', required: true },
1714
- name: { source: 'location.name', required: true },
1715
- type: { source: 'location.@type', required: true },
1716
-
1717
- // Nested primaryAddress object
1718
- 'primaryAddress.ref': { source: 'location.@ref' },
1719
- 'primaryAddress.street': { source: 'location.address.street1' },
1720
- 'primaryAddress.city': { source: 'location.address.city' },
1721
- 'primaryAddress.latitude': { source: 'location.coordinates.@lat', resolver: 'sdk.parseFloat' },
1722
-
1723
- // Nested openingSchedule object
1724
- 'openingSchedule.allHours': {
1725
- source: 'location.openingSchedule.allHours',
1726
- resolver: 'sdk.boolean',
1727
- },
1728
- 'openingSchedule.monStart': {
1729
- source: 'location.openingSchedule.monStart',
1730
- resolver: 'sdk.parseInt',
1731
- },
1732
-
1733
- // Note: retailer.id not shown - standard createLocation does not have this field
1734
- // If your schema requires it, add: 'retailer.id': { value: parseInt(retailerId) }
1735
- },
1736
- };
1737
- ```
1738
-
1739
- ---
1740
-
1741
- ## Sample XML Files
1742
-
1743
- ### Minimal Test File
1744
-
1745
- **File: test-location.xml**
1746
-
1747
- ```xml
1748
- <?xml version="1.0" encoding="UTF-8"?>
1749
- <locations>
1750
- <location ref="TEST-001" type="WAREHOUSE">
1751
- <name>Test Warehouse</name>
1752
- <address country="USA">
1753
- <street1>123 Test St</street1>
1754
- <city>TestCity</city>
1755
- <state>TC</state>
1756
- <postalCode>12345</postalCode>
1757
- </address>
1758
- <coordinates lat="40.7128" lon="-74.0060"/>
1759
- <timeZone>America/New_York</timeZone>
1760
- <openingSchedule>
1761
- <allHours>false</allHours>
1762
- <monStart>800</monStart>
1763
- <monEnd>1800</monEnd>
1764
- <tueStart>800</tueStart>
1765
- <tueEnd>1800</tueEnd>
1766
- <wedStart>800</wedStart>
1767
- <wedEnd>1800</wedEnd>
1768
- <thuStart>800</thuStart>
1769
- <thuEnd>1800</thuEnd>
1770
- <friStart>800</friStart>
1771
- <friEnd>1800</friEnd>
1772
- <satStart>0</satStart>
1773
- <satEnd>0</satEnd>
1774
- <sunStart>0</sunStart>
1775
- <sunEnd>0</sunEnd>
1776
- </openingSchedule>
1777
- </location>
1778
- </locations>
1779
- ```
1780
-
1781
- ---
1782
-
1783
- ## Service Functions Deep Dive
1784
-
1785
- This template demonstrates service function composition for maintainable workflows.
1786
-
1787
- ### Function 1: processFile()
1788
-
1789
- **Purpose**: Download, parse, normalize, and map XML data
1790
-
1791
- **Inputs**:
1792
- - `s3: S3DataSource` - For file download
1793
- - `parser: XMLParserService` - For XML parsing
1794
- - `mapper: UniversalMapper` - For field mapping
1795
- - `filePath: string` - S3 path to file
1796
- - `fileName: string` - File name for logging
1797
- - `log: any` - Logger instance
1798
-
1799
- **Outputs**: `FileProcessingResult`
1800
- ```typescript
1801
- {
1802
- success: boolean; // Overall success
1803
- locations: any[]; // Mapped location objects
1804
- errors: string[]; // Mapping error messages
1805
- }
1806
- ```
1807
-
1808
- **Key Operations**:
1809
- 1. Downloads file with retry logic
1810
- 2. Parses XML using XMLParserService
1811
- 3. **Normalizes arrays** - Handles single vs multiple `<location>` elements
1812
- 4. Maps each location using UniversalMapper
1813
- 5. Collects errors without stopping processing
1814
- 6. Returns all mapped locations + errors
1815
-
1816
- **Why separate function?**
1817
- - Testable independently with mock data
1818
- - Reusable across workflows (scheduled, webhook, adhoc)
1819
- - Clear error boundaries - file-level errors vs record-level errors
1820
- - Easy to add XML validation or schema checks
1821
-
1822
- ### Function 2: executeMutations()
1823
-
1824
- **Purpose**: Execute GraphQL createLocation mutations with rate limiting
1825
-
1826
- **Inputs**:
1827
- - `client: FluentClient` - For GraphQL mutations
1828
- - `locations: any[]` - Mapped location data
1829
- - `mutationDelayMs: number` - Rate limit delay
1830
- - `fileName: string` - For logging context
1831
- - `log: any` - Logger instance
1832
-
1833
- **Outputs**: `MutationResult`
1834
- ```typescript
1835
- {
1836
- successful: number; // Count of successful mutations
1837
- failed: number; // Count of failed mutations
1838
- errors: string[]; // Error messages with location refs
1839
- }
1840
- ```
1841
-
1842
- **Key Operations**:
1843
- 1. Loops through locations
1844
- 2. Builds GraphQL mutation input
1845
- 3. Executes with retry logic + rate limiting
1846
- 4. Tracks success/failure per location
1847
- 5. Returns summary counts + errors
1848
-
1849
- **Why separate function?**
1850
- - Clear separation: mapping vs mutation execution
1851
- - Rate limiting logic isolated and configurable
1852
- - Easy to swap mutation types (create vs update)
1853
- - Testable with mock FluentClient
1854
- - Can parallelize in future (batch mutations)
1855
-
1856
- ### Function 3: writeMutationLog()
1857
-
1858
- **Purpose**: Write detailed mutation results to S3 as JSON log
1859
-
1860
- **Inputs**:
1861
- - `s3: S3DataSource` - For log upload
1862
- - `logEntries: MutationLogEntry[]` - Mutation status per location
1863
- - `fileName: string` - Original file name
1864
- - `logPrefix: string` - S3 prefix for logs (e.g., `logs/`)
1865
- - `log: any` - Logger instance
1866
-
1867
- **Outputs**: `void` (non-blocking - errors logged but not thrown)
1868
-
1869
- **Key Operations**:
1870
- 1. Creates timestamped log file name
1871
- 2. Builds JSON log with summary + entries
1872
- 3. **Uses Buffer.from()** - Required for Deno/Versori runtime
1873
- 4. Writes to S3 with `uploadFile()`
1874
- 5. Errors don't stop workflow (logging is non-critical)
1875
-
1876
- **Why separate function?**
1877
- - Optional feature - can be disabled via config
1878
- - Non-blocking - logging failure doesn't fail workflow
1879
- - Structured logging for audit trails
1880
- - Easy to change log format (JSON, CSV, XML)
1881
- - Can add log rotation/cleanup logic later
1882
-
1883
- ### Service Function Composition Benefits
1884
-
1885
- **1. Maintainability**
1886
- ```typescript
1887
- // Clear workflow orchestration
1888
- const processingResult = await processFile(...);
1889
- const mutationResult = await executeMutations(...);
1890
- await writeMutationLog(...);
1891
- ```
1892
-
1893
- **2. Testability**
1894
- ```typescript
1895
- // Unit test processFile() with mock XML
1896
- const mockS3 = { downloadFile: jest.fn() };
1897
- const result = await processFile(mockS3, parser, mapper, ...);
1898
- expect(result.locations).toHaveLength(5);
1899
- ```
1900
-
1901
- **3. Reusability**
1902
- ```typescript
1903
- // Use processFile() in different workflows
1904
- export const webhook = webhook('location-webhook').then(async ctx => {
1905
- const result = await processFile(s3, parser, mapper, filePath, fileName, log);
1906
- return { locations: result.locations };
1907
- });
1908
- ```
1909
-
1910
- **4. Error Isolation**
1911
- ```typescript
1912
- // Each function has clear error boundaries
1913
- try {
1914
- const processingResult = await processFile(...);
1915
- // File-level errors caught here
1916
- } catch (error) {
1917
- // Handle file processing failure
1918
- }
1919
-
1920
- // Mutation errors don't stop file processing
1921
- const mutationResult = await executeMutations(...);
1922
- // mutationResult.errors contains per-location failures
1923
- ```
1924
-
1925
- **5. Progressive Enhancement**
1926
- ```typescript
1927
- // Easy to add features without touching core logic
1928
- async function validateLocations(locations: any[]) {
1929
- // Add validation step
1930
- }
1931
-
1932
- const processingResult = await processFile(...);
1933
- await validateLocations(processingResult.locations); // New step
1934
- const mutationResult = await executeMutations(...);
1935
- ```
1936
-
1937
- ---
1938
-
1939
- ## Versori Environment Variables
1940
-
1941
- **Activation Variables:**
1942
-
1943
- ```bash
1944
- # ============================================================================
1945
- # Required Variables
1946
- # ============================================================================
1947
- s3BucketName=my-location-bucket
1948
- awsAccessKeyId=AKIAXXXXXXXXXXXX
1949
- awsSecretAccessKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
1950
-
1951
- # ============================================================================
1952
- # S3 Configuration (Optional - with defaults)
1953
- # ============================================================================
1954
- awsRegion=us-east-1
1955
- s3Prefix=locations/
1956
- archivePrefix=processed/
1957
- errorPrefix=errors/
1958
- logPrefix=logs/
1959
- filePattern=.xml
1960
- maxFilesToProcess=10
1961
-
1962
- # ============================================================================
1963
- # Feature Toggles (Optional - with defaults)
1964
- # ============================================================================
1965
- # Enable S3 archival (move files to processed/errors directories)
1966
- enableArchival=true
1967
-
1968
- # Enable mutation logs (write detailed mutation results to S3)
1969
- enableMutationLogs=true
1970
-
1971
- # Enable file tracking via StateService + KV store
1972
- # When disabled, relies on S3 archival only for deduplication
1973
- enableFileTracking=true
1974
-
1975
- # ============================================================================
1976
- # Mutation Configuration (Optional - with defaults)
1977
- # ============================================================================
1978
- # Mutation batch size (concurrent requests)
1979
- # - 1 = Sequential (default, safest)
1980
- # - 5 = Process 5 mutations in parallel
1981
- # - 10 = Process 10 mutations in parallel
1982
- mutationBatchSize=1
1983
-
1984
- # Alias batching (combine multiple mutations into single request)
1985
- # - undefined = Disabled (default, use separate requests)
1986
- # - 5 = Combine 5 mutations per aliased request
1987
- # - 10 = Combine 10 mutations per aliased request
1988
- mutationsPerAliasBatch=
1989
-
1990
- # ============================================================================
1991
- # Fluent Commerce Configuration (Optional)
1992
- # ============================================================================
1993
- # Retailer ID - Only if mutation schema requires retailerId in input
1994
- # Standard createLocation does NOT require this field
1995
- # Check your GraphQL schema to determine if needed
1996
- retailerId=my-retailer-id
1997
- ```
1998
-
1999
- **Notes:**
2000
- - Webhook security is handled by Versori's native connection authentication. No manual API key configuration needed.
2001
- - `retailerId` - Standard createLocation does not have this field. Only use if YOUR custom schema requires it.
2002
- - See `fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md` for details.
2003
-
2004
- ---
2005
-
2006
- ## Schema Validation CLI Commands
2007
-
2008
- Before deploying, validate your field mappings against the Fluent GraphQL schema:
2009
-
2010
- ### 1. Introspect Schema
2011
-
2012
- ```bash
2013
- # Generate schema.json from live Fluent API
2014
- npx fc-connect introspect-schema \
2015
- --url https://api.fluentcommerce.com/graphql \
2016
- --client-id your-client-id \
2017
- --client-secret your-client-secret \
2018
- --output schema.json
2019
- ```
2020
-
2021
- ### 2. Create Mapping Config File
2022
-
2023
- **File: location-mapping.json**
2024
-
2025
- ```json
2026
- {
2027
- "version": "1.0.0",
2028
- "mutation": "createLocation",
2029
- "sourceFormat": "xml",
2030
- "returnFields": ["id", "ref", "name", "type", "status"],
2031
- "description": "XML location to Fluent Commerce GraphQL mapping",
2032
- "fields": {
2033
- "ref": { "source": "location.@ref", "required": true, "resolver": "sdk.trim" },
2034
- "name": { "source": "location.name", "required": true, "resolver": "sdk.trim" },
2035
- "type": { "source": "location.@type", "required": true, "resolver": "sdk.uppercase" },
2036
- "primaryAddress.ref": { "source": "location.@ref", "required": true },
2037
- "primaryAddress.street": { "source": "location.address.street1" },
2038
- "primaryAddress.latitude": {
2039
- "source": "location.coordinates.@lat",
2040
- "resolver": "sdk.parseFloat"
2041
- },
2042
- "primaryAddress.longitude": {
2043
- "source": "location.coordinates.@lon",
2044
- "resolver": "sdk.parseFloat"
2045
- }
2046
- }
2047
- }
2048
- ```
2049
-
2050
- ### 3. Validate Mapping
2051
-
2052
- ```bash
2053
- # Validate that all target fields exist in schema
2054
- npx fc-connect validate-schema \
2055
- --mapping location-mapping.json \
2056
- --schema schema.json
2057
- ```
2058
-
2059
- ### 4. Analyze Coverage
2060
-
2061
- ```bash
2062
- # Check which Location fields are mapped vs available
2063
- npx fc-connect analyze-coverage \
2064
- --mapping location-mapping.json \
2065
- --schema schema.json \
2066
- --type CreateLocationInput
2067
- ```
2068
-
2069
- **Output:**
2070
-
2071
- ```
2072
- ✅ Mapped: 15/42 fields (35%)
2073
- ❌ Missing required: timezone (String!)
2074
- ⚠️ Optional unmapped: supportPhoneNumber, networkId, attributes
2075
- ```
2076
-
2077
- ---
2078
-
2079
- ## Testing Locally
2080
-
2081
- ### 1. Upload Test XML to S3
2082
-
2083
- ```bash
2084
- aws s3 cp test-location.xml s3://my-location-bucket/locations/test-location.xml
2085
- ```
2086
-
2087
- ### 2. Deploy to Versori
2088
-
2089
- ```bash
2090
- npm run deploy
2091
- ```
2092
-
2093
- ### 3. Manual Testing
2094
-
2095
- ```bash
2096
- # Trigger manual sync (auth handled by Versori connection)
2097
- curl -X POST https://your-workspace.versori.run/location-xml-adhoc
2098
-
2099
- # Check job status
2100
- curl -X POST https://your-workspace.versori.run/location-xml-job-status \
2101
- -H "Content-Type: application/json" \
2102
- -d '{"jobId": "location-xml-adhoc-1737525600000"}'
2103
- ```
2104
-
2105
- ### 4. Verify Processing
2106
-
2107
- - Upload a small XML to S3 (2-3 locations) and trigger `adhoc` webhook
2108
- - Verify GraphQL mutations are executed for each location
2109
- - Confirm file moved from `locations/` to `processed/` in S3
2110
- - Check KV state: errors and last processed metadata
2111
- - Monitor rate limiting: verify mutations respect configured rate
2112
-
2113
- ---
2114
-
2115
- ## Deployment
2116
-
2117
- ```bash
2118
- # Deploy to Versori
2119
- npm run deploy
2120
-
2121
- # View logs
2122
- npm run logs
2123
-
2124
- # Monitor execution
2125
- versori logs --follow
2126
- ```
2127
-
2128
- ---
2129
-
2130
- ## Monitoring
2131
-
2132
- ### Success Response
2133
-
2134
- ```json
2135
- {
2136
- "success": true,
2137
- "jobId": "location-xml-scheduled-1737525600000",
2138
- "processed": 3,
2139
- "skipped": 0,
2140
- "failed": 0,
2141
- "totalRecords": 12,
2142
- "errors": []
2143
- }
2144
- ```
2145
-
2146
- ### Partial Success Response
2147
-
2148
- ```json
2149
- {
2150
- "success": true,
2151
- "jobId": "location-xml-scheduled-1737525600000",
2152
- "processed": 3,
2153
- "skipped": 1,
2154
- "failed": 0,
2155
- "totalRecords": 10,
2156
- "errors": ["locations-003.xml: 2 mapping errors"]
2157
- }
2158
- ```
2159
-
2160
- ### Error Response
2161
-
2162
- ```json
2163
- {
2164
- "success": false,
2165
- "jobId": "location-xml-scheduled-1737525600000",
2166
- "processed": 0,
2167
- "skipped": 0,
2168
- "failed": 1,
2169
- "totalRecords": 0,
2170
- "errors": ["locations-001.xml: Invalid XML structure"]
2171
- }
2172
- ```
2173
-
2174
- ---
2175
-
2176
- ## Common Pitfalls and Solutions
2177
-
2178
- ### 1. XML Attribute Not Found
2179
-
2180
- **Symptoms**: Mapping errors like "field not found"
2181
-
2182
- **Solution**:
2183
-
2184
- ```typescript
2185
- // ❌ WRONG - Missing @ prefix for attribute
2186
- ref: {
2187
- source: 'location.ref';
2188
- }
2189
-
2190
- // ✅ CORRECT - Use @ prefix for XML attributes
2191
- ref: {
2192
- source: 'location.@ref';
2193
- }
2194
- ```
2195
-
2196
- ### 2. Single Element Not Array
2197
-
2198
- **Symptoms**: "Cannot read property forEach of undefined"
2199
-
2200
- **Solution**:
2201
-
2202
- ```typescript
2203
- // Always normalize to array
2204
- const locationsData = xmlData.locations?.location;
2205
- const locations = Array.isArray(locationsData) ? locationsData : [locationsData];
2206
- ```
2207
-
2208
- ### 3. Rate Limiting Too Aggressive
2209
-
2210
- **Symptoms**: Slow processing, mutations taking too long
2211
-
2212
- **Solution**:
2213
-
2214
- ```bash
2215
- # Increase rate limit (mutations per second)
2216
- mutationRateLimit=10 # Default is 5
2217
- ```
2218
-
2219
- ### 4. Empty Element vs Missing Element
2220
-
2221
- **Solution**:
2222
-
2223
- ```typescript
2224
- // Use required: false and defaultValue for optional fields
2225
- 'primaryAddress.street2': {
2226
- source: 'location.address.street2',
2227
- required: false,
2228
- defaultValue: ''
2229
- }
2230
- ```
2231
-
2232
- ### 5. S3 Access Denied
2233
-
2234
- **Symptoms**: S3 operations fail with 403 errors
2235
-
2236
- **Solution**: Validate IAM permissions
2237
-
2238
- **Required IAM Permissions:**
2239
-
2240
- ```json
2241
- {
2242
- "Version": "2012-10-17",
2243
- "Statement": [
2244
- {
2245
- "Effect": "Allow",
2246
- "Action": ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
2247
- "Resource": ["arn:aws:s3:::my-location-bucket", "arn:aws:s3:::my-location-bucket/*"]
2248
- }
2249
- ]
2250
- }
2251
- ```
2252
-
2253
- ### 6. GraphQL Schema Mismatch
2254
-
2255
- **Symptoms**: Mutation errors like "Unknown field", "Invalid input type"
2256
-
2257
- **Solution**: Use CLI tools to validate mappings
2258
-
2259
- ```bash
2260
- npx fc-connect validate-schema --mapping location-mapping.json --schema schema.json
2261
- ```
2262
-
2263
- ### 7. Nested Object Mapping Errors
2264
-
2265
- **Symptoms**: Flat structure instead of nested objects in mutation input
2266
-
2267
- **Solution**: Use dot notation in field mapping
2268
-
2269
- ```typescript
2270
- // ✅ CORRECT - Creates nested structure
2271
- 'primaryAddress.city': { source: 'location.address.city' }
2272
-
2273
- // ❌ WRONG - Creates flat structure
2274
- primaryAddress_city: { source: 'location.address.city' }
2275
- ```
2276
-
2277
- ### 8. retailerId Configuration Errors
2278
-
2279
- **Symptoms**: "retailerId is required" errors or confusion about when to use `setRetailerId()`
2280
-
2281
- **Solution**: Understand the correct pattern
2282
-
2283
- ```typescript
2284
- // ✅ CORRECT - GraphQL mutations don't need setRetailerId()
2285
- // Check your GraphQL schema to determine retailerId handling:
2286
- // - Mandatory retailerId → Must pass it in mutation input
2287
- // - Optional retailerId → Can pass it if needed
2288
- // - No retailerId field → Don't pass it
2289
- // Standard createLocation does not have retailerId field in schema
2290
-
2291
- // ✅ IF mutation schema requires retailerId (mandatory):
2292
- const { query, variables } = await mapper.map(location);
2293
- if (retailerId && variables.input) {
2294
- variables.input.retailer = { id: parseInt(retailerId) };
2295
- }
2296
- await client.graphql({ query, variables });
2297
- ```
2298
-
2299
- **Reference:** `fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md`
2300
-
2301
- ---
2302
-
2303
- ## Key Takeaways
2304
-
2305
- ### Architecture Patterns
2306
-
2307
- - **Service Function Composition**: Workflow broken into 3 service functions - `processFile()`, `executeMutations()`, `writeMutationLog()`
2308
- - **Per-File Processing**: Main workflow orchestrates service functions for each file
2309
- - **Clear Separation of Concerns**: Parse/map, execute mutations, logging are independent functions
2310
-
2311
- ### SDK Usage
2312
-
2313
- - **Buffer Import (CRITICAL)**: Always `import { Buffer } from 'node:buffer'` for Deno/Versori runtime
2314
- - **S3 Archival Deduplication**: Use `s3.moveFile()` to `processed/` subdirectory - PRIMARY deduplication mechanism
2315
- - **NO VersoriFileTracker for S3**: S3 archival is simpler and more reliable
2316
- - **StateService Role**: SECONDARY - provides metadata/history, not primary deduplication
2317
-
2318
- ### XML Processing
2319
-
2320
- - **XML @ Prefix**: Always use `@` prefix for XML attributes (`location.@ref`)
2321
- - **Array Normalization (CRITICAL)**: Handle single vs multiple elements with `Array.isArray()` check - single `<location>` becomes object, not array
2322
- - **Nested Mapping**: Use dot notation for nested objects (`primaryAddress.street`, `openingSchedule.monStart`)
2323
-
2324
- ### GraphQL Mutations
2325
-
2326
- - **NO setRetailerId() Required**: GraphQL mutations do NOT need `client.setRetailerId()` - only Job/Event API needs it
2327
- - **retailerId in Input**: Check your GraphQL schema to determine retailerId handling:
2328
- - **Mandatory retailerId** - Field exists and is required (`!`) → Must pass it
2329
- - **Optional retailerId** - Field exists and is optional → Can pass it if needed
2330
- - **No retailerId field** - Field doesn't exist → Don't pass it
2331
- - **Rate Limiting**: Implement configurable delays between mutations to avoid API throttling
2332
- - **Retry Logic**: Exponential backoff for failed mutations with `retryWithBackoff()`
2333
- - **Direct Mutations**: Use `client.graphql()` for location upserts (NOT Batch API)
2334
- - **GraphQL vs Batch API**: Use GraphQL for low-volume master data, Batch API for high-volume inventory
2335
-
2336
- ### Error Handling & Monitoring
2337
-
2338
- - **Error Recovery**: Exponential backoff for error state tracking with retry timestamps
2339
- - **Mutation Logging**: Optional S3 JSON logs with detailed per-location mutation status
2340
- - **Schema Validation**: Use CLI tools before deployment to catch mapping errors
2341
- - **Archival Order**: Archive FIRST (deduplication), then KV tracking (metadata)
2342
-
2343
- ---
2344
-
2345
- ## Related Documentation
2346
-
2347
- ### Core Guides
2348
-
2349
- - **GraphQL Mutation Mapping**: `fc-connect-sdk/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/readme.md`
2350
- - **Universal Mapping**: `fc-connect-sdk/docs/02-CORE-GUIDES/mapping/modules/readme.md`
2351
- - **XML Parser**: `fc-connect-sdk/docs/02-CORE-GUIDES/parsers/modules/05-xml-parser.md`
2352
- - **Data Sources**: `fc-connect-sdk/docs/02-CORE-GUIDES/data-sources/readme.md`
2353
- - **State Management**: `fc-connect-sdk/docs/03-PATTERN-GUIDES/file-operations/state-duplicate-prevention.md`
2354
-
2355
- ### Related Templates
2356
-
2357
- - **CSV Version**: `template-ingestion-s3-csv-location-graphql.md`
2358
- - **JSON Version**: `template-ingestion-s3-json-location-graphql.md`
2359
- - **SFTP XML Version**: `template-ingestion-sftp-xml-location-graphql.md`
2360
- - **Event API Pattern**: `../event-api/template-ingestion-s3-xml-product-event.md`
2361
- - **Batch API Pattern**: `../batch-api/template-ingestion-s3-xml-inventory-batch.md`
2362
-
2363
- ### CLI Tools
2364
-
2365
- - **Schema Introspection**: `fc-connect-sdk/bin/readme.md#introspect-schema`
2366
- - **Mapping Validation**: `fc-connect-sdk/bin/readme.md#validate-schema`
2367
- - **Coverage Analysis**: `fc-connect-sdk/bin/readme.md#analyze-coverage`
2368
-
2369
- ### Patterns
2370
-
2371
- - **Error Handling**: `fc-connect-sdk/docs/01-TEMPLATES/patterns/error-handling-retry.md`
2372
- - **Rate Limiting**: `fc-connect-sdk/docs/03-PATTERN-GUIDES/integration-patterns/rate-limiting.md`
2373
- - **XML Patterns**: `fc-connect-sdk/docs/01-TEMPLATES/versori/patterns/xml-response-patterns.md`
1
+ ---
2
+ template_id: tpl-ingest-s3-xml-to-location-graphql
3
+ canonical_filename: template-ingestion-s3-xml-location-graphql.md
4
+ version: 2.0.0
5
+ sdk_version: ^0.1.39
6
+ runtime: versori
7
+ direction: ingestion
8
+ source: s3-xml
9
+ destination: fluent-graphql
10
+ entity: location
11
+ format: xml
12
+ logging: versori
13
+ status: stable
14
+ compliance: gold-standard
15
+ features:
16
+ - graphql-mutation-mapper
17
+ - memory-management
18
+ - enhanced-logging
19
+ - attribute-transformation
20
+ ---
21
+
22
+ Template: Ingestion - S3 XML to Location GraphQL
23
+
24
+ **Template Version:** 2.0.0
25
+ **SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
26
+ **Last Updated:** 2025-01-24
27
+
28
+ **🆕 Version 2.0.0 Enhancements:**
29
+ - ✅ **GraphQL Mutation Mapper** - Direct field mapping to mutation variables
30
+ - ✅ **Memory Management** - Clear large arrays after processing batches
31
+ - ✅ **Enhanced Logging** - Track mutation execution with emoji indicators
32
+ - ✅ **Attribute Transformation** - Handle complex nested data structures
33
+
34
+ ---
35
+
36
+ ## Implementation Prompt
37
+
38
+ "Create a Versori scheduled workflow that reads location XML files from S3, transforms the data using GraphQLMutationMapper with nested object mapping, and creates/updates Fluent Commerce locations via direct GraphQL mutations with alias batching support.
39
+
40
+ **Requirements:**
41
+
42
+ 1. **S3 Source**:
43
+ - List and download XML files with configurable prefix and pattern
44
+ - Archive processed files to `processed/` directory (deduplication via archiving)
45
+ - Move failed files to `errors/` directory
46
+ - Use S3DataSource with retry logic and built-in archival
47
+ - Support configurable bucket, region, credentials
48
+
49
+ 2. **XML Parsing**:
50
+ - Use XMLParserService with @ prefix for attribute access
51
+ - Handle both single and multiple `<location>` elements (array normalization)
52
+ - Support nested XML paths (`location.address.street1`, `location.coordinates.@lat`)
53
+ - Parse complex structures (addresses, coordinates, opening schedules)
54
+
55
+ 3. **Field Mapping**:
56
+ - Map XML fields to Location GraphQL input type with nested objects:
57
+ - `ref`, `name`, `type` (root fields)
58
+ - `primaryAddress.*` (nested address object with coordinates)
59
+ - `openingSchedule.*` (nested schedule object with 7-day hours)
60
+ - Use SDK resolvers (trim, uppercase, parseFloat, parseInt, boolean)
61
+ - Support custom resolvers for complex transformations
62
+ - Validate required fields
63
+ - Note: Check your GraphQL schema to determine if `retailer.id` field exists and is mandatory/optional
64
+
65
+ 4. **GraphQL Mutations** (Direct - NO Batch API):
66
+ - Execute `createLocation` mutation directly for each location
67
+ - **NO BPP (Batch Pre-Processing)** - Not applicable for direct GraphQL mutations
68
+ - **NO Batch API** - Use direct GraphQL mutations with rate limiting instead
69
+ - Use rate limiting (configurable mutations per second)
70
+ - Add delay between mutations to avoid API throttling
71
+ - Retry failed mutations with exponential backoff
72
+ - Track successful vs failed mutations per file
73
+
74
+ 5. **Job Tracking & State Management**:
75
+ - **Use JobTracker** - Track job lifecycle (start, complete, fail) in KV store
76
+ - Use StateService + VersoriKVAdapter for duplicate file prevention
77
+ - Track processed files in KV store with metadata
78
+ - Store error state with exponential backoff tracking
79
+ - Support distributed state across workflow runs
80
+ - Provide job status endpoint for monitoring
81
+
82
+ 6. **Error Handling**:
83
+ - File-level errors archived to `/errors/` subdirectory
84
+ - Record-level errors tracked but don't stop file processing
85
+ - Mapping errors logged with specific location context
86
+ - Mutation errors retried with exponential backoff
87
+ - Error state tracking with next retry timestamp
88
+
89
+ 7. **Advanced Features**:
90
+ - Configurable rate limiting (mutations per second)
91
+ - Empty file detection and archival
92
+ - Timestamp-based error tracking
93
+ - Comprehensive monitoring and logging
94
+ - Manual webhook trigger with job tracking
95
+ - Job status query endpoint
96
+
97
+ **Use SDK Components:**
98
+
99
+ - `createClient()` - Universal client factory for Versori
100
+ - `S3DataSource` - S3 operations with retry logic
101
+ - `XMLParserService` - XML parsing with @ prefix attribute support
102
+ - `GraphQLMutationMapper` - Field transformation with schema validation and nested object support
103
+ - `StateService` + `VersoriKVAdapter` - Duplicate prevention with KV storage
104
+ - Native Versori `log` - Structured logging
105
+
106
+ **Configuration Variables** (from Versori activation):
107
+
108
+ ```typescript
109
+ {
110
+ s3: {
111
+ bucketName: string;
112
+ region: string;
113
+ accessKeyId: string;
114
+ secretAccessKey: string;
115
+ prefix: string; // e.g., 'locations/'
116
+ archivePrefix: string; // e.g., 'processed/'
117
+ errorPrefix: string; // e.g., 'errors/'
118
+ filePattern: string; // e.g., '.xml'
119
+ maxFilesToProcess: number;
120
+ enableArchival: boolean;
121
+ },
122
+ fluent: {
123
+ retailerId: string; // Optional: Only if mutation schema requires retailerId in input
124
+ mutationRateLimit: number; // mutations per second (e.g., 5)
125
+ }
126
+ }
127
+ ```
128
+
129
+ **Architecture Pattern:**
130
+
131
+ ```
132
+ S3 Bucket → List Files → Download XML → Parse → Map → GraphQL Mutation → Archive/Move
133
+ ↓ ↓ ↓ ↓ ↓ ↓ ↓
134
+ Configure Filter by S3DataSource XML Universal Direct S3 moveFile()
135
+ Connection Pattern + Retry Parser Mapper createLocation to processed/
136
+ (@) (nested) + Rate Limit or errors/
137
+ (deduplication)
138
+ ```
139
+
140
+ **Deliverables:**
141
+
142
+ 1. Complete Versori workflow with package.json
143
+ 2. Main workflow logic with S3 + XML + GraphQL patterns
144
+ 3. Helper functions for rate limiting and retry
145
+ 4. XML path resolution examples
146
+ 5. Sample XML files with nested structures
147
+ 6. Schema validation CLI commands
148
+ 7. Testing and deployment instructions
149
+ 8. Monitoring and troubleshooting guidance"
150
+
151
+ ---
152
+
153
+ # STEP 3: Complete Implementation
154
+
155
+ ## Versori Scheduled: S3 XML → Location GraphQL
156
+
157
+ **FC Connect SDK Use Case Guide**
158
+
159
+ > **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
160
+ > **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
161
+
162
+ **Context**: Versori scheduled workflow that reads location XML files from S3 and creates/updates Fluent Commerce locations via GraphQL mutations with XML path resolution and rate limiting
163
+
164
+ **Complexity**: Medium
165
+
166
+ **Runtime**: Versori Platform
167
+
168
+ **Estimated Lines**: ~850 lines (with comprehensive documentation)
169
+
170
+ ---
171
+
172
+ ## What You'll Build
173
+
174
+ - Scheduled Versori workflow (daily location sync)
175
+ - S3 file listing, download, and archival/move
176
+ - XML parsing with @ prefix for attributes (XPath-style)
177
+ - Array normalization (single element → array conversion)
178
+ - GraphQLMutationMapper-based field transformations with nested objects
179
+ - GraphQL mutations for location upserts with alias batching support
180
+ - Retry logic with exponential backoff
181
+ - StateService duplicate prevention (KV-backed)
182
+ - Error state tracking and file error archival
183
+ - Manual webhook trigger and job status endpoint
184
+
185
+ ---
186
+
187
+ ## When to Use GraphQL Mutations vs Batch API vs Event API
188
+
189
+ ### ✅ Use GraphQL Mutations For:
190
+
191
+ | Entity Type | Use Case | Why GraphQL |
192
+ | -------------- | ---------------------------------------- | ------------------------------------- |
193
+ | **Locations** | Store/warehouse master data (low volume) | Direct control, immediate validation |
194
+ | **Controls** | System configuration, settings | Single operations, complex queries |
195
+ | **Prices** | Price updates (moderate volume) | Immediate feedback, custom logic |
196
+ | **Single Ops** | One-off creates/updates | Testing, debugging, direct API access |
197
+
198
+ ### ❌ Use Event API Instead For:
199
+
200
+ | Entity Type | Use Case | Why Event API |
201
+ | ------------------- | -------------------------------------- | -------------------------------------------- |
202
+ | **Products** | Product catalog sync, variant updates | Triggers workflows, validates business rules |
203
+ | **Customers** | Customer registration, profile updates | Needs workflow for downstream systems |
204
+ | **Orders** | Order creation, status updates | Event-driven fulfillment workflows |
205
+ | **Custom Entities** | Any entity needing workflow triggers | Full Rubix workflow support |
206
+
207
+ ### 🔄 Use Batch API For:
208
+
209
+ | Entity Type | Use Case | Why Batch API |
210
+ | ------------------ | ---------------------------------- | ----------------------------------------------- |
211
+ | **Inventory ONLY** | Bulk inventory updates, daily sync | Optimized for high-volume, BPP change detection |
212
+
213
+ ---
214
+
215
+ ## XML File Format
216
+
217
+ ### Sample: locations.xml
218
+
219
+ ```xml
220
+ <?xml version="1.0" encoding="UTF-8"?>
221
+ <locations>
222
+ <location ref="LOC-001" type="WAREHOUSE">
223
+ <name>Downtown Warehouse</name>
224
+ <address country="USA">
225
+ <street1>123 Main St</street1>
226
+ <street2>Building A</street2>
227
+ <city>New York</city>
228
+ <state>NY</state>
229
+ <postalCode>10001</postalCode>
230
+ </address>
231
+ <coordinates lat="40.7128" lon="-74.0060"/>
232
+ <timeZone>America/New_York</timeZone>
233
+ <openingSchedule>
234
+ <allHours>false</allHours>
235
+ <monStart>800</monStart>
236
+ <monEnd>1800</monEnd>
237
+ <tueStart>800</tueStart>
238
+ <tueEnd>1800</tueEnd>
239
+ <wedStart>800</wedStart>
240
+ <wedEnd>1800</wedEnd>
241
+ <thuStart>800</thuStart>
242
+ <thuEnd>1800</thuEnd>
243
+ <friStart>800</friStart>
244
+ <friEnd>1800</friEnd>
245
+ <satStart>0</satStart>
246
+ <satEnd>0</satEnd>
247
+ <sunStart>0</sunStart>
248
+ <sunEnd>0</sunEnd>
249
+ </openingSchedule>
250
+ </location>
251
+
252
+ <location ref="LOC-002" type="DC">
253
+ <name>Regional DC</name>
254
+ <address country="USA">
255
+ <street1>456 Industrial Pkwy</street1>
256
+ <city>Los Angeles</city>
257
+ <state>CA</state>
258
+ <postalCode>90001</postalCode>
259
+ </address>
260
+ <coordinates lat="34.0522" lon="-118.2437"/>
261
+ <timeZone>America/Los_Angeles</timeZone>
262
+ <openingSchedule>
263
+ <allHours>true</allHours>
264
+ <monStart>0</monStart>
265
+ <monEnd>0</monEnd>
266
+ <tueStart>0</tueStart>
267
+ <tueEnd>0</tueEnd>
268
+ <wedStart>0</wedStart>
269
+ <wedEnd>0</wedEnd>
270
+ <thuStart>0</thuStart>
271
+ <thuEnd>0</thuEnd>
272
+ <friStart>0</friStart>
273
+ <friEnd>0</friEnd>
274
+ <satStart>0</satStart>
275
+ <satEnd>0</satEnd>
276
+ <sunStart>0</sunStart>
277
+ <sunEnd>0</sunEnd>
278
+ </openingSchedule>
279
+ </location>
280
+ </locations>
281
+ ```
282
+
283
+ **XML Path Syntax (with @ prefix for attributes):**
284
+
285
+ - `location.@ref` → XML attribute `ref` on `<location>` element
286
+ - `location.name` → Text content of `<name>` element
287
+ - `location.address.street1` → Nested element path
288
+ - `location.address.@country` → XML attribute on nested `<address>` element
289
+ - `location.coordinates.@lat` → XML attribute for latitude
290
+ - `location.openingSchedule.monStart` → Deeply nested element
291
+
292
+ **Note**: The SDK's `XMLParserService` automatically handles XML attributes using `@` prefix notation.
293
+
294
+ ---
295
+
296
+ ## Versori Workflows Structure
297
+
298
+ **Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
299
+
300
+ **Trigger Types:**
301
+ - **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
302
+ - **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
303
+ - **`workflow()`** → Durable workflows (advanced, rarely used)
304
+
305
+ **Execution Steps (chained to triggers):**
306
+ - **`http()`** → External API calls (chained from schedule/webhook)
307
+ - **`fn()`** → Internal processing (chained from schedule/webhook)
308
+
309
+ ### Recommended Project Structure
310
+
311
+ ```
312
+ s3-xml-location-graphql/
313
+ ├── index.ts # Entry point - exports all workflows
314
+ └── src/
315
+ ├── workflows/
316
+ │ ├── scheduled/
317
+ │ │ └── daily-location-sync.ts # Scheduled: Daily location sync
318
+ │ │
319
+ │ └── webhook/
320
+ │ ├── adhoc-location-sync.ts # Webhook: Manual trigger
321
+ │ └── job-status-check.ts # Webhook: Status query
322
+
323
+ ├── services/
324
+ │ └── location-sync.service.ts # Shared orchestration logic (reusable)
325
+
326
+ └── config/
327
+ └── location-mapping.json # GraphQL mapping config
328
+ ```
329
+
330
+ ---
331
+
332
+ ## Complete Versori Workflow
333
+
334
+ ### Step 1: Package Configuration
335
+
336
+ **File: package.json**
337
+
338
+ ```json
339
+ {
340
+ "name": "versori-s3-xml-location-sync",
341
+ "version": "1.0.0",
342
+ "description": "Versori workflow: S3 XML location sync to Fluent GraphQL",
343
+ "versori": {
344
+ "workflows": "./index.ts"
345
+ },
346
+ "type": "module",
347
+ "scripts": {
348
+ "deploy": "versori deploy",
349
+ "logs": "versori logs"
350
+ },
351
+ "dependencies": {
352
+ "@fluentcommerce/fc-connect-sdk": "^0.1.39",
353
+ "@versori/run": "latest"
354
+ },
355
+ "devDependencies": {
356
+ "typescript": "^5.0.0",
357
+ "@types/node": "^20.0.0"
358
+ }
359
+ }
360
+ ```
361
+
362
+ ### Step 2: Workflow Entry Point (`index.ts`)
363
+
364
+ **Purpose**: Register all workflows with Versori platform
365
+
366
+ ```typescript
367
+ /**
368
+ * Entry Point - Registers all workflows with Versori platform
369
+ *
370
+ * Versori automatically discovers and registers exported workflows
371
+ *
372
+ * File Structure:
373
+ * - src/workflows/scheduled/ → Time-based triggers (cron)
374
+ * - src/workflows/webhook/ → HTTP-based triggers (webhooks)
375
+ */
376
+
377
+ // Scheduled workflows
378
+ export { dailyLocationSync } from './src/workflows/scheduled/daily-location-sync';
379
+
380
+ // Webhook workflows
381
+ export { adhocLocationSync } from './src/workflows/webhook/adhoc-location-sync';
382
+ export { locationSyncJobStatus } from './src/workflows/webhook/job-status-check';
383
+ ```
384
+
385
+ **What Gets Exposed:**
386
+ - ✅ `adhocLocationSync` → `https://{workspace}.versori.run/location-sync-adhoc`
387
+ - ✅ `locationSyncJobStatus` → `https://{workspace}.versori.run/location-sync-job-status`
388
+ - ❌ `dailyLocationSync` → NOT exposed (runs automatically on cron)
389
+
390
+ ---
391
+
392
+ ### Step 3: Workflow Files
393
+
394
+ #### `src/workflows/scheduled/daily-location-sync.ts`
395
+
396
+ **Purpose**: Automatic daily location sync
397
+ **Trigger**: Cron schedule (`0 2 * * *`)
398
+ **Exposed as Endpoint**: ❌ NO - Runs automatically
399
+
400
+ ```typescript
401
+ import { schedule, http } from '@versori/run';
402
+ import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
403
+ import { executeLocationSync } from '../../services/location-sync.service';
404
+
405
+ /**
406
+ * Scheduled Workflow: Daily Location Sync
407
+ *
408
+ * Runs automatically daily at 2 AM UTC
409
+ * NOT exposed as HTTP endpoint - Versori executes on schedule
410
+ *
411
+ * Uses shared service: location-sync.service.ts
412
+ */
413
+ export const dailyLocationSync = schedule(
414
+ 'location-sync-scheduled',
415
+ '0 2 * * *' // Daily at 2 AM UTC
416
+ ).then(
417
+ http('run-location-sync', { connection: 'fluent_commerce' }, async ctx => {
418
+ const startTime = Date.now();
419
+ const { log, openKv } = ctx;
420
+ const jobId = `location-sync-${Date.now()}`;
421
+ const tracker = new JobTracker(openKv(':project:'), log);
422
+
423
+ log.info('🚀 [START] Daily location sync initiated', { jobId, trigger: 'schedule' });
424
+
425
+ await tracker.createJob(jobId, { triggeredBy: 'schedule', stage: 'initialization' });
426
+ await tracker.updateJob(jobId, { status: 'processing' });
427
+
428
+ try {
429
+ log.info('⚙️ [PROCESSING] Starting location synchronization workflow', { jobId });
430
+ const result = await executeLocationSync(ctx, jobId, tracker);
431
+ await tracker.markCompleted(jobId, result);
432
+
433
+ const duration = Date.now() - startTime;
434
+ log.info('✅ [SUCCESS] Daily location sync completed', {
435
+ jobId,
436
+ duration: `${duration}ms`,
437
+ processed: result.processed,
438
+ totalRecords: result.totalRecords
439
+ });
440
+
441
+ return { success: true, jobId, duration, ...result };
442
+ } catch (e: any) {
443
+ await tracker.markFailed(jobId, e);
444
+ const duration = Date.now() - startTime;
445
+
446
+ log.error('❌ [FAILED] Daily location sync failed', {
447
+ jobId,
448
+ duration: `${duration}ms`,
449
+ error: e?.message,
450
+ errorType: e?.name
451
+ });
452
+
453
+ return { success: false, jobId, duration, error: e?.message };
454
+ }
455
+ })
456
+ );
457
+ ```
458
+
459
+ ---
460
+
461
+ #### `src/workflows/webhook/adhoc-location-sync.ts`
462
+
463
+ **Purpose**: Manual location sync trigger (on-demand)
464
+ **Trigger**: HTTP POST
465
+ **Endpoint**: `POST https://{workspace}.versori.run/location-sync-adhoc`
466
+ **Use Cases**: Testing, priority processing, ad-hoc runs
467
+
468
+ ```typescript
469
+ import { webhook, http } from '@versori/run';
470
+ import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
471
+ import { executeLocationSync } from '../../services/location-sync.service';
472
+
473
+ /**
474
+ * Webhook: Manual Location Sync Trigger
475
+ *
476
+ * Endpoint: POST https://{workspace}.versori.run/location-sync-adhoc
477
+ * Request body (optional): { filePattern: "urgent_*.xml", maxFiles: 5 }
478
+ *
479
+ * Pattern: webhook().then(http()) - needs Fluent API access
480
+ * Uses shared service: location-sync.service.ts
481
+ *
482
+ * SECURITY: Authentication handled via connection parameter
483
+ * No manual API key validation needed - Versori manages this via connection auth
484
+ */
485
+ export const adhocLocationSync = webhook('location-sync-adhoc', {
486
+ response: { mode: 'sync' },
487
+ connection: 'location-sync-adhoc', // Versori validates API key
488
+ }).then(
489
+ http('run-location-sync-adhoc', { connection: 'fluent_commerce' }, async ctx => {
490
+ const startTime = Date.now();
491
+ const { log, openKv, data } = ctx;
492
+ const jobId = `location-sync-adhoc-${Date.now()}`;
493
+ const tracker = new JobTracker(openKv(':project:'), log);
494
+
495
+ log.info('🚀 [START] Manual location sync triggered', { jobId, trigger: 'webhook', options: data });
496
+
497
+ await tracker.createJob(jobId, {
498
+ triggeredBy: 'manual',
499
+ stage: 'initialization',
500
+ options: data // Optional: filePattern, maxFiles, etc.
501
+ });
502
+ await tracker.updateJob(jobId, { status: 'processing' });
503
+
504
+ try {
505
+ log.info('⚙️ [PROCESSING] Starting manual location synchronization', { jobId });
506
+ const result = await executeLocationSync(ctx, jobId, tracker);
507
+ await tracker.markCompleted(jobId, result);
508
+
509
+ const duration = Date.now() - startTime;
510
+ log.info('✅ [SUCCESS] Manual location sync completed', {
511
+ jobId,
512
+ duration: `${duration}ms`,
513
+ processed: result.processed,
514
+ totalRecords: result.totalRecords
515
+ });
516
+
517
+ return { success: true, jobId, duration, ...result };
518
+ } catch (e: any) {
519
+ await tracker.markFailed(jobId, e);
520
+ const duration = Date.now() - startTime;
521
+
522
+ log.error('❌ [FAILED] Manual location sync failed', {
523
+ jobId,
524
+ duration: `${duration}ms`,
525
+ error: e?.message,
526
+ errorType: e?.name
527
+ });
528
+
529
+ return { success: false, jobId, duration, error: e?.message };
530
+ }
531
+ })
532
+ );
533
+ ```
534
+
535
+ ---
536
+
537
+ #### `src/workflows/webhook/job-status-check.ts`
538
+
539
+ **Purpose**: Query job status
540
+ **Trigger**: HTTP POST
541
+ **Endpoint**: `POST https://{workspace}.versori.run/location-sync-job-status`
542
+ **Request body**: `{ "jobId": "location-sync-1234567890" }`
543
+
544
+ ```typescript
545
+ import { webhook, fn } from '@versori/run';
546
+ import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
547
+
548
+ /**
549
+ * Webhook: Job Status Check
550
+ *
551
+ * Endpoint: POST https://{workspace}.versori.run/location-sync-job-status
552
+ * Request body: { "jobId": "location-sync-1234567890" }
553
+ *
554
+ * Pattern: webhook().then(fn()) - no external API needed, only KV storage
555
+ * Lightweight: Only queries KV store, no Fluent API calls
556
+ *
557
+ * SECURITY: Authentication handled via connection parameter
558
+ * No manual API key validation needed - Versori manages this via connection auth
559
+ */
560
+ export const locationSyncJobStatus = webhook('location-sync-job-status', {
561
+ response: { mode: 'sync' },
562
+ connection: 'location-sync-job-status',
563
+ }).then(
564
+ fn('status', async ctx => {
565
+ const { data, log, openKv } = ctx;
566
+ const jobId = data?.jobId as string;
567
+
568
+ if (!jobId) {
569
+ return { success: false, error: 'jobId required' };
570
+ }
571
+
572
+ const tracker = new JobTracker(openKv(':project:'), log);
573
+ const status = await tracker.getJob(jobId);
574
+
575
+ return status
576
+ ? { success: true, jobId, ...status }
577
+ : { success: false, error: 'Job not found', jobId };
578
+ })
579
+ );
580
+ ```
581
+
582
+ ---
583
+
584
+ ### Step 4: Main Orchestration Service (`src/services/location-sync.service.ts`)
585
+
586
+ **Note:** This service file should contain the `executeLocationSync` function (renamed from `runLocationXmlWorkflow`). The main workflow logic should be moved here.
587
+
588
+ ```typescript
589
+ /**
590
+ * Main Orchestration Service: Location Sync
591
+ *
592
+ * This service contains the core business logic for location synchronization.
593
+ *
594
+ * Features:
595
+ * - S3 file operations with archival-based deduplication (moveFile to processed/)
596
+ * - XML parsing with @ prefix for attributes
597
+ * - Single element → array normalization
598
+ * - GraphQLMutationMapper for nested field transformations
599
+ * - GraphQL mutations with alias batching support
600
+ * - StateService for metadata tracking (secondary to archival)
601
+ * - Error state tracking with exponential backoff
602
+ *
603
+ * Deduplication Strategy:
604
+ * - PRIMARY: S3 archival via moveFile() - Files in processed/ won't be re-listed
605
+ * - SECONDARY: StateService KV tracking - Provides metadata and processing history
606
+ */
607
+ import { Buffer } from 'node:buffer'; // Required for Deno/Versori runtime
608
+ import {
609
+ createClient,
610
+ S3DataSource,
611
+ XMLParserService,
612
+ GraphQLMutationMapper,
613
+ StateService,
614
+ VersoriKVAdapter,
615
+ JobTracker,
616
+ FluentClient,
617
+ } from '@fluentcommerce/fc-connect-sdk';
618
+
619
+ // ============================================================================
620
+ // Type Definitions
621
+ // ============================================================================
622
+
623
+ interface FileProcessingResult {
624
+ success: boolean;
625
+ locations: any[];
626
+ errors: string[];
627
+ }
628
+
629
+ interface MutationResult {
630
+ successful: number;
631
+ failed: number;
632
+ errors: string[];
633
+ }
634
+
635
+ interface MutationLogEntry {
636
+ timestamp: string;
637
+ fileName: string;
638
+ locationRef: string;
639
+ status: 'success' | 'failure';
640
+ error?: string;
641
+ }
642
+
643
+ // ============================================================================
644
+ // Utility Functions
645
+ // ============================================================================
646
+
647
+ /**
648
+ * Retry utility with exponential backoff
649
+ * (User-defined - not part of SDK public API)
650
+ */
651
+ async function retryWithBackoff<T>(
652
+ operation: () => Promise<T>,
653
+ maxRetries = 3,
654
+ baseDelayMs = 1000
655
+ ): Promise<T> {
656
+ let lastError: any;
657
+ for (let attempt = 0; attempt < maxRetries; attempt++) {
658
+ try {
659
+ return await operation();
660
+ } catch (error) {
661
+ lastError = error;
662
+ if (attempt < maxRetries - 1) {
663
+ const delay = baseDelayMs * Math.pow(2, attempt) + Math.floor(Math.random() * 200);
664
+ await new Promise(resolve => setTimeout(resolve, delay));
665
+ }
666
+ }
667
+ }
668
+ throw lastError;
669
+ }
670
+
671
+ /**
672
+ * Rate limiter for GraphQL mutations
673
+ */
674
+ async function rateLimitedMutation(operation: () => Promise<any>, delayMs: number): Promise<any> {
675
+ const result = await operation();
676
+ await new Promise(resolve => setTimeout(resolve, delayMs));
677
+ return result;
678
+ }
679
+
680
+ // ============================================================================
681
+ // Service Functions
682
+ // ============================================================================
683
+
684
+ /**
685
+ * Service Function 1: Process File
686
+ *
687
+ * Downloads XML from S3, parses with XMLParserService, normalizes arrays,
688
+ * and maps fields with GraphQLMutationMapper.
689
+ *
690
+ * @param s3 - S3DataSource instance
691
+ * @param parser - XMLParserService instance
692
+ * @param mapper - GraphQLMutationMapper instance
693
+ * @param filePath - Full S3 path to file
694
+ * @param fileName - File name only (for logging)
695
+ * @param log - Logger instance
696
+ * @returns FileProcessingResult with locations array and errors
697
+ */
698
+ async function processFile(
699
+ s3: S3DataSource,
700
+ parser: XMLParserService,
701
+ mapper: GraphQLMutationMapper,
702
+ filePath: string,
703
+ fileName: string,
704
+ log: any
705
+ ): Promise<FileProcessingResult> {
706
+ try {
707
+ log.info('Processing file', { fileName });
708
+
709
+ // Download with retry
710
+ const content = await retryWithBackoff(
711
+ () => s3.downloadFile(filePath, { encoding: 'utf8' }) as Promise<string>
712
+ );
713
+
714
+ // Parse XML
715
+ const xmlData = await parser.parse(content);
716
+
717
+ // Extract location array (handle both single and multiple locations)
718
+ // CRITICAL: XML array normalization - single <location> becomes object, not array
719
+ const locationsData = xmlData.locations?.location;
720
+ const locationsArray = Array.isArray(locationsData) ? locationsData : [locationsData];
721
+
722
+ if (!locationsArray || locationsArray.length === 0) {
723
+ log.warn('Empty file (no locations)', { fileName });
724
+ return {
725
+ success: true,
726
+ locations: [],
727
+ errors: [],
728
+ };
729
+ }
730
+
731
+ // Map each location using GraphQLMutationMapper
732
+ const mappedLocations: Array<{ query: string; variables: any; input: any }> = [];
733
+ const mappingErrors: string[] = [];
734
+
735
+ // ✅ PRODUCTION ENHANCEMENT: Log transformation start
736
+ log.info('Transforming locations to GraphQL mutations', {
737
+ fileName,
738
+ totalLocations: locationsArray.length,
739
+ });
740
+
741
+ for (let i = 0; i < locationsArray.length; i++) {
742
+ const locationNumber = i + 1;
743
+
744
+ // ✅ PRODUCTION ENHANCEMENT: Log progress every 50 locations
745
+ if (locationNumber % 50 === 0) {
746
+ log.info(`📤 Transforming location ${locationNumber}/${locationsArray.length}`, {
747
+ fileName,
748
+ locationNumber,
749
+ totalLocations: locationsArray.length,
750
+ validSoFar: mappedLocations.length,
751
+ errorsSoFar: mappingErrors.length,
752
+ progress: `${((locationNumber / locationsArray.length) * 100).toFixed(1)}%`,
753
+ });
754
+ }
755
+
756
+ // Wrap location in context for mapping
757
+ const record = { location: locationsArray[i] };
758
+ try {
759
+ // GraphQLMutationMapper returns { query, variables } directly
760
+ const mappingResult = await mapper.map(record);
761
+
762
+ mappedLocations.push({
763
+ query: mappingResult.query,
764
+ variables: mappingResult.variables,
765
+ input: mappingResult.variables.input || mappingResult.variables,
766
+ });
767
+ } catch (error: unknown) {
768
+ const errorMsg = error instanceof Error ? error.message : String(error);
769
+ mappingErrors.push(`Location ${locationNumber}: ${errorMsg}`);
770
+ log.warn('Mapping failed for location', { fileName, index: i, error: errorMsg });
771
+ }
772
+ }
773
+
774
+ log.info('File processed', {
775
+ fileName,
776
+ total: locationsArray.length,
777
+ mapped: mappedLocations.length,
778
+ errors: mappingErrors.length,
779
+ successRate: `${((mappedLocations.length / locationsArray.length) * 100).toFixed(1)}%`,
780
+ });
781
+
782
+ return {
783
+ success: true,
784
+ locations: mappedLocations,
785
+ errors: mappingErrors,
786
+ };
787
+ } catch (error: any) {
788
+ // ✅ Enhanced error logging: Extract all error details for visibility
789
+ const errorDetails = {
790
+ message: error?.message || 'Unknown error',
791
+ stack: error?.stack,
792
+ fileName: error?.fileName,
793
+ lineNumber: error?.lineNumber,
794
+ originalError: error?.context?.originalError?.message,
795
+ errorType: error?.name || 'Error',
796
+ };
797
+ log.error('File processing failed', errorDetails, { fileName });
798
+ return {
799
+ success: false,
800
+ locations: [],
801
+ errors: [error.message || 'Unknown error'],
802
+ };
803
+ }
804
+ }
805
+
806
+ /**
807
+ * Service Function 2: Execute Mutations
808
+ *
809
+ * Executes GraphQL createLocation mutations with alias batching support.
810
+ *
811
+ * @param client - FluentClient instance
812
+ * @param mapper - GraphQLMutationMapper instance
813
+ * @param locations - Array of mapped location objects with query and variables
814
+ * @param log - Logger instance
815
+ * @param retailerId - Fluent retailer ID
816
+ * @param batchSize - Number of concurrent requests (default: 1)
817
+ * @param mutationsPerAliasBatch - Optional: Number of mutations per aliased request (default: undefined)
818
+ * @returns MutationResult with success/failure counts
819
+ */
820
+ async function executeMutations(
821
+ client: FluentClient,
822
+ mapper: GraphQLMutationMapper,
823
+ locations: Array<{ query: string; variables: any; input: any }>,
824
+ log: any,
825
+ retailerId: string,
826
+ batchSize: number = 1, // ✅ Default: 1 (sequential)
827
+ mutationsPerAliasBatch?: number // ✅ NEW: Alias batching parameter (default: undefined = disabled)
828
+ ): Promise<MutationResult> {
829
+ // Determine mode: use aliases if mutationsPerAliasBatch is set and > 1
830
+ const useAliases = mutationsPerAliasBatch && mutationsPerAliasBatch > 1;
831
+
832
+ if (useAliases) {
833
+ return await executeMutationsWithAliases(
834
+ client,
835
+ mapper,
836
+ locations,
837
+ log,
838
+ retailerId,
839
+ batchSize,
840
+ mutationsPerAliasBatch!
841
+ );
842
+ } else {
843
+ return await executeMutationsSeparate(
844
+ client,
845
+ locations,
846
+ log,
847
+ batchSize
848
+ );
849
+ }
850
+ }
851
+
852
+ /**
853
+ * Execute mutations using separate concurrent requests (current mode)
854
+ */
855
+ async function executeMutationsSeparate(
856
+ client: FluentClient,
857
+ locations: Array<{ query: string; variables: any; input: any }>,
858
+ log: any,
859
+ batchSize: number
860
+ ): Promise<MutationResult> {
861
+ const result: MutationResult = {
862
+ successful: 0,
863
+ failed: 0,
864
+ errors: [],
865
+ };
866
+
867
+ const safeConc = Math.max(1, Math.floor(batchSize));
868
+
869
+ // Sequential mode
870
+ if (safeConc === 1) {
871
+ for (const location of locations) {
872
+ try {
873
+ await retryWithBackoff(() =>
874
+ client.graphql({
875
+ query: location.query,
876
+ variables: location.variables,
877
+ })
878
+ );
879
+ result.successful++;
880
+ } catch (error: unknown) {
881
+ result.failed++;
882
+ const errorMsg = error instanceof Error ? error.message : String(error);
883
+ result.errors.push(errorMsg);
884
+ }
885
+ }
886
+ return result;
887
+ }
888
+
889
+ // Parallel mode
890
+ for (let i = 0; i < locations.length; i += safeConc) {
891
+ const chunk = locations.slice(i, i + safeConc);
892
+ const results = await Promise.allSettled(
893
+ chunk.map(loc =>
894
+ retryWithBackoff(() =>
895
+ client.graphql({
896
+ query: loc.query,
897
+ variables: loc.variables,
898
+ })
899
+ )
900
+ )
901
+ );
902
+
903
+ results.forEach((settledResult, idx) => {
904
+ if (settledResult.status === 'fulfilled') {
905
+ result.successful++;
906
+ } else {
907
+ result.failed++;
908
+ result.errors.push(
909
+ settledResult.reason instanceof Error ? settledResult.reason.message : String(settledResult.reason)
910
+ );
911
+ }
912
+ });
913
+ }
914
+
915
+ return result;
916
+ }
917
+
918
+ /**
919
+ * ✅ NEW: Execute mutations using GraphQL aliases (batched requests)
920
+ */
921
+ async function executeMutationsWithAliases(
922
+ client: FluentClient,
923
+ mapper: GraphQLMutationMapper,
924
+ locations: Array<{ query: string; variables: any; input: any }>,
925
+ log: any,
926
+ retailerId: string,
927
+ maxParallel: number,
928
+ mutationsPerAliasBatch: number
929
+ ): Promise<MutationResult> {
930
+ const result: MutationResult = { successful: 0, failed: 0, errors: [] };
931
+
932
+ const mutationName = (mapper as any).config.mutation || 'createLocation';
933
+ const aliasBatches: Array<Array<typeof locations[0]>> = [];
934
+
935
+ for (let i = 0; i < locations.length; i += mutationsPerAliasBatch) {
936
+ aliasBatches.push(locations.slice(i, i + mutationsPerAliasBatch));
937
+ }
938
+
939
+ // Process batches with concurrency control
940
+ for (let i = 0; i < aliasBatches.length; i += maxParallel) {
941
+ const concurrentBatches = aliasBatches.slice(i, i + maxParallel);
942
+
943
+ const batchResults = await Promise.allSettled(
944
+ concurrentBatches.map(async (batch) => {
945
+ const { query, variables } = buildAliasedBatch(batch, mutationName, retailerId);
946
+ const response = await retryWithBackoff(() => client.graphql({ query, variables }));
947
+ return parseAliasResponse(response, batch, mutationName);
948
+ })
949
+ );
950
+
951
+ batchResults.forEach((batchResult, idx) => {
952
+ if (batchResult.status === 'fulfilled') {
953
+ const batchRes = batchResult.value;
954
+ result.successful += batchRes.executed;
955
+ result.failed += batchRes.failed;
956
+ result.errors.push(...batchRes.errors);
957
+ } else {
958
+ const batch = concurrentBatches[idx];
959
+ const errorMsg = batchResult.reason instanceof Error ? batchResult.reason.message : String(batchResult.reason);
960
+ batch.forEach(loc => {
961
+ result.failed++;
962
+ result.errors.push(`Batch execution failed: ${errorMsg}`);
963
+ });
964
+ }
965
+ });
966
+
967
+ if (i + maxParallel < aliasBatches.length) {
968
+ await new Promise(resolve => setTimeout(resolve, 500));
969
+ }
970
+ }
971
+
972
+ return result;
973
+ }
974
+
975
+ /**
976
+ * ✅ NEW: Build aliased batch query and variables
977
+ */
978
+ function buildAliasedBatch(
979
+ batch: Array<{ query: string; variables: any; input: any }>,
980
+ mutationName: string,
981
+ retailerId: string
982
+ ): { query: string; variables: Record<string, any> } {
983
+ const batchSize = batch.length;
984
+ const inputTypeName = `${mutationName.charAt(0).toUpperCase() + mutationName.slice(1)}Input`;
985
+
986
+ const variables = Array.from({ length: batchSize }, (_, i) =>
987
+ `$input${i + 1}: ${inputTypeName}!`
988
+ ).join(', ');
989
+
990
+ const aliasedMutations = Array.from({ length: batchSize }, (_, i) => {
991
+ const alias = `${mutationName}${i + 1}`;
992
+ return ` ${alias}: ${mutationName}(input: $input${i + 1}) { id ref name }`;
993
+ }).join('\n');
994
+
995
+ const operationName = `Batch${mutationName.charAt(0).toUpperCase() + mutationName.slice(1)}s`;
996
+ const query = `mutation ${operationName}(${variables}) {\n${aliasedMutations}\n}`;
997
+
998
+ const variablesObj: Record<string, any> = {};
999
+ batch.forEach((loc, index) => {
1000
+ const input = loc.variables.input || loc.variables;
1001
+ if (input && !input.retailer) {
1002
+ input.retailer = { id: parseInt(retailerId) };
1003
+ }
1004
+ variablesObj[`input${index + 1}`] = input;
1005
+ });
1006
+
1007
+ return { query, variables: variablesObj };
1008
+ }
1009
+
1010
+ /**
1011
+ * ✅ NEW: Parse aliased GraphQL response
1012
+ */
1013
+ function parseAliasResponse(
1014
+ response: any,
1015
+ batch: Array<{ query: string; variables: any; input: any }>,
1016
+ mutationName: string
1017
+ ): { executed: number; failed: number; errors: string[] } {
1018
+ const result = { executed: 0, failed: 0, errors: [] as string[] };
1019
+
1020
+ const data = response.data || {};
1021
+ const errors = response.errors || [];
1022
+
1023
+ batch.forEach((loc, index) => {
1024
+ const alias = `${mutationName}${index + 1}`;
1025
+ const aliasData = data[alias];
1026
+ const aliasErrors = errors.filter((e: any) =>
1027
+ e.path && Array.isArray(e.path) && e.path.includes(alias)
1028
+ );
1029
+
1030
+ if (aliasData && !aliasErrors.length) {
1031
+ result.executed++;
1032
+ } else {
1033
+ result.failed++;
1034
+ const errorMsg = aliasErrors[0]?.message || 'Mutation failed';
1035
+ result.errors.push(`${loc.input?.ref || 'unknown'}: ${errorMsg}`);
1036
+ }
1037
+ });
1038
+
1039
+ return result;
1040
+ }
1041
+
1042
+ /**
1043
+ * Service Function 3: Write Mutation Log
1044
+ *
1045
+ * Writes mutation results to S3 as a JSON log file.
1046
+ *
1047
+ * @param s3 - S3DataSource instance
1048
+ * @param logEntries - Array of mutation log entries
1049
+ * @param fileName - Original file name (used to generate log path)
1050
+ * @param logPrefix - S3 prefix for logs (e.g., 'logs/')
1051
+ * @param log - Logger instance
1052
+ */
1053
+ async function writeMutationLog(
1054
+ s3: S3DataSource,
1055
+ logEntries: MutationLogEntry[],
1056
+ fileName: string,
1057
+ logPrefix: string,
1058
+ log: any
1059
+ ): Promise<void> {
1060
+ try {
1061
+ const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
1062
+ const logFileName = `${logPrefix}${fileName.replace('.xml', '')}_${timestamp}.json`;
1063
+
1064
+ const logContent = JSON.stringify(
1065
+ {
1066
+ fileName,
1067
+ timestamp: new Date().toISOString(),
1068
+ totalMutations: logEntries.length,
1069
+ successful: logEntries.filter(e => e.status === 'success').length,
1070
+ failed: logEntries.filter(e => e.status === 'failure').length,
1071
+ entries: logEntries,
1072
+ },
1073
+ null,
1074
+ 2
1075
+ );
1076
+
1077
+ // Write to S3 (uploadFile accepts string or Buffer)
1078
+ await s3.uploadFile(logFileName, logContent);
1079
+
1080
+ log.info('Mutation log written', { logFileName, entries: logEntries.length });
1081
+ } catch (error: any) {
1082
+ // ✅ Enhanced error logging: Extract all error details for visibility
1083
+ const errorDetails = {
1084
+ message: error?.message || 'Unknown error',
1085
+ stack: error?.stack,
1086
+ errorType: error?.name || 'Error',
1087
+ };
1088
+ log.error('Failed to write mutation log', errorDetails, { fileName });
1089
+ // Don't throw - logging failure shouldn't stop workflow
1090
+ }
1091
+ }
1092
+
1093
+ // ============================================================================
1094
+ // Main Workflow Function
1095
+ // ============================================================================
1096
+
1097
+ /**
1098
+ * Main Orchestration Function: Execute Location Sync
1099
+ *
1100
+ * This function orchestrates the location synchronization workflow.
1101
+ *
1102
+ * Architecture:
1103
+ * 1. List files from S3
1104
+ * 2. For each file:
1105
+ * a. processFile() - Download, parse, map
1106
+ * b. executeMutations() - Send GraphQL mutations with rate limiting
1107
+ * c. writeMutationLog() - Log results to S3
1108
+ * d. Archive file (primary deduplication)
1109
+ * e. Mark processed in KV (metadata tracking)
1110
+ *
1111
+ * @param ctx - Versori context
1112
+ * @param jobId - Job identifier
1113
+ * @param tracker - JobTracker instance
1114
+ * @returns Processing result
1115
+ */
1116
+ export async function executeLocationSync(ctx: any, jobId: string, tracker: JobTracker) {
1117
+ const { log, activation } = ctx;
1118
+
1119
+ log.info('📋 [INIT] Reading activation variables', { jobId });
1120
+
1121
+ // Read activation variables
1122
+ const s3Bucket = activation?.getVariable('s3BucketName');
1123
+ const s3Region = activation?.getVariable('awsRegion') || 'us-east-1';
1124
+ const s3AccessKeyId = activation?.getVariable('awsAccessKeyId');
1125
+ const s3SecretAccessKey = activation?.getVariable('awsSecretAccessKey');
1126
+ const s3Prefix = activation?.getVariable('s3Prefix') || 'locations/';
1127
+ const archivePrefix = activation?.getVariable('archivePrefix') || 'processed/';
1128
+ const errorPrefix = activation?.getVariable('errorPrefix') || 'errors/';
1129
+ const logPrefix = activation?.getVariable('logPrefix') || 'logs/';
1130
+ const filePattern = (activation?.getVariable('filePattern') || '.xml').toLowerCase();
1131
+ const maxFiles = parseInt(activation?.getVariable('maxFilesToProcess') || '10', 10);
1132
+ const retailerId = activation?.getVariable('retailerId'); // Optional: Only if mutation schema requires it
1133
+ const enableArchival = activation?.getVariable('enableArchival') !== 'false';
1134
+ const enableMutationLogs = activation?.getVariable('enableMutationLogs') !== 'false';
1135
+ const enableFileTracking = activation?.getVariable('enableFileTracking') !== 'false';
1136
+
1137
+ // ✅ Configuration with defaults
1138
+ const mutationBatchSize = parseInt(
1139
+ activation?.getVariable('mutationBatchSize') || '1', // ✅ Default: 1 (sequential)
1140
+ 10
1141
+ );
1142
+
1143
+ const mutationsPerAliasBatch = activation?.getVariable('mutationsPerAliasBatch')
1144
+ ? parseInt(activation?.getVariable('mutationsPerAliasBatch') || '1', 10)
1145
+ : undefined; // ✅ Default: undefined (disabled, use separate requests)
1146
+
1147
+ // Validate required variables
1148
+ const missingVars: string[] = [];
1149
+ if (!s3Bucket) missingVars.push('s3BucketName');
1150
+ if (!s3AccessKeyId) missingVars.push('awsAccessKeyId');
1151
+ if (!s3SecretAccessKey) missingVars.push('awsSecretAccessKey');
1152
+ // Note: retailerId is optional - only needed if mutation schema requires it
1153
+
1154
+ if (missingVars.length > 0) {
1155
+ const errorMsg = `Missing required variables: ${missingVars.join(', ')}`;
1156
+ log.error('❌ [VALIDATION] Missing required activation variables', {
1157
+ missingVars,
1158
+ recommendation: 'Add missing variables in Versori activation settings'
1159
+ });
1160
+ return { success: false, error: errorMsg, processed: 0 };
1161
+ }
1162
+
1163
+ log.info('✅ [VALIDATION] All required variables present', {
1164
+ s3Bucket,
1165
+ s3Region,
1166
+ s3Prefix,
1167
+ enableFileTracking,
1168
+ mutationBatchSize
1169
+ });
1170
+
1171
+ try {
1172
+ log.info('🔧 [INIT] Initializing Fluent Commerce client', { jobId });
1173
+
1174
+ // Initialize services with validateConnection
1175
+ const client = await createClient(ctx, { validateConnection: true });
1176
+ if (!client) {
1177
+ throw new Error('Failed to create Fluent Commerce client');
1178
+ }
1179
+
1180
+ log.info('✅ [INIT] Fluent Commerce client validated and ready', { jobId });
1181
+
1182
+ // ✅ CORRECT: GraphQL mutations do NOT need client.setRetailerId()
1183
+ // setRetailerId() is only for Job/Event API, NOT GraphQL
1184
+ // Check your GraphQL schema to determine retailerId handling:
1185
+ // - Mandatory retailerId → Must pass it in mutation input
1186
+ // - Optional retailerId → Can pass it if needed
1187
+ // - No retailerId field → Don't pass it
1188
+ // See: fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md
1189
+
1190
+ log.info('🗄️ [INIT] Initializing S3 data source', { s3Bucket, s3Region, s3Prefix });
1191
+
1192
+ const s3 = new S3DataSource(
1193
+ {
1194
+ type: 'S3_XML',
1195
+ connectionId: 's3-location-sync',
1196
+ name: 'Source S3',
1197
+ s3Config: {
1198
+ bucket: s3Bucket,
1199
+ region: s3Region,
1200
+ accessKeyId: s3AccessKeyId,
1201
+ secretAccessKey: s3SecretAccessKey,
1202
+ },
1203
+ },
1204
+ log
1205
+ );
1206
+
1207
+ const parser = new XMLParserService();
1208
+
1209
+ // Initialize state tracking (only if enabled)
1210
+ let stateService: StateService | null = null;
1211
+ if (enableFileTracking) {
1212
+ log.info('🔄 [INIT] Enabling file tracking with StateService', { jobId });
1213
+ const stateKV = new VersoriKVAdapter(ctx);
1214
+ stateService = new StateService(stateKV);
1215
+ } else {
1216
+ log.info('⏭️ [INIT] File tracking disabled - relying on S3 archival only', { jobId });
1217
+ }
1218
+
1219
+ log.info('📝 [INIT] Loading mapping configuration', { jobId });
1220
+
1221
+ // ✅ CRITICAL: Load mapping config from external JSON file
1222
+ // Mapping config uses GraphQLMutationMapper structure (nested objects, not dot notation)
1223
+ // File: src/config/location-mapping.json
1224
+ const mappingConfigJson = await import('../config/location-mapping.json', { assert: { type: 'json' } });
1225
+ const mappingConfig = mappingConfigJson.default;
1226
+
1227
+ // Initialize GraphQLMutationMapper with client for schema introspection
1228
+ const mapper = new GraphQLMutationMapper(mappingConfig, log, { fluentClient: client });
1229
+
1230
+ log.info('📂 [S3] Listing files from S3', { s3Bucket, s3Prefix, filePattern });
1231
+
1232
+ // List files (pattern filtering handled by listFiles)
1233
+ const files = await s3.listFiles({
1234
+ prefix: s3Prefix,
1235
+ pattern: filePattern,
1236
+ maxKeys: 1000
1237
+ });
1238
+
1239
+ // Newest-first ordering
1240
+ const xmlFiles = files
1241
+ .sort((a: any, b: any) => {
1242
+ const aTime = a.lastModified ? new Date(a.lastModified).getTime() : 0;
1243
+ const bTime = b.lastModified ? new Date(b.lastModified).getTime() : 0;
1244
+ return bTime - aTime;
1245
+ })
1246
+ .slice(0, maxFiles);
1247
+
1248
+ log.info('📊 [S3] File discovery complete', {
1249
+ totalFiles: files.length,
1250
+ xmlFiles: xmlFiles.length,
1251
+ maxFiles,
1252
+ selectedFiles: xmlFiles.map(f => f.name)
1253
+ });
1254
+
1255
+ const results = {
1256
+ processed: 0,
1257
+ skipped: 0,
1258
+ failed: 0,
1259
+ totalRecords: 0,
1260
+ errors: [] as string[],
1261
+ };
1262
+
1263
+ log.info('🔄 [PROCESSING] Starting file processing loop', {
1264
+ fileCount: xmlFiles.length,
1265
+ jobId
1266
+ });
1267
+
1268
+ // Per-file processing loop
1269
+ for (const file of xmlFiles) {
1270
+ const filePath = file.path;
1271
+ const fileName = file.name;
1272
+
1273
+ log.info('📄 [FILE] Processing file', { fileName, filePath });
1274
+
1275
+ // Duplicate prevention (secondary check - files in processed/ won't be listed)
1276
+ // Primary deduplication: S3 archival (files moved to processed/ subdirectory)
1277
+ if (enableFileTracking && stateService) {
1278
+ const wasProcessed = await stateService.isFileProcessed(fileName);
1279
+ if (wasProcessed) {
1280
+ log.info('⏭️ [SKIP] File already processed (KV check)', { fileName });
1281
+ results.skipped++;
1282
+ continue;
1283
+ }
1284
+ }
1285
+
1286
+ try {
1287
+ // Step 1: Process file (download, parse, map)
1288
+ log.info('📥 [DOWNLOAD] Downloading and parsing file', { fileName });
1289
+ const processingResult = await processFile(s3, parser, mapper, filePath, fileName, log);
1290
+
1291
+ if (!processingResult.success) {
1292
+ throw new Error(`File processing failed: ${processingResult.errors.join(', ')}`);
1293
+ }
1294
+
1295
+ if (processingResult.locations.length === 0) {
1296
+ log.warn('⚠️ [SKIP] Empty file detected, archiving', { fileName });
1297
+ if (enableArchival) {
1298
+ await s3.moveFile(filePath, `${archivePrefix}${fileName}`);
1299
+ log.info('📦 [ARCHIVE] Empty file archived', { fileName, destination: `${archivePrefix}${fileName}` });
1300
+ }
1301
+ results.skipped++;
1302
+ continue;
1303
+ }
1304
+
1305
+ log.info('✅ [PARSE] File parsed successfully', {
1306
+ fileName,
1307
+ locationCount: processingResult.locations.length,
1308
+ mappingErrors: processingResult.errors.length
1309
+ });
1310
+
1311
+ // Step 2: Execute mutations
1312
+ // Step 2: Execute mutations with alias batching support
1313
+ // ? Enhanced: Extract context for progress logging
1314
+ const sampleLocationRefs = processingResult.locations.slice(0, 5).map((loc: any) => loc.input?.ref || loc.ref || 'unknown');
1315
+ const mutationType = mapper?.mutationName || 'createLocation';
1316
+
1317
+ // ? Enhanced: Start logging with context
1318
+ log.info(`[GraphQLMutations] Sending mutations for file "${fileName}"`, {
1319
+ totalMutations: processingResult.locations.length,
1320
+ mutationType,
1321
+ batchSize: mutationBatchSize,
1322
+ batchMode: mutationBatchSize === 1 ? 'sequential' : `parallel (${mutationBatchSize})`,
1323
+ sampleLocationRefs: sampleLocationRefs.join(', '),
1324
+ aliasBatching: mutationsPerAliasBatch ? `enabled (${mutationsPerAliasBatch} per alias)` : 'disabled'
1325
+ });
1326
+
1327
+ const mutationResult = await executeMutations(
1328
+ client,
1329
+ mapper,
1330
+ processingResult.locations,
1331
+ log,
1332
+ retailerId, // Pass retailerId for mutations that require it in input
1333
+ mutationBatchSize, // Concurrency control (default: 1)
1334
+ mutationsPerAliasBatch // ✅ NEW: Alias batching (default: undefined)
1335
+ );
1336
+
1337
+ // ? Enhanced: Completion logging with summary
1338
+ log.info(`[GraphQLMutations] Mutation submission completed for file "${fileName}"`, {
1339
+ totalMutations: processingResult.locations.length,
1340
+ successful: mutationResult.successful,
1341
+ failed: mutationResult.failed,
1342
+ successRate: processingResult.locations.length > 0 ? `${Math.round((mutationResult.successful / processingResult.locations.length) * 100)}%` : '0%',
1343
+ mutationType
1344
+ });
1345
+
1346
+ // Step 3: Write mutation log (if enabled)
1347
+ if (enableMutationLogs) {
1348
+ const logEntries: MutationLogEntry[] = processingResult.locations.map(loc => {
1349
+ const failed = mutationResult.errors.find(e => e.startsWith(loc.ref));
1350
+ return {
1351
+ timestamp: new Date().toISOString(),
1352
+ fileName,
1353
+ locationRef: loc.ref,
1354
+ status: failed ? 'failure' : 'success',
1355
+ error: failed,
1356
+ };
1357
+ });
1358
+
1359
+ await writeMutationLog(s3, logEntries, fileName, logPrefix, log);
1360
+ }
1361
+
1362
+ // Step 4: Archive file (PRIMARY deduplication - file won't be listed again)
1363
+ if (enableArchival) {
1364
+ log.info('📦 [ARCHIVE] Moving file to processed directory', { fileName });
1365
+ await s3.moveFile(filePath, `${archivePrefix}${fileName}`);
1366
+ log.info('✅ [ARCHIVE] File archived successfully', {
1367
+ fileName,
1368
+ destination: `${archivePrefix}${fileName}`
1369
+ });
1370
+ }
1371
+
1372
+ // Step 5: Mark processed in KV (SECONDARY - provides metadata/history)
1373
+ if (enableFileTracking && stateService) {
1374
+ log.info('💾 [STATE] Recording file processing metadata', { fileName });
1375
+ await stateService.markFileProcessed(fileName, {
1376
+ recordCount: processingResult.locations.length,
1377
+ successful: mutationResult.successful,
1378
+ failed: mutationResult.failed,
1379
+ mappingErrors: processingResult.errors.length,
1380
+ timestamp: new Date().toISOString(),
1381
+ });
1382
+ }
1383
+
1384
+ results.processed++;
1385
+ results.totalRecords += mutationResult.successful;
1386
+
1387
+ log.info('✅ [COMPLETE] File processing complete', {
1388
+ fileName,
1389
+ locations: processingResult.locations.length,
1390
+ successful: mutationResult.successful,
1391
+ failed: mutationResult.failed,
1392
+ mappingErrors: processingResult.errors.length,
1393
+ });
1394
+
1395
+ if (processingResult.errors.length > 0 || mutationResult.errors.length > 0) {
1396
+ results.errors.push(
1397
+ `${fileName}: ${processingResult.errors.length} mapping errors, ${mutationResult.errors.length} mutation errors`
1398
+ );
1399
+ }
1400
+ } catch (error: any) {
1401
+ // ✅ Enhanced error logging: Extract all error details for visibility
1402
+ const errorDetails = {
1403
+ message: error?.message || 'Unknown error',
1404
+ stack: error?.stack,
1405
+ fileName: error?.fileName,
1406
+ lineNumber: error?.lineNumber,
1407
+ originalError: error?.context?.originalError?.message,
1408
+ errorType: error?.name || 'Error',
1409
+ };
1410
+
1411
+ log.error('❌ [ERROR] File processing failed', errorDetails, { fileName });
1412
+
1413
+ // Provide error recommendations based on error type
1414
+ const recommendation = getErrorRecommendation(error);
1415
+ if (recommendation) {
1416
+ log.warn('💡 [RECOMMENDATION]', { fileName, recommendation });
1417
+ }
1418
+
1419
+ results.failed++;
1420
+ results.errors.push(`${fileName}: ${error.message}`);
1421
+
1422
+ // Attempt to move to error directory, ignore failures
1423
+ try {
1424
+ log.info('🗂️ [ERROR] Moving failed file to error directory', { fileName });
1425
+ await s3.moveFile(filePath, `${errorPrefix}${fileName}`);
1426
+ log.info('✅ [ERROR] Failed file moved to error directory', {
1427
+ fileName,
1428
+ destination: `${errorPrefix}${fileName}`
1429
+ });
1430
+ } catch (moveError) {
1431
+ log.warn('⚠️ [ERROR] Failed to move file to error directory', {
1432
+ fileName,
1433
+ moveError: moveError instanceof Error ? moveError.message : String(moveError)
1434
+ });
1435
+ }
1436
+
1437
+ // Track error state with exponential backoff (only if file tracking enabled)
1438
+ if (enableFileTracking && stateService) {
1439
+ try {
1440
+ const stateKV = new VersoriKVAdapter(ctx);
1441
+ const key = ['error-state', fileName];
1442
+ const prev = (await stateKV.get(key))?.value as any;
1443
+ const attempts = (prev?.attemptCount || 0) + 1;
1444
+ const backoffMinutes = Math.min(Math.pow(2, attempts) * 5, 24 * 60);
1445
+ const nextRetryAt = new Date(Date.now() + backoffMinutes * 60000).toISOString();
1446
+
1447
+ await stateKV.set(key, {
1448
+ fileName,
1449
+ attemptCount: attempts,
1450
+ lastError: error?.message || 'unknown',
1451
+ lastAttemptAt: new Date().toISOString(),
1452
+ firstFailedAt: prev?.firstFailedAt || new Date().toISOString(),
1453
+ nextRetryAt,
1454
+ });
1455
+
1456
+ log.info('💾 [ERROR] Error state tracked with exponential backoff', {
1457
+ fileName,
1458
+ attempts,
1459
+ nextRetryAt
1460
+ });
1461
+ } catch (stateError) {
1462
+ log.warn('⚠️ [ERROR] Failed to track error state', {
1463
+ fileName,
1464
+ stateError: stateError instanceof Error ? stateError.message : String(stateError)
1465
+ });
1466
+ }
1467
+ }
1468
+ }
1469
+ }
1470
+
1471
+ log.info('🏁 [COMPLETE] File processing loop finished', {
1472
+ processed: results.processed,
1473
+ skipped: results.skipped,
1474
+ failed: results.failed,
1475
+ totalRecords: results.totalRecords
1476
+ });
1477
+
1478
+ return results;
1479
+ } catch (error: any) {
1480
+ // ✅ Enhanced error logging: Extract all error details for visibility
1481
+ const errorDetails = {
1482
+ message: error?.message || 'Unknown error',
1483
+ stack: error?.stack,
1484
+ errorType: error?.name || 'Error',
1485
+ };
1486
+
1487
+ log.error('❌ [FATAL] Location sync failed', errorDetails);
1488
+
1489
+ // Provide fatal error recommendations
1490
+ const recommendation = getErrorRecommendation(error);
1491
+ if (recommendation) {
1492
+ log.warn('💡 [RECOMMENDATION]', { recommendation });
1493
+ }
1494
+
1495
+ return {
1496
+ success: false,
1497
+ error: error.message,
1498
+ processed: 0,
1499
+ timestamp: new Date().toISOString(),
1500
+ };
1501
+ }
1502
+ }
1503
+
1504
+ /**
1505
+ * Get error recommendation based on error type
1506
+ */
1507
+ function getErrorRecommendation(error: any): string | null {
1508
+ const message = error?.message?.toLowerCase() || '';
1509
+
1510
+ if (message.includes('s3') || message.includes('access denied')) {
1511
+ return 'Check S3 credentials and bucket permissions. Verify IAM policy includes s3:ListBucket, s3:GetObject, s3:PutObject, s3:DeleteObject.';
1512
+ }
1513
+
1514
+ if (message.includes('xml') || message.includes('parse')) {
1515
+ return 'Verify XML file structure matches expected schema. Check for malformed XML or encoding issues.';
1516
+ }
1517
+
1518
+ if (message.includes('mapping') || message.includes('field')) {
1519
+ return 'Review field mapping configuration. Ensure all required fields are present and source paths are correct.';
1520
+ }
1521
+
1522
+ if (message.includes('graphql') || message.includes('mutation')) {
1523
+ return 'Check GraphQL schema and mutation input. Verify all required fields are provided and types match schema.';
1524
+ }
1525
+
1526
+ if (message.includes('auth') || message.includes('401') || message.includes('403')) {
1527
+ return 'Verify Fluent Commerce credentials. Check OAuth2 client ID/secret and ensure connection is active.';
1528
+ }
1529
+
1530
+ if (message.includes('timeout') || message.includes('econnrefused')) {
1531
+ return 'Check network connectivity. Verify API endpoints are accessible and not rate-limited.';
1532
+ }
1533
+
1534
+ return null;
1535
+ }
1536
+ ```
1537
+
1538
+ **Note:** The `executeLocationSync` function should contain the full implementation of `runLocationXmlWorkflow` (renamed to `executeLocationSync`), including all the logic for processing files, executing mutations, and logging. The implementation details are shown in the service function code above.
1539
+
1540
+ ---
1541
+
1542
+ ### Step 5: TypeScript Configuration
1543
+
1544
+ **File: tsconfig.json**
1545
+
1546
+ ```json
1547
+ {
1548
+ "compilerOptions": {
1549
+ "module": "ES2022",
1550
+ "target": "ES2024",
1551
+ "moduleResolution": "node"
1552
+ }
1553
+ }
1554
+ ```
1555
+
1556
+ ---
1557
+
1558
+ ## Code Flow Explanation
1559
+
1560
+ ### Initialization Phase
1561
+
1562
+ 1. **Read activation variables** - S3 config, Fluent config, rate limiting, logging options
1563
+ 2. **Validate required variables** - Fail fast if missing credentials
1564
+ 3. **Initialize SDK services** - FluentClient, S3DataSource, XMLParserService, StateService
1565
+ 4. **Create mapping configuration** - Define XML → GraphQL field mappings with nested objects
1566
+ 5. **Calculate rate limit delay** - Convert mutations/second to delay in milliseconds
1567
+
1568
+ ### File Discovery Phase
1569
+
1570
+ 1. **List S3 files** - Use `s3.listFiles()` with prefix filter (excludes processed/ subdirectory)
1571
+ 2. **Filter by pattern** - Match file extension (e.g., `.xml`)
1572
+ 3. **Sort newest-first** - Process most recent files first
1573
+ 4. **Apply max files limit** - Prevent overwhelming the workflow
1574
+ 5. **Note**: Files in `processed/` subdirectory won't be listed (primary deduplication)
1575
+
1576
+ ### Per-File Processing (Service Functions)
1577
+
1578
+ **Step 1: processFile()** - Download, parse, map
1579
+
1580
+ 1. **Download file** - Use S3DataSource with retry logic
1581
+ 2. **Parse XML** - XMLParserService converts to JavaScript object
1582
+ 3. **Normalize array** - Handle single vs multiple `<location>` elements (CRITICAL for XML)
1583
+ 4. **Map locations** - Use UniversalMapper with nested field mapping
1584
+ 5. **Collect errors** - Track mapping errors without stopping
1585
+ 6. **Return result** - FileProcessingResult with locations array and errors
1586
+
1587
+ **Step 2: executeMutations()** - GraphQL mutations with rate limiting
1588
+
1589
+ 1. **Loop through locations** - Process each location individually
1590
+ 2. **Build mutation input** - Extract nested fields (primaryAddress, openingSchedule)
1591
+ 3. **Execute mutation** - Direct GraphQL `createLocation` with retry logic
1592
+ 4. **Apply rate limiting** - Add configurable delay between mutations
1593
+ 5. **Track results** - Count successful vs failed, collect error messages
1594
+ 6. **Return result** - MutationResult with counts and errors
1595
+
1596
+ **Step 3: writeMutationLog()** - S3 log file (optional)
1597
+
1598
+ 1. **Create log entries** - Map locations to log entries with status
1599
+ 2. **Build JSON log** - Include timestamp, summary, detailed entries
1600
+ 3. **Write to S3** - Use Buffer.from() for Deno/Versori runtime compatibility
1601
+ 4. **Non-blocking** - Logging failure doesn't stop workflow
1602
+ 5. **Timestamped naming** - Unique log file per processed file
1603
+
1604
+ ### Cleanup Phase
1605
+
1606
+ 1. **Archive file FIRST** - Move to `processed/` or `errors/` (PRIMARY deduplication)
1607
+ 2. **Mark processed in KV** - StateService tracks metadata and processing history
1608
+ 3. **Error state tracking** - Store error info with exponential backoff timestamp
1609
+ 4. **Return results** - Summary of processed, skipped, failed files
1610
+
1611
+ **Note**: The order matters! Archive first (primary deduplication), then KV tracking (metadata/history).
1612
+
1613
+ ### Service Function Benefits
1614
+
1615
+ - **Composability**: Each function has single responsibility and can be reused
1616
+ - **Testability**: Service functions can be unit tested independently
1617
+ - **Clarity**: Main workflow shows high-level orchestration
1618
+ - **Error Handling**: Isolated error handling per service function
1619
+ - **Logging**: Detailed logging at each step with structured context
1620
+
1621
+ ---
1622
+
1623
+ ## S3 Archival Deduplication Pattern
1624
+
1625
+ This template uses **S3 archival** as the primary deduplication mechanism, NOT VersoriFileTracker.
1626
+
1627
+ ### How It Works
1628
+
1629
+ **S3 Directory Structure:**
1630
+
1631
+ ```
1632
+ s3://my-bucket/
1633
+ ├── locations/ ← listFiles() reads from here
1634
+ │ ├── new-file-1.xml
1635
+ │ └── new-file-2.xml
1636
+ ├── processed/ ← Successfully processed files
1637
+ │ ├── old-file-1.xml
1638
+ │ └── old-file-2.xml
1639
+ └── errors/ ← Failed files
1640
+ └── bad-file.xml
1641
+ ```
1642
+
1643
+ **Deduplication Flow:**
1644
+
1645
+ 1. **List files**: `s3.listFiles({ prefix: 'locations/' })` - Only lists `locations/` subdirectory
1646
+ 2. **Process file**: Download, parse, transform, send mutations
1647
+ 3. **Archive**: `s3.moveFile(filePath, 'processed/new-file-1.xml')` - Moves file out of `locations/`
1648
+ 4. **Next run**: File is now in `processed/`, won't be listed again
1649
+
1650
+ **Why This Works:**
1651
+
1652
+ - Files in `processed/` subdirectory are **never listed** when prefix is `locations/`
1653
+ - No need to track file state in KV store for deduplication
1654
+ - S3 is the single source of truth for file status
1655
+ - Simple, reliable, scales to millions of files
1656
+
1657
+ **StateService Role (Secondary):**
1658
+
1659
+ - Provides metadata and processing history
1660
+ - Backup check in case archival fails mid-process
1661
+ - Useful for monitoring and debugging
1662
+ - NOT the primary deduplication mechanism
1663
+
1664
+ **When to Use VersoriFileTracker:**
1665
+
1666
+ - **NEVER for S3 sources** - Use archival pattern instead
1667
+ - **Only for SFTP sources** - Where archival might not be possible
1668
+ - See SFTP templates for VersoriFileTracker usage
1669
+
1670
+ ---
1671
+
1672
+ ## XML Path Resolution Patterns
1673
+
1674
+ ### Pattern 1: XML Attribute Access with @ Prefix
1675
+
1676
+ ```typescript
1677
+ const mappingConfig = {
1678
+ fields: {
1679
+ // XML attribute access
1680
+ ref: { source: 'location.@ref' }, // <location ref="LOC-001">
1681
+ type: { source: 'location.@type' }, // <location type="WAREHOUSE">
1682
+ country: { source: 'location.address.@country' }, // <address country="USA">
1683
+
1684
+ // XML element text content
1685
+ name: { source: 'location.name' }, // <name>Downtown</name>
1686
+ city: { source: 'location.address.city' }, // <address><city>NYC</city></address>
1687
+ },
1688
+ };
1689
+ ```
1690
+
1691
+ ### Pattern 2: Handling Single vs Multiple Elements
1692
+
1693
+ ```typescript
1694
+ // XML can have single or multiple <location> elements
1695
+ const locationsData = xmlData.locations?.location;
1696
+
1697
+ // Normalize to array
1698
+ const locations = Array.isArray(locationsData) ? locationsData : [locationsData];
1699
+
1700
+ // Process each location
1701
+ for (const loc of locations) {
1702
+ const record = { location: loc }; // Wrap for mapping
1703
+ const result = await mapper.map(record);
1704
+ }
1705
+ ```
1706
+
1707
+ ### Pattern 3: Nested Object Mapping
1708
+
1709
+ ```typescript
1710
+ const mappingConfig = {
1711
+ fields: {
1712
+ // Root fields
1713
+ ref: { source: 'location.@ref', required: true },
1714
+ name: { source: 'location.name', required: true },
1715
+ type: { source: 'location.@type', required: true },
1716
+
1717
+ // Nested primaryAddress object
1718
+ 'primaryAddress.ref': { source: 'location.@ref' },
1719
+ 'primaryAddress.street': { source: 'location.address.street1' },
1720
+ 'primaryAddress.city': { source: 'location.address.city' },
1721
+ 'primaryAddress.latitude': { source: 'location.coordinates.@lat', resolver: 'sdk.parseFloat' },
1722
+
1723
+ // Nested openingSchedule object
1724
+ 'openingSchedule.allHours': {
1725
+ source: 'location.openingSchedule.allHours',
1726
+ resolver: 'sdk.boolean',
1727
+ },
1728
+ 'openingSchedule.monStart': {
1729
+ source: 'location.openingSchedule.monStart',
1730
+ resolver: 'sdk.parseInt',
1731
+ },
1732
+
1733
+ // Note: retailer.id not shown - standard createLocation does not have this field
1734
+ // If your schema requires it, add: 'retailer.id': { value: parseInt(retailerId) }
1735
+ },
1736
+ };
1737
+ ```
1738
+
1739
+ ---
1740
+
1741
+ ## Sample XML Files
1742
+
1743
+ ### Minimal Test File
1744
+
1745
+ **File: test-location.xml**
1746
+
1747
+ ```xml
1748
+ <?xml version="1.0" encoding="UTF-8"?>
1749
+ <locations>
1750
+ <location ref="TEST-001" type="WAREHOUSE">
1751
+ <name>Test Warehouse</name>
1752
+ <address country="USA">
1753
+ <street1>123 Test St</street1>
1754
+ <city>TestCity</city>
1755
+ <state>TC</state>
1756
+ <postalCode>12345</postalCode>
1757
+ </address>
1758
+ <coordinates lat="40.7128" lon="-74.0060"/>
1759
+ <timeZone>America/New_York</timeZone>
1760
+ <openingSchedule>
1761
+ <allHours>false</allHours>
1762
+ <monStart>800</monStart>
1763
+ <monEnd>1800</monEnd>
1764
+ <tueStart>800</tueStart>
1765
+ <tueEnd>1800</tueEnd>
1766
+ <wedStart>800</wedStart>
1767
+ <wedEnd>1800</wedEnd>
1768
+ <thuStart>800</thuStart>
1769
+ <thuEnd>1800</thuEnd>
1770
+ <friStart>800</friStart>
1771
+ <friEnd>1800</friEnd>
1772
+ <satStart>0</satStart>
1773
+ <satEnd>0</satEnd>
1774
+ <sunStart>0</sunStart>
1775
+ <sunEnd>0</sunEnd>
1776
+ </openingSchedule>
1777
+ </location>
1778
+ </locations>
1779
+ ```
1780
+
1781
+ ---
1782
+
1783
+ ## Service Functions Deep Dive
1784
+
1785
+ This template demonstrates service function composition for maintainable workflows.
1786
+
1787
+ ### Function 1: processFile()
1788
+
1789
+ **Purpose**: Download, parse, normalize, and map XML data
1790
+
1791
+ **Inputs**:
1792
+ - `s3: S3DataSource` - For file download
1793
+ - `parser: XMLParserService` - For XML parsing
1794
+ - `mapper: UniversalMapper` - For field mapping
1795
+ - `filePath: string` - S3 path to file
1796
+ - `fileName: string` - File name for logging
1797
+ - `log: any` - Logger instance
1798
+
1799
+ **Outputs**: `FileProcessingResult`
1800
+ ```typescript
1801
+ {
1802
+ success: boolean; // Overall success
1803
+ locations: any[]; // Mapped location objects
1804
+ errors: string[]; // Mapping error messages
1805
+ }
1806
+ ```
1807
+
1808
+ **Key Operations**:
1809
+ 1. Downloads file with retry logic
1810
+ 2. Parses XML using XMLParserService
1811
+ 3. **Normalizes arrays** - Handles single vs multiple `<location>` elements
1812
+ 4. Maps each location using UniversalMapper
1813
+ 5. Collects errors without stopping processing
1814
+ 6. Returns all mapped locations + errors
1815
+
1816
+ **Why separate function?**
1817
+ - Testable independently with mock data
1818
+ - Reusable across workflows (scheduled, webhook, adhoc)
1819
+ - Clear error boundaries - file-level errors vs record-level errors
1820
+ - Easy to add XML validation or schema checks
1821
+
1822
+ ### Function 2: executeMutations()
1823
+
1824
+ **Purpose**: Execute GraphQL createLocation mutations with rate limiting
1825
+
1826
+ **Inputs**:
1827
+ - `client: FluentClient` - For GraphQL mutations
1828
+ - `locations: any[]` - Mapped location data
1829
+ - `mutationDelayMs: number` - Rate limit delay
1830
+ - `fileName: string` - For logging context
1831
+ - `log: any` - Logger instance
1832
+
1833
+ **Outputs**: `MutationResult`
1834
+ ```typescript
1835
+ {
1836
+ successful: number; // Count of successful mutations
1837
+ failed: number; // Count of failed mutations
1838
+ errors: string[]; // Error messages with location refs
1839
+ }
1840
+ ```
1841
+
1842
+ **Key Operations**:
1843
+ 1. Loops through locations
1844
+ 2. Builds GraphQL mutation input
1845
+ 3. Executes with retry logic + rate limiting
1846
+ 4. Tracks success/failure per location
1847
+ 5. Returns summary counts + errors
1848
+
1849
+ **Why separate function?**
1850
+ - Clear separation: mapping vs mutation execution
1851
+ - Rate limiting logic isolated and configurable
1852
+ - Easy to swap mutation types (create vs update)
1853
+ - Testable with mock FluentClient
1854
+ - Can parallelize in future (batch mutations)
1855
+
1856
+ ### Function 3: writeMutationLog()
1857
+
1858
+ **Purpose**: Write detailed mutation results to S3 as JSON log
1859
+
1860
+ **Inputs**:
1861
+ - `s3: S3DataSource` - For log upload
1862
+ - `logEntries: MutationLogEntry[]` - Mutation status per location
1863
+ - `fileName: string` - Original file name
1864
+ - `logPrefix: string` - S3 prefix for logs (e.g., `logs/`)
1865
+ - `log: any` - Logger instance
1866
+
1867
+ **Outputs**: `void` (non-blocking - errors logged but not thrown)
1868
+
1869
+ **Key Operations**:
1870
+ 1. Creates timestamped log file name
1871
+ 2. Builds JSON log with summary + entries
1872
+ 3. **Uses Buffer.from()** - Required for Deno/Versori runtime
1873
+ 4. Writes to S3 with `uploadFile()`
1874
+ 5. Errors don't stop workflow (logging is non-critical)
1875
+
1876
+ **Why separate function?**
1877
+ - Optional feature - can be disabled via config
1878
+ - Non-blocking - logging failure doesn't fail workflow
1879
+ - Structured logging for audit trails
1880
+ - Easy to change log format (JSON, CSV, XML)
1881
+ - Can add log rotation/cleanup logic later
1882
+
1883
+ ### Service Function Composition Benefits
1884
+
1885
+ **1. Maintainability**
1886
+ ```typescript
1887
+ // Clear workflow orchestration
1888
+ const processingResult = await processFile(...);
1889
+ const mutationResult = await executeMutations(...);
1890
+ await writeMutationLog(...);
1891
+ ```
1892
+
1893
+ **2. Testability**
1894
+ ```typescript
1895
+ // Unit test processFile() with mock XML
1896
+ const mockS3 = { downloadFile: jest.fn() };
1897
+ const result = await processFile(mockS3, parser, mapper, ...);
1898
+ expect(result.locations).toHaveLength(5);
1899
+ ```
1900
+
1901
+ **3. Reusability**
1902
+ ```typescript
1903
+ // Use processFile() in different workflows
1904
+ export const webhook = webhook('location-webhook').then(async ctx => {
1905
+ const result = await processFile(s3, parser, mapper, filePath, fileName, log);
1906
+ return { locations: result.locations };
1907
+ });
1908
+ ```
1909
+
1910
+ **4. Error Isolation**
1911
+ ```typescript
1912
+ // Each function has clear error boundaries
1913
+ try {
1914
+ const processingResult = await processFile(...);
1915
+ // File-level errors caught here
1916
+ } catch (error) {
1917
+ // Handle file processing failure
1918
+ }
1919
+
1920
+ // Mutation errors don't stop file processing
1921
+ const mutationResult = await executeMutations(...);
1922
+ // mutationResult.errors contains per-location failures
1923
+ ```
1924
+
1925
+ **5. Progressive Enhancement**
1926
+ ```typescript
1927
+ // Easy to add features without touching core logic
1928
+ async function validateLocations(locations: any[]) {
1929
+ // Add validation step
1930
+ }
1931
+
1932
+ const processingResult = await processFile(...);
1933
+ await validateLocations(processingResult.locations); // New step
1934
+ const mutationResult = await executeMutations(...);
1935
+ ```
1936
+
1937
+ ---
1938
+
1939
+ ## Versori Environment Variables
1940
+
1941
+ **Activation Variables:**
1942
+
1943
+ ```bash
1944
+ # ============================================================================
1945
+ # Required Variables
1946
+ # ============================================================================
1947
+ s3BucketName=my-location-bucket
1948
+ awsAccessKeyId=AKIAXXXXXXXXXXXX
1949
+ awsSecretAccessKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
1950
+
1951
+ # ============================================================================
1952
+ # S3 Configuration (Optional - with defaults)
1953
+ # ============================================================================
1954
+ awsRegion=us-east-1
1955
+ s3Prefix=locations/
1956
+ archivePrefix=processed/
1957
+ errorPrefix=errors/
1958
+ logPrefix=logs/
1959
+ filePattern=.xml
1960
+ maxFilesToProcess=10
1961
+
1962
+ # ============================================================================
1963
+ # Feature Toggles (Optional - with defaults)
1964
+ # ============================================================================
1965
+ # Enable S3 archival (move files to processed/errors directories)
1966
+ enableArchival=true
1967
+
1968
+ # Enable mutation logs (write detailed mutation results to S3)
1969
+ enableMutationLogs=true
1970
+
1971
+ # Enable file tracking via StateService + KV store
1972
+ # When disabled, relies on S3 archival only for deduplication
1973
+ enableFileTracking=true
1974
+
1975
+ # ============================================================================
1976
+ # Mutation Configuration (Optional - with defaults)
1977
+ # ============================================================================
1978
+ # Mutation batch size (concurrent requests)
1979
+ # - 1 = Sequential (default, safest)
1980
+ # - 5 = Process 5 mutations in parallel
1981
+ # - 10 = Process 10 mutations in parallel
1982
+ mutationBatchSize=1
1983
+
1984
+ # Alias batching (combine multiple mutations into single request)
1985
+ # - undefined = Disabled (default, use separate requests)
1986
+ # - 5 = Combine 5 mutations per aliased request
1987
+ # - 10 = Combine 10 mutations per aliased request
1988
+ mutationsPerAliasBatch=
1989
+
1990
+ # ============================================================================
1991
+ # Fluent Commerce Configuration (Optional)
1992
+ # ============================================================================
1993
+ # Retailer ID - Only if mutation schema requires retailerId in input
1994
+ # Standard createLocation does NOT require this field
1995
+ # Check your GraphQL schema to determine if needed
1996
+ retailerId=my-retailer-id
1997
+ ```
1998
+
1999
+ **Notes:**
2000
+ - Webhook security is handled by Versori's native connection authentication. No manual API key configuration needed.
2001
+ - `retailerId` - Standard createLocation does not have this field. Only use if YOUR custom schema requires it.
2002
+ - See `fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md` for details.
2003
+
2004
+ ---
2005
+
2006
+ ## Schema Validation CLI Commands
2007
+
2008
+ Before deploying, validate your field mappings against the Fluent GraphQL schema:
2009
+
2010
+ ### 1. Introspect Schema
2011
+
2012
+ ```bash
2013
+ # Generate schema.json from live Fluent API
2014
+ npx fc-connect introspect-schema \
2015
+ --url https://api.fluentcommerce.com/graphql \
2016
+ --client-id your-client-id \
2017
+ --client-secret your-client-secret \
2018
+ --output schema.json
2019
+ ```
2020
+
2021
+ ### 2. Create Mapping Config File
2022
+
2023
+ **File: location-mapping.json**
2024
+
2025
+ ```json
2026
+ {
2027
+ "version": "1.0.0",
2028
+ "mutation": "createLocation",
2029
+ "sourceFormat": "xml",
2030
+ "returnFields": ["id", "ref", "name", "type", "status"],
2031
+ "description": "XML location to Fluent Commerce GraphQL mapping",
2032
+ "fields": {
2033
+ "ref": { "source": "location.@ref", "required": true, "resolver": "sdk.trim" },
2034
+ "name": { "source": "location.name", "required": true, "resolver": "sdk.trim" },
2035
+ "type": { "source": "location.@type", "required": true, "resolver": "sdk.uppercase" },
2036
+ "primaryAddress.ref": { "source": "location.@ref", "required": true },
2037
+ "primaryAddress.street": { "source": "location.address.street1" },
2038
+ "primaryAddress.latitude": {
2039
+ "source": "location.coordinates.@lat",
2040
+ "resolver": "sdk.parseFloat"
2041
+ },
2042
+ "primaryAddress.longitude": {
2043
+ "source": "location.coordinates.@lon",
2044
+ "resolver": "sdk.parseFloat"
2045
+ }
2046
+ }
2047
+ }
2048
+ ```
2049
+
2050
+ ### 3. Validate Mapping
2051
+
2052
+ ```bash
2053
+ # Validate that all target fields exist in schema
2054
+ npx fc-connect validate-schema \
2055
+ --mapping location-mapping.json \
2056
+ --schema schema.json
2057
+ ```
2058
+
2059
+ ### 4. Analyze Coverage
2060
+
2061
+ ```bash
2062
+ # Check which Location fields are mapped vs available
2063
+ npx fc-connect analyze-coverage \
2064
+ --mapping location-mapping.json \
2065
+ --schema schema.json \
2066
+ --type CreateLocationInput
2067
+ ```
2068
+
2069
+ **Output:**
2070
+
2071
+ ```
2072
+ ✅ Mapped: 15/42 fields (35%)
2073
+ ❌ Missing required: timezone (String!)
2074
+ ⚠️ Optional unmapped: supportPhoneNumber, networkId, attributes
2075
+ ```
2076
+
2077
+ ---
2078
+
2079
+ ## Testing Locally
2080
+
2081
+ ### 1. Upload Test XML to S3
2082
+
2083
+ ```bash
2084
+ aws s3 cp test-location.xml s3://my-location-bucket/locations/test-location.xml
2085
+ ```
2086
+
2087
+ ### 2. Deploy to Versori
2088
+
2089
+ ```bash
2090
+ npm run deploy
2091
+ ```
2092
+
2093
+ ### 3. Manual Testing
2094
+
2095
+ ```bash
2096
+ # Trigger manual sync (auth handled by Versori connection)
2097
+ curl -X POST https://your-workspace.versori.run/location-xml-adhoc
2098
+
2099
+ # Check job status
2100
+ curl -X POST https://your-workspace.versori.run/location-xml-job-status \
2101
+ -H "Content-Type: application/json" \
2102
+ -d '{"jobId": "location-xml-adhoc-1737525600000"}'
2103
+ ```
2104
+
2105
+ ### 4. Verify Processing
2106
+
2107
+ - Upload a small XML to S3 (2-3 locations) and trigger `adhoc` webhook
2108
+ - Verify GraphQL mutations are executed for each location
2109
+ - Confirm file moved from `locations/` to `processed/` in S3
2110
+ - Check KV state: errors and last processed metadata
2111
+ - Monitor rate limiting: verify mutations respect configured rate
2112
+
2113
+ ---
2114
+
2115
+ ## Deployment
2116
+
2117
+ ```bash
2118
+ # Deploy to Versori
2119
+ npm run deploy
2120
+
2121
+ # View logs
2122
+ npm run logs
2123
+
2124
+ # Monitor execution
2125
+ versori logs --follow
2126
+ ```
2127
+
2128
+ ---
2129
+
2130
+ ## Monitoring
2131
+
2132
+ ### Success Response
2133
+
2134
+ ```json
2135
+ {
2136
+ "success": true,
2137
+ "jobId": "location-xml-scheduled-1737525600000",
2138
+ "processed": 3,
2139
+ "skipped": 0,
2140
+ "failed": 0,
2141
+ "totalRecords": 12,
2142
+ "errors": []
2143
+ }
2144
+ ```
2145
+
2146
+ ### Partial Success Response
2147
+
2148
+ ```json
2149
+ {
2150
+ "success": true,
2151
+ "jobId": "location-xml-scheduled-1737525600000",
2152
+ "processed": 3,
2153
+ "skipped": 1,
2154
+ "failed": 0,
2155
+ "totalRecords": 10,
2156
+ "errors": ["locations-003.xml: 2 mapping errors"]
2157
+ }
2158
+ ```
2159
+
2160
+ ### Error Response
2161
+
2162
+ ```json
2163
+ {
2164
+ "success": false,
2165
+ "jobId": "location-xml-scheduled-1737525600000",
2166
+ "processed": 0,
2167
+ "skipped": 0,
2168
+ "failed": 1,
2169
+ "totalRecords": 0,
2170
+ "errors": ["locations-001.xml: Invalid XML structure"]
2171
+ }
2172
+ ```
2173
+
2174
+ ---
2175
+
2176
+ ## Common Pitfalls and Solutions
2177
+
2178
+ ### 1. XML Attribute Not Found
2179
+
2180
+ **Symptoms**: Mapping errors like "field not found"
2181
+
2182
+ **Solution**:
2183
+
2184
+ ```typescript
2185
+ // ❌ WRONG - Missing @ prefix for attribute
2186
+ ref: {
2187
+ source: 'location.ref';
2188
+ }
2189
+
2190
+ // ✅ CORRECT - Use @ prefix for XML attributes
2191
+ ref: {
2192
+ source: 'location.@ref';
2193
+ }
2194
+ ```
2195
+
2196
+ ### 2. Single Element Not Array
2197
+
2198
+ **Symptoms**: "Cannot read property forEach of undefined"
2199
+
2200
+ **Solution**:
2201
+
2202
+ ```typescript
2203
+ // Always normalize to array
2204
+ const locationsData = xmlData.locations?.location;
2205
+ const locations = Array.isArray(locationsData) ? locationsData : [locationsData];
2206
+ ```
2207
+
2208
+ ### 3. Rate Limiting Too Aggressive
2209
+
2210
+ **Symptoms**: Slow processing, mutations taking too long
2211
+
2212
+ **Solution**:
2213
+
2214
+ ```bash
2215
+ # Increase rate limit (mutations per second)
2216
+ mutationRateLimit=10 # Default is 5
2217
+ ```
2218
+
2219
+ ### 4. Empty Element vs Missing Element
2220
+
2221
+ **Solution**:
2222
+
2223
+ ```typescript
2224
+ // Use required: false and defaultValue for optional fields
2225
+ 'primaryAddress.street2': {
2226
+ source: 'location.address.street2',
2227
+ required: false,
2228
+ defaultValue: ''
2229
+ }
2230
+ ```
2231
+
2232
+ ### 5. S3 Access Denied
2233
+
2234
+ **Symptoms**: S3 operations fail with 403 errors
2235
+
2236
+ **Solution**: Validate IAM permissions
2237
+
2238
+ **Required IAM Permissions:**
2239
+
2240
+ ```json
2241
+ {
2242
+ "Version": "2012-10-17",
2243
+ "Statement": [
2244
+ {
2245
+ "Effect": "Allow",
2246
+ "Action": ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
2247
+ "Resource": ["arn:aws:s3:::my-location-bucket", "arn:aws:s3:::my-location-bucket/*"]
2248
+ }
2249
+ ]
2250
+ }
2251
+ ```
2252
+
2253
+ ### 6. GraphQL Schema Mismatch
2254
+
2255
+ **Symptoms**: Mutation errors like "Unknown field", "Invalid input type"
2256
+
2257
+ **Solution**: Use CLI tools to validate mappings
2258
+
2259
+ ```bash
2260
+ npx fc-connect validate-schema --mapping location-mapping.json --schema schema.json
2261
+ ```
2262
+
2263
+ ### 7. Nested Object Mapping Errors
2264
+
2265
+ **Symptoms**: Flat structure instead of nested objects in mutation input
2266
+
2267
+ **Solution**: Use dot notation in field mapping
2268
+
2269
+ ```typescript
2270
+ // ✅ CORRECT - Creates nested structure
2271
+ 'primaryAddress.city': { source: 'location.address.city' }
2272
+
2273
+ // ❌ WRONG - Creates flat structure
2274
+ primaryAddress_city: { source: 'location.address.city' }
2275
+ ```
2276
+
2277
+ ### 8. retailerId Configuration Errors
2278
+
2279
+ **Symptoms**: "retailerId is required" errors or confusion about when to use `setRetailerId()`
2280
+
2281
+ **Solution**: Understand the correct pattern
2282
+
2283
+ ```typescript
2284
+ // ✅ CORRECT - GraphQL mutations don't need setRetailerId()
2285
+ // Check your GraphQL schema to determine retailerId handling:
2286
+ // - Mandatory retailerId → Must pass it in mutation input
2287
+ // - Optional retailerId → Can pass it if needed
2288
+ // - No retailerId field → Don't pass it
2289
+ // Standard createLocation does not have retailerId field in schema
2290
+
2291
+ // ✅ IF mutation schema requires retailerId (mandatory):
2292
+ const { query, variables } = await mapper.map(location);
2293
+ if (retailerId && variables.input) {
2294
+ variables.input.retailer = { id: parseInt(retailerId) };
2295
+ }
2296
+ await client.graphql({ query, variables });
2297
+ ```
2298
+
2299
+ **Reference:** `fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md`
2300
+
2301
+ ---
2302
+
2303
+ ## Key Takeaways
2304
+
2305
+ ### Architecture Patterns
2306
+
2307
+ - **Service Function Composition**: Workflow broken into 3 service functions - `processFile()`, `executeMutations()`, `writeMutationLog()`
2308
+ - **Per-File Processing**: Main workflow orchestrates service functions for each file
2309
+ - **Clear Separation of Concerns**: Parse/map, execute mutations, logging are independent functions
2310
+
2311
+ ### SDK Usage
2312
+
2313
+ - **Buffer Import (CRITICAL)**: Always `import { Buffer } from 'node:buffer'` for Deno/Versori runtime
2314
+ - **S3 Archival Deduplication**: Use `s3.moveFile()` to `processed/` subdirectory - PRIMARY deduplication mechanism
2315
+ - **NO VersoriFileTracker for S3**: S3 archival is simpler and more reliable
2316
+ - **StateService Role**: SECONDARY - provides metadata/history, not primary deduplication
2317
+
2318
+ ### XML Processing
2319
+
2320
+ - **XML @ Prefix**: Always use `@` prefix for XML attributes (`location.@ref`)
2321
+ - **Array Normalization (CRITICAL)**: Handle single vs multiple elements with `Array.isArray()` check - single `<location>` becomes object, not array
2322
+ - **Nested Mapping**: Use dot notation for nested objects (`primaryAddress.street`, `openingSchedule.monStart`)
2323
+
2324
+ ### GraphQL Mutations
2325
+
2326
+ - **NO setRetailerId() Required**: GraphQL mutations do NOT need `client.setRetailerId()` - only Job/Event API needs it
2327
+ - **retailerId in Input**: Check your GraphQL schema to determine retailerId handling:
2328
+ - **Mandatory retailerId** - Field exists and is required (`!`) → Must pass it
2329
+ - **Optional retailerId** - Field exists and is optional → Can pass it if needed
2330
+ - **No retailerId field** - Field doesn't exist → Don't pass it
2331
+ - **Rate Limiting**: Implement configurable delays between mutations to avoid API throttling
2332
+ - **Retry Logic**: Exponential backoff for failed mutations with `retryWithBackoff()`
2333
+ - **Direct Mutations**: Use `client.graphql()` for location upserts (NOT Batch API)
2334
+ - **GraphQL vs Batch API**: Use GraphQL for low-volume master data, Batch API for high-volume inventory
2335
+
2336
+ ### Error Handling & Monitoring
2337
+
2338
+ - **Error Recovery**: Exponential backoff for error state tracking with retry timestamps
2339
+ - **Mutation Logging**: Optional S3 JSON logs with detailed per-location mutation status
2340
+ - **Schema Validation**: Use CLI tools before deployment to catch mapping errors
2341
+ - **Archival Order**: Archive FIRST (deduplication), then KV tracking (metadata)
2342
+
2343
+ ---
2344
+
2345
+ ## Related Documentation
2346
+
2347
+ ### Core Guides
2348
+
2349
+ - **GraphQL Mutation Mapping**: `fc-connect-sdk/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/readme.md`
2350
+ - **Universal Mapping**: `fc-connect-sdk/docs/02-CORE-GUIDES/mapping/modules/readme.md`
2351
+ - **XML Parser**: `fc-connect-sdk/docs/02-CORE-GUIDES/parsers/modules/05-xml-parser.md`
2352
+ - **Data Sources**: `fc-connect-sdk/docs/02-CORE-GUIDES/data-sources/readme.md`
2353
+ - **State Management**: `fc-connect-sdk/docs/03-PATTERN-GUIDES/file-operations/state-duplicate-prevention.md`
2354
+
2355
+ ### Related Templates
2356
+
2357
+ - **CSV Version**: `template-ingestion-s3-csv-location-graphql.md`
2358
+ - **JSON Version**: `template-ingestion-s3-json-location-graphql.md`
2359
+ - **SFTP XML Version**: `template-ingestion-sftp-xml-location-graphql.md`
2360
+ - **Event API Pattern**: `../event-api/template-ingestion-s3-xml-product-event.md`
2361
+ - **Batch API Pattern**: `../batch-api/template-ingestion-s3-xml-inventory-batch.md`
2362
+
2363
+ ### CLI Tools
2364
+
2365
+ - **Schema Introspection**: `fc-connect-sdk/bin/readme.md#introspect-schema`
2366
+ - **Mapping Validation**: `fc-connect-sdk/bin/readme.md#validate-schema`
2367
+ - **Coverage Analysis**: `fc-connect-sdk/bin/readme.md#analyze-coverage`
2368
+
2369
+ ### Patterns
2370
+
2371
+ - **Error Handling**: `fc-connect-sdk/docs/01-TEMPLATES/patterns/error-handling-retry.md`
2372
+ - **Rate Limiting**: `fc-connect-sdk/docs/03-PATTERN-GUIDES/integration-patterns/rate-limiting.md`
2373
+ - **XML Patterns**: `fc-connect-sdk/docs/01-TEMPLATES/versori/patterns/xml-response-patterns.md`