@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.56

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (476) hide show
  1. package/CHANGELOG.md +12 -0
  2. package/README.md +11 -0
  3. package/dist/cjs/clients/fluent-client.js +13 -6
  4. package/dist/cjs/utils/pagination-helpers.js +38 -2
  5. package/dist/cjs/versori/fluent-versori-client.js +11 -5
  6. package/dist/esm/clients/fluent-client.js +13 -6
  7. package/dist/esm/utils/pagination-helpers.js +38 -2
  8. package/dist/esm/versori/fluent-versori-client.js +11 -5
  9. package/dist/tsconfig.esm.tsbuildinfo +1 -1
  10. package/dist/tsconfig.tsbuildinfo +1 -1
  11. package/dist/tsconfig.types.tsbuildinfo +1 -1
  12. package/docs/00-START-HERE/EXPORT-VALIDATION.md +158 -158
  13. package/docs/00-START-HERE/cli-analyze-source-structure-guide.md +655 -655
  14. package/docs/00-START-HERE/cli-documentation-index.md +202 -202
  15. package/docs/00-START-HERE/cli-quick-reference.md +252 -252
  16. package/docs/00-START-HERE/decision-tree.md +552 -552
  17. package/docs/00-START-HERE/getting-started.md +1070 -1070
  18. package/docs/00-START-HERE/mapper-quick-decision-guide.md +235 -235
  19. package/docs/00-START-HERE/readme.md +237 -237
  20. package/docs/00-START-HERE/retailerid-configuration.md +404 -404
  21. package/docs/00-START-HERE/sdk-philosophy.md +794 -794
  22. package/docs/00-START-HERE/troubleshooting-quick-reference.md +1086 -1086
  23. package/docs/01-TEMPLATES/faq.md +686 -686
  24. package/docs/01-TEMPLATES/patterns/pattern-templates-guide.md +68 -68
  25. package/docs/01-TEMPLATES/patterns/patterns-csv-schema-validation-and-rejection-report.md +233 -233
  26. package/docs/01-TEMPLATES/patterns/patterns-custom-resolvers.md +407 -407
  27. package/docs/01-TEMPLATES/patterns/patterns-error-handling-retry.md +511 -511
  28. package/docs/01-TEMPLATES/patterns/patterns-field-mapping-universal.md +701 -701
  29. package/docs/01-TEMPLATES/patterns/patterns-large-file-splitting.md +1430 -1430
  30. package/docs/01-TEMPLATES/patterns/patterns-master-data-etl.md +2399 -2399
  31. package/docs/01-TEMPLATES/patterns/patterns-pagination-streaming.md +447 -447
  32. package/docs/01-TEMPLATES/patterns/patterns-state-duplicate-prevention.md +385 -385
  33. package/docs/01-TEMPLATES/readme.md +957 -957
  34. package/docs/01-TEMPLATES/standalone/standalone-asn-inbound-processing.md +1209 -1209
  35. package/docs/01-TEMPLATES/standalone/standalone-graphql-query-export.md +1140 -1140
  36. package/docs/01-TEMPLATES/standalone/standalone-graphql-to-parquet-partitioned-s3.md +432 -432
  37. package/docs/01-TEMPLATES/standalone/standalone-multi-channel-inventory-sync.md +1185 -1185
  38. package/docs/01-TEMPLATES/standalone/standalone-multi-source-aggregation.md +1462 -1462
  39. package/docs/01-TEMPLATES/standalone/standalone-s3-csv-batch-api.md +1390 -1390
  40. package/docs/01-TEMPLATES/standalone/standalone-s3-csv-inventory-to-batch.md +330 -330
  41. package/docs/01-TEMPLATES/standalone/standalone-scripts-guide.md +87 -87
  42. package/docs/01-TEMPLATES/standalone/standalone-sftp-xml-graphql.md +1444 -1444
  43. package/docs/01-TEMPLATES/standalone/standalone-webhook-payload-processing.md +688 -688
  44. package/docs/01-TEMPLATES/versori/business-examples/business-examples-dropship-order-routing.md +193 -193
  45. package/docs/01-TEMPLATES/versori/business-examples/business-examples-graphql-parquet-extraction.md +518 -518
  46. package/docs/01-TEMPLATES/versori/business-examples/business-examples-inter-location-transfers.md +2162 -2162
  47. package/docs/01-TEMPLATES/versori/business-examples/business-examples-pre-order-allocation.md +2226 -2226
  48. package/docs/01-TEMPLATES/versori/business-examples/business-scenarios-guide.md +87 -87
  49. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-connection-validation-pattern.md +656 -656
  50. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-dual-workflow-connector.md +835 -835
  51. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-guide.md +108 -108
  52. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-kv-state-management.md +1533 -1533
  53. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-xml-response-patterns.md +1160 -1160
  54. package/docs/01-TEMPLATES/versori/versori-platform-guide.md +201 -201
  55. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-asn-purchase-order.md +1906 -1906
  56. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-dropship-routing.md +1074 -1074
  57. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-flash-sale-reserve.md +1395 -1395
  58. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-generic-xml-order.md +888 -888
  59. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-payment-gateway-integration.md +2478 -2478
  60. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-rma-returns-comprehensive.md +2240 -2240
  61. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-xml-order-ingestion.md +2029 -2029
  62. package/docs/01-TEMPLATES/versori/webhooks/webhook-templates-guide.md +140 -140
  63. package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/inventory-mapping.json +20 -20
  64. package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/products_2025-01-22.csv +11 -11
  65. package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/sample-data-guide.md +34 -34
  66. package/docs/01-TEMPLATES/versori/workflows/_examples/workflow-examples-guide.md +36 -36
  67. package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-modes-guide.md +1038 -1038
  68. package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-workflows-guide.md +138 -138
  69. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/graphql-extraction-guide.md +63 -63
  70. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-csv.md +2062 -2062
  71. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-xml.md +2294 -2294
  72. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-s3-csv.md +2461 -2461
  73. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-sftp-xml.md +2529 -2529
  74. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-csv.md +2464 -2464
  75. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-json.md +1959 -1959
  76. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-s3-csv.md +1953 -1953
  77. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-sftp-xml.md +2541 -2541
  78. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-s3-json.md +2384 -2384
  79. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-sftp-xml.md +2445 -2445
  80. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-csv.md +2355 -2355
  81. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-json.md +2042 -2042
  82. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-sftp-xml.md +2726 -2726
  83. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/batch-api-guide.md +206 -206
  84. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-cycle-count-reconciliation.md +2030 -2030
  85. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-multi-channel-inventory-sync.md +1882 -1882
  86. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-csv-inventory-batch.md +2827 -2827
  87. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-json-inventory-batch.md +1952 -1952
  88. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-xml-inventory-batch.md +3289 -3289
  89. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-csv-inventory-batch.md +3064 -3064
  90. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-json-inventory-batch.md +3238 -3238
  91. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-xml-inventory-batch.md +2977 -2977
  92. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/event-api-guide.md +321 -321
  93. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-json-order-cancel-event.md +959 -959
  94. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-xml-order-cancel-event.md +1170 -1170
  95. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-csv-product-event.md +2312 -2312
  96. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-json-product-event.md +2999 -2999
  97. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-parquet-product-event.md +2836 -2836
  98. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-xml-product-event.md +2395 -2395
  99. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-csv-product-event.md +2295 -2295
  100. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-json-product-event.md +2602 -2602
  101. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-parquet-product-event.md +2589 -2589
  102. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-xml-product-event.md +3578 -3578
  103. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/graphql-mutations-guide.md +93 -93
  104. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-json-order-update-graphql.md +1260 -1260
  105. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-xml-order-update-graphql.md +1472 -1472
  106. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-control-graphql.md +2417 -2417
  107. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-location-graphql.md +2811 -2811
  108. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-price-graphql.md +2619 -2619
  109. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-json-location-graphql.md +2807 -2807
  110. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-xml-location-graphql.md +2373 -2373
  111. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-control-graphql.md +2740 -2740
  112. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-location-graphql.md +2760 -2760
  113. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-json-location-graphql.md +1710 -1710
  114. package/docs/01-TEMPLATES/versori/workflows/ingestion/ingestion-workflows-guide.md +136 -136
  115. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/rubix-webhooks-guide.md +520 -520
  116. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-inline.md +1418 -1418
  117. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-universal-mapper.md +1785 -1785
  118. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-order-attribute-update.md +824 -824
  119. package/docs/01-TEMPLATES/versori/workflows/workflows-overview-guide.md +646 -646
  120. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-batch-archival.md +724 -724
  121. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-job-tracker.md +627 -627
  122. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-partial-batch-recovery.md +561 -561
  123. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-quick-reference.md +367 -367
  124. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-readme.md +407 -407
  125. package/docs/02-CORE-GUIDES/advanced-services/readme.md +49 -49
  126. package/docs/02-CORE-GUIDES/api-reference/api-reference-quick-reference.md +548 -548
  127. package/docs/02-CORE-GUIDES/api-reference/event-api-input-output-reference.md +702 -1171
  128. package/docs/02-CORE-GUIDES/api-reference/examples/client-initialization.ts +286 -286
  129. package/docs/02-CORE-GUIDES/api-reference/graphql-error-classification.md +337 -337
  130. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-01-client-api.md +399 -520
  131. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-03-authentication.md +199 -199
  132. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-04-graphql-mapping.md +925 -925
  133. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-05-services.md +1198 -1198
  134. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-06-data-sources.md +1083 -1083
  135. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-07-parsers.md +1097 -1097
  136. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-pagination.md +513 -513
  137. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-types.md +545 -597
  138. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-error-handling.md +527 -527
  139. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-webhook-validation.md +514 -514
  140. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-extraction.md +557 -557
  141. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-utilities.md +412 -412
  142. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-cli-tools.md +423 -423
  143. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-error-handling.md +716 -716
  144. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-analyze-source-structure.md +518 -518
  145. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-partial-responses.md +212 -212
  146. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-testing.md +300 -300
  147. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-13-resolver-builder.md +322 -322
  148. package/docs/02-CORE-GUIDES/api-reference/readme.md +279 -279
  149. package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-quick-reference.md +351 -351
  150. package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-readme.md +277 -277
  151. package/docs/02-CORE-GUIDES/auto-pagination/examples/auto-pagination-readme.md +178 -178
  152. package/docs/02-CORE-GUIDES/auto-pagination/examples/common-patterns.ts +351 -351
  153. package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-products.ts +384 -384
  154. package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-virtual-positions.ts +308 -308
  155. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-01-foundations.md +470 -470
  156. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-02-quick-start.md +713 -713
  157. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-03-configuration.md +754 -754
  158. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-04-advanced-patterns.md +732 -732
  159. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-05-sdk-integration.md +847 -847
  160. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-06-troubleshooting.md +359 -359
  161. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-07-api-reference.md +462 -462
  162. package/docs/02-CORE-GUIDES/auto-pagination/readme.md +54 -54
  163. package/docs/02-CORE-GUIDES/data-sources/data-sources-file-operations-error-handling.md +1487 -1487
  164. package/docs/02-CORE-GUIDES/data-sources/data-sources-quick-reference.md +836 -836
  165. package/docs/02-CORE-GUIDES/data-sources/data-sources-readme.md +276 -276
  166. package/docs/02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md +553 -553
  167. package/docs/02-CORE-GUIDES/data-sources/examples/common-patterns.ts +409 -409
  168. package/docs/02-CORE-GUIDES/data-sources/examples/data-sources-readme.md +178 -178
  169. package/docs/02-CORE-GUIDES/data-sources/examples/s3-operations.ts +308 -308
  170. package/docs/02-CORE-GUIDES/data-sources/examples/sftp-operations.ts +371 -371
  171. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-01-foundations.md +735 -735
  172. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-02-s3-operations.md +1302 -1302
  173. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-03-sftp-operations.md +1379 -1379
  174. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-04-file-patterns.md +941 -941
  175. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-05-advanced-topics.md +813 -813
  176. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-06-integration-patterns.md +486 -486
  177. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-07-troubleshooting.md +387 -387
  178. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-08-api-reference.md +417 -417
  179. package/docs/02-CORE-GUIDES/data-sources/readme.md +77 -77
  180. package/docs/02-CORE-GUIDES/error-handling-guide.md +936 -936
  181. package/docs/02-CORE-GUIDES/extraction/examples/02-core-guides-extraction-readme.md +116 -116
  182. package/docs/02-CORE-GUIDES/extraction/examples/common-patterns.ts +428 -428
  183. package/docs/02-CORE-GUIDES/extraction/examples/extract-inventory-basic.ts +187 -187
  184. package/docs/02-CORE-GUIDES/extraction/extraction-quick-reference.md +596 -596
  185. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-01-foundations.md +514 -514
  186. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-02-basic-extraction.md +823 -823
  187. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-03-parquet-processing.md +507 -507
  188. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-04-data-enrichment.md +546 -546
  189. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-05-transformation.md +494 -494
  190. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-export-formats.md +458 -458
  191. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-performance.md +138 -138
  192. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-api-reference.md +148 -148
  193. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-optimization.md +692 -692
  194. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-08-extraction-orchestrator.md +1008 -1008
  195. package/docs/02-CORE-GUIDES/extraction/readme.md +151 -151
  196. package/docs/02-CORE-GUIDES/ingestion/examples/_simple-kv-store.ts +40 -40
  197. package/docs/02-CORE-GUIDES/ingestion/examples/error-recovery.ts +728 -728
  198. package/docs/02-CORE-GUIDES/ingestion/examples/event-driven.ts +501 -501
  199. package/docs/02-CORE-GUIDES/ingestion/examples/local-file-ingestion.ts +88 -88
  200. package/docs/02-CORE-GUIDES/ingestion/examples/parquet-ingestion.ts +117 -117
  201. package/docs/02-CORE-GUIDES/ingestion/examples/performance-optimized.ts +647 -647
  202. package/docs/02-CORE-GUIDES/ingestion/examples/s3-csv-ingestion.ts +169 -169
  203. package/docs/02-CORE-GUIDES/ingestion/examples/sftp-csv-ingestion.ts +134 -134
  204. package/docs/02-CORE-GUIDES/ingestion/ingestion-quick-reference.md +546 -546
  205. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-01-introduction.md +626 -626
  206. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-02-quick-start.md +658 -658
  207. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-03-data-sources.md +1052 -1052
  208. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-04-field-mapping.md +763 -763
  209. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-05-advanced-parsers.md +676 -676
  210. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-06-batch-api.md +1295 -1295
  211. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-api-reference.md +138 -138
  212. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-state-management.md +1037 -1037
  213. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-08-performance-optimization.md +1349 -1349
  214. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-09-best-practices.md +1893 -1893
  215. package/docs/02-CORE-GUIDES/ingestion/readme.md +160 -160
  216. package/docs/02-CORE-GUIDES/logging-guide.md +585 -585
  217. package/docs/02-CORE-GUIDES/mapping/error-handling-patterns.md +401 -401
  218. package/docs/02-CORE-GUIDES/mapping/examples/02-core-guides-mapping-readme.md +128 -128
  219. package/docs/02-CORE-GUIDES/mapping/examples/common-patterns.ts +273 -273
  220. package/docs/02-CORE-GUIDES/mapping/examples/csv-location-ingestion.json +36 -36
  221. package/docs/02-CORE-GUIDES/mapping/examples/csv-mapping.ts +242 -242
  222. package/docs/02-CORE-GUIDES/mapping/examples/graphql-to-parquet-extraction.json +36 -36
  223. package/docs/02-CORE-GUIDES/mapping/examples/json-mapping.ts +213 -213
  224. package/docs/02-CORE-GUIDES/mapping/examples/json-product-to-mutation.json +48 -48
  225. package/docs/02-CORE-GUIDES/mapping/examples/xml-mapping.ts +291 -291
  226. package/docs/02-CORE-GUIDES/mapping/examples/xml-order-to-mutation.json +45 -45
  227. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-quick-reference.md +463 -463
  228. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-readme.md +227 -227
  229. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-01-introduction.md +222 -222
  230. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-02-quick-start.md +351 -351
  231. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-03-schema-validation.md +569 -569
  232. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-04-mapping-patterns.md +471 -471
  233. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-05-configuration-reference.md +611 -611
  234. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-advanced-xpath.md +148 -148
  235. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-path-syntax.md +464 -464
  236. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-api-reference.md +94 -94
  237. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-array-handling.md +307 -307
  238. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-08-custom-resolvers.md +544 -544
  239. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-09-advanced-patterns.md +427 -427
  240. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-10-hooks-and-variables.md +336 -336
  241. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-11-error-handling.md +488 -488
  242. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-12-arguments-vs-nodes.md +383 -383
  243. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-13-best-practices.md +477 -477
  244. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/readme.md +62 -62
  245. package/docs/02-CORE-GUIDES/mapping/mapping-format-decision-tree.md +480 -480
  246. package/docs/02-CORE-GUIDES/mapping/mapping-graphql-alias-batching-guide.md +820 -820
  247. package/docs/02-CORE-GUIDES/mapping/mapping-javascript-objects.md +2369 -2369
  248. package/docs/02-CORE-GUIDES/mapping/mapping-mapper-comparison-guide.md +682 -682
  249. package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-07-api-reference.md +1327 -1327
  250. package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-08-error-handling.md +1142 -1142
  251. package/docs/02-CORE-GUIDES/mapping/modules/mapping-04-use-cases.md +891 -891
  252. package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-helpers-resolvers.md +1126 -1126
  253. package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-sdk-resolvers.md +199 -199
  254. package/docs/02-CORE-GUIDES/mapping/modules/mapping-07-api-reference.md +1319 -1319
  255. package/docs/02-CORE-GUIDES/mapping/readme.md +178 -178
  256. package/docs/02-CORE-GUIDES/mapping/resolver-registration.md +410 -410
  257. package/docs/02-CORE-GUIDES/mapping/resolvers/examples/common-patterns.ts +226 -226
  258. package/docs/02-CORE-GUIDES/mapping/resolvers/examples/custom-resolvers.ts +227 -227
  259. package/docs/02-CORE-GUIDES/mapping/resolvers/examples/sdk-resolvers-usage.ts +203 -203
  260. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-readme.md +274 -274
  261. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-api-reference.md +679 -679
  262. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-cookbook.md +826 -826
  263. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-guide.md +1330 -1330
  264. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-helpers-reference.md +1437 -1437
  265. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-parameters-reference.md +553 -553
  266. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-troubleshooting.md +854 -854
  267. package/docs/02-CORE-GUIDES/mapping/resolvers/readme.md +75 -75
  268. package/docs/02-CORE-GUIDES/parsers/examples/02-core-guides-parsers-readme.md +161 -161
  269. package/docs/02-CORE-GUIDES/parsers/examples/csv-parser-examples.ts +110 -110
  270. package/docs/02-CORE-GUIDES/parsers/examples/json-parser-examples.ts +33 -33
  271. package/docs/02-CORE-GUIDES/parsers/examples/parquet-parser-examples.ts +47 -47
  272. package/docs/02-CORE-GUIDES/parsers/examples/xml-parser-examples.ts +38 -38
  273. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-01-foundations.md +355 -355
  274. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-02-csv-parser.md +772 -772
  275. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-03-json-parser.md +789 -789
  276. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-04-xml-parser.md +857 -857
  277. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-05-parquet-parser.md +603 -603
  278. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-integration-patterns.md +702 -702
  279. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-streaming.md +121 -121
  280. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-api-reference.md +89 -89
  281. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-troubleshooting.md +727 -727
  282. package/docs/02-CORE-GUIDES/parsers/parsers-quick-reference.md +482 -482
  283. package/docs/02-CORE-GUIDES/parsers/parsers-readme.md +258 -258
  284. package/docs/02-CORE-GUIDES/parsers/readme.md +65 -65
  285. package/docs/02-CORE-GUIDES/readme.md +194 -194
  286. package/docs/02-CORE-GUIDES/webhook-validation/examples/basic-validation.ts +108 -108
  287. package/docs/02-CORE-GUIDES/webhook-validation/examples/common-patterns.ts +316 -316
  288. package/docs/02-CORE-GUIDES/webhook-validation/examples/webhook-validation-readme.md +61 -61
  289. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-01-foundations.md +440 -440
  290. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-02-quick-start.md +525 -525
  291. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-03-versori-integration.md +741 -741
  292. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-04-platform-integration.md +629 -629
  293. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-05-configuration.md +535 -535
  294. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-error-handling.md +611 -611
  295. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-troubleshooting.md +124 -124
  296. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-07-api-reference.md +511 -511
  297. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-08-rubix-webhooks.md +590 -590
  298. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-09-rubix-event-vs-http-call.md +432 -432
  299. package/docs/02-CORE-GUIDES/webhook-validation/readme.md +239 -239
  300. package/docs/02-CORE-GUIDES/webhook-validation/webhook-validation-quick-reference.md +392 -392
  301. package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-quick-reference.md +498 -498
  302. package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-readme.md +313 -313
  303. package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/common-patterns.ts +612 -612
  304. package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/connector-scenarios-readme.md +253 -253
  305. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-01-foundations.md +452 -452
  306. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-02-simple-scenarios.md +681 -681
  307. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-03-intermediate-scenarios.md +637 -637
  308. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-04-advanced-scenarios.md +650 -650
  309. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-05-bidirectional-sync.md +233 -233
  310. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-06-production-patterns.md +442 -442
  311. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-07-reference.md +445 -445
  312. package/docs/03-PATTERN-GUIDES/connector-scenarios/readme.md +31 -31
  313. package/docs/03-PATTERN-GUIDES/enterprise-integration-patterns.md +1528 -1528
  314. package/docs/03-PATTERN-GUIDES/error-handling/comprehensive-error-handling-guide.md +1437 -1437
  315. package/docs/03-PATTERN-GUIDES/error-handling/error-handling-quick-reference.md +390 -390
  316. package/docs/03-PATTERN-GUIDES/error-handling/examples/common-patterns.ts +438 -438
  317. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-01-foundations.md +362 -362
  318. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-02-error-types.md +850 -850
  319. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-03-utf8-handling.md +456 -456
  320. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-04-error-scenarios.md +658 -658
  321. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-05-calling-patterns.md +671 -671
  322. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-06-retry-strategies.md +1034 -1034
  323. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-07-monitoring.md +653 -653
  324. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-08-api-reference.md +847 -847
  325. package/docs/03-PATTERN-GUIDES/error-handling/readme.md +36 -36
  326. package/docs/03-PATTERN-GUIDES/examples/__tests__/readme.md +40 -40
  327. package/docs/03-PATTERN-GUIDES/examples/__tests__/resolver-examples.test.js +282 -282
  328. package/docs/03-PATTERN-GUIDES/examples/test-data/03-pattern-guides-readme.md +110 -110
  329. package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-inventory.json +123 -123
  330. package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-order.json +171 -171
  331. package/docs/03-PATTERN-GUIDES/examples/test-data/readme.md +28 -28
  332. package/docs/03-PATTERN-GUIDES/extraction/extraction-readme.md +15 -15
  333. package/docs/03-PATTERN-GUIDES/extraction/readme.md +25 -25
  334. package/docs/03-PATTERN-GUIDES/file-operations/examples/common-patterns.ts +407 -407
  335. package/docs/03-PATTERN-GUIDES/file-operations/examples/file-operations-readme.md +142 -142
  336. package/docs/03-PATTERN-GUIDES/file-operations/file-operations-quick-reference.md +462 -462
  337. package/docs/03-PATTERN-GUIDES/file-operations/file-operations-readme.md +379 -379
  338. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-01-foundations.md +430 -430
  339. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-02-quick-start.md +484 -484
  340. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-03-s3-operations.md +507 -507
  341. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-04-sftp-operations.md +963 -963
  342. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-05-streaming-performance.md +503 -503
  343. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-archive-patterns.md +386 -386
  344. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-error-handling.md +117 -117
  345. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-api-reference.md +78 -78
  346. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-testing-troubleshooting.md +567 -567
  347. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-08-api-reference.md +1055 -1055
  348. package/docs/03-PATTERN-GUIDES/file-operations/readme.md +32 -32
  349. package/docs/03-PATTERN-GUIDES/ingestion/ingestion-readme.md +15 -15
  350. package/docs/03-PATTERN-GUIDES/ingestion/readme.md +25 -25
  351. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/batch-processing.ts +130 -130
  352. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/common-patterns.ts +360 -360
  353. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/delta-sync.ts +130 -130
  354. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/integration-patterns-readme.md +100 -100
  355. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/real-time-webhook.ts +398 -398
  356. package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-quick-reference.md +962 -962
  357. package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-readme.md +134 -134
  358. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-01-real-time-processing.md +991 -991
  359. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-02-batch-processing.md +1547 -1547
  360. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-03-delta-sync.md +1108 -1108
  361. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-04-webhook-patterns.md +1181 -1181
  362. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-05-error-handling.md +1061 -1061
  363. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-advanced-integration-services.md +1547 -1547
  364. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-performance.md +109 -109
  365. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-07-api-reference.md +34 -34
  366. package/docs/03-PATTERN-GUIDES/integration-patterns/readme.md +30 -30
  367. package/docs/03-PATTERN-GUIDES/logging-minimal-mode.md +128 -128
  368. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/common-patterns.ts +380 -380
  369. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/multiple-connections-readme.md +139 -139
  370. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/parallel-root-connections.ts +149 -149
  371. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/real-world-scenarios.ts +405 -405
  372. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-01-foundations.md +378 -378
  373. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-02-quick-start.md +566 -566
  374. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-03-targeting-connections.md +659 -659
  375. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-04-parallel-queries.md +656 -656
  376. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-05-best-practices.md +624 -624
  377. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-api-reference.md +824 -824
  378. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-versori.md +119 -119
  379. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-07-api-reference.md +87 -87
  380. package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-quick-reference.md +353 -353
  381. package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-readme.md +270 -270
  382. package/docs/03-PATTERN-GUIDES/multiple-connections/readme.md +30 -30
  383. package/docs/03-PATTERN-GUIDES/pagination/pagination-readme.md +14 -14
  384. package/docs/03-PATTERN-GUIDES/pagination/readme.md +24 -24
  385. package/docs/03-PATTERN-GUIDES/parquet/examples/common-patterns.ts +180 -180
  386. package/docs/03-PATTERN-GUIDES/parquet/examples/read-parquet.ts +48 -48
  387. package/docs/03-PATTERN-GUIDES/parquet/examples/write-parquet.ts +65 -65
  388. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-01-introduction.md +393 -393
  389. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-02-quick-start.md +572 -572
  390. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-03-reading-parquet.md +525 -525
  391. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-04-writing-parquet.md +554 -554
  392. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-05-graphql-extraction.md +405 -405
  393. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-performance.md +104 -104
  394. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-s3-integration.md +511 -511
  395. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-api-reference.md +90 -90
  396. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-performance-optimization.md +525 -525
  397. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-08-best-practices.md +712 -712
  398. package/docs/03-PATTERN-GUIDES/parquet/parquet-quick-reference.md +683 -683
  399. package/docs/03-PATTERN-GUIDES/parquet/parquet-readme.md +248 -248
  400. package/docs/03-PATTERN-GUIDES/parquet/readme.md +32 -32
  401. package/docs/03-PATTERN-GUIDES/parsers/parsers-readme.md +12 -12
  402. package/docs/03-PATTERN-GUIDES/parsers/readme.md +24 -24
  403. package/docs/03-PATTERN-GUIDES/readme.md +159 -159
  404. package/docs/03-PATTERN-GUIDES/webhooks/readme.md +24 -24
  405. package/docs/03-PATTERN-GUIDES/webhooks/webhooks-readme.md +8 -8
  406. package/docs/04-REFERENCE/architecture/architecture-01-overview.md +427 -427
  407. package/docs/04-REFERENCE/architecture/architecture-02-client-architecture.md +424 -424
  408. package/docs/04-REFERENCE/architecture/architecture-03-data-flow.md +690 -690
  409. package/docs/04-REFERENCE/architecture/architecture-04-service-layer.md +834 -834
  410. package/docs/04-REFERENCE/architecture/architecture-05-integration-architecture.md +655 -655
  411. package/docs/04-REFERENCE/architecture/architecture-06-state-management.md +653 -653
  412. package/docs/04-REFERENCE/architecture/architecture-adding-new-data-sources.md +686 -686
  413. package/docs/04-REFERENCE/architecture/readme.md +279 -279
  414. package/docs/04-REFERENCE/platforms/deno/readme.md +117 -117
  415. package/docs/04-REFERENCE/platforms/nodejs/readme.md +146 -146
  416. package/docs/04-REFERENCE/platforms/readme.md +135 -135
  417. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-01-introduction.md +398 -398
  418. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-02-quick-start.md +560 -560
  419. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-03-authentication.md +757 -757
  420. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-04-workflows.md +2476 -2476
  421. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-05-connections.md +1167 -1167
  422. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-kv-storage.md +990 -990
  423. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-state-management.md +121 -121
  424. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-api-reference.md +68 -68
  425. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-deployment.md +731 -731
  426. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-08-best-practices.md +1111 -1111
  427. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-09-signature-reference.md +766 -766
  428. package/docs/04-REFERENCE/platforms/versori/platforms-versori-readme.md +299 -299
  429. package/docs/04-REFERENCE/platforms/versori/platforms-versori-s3-sftp-configuration-guide.md +1425 -1425
  430. package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-api-key-security.md +816 -816
  431. package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-connection-security.md +681 -681
  432. package/docs/04-REFERENCE/platforms/versori/platforms-versori-workflow-task-types.md +708 -708
  433. package/docs/04-REFERENCE/platforms/versori/readme.md +108 -108
  434. package/docs/04-REFERENCE/readme.md +148 -148
  435. package/docs/04-REFERENCE/resolver-signature/examples/advanced-resolvers.ts +482 -482
  436. package/docs/04-REFERENCE/resolver-signature/examples/async-resolvers.ts +496 -496
  437. package/docs/04-REFERENCE/resolver-signature/examples/basic-resolvers.ts +343 -343
  438. package/docs/04-REFERENCE/resolver-signature/examples/resolver-signature-readme.md +188 -188
  439. package/docs/04-REFERENCE/resolver-signature/examples/testing-resolvers.ts +463 -463
  440. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-01-foundations.md +286 -286
  441. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-02-parameter-reference.md +643 -643
  442. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-03-basic-examples.md +521 -521
  443. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-04-advanced-patterns.md +739 -739
  444. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-05-sdk-resolvers.md +531 -531
  445. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-migration-guide.md +650 -650
  446. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-testing.md +125 -125
  447. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-07-api-reference.md +794 -794
  448. package/docs/04-REFERENCE/resolver-signature/readme.md +64 -64
  449. package/docs/04-REFERENCE/resolver-signature/resolver-signature-quick-reference.md +270 -270
  450. package/docs/04-REFERENCE/resolver-signature/resolver-signature-readme.md +351 -351
  451. package/docs/04-REFERENCE/schema/fluent-commerce-schema.json +764 -764
  452. package/docs/04-REFERENCE/schema/readme.md +141 -141
  453. package/docs/04-REFERENCE/testing/examples/04-reference-testing-readme.md +158 -158
  454. package/docs/04-REFERENCE/testing/examples/fluent-testing.ts +62 -62
  455. package/docs/04-REFERENCE/testing/examples/health-check.ts +155 -155
  456. package/docs/04-REFERENCE/testing/examples/integration-test.ts +119 -119
  457. package/docs/04-REFERENCE/testing/examples/performance-test.ts +183 -183
  458. package/docs/04-REFERENCE/testing/examples/s3-testing.ts +127 -127
  459. package/docs/04-REFERENCE/testing/modules/04-reference-testing-01-foundations.md +267 -267
  460. package/docs/04-REFERENCE/testing/modules/04-reference-testing-02-s3-testing.md +599 -599
  461. package/docs/04-REFERENCE/testing/modules/04-reference-testing-03-fluent-testing.md +589 -589
  462. package/docs/04-REFERENCE/testing/modules/04-reference-testing-04-integration-testing.md +699 -699
  463. package/docs/04-REFERENCE/testing/modules/04-reference-testing-05-debugging.md +478 -478
  464. package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-cicd-integration.md +463 -463
  465. package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-preflight-validation.md +131 -131
  466. package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-best-practices.md +499 -499
  467. package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-coverage-ci.md +165 -165
  468. package/docs/04-REFERENCE/testing/modules/04-reference-testing-08-api-reference.md +634 -634
  469. package/docs/04-REFERENCE/testing/readme.md +86 -86
  470. package/docs/04-REFERENCE/testing/testing-quick-reference.md +667 -667
  471. package/docs/04-REFERENCE/testing/testing-readme.md +286 -286
  472. package/docs/04-REFERENCE/troubleshooting/readme.md +144 -144
  473. package/docs/04-REFERENCE/troubleshooting/troubleshooting-deno-sftp-compatibility.md +392 -392
  474. package/docs/template-loading-matrix.md +242 -242
  475. package/package.json +5 -3
  476. package/docs/02-CORE-GUIDES/api-reference/cli-profile-integration.md +0 -377
@@ -1,2373 +1,2373 @@
1
- ---
2
- template_id: tpl-ingest-s3-xml-to-location-graphql
3
- canonical_filename: template-ingestion-s3-xml-location-graphql.md
4
- version: 2.0.0
5
- sdk_version: ^0.1.39
6
- runtime: versori
7
- direction: ingestion
8
- source: s3-xml
9
- destination: fluent-graphql
10
- entity: location
11
- format: xml
12
- logging: versori
13
- status: stable
14
- compliance: gold-standard
15
- features:
16
- - graphql-mutation-mapper
17
- - memory-management
18
- - enhanced-logging
19
- - attribute-transformation
20
- ---
21
-
22
- Template: Ingestion - S3 XML to Location GraphQL
23
-
24
- **Template Version:** 2.0.0
25
- **SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
26
- **Last Updated:** 2025-01-24
27
-
28
- **🆕 Version 2.0.0 Enhancements:**
29
- - ✅ **GraphQL Mutation Mapper** - Direct field mapping to mutation variables
30
- - ✅ **Memory Management** - Clear large arrays after processing batches
31
- - ✅ **Enhanced Logging** - Track mutation execution with emoji indicators
32
- - ✅ **Attribute Transformation** - Handle complex nested data structures
33
-
34
- ---
35
-
36
- ## Implementation Prompt
37
-
38
- "Create a Versori scheduled workflow that reads location XML files from S3, transforms the data using GraphQLMutationMapper with nested object mapping, and creates/updates Fluent Commerce locations via direct GraphQL mutations with alias batching support.
39
-
40
- **Requirements:**
41
-
42
- 1. **S3 Source**:
43
- - List and download XML files with configurable prefix and pattern
44
- - Archive processed files to `processed/` directory (deduplication via archiving)
45
- - Move failed files to `errors/` directory
46
- - Use S3DataSource with retry logic and built-in archival
47
- - Support configurable bucket, region, credentials
48
-
49
- 2. **XML Parsing**:
50
- - Use XMLParserService with @ prefix for attribute access
51
- - Handle both single and multiple `<location>` elements (array normalization)
52
- - Support nested XML paths (`location.address.street1`, `location.coordinates.@lat`)
53
- - Parse complex structures (addresses, coordinates, opening schedules)
54
-
55
- 3. **Field Mapping**:
56
- - Map XML fields to Location GraphQL input type with nested objects:
57
- - `ref`, `name`, `type` (root fields)
58
- - `primaryAddress.*` (nested address object with coordinates)
59
- - `openingSchedule.*` (nested schedule object with 7-day hours)
60
- - Use SDK resolvers (trim, uppercase, parseFloat, parseInt, boolean)
61
- - Support custom resolvers for complex transformations
62
- - Validate required fields
63
- - Note: Check your GraphQL schema to determine if `retailer.id` field exists and is mandatory/optional
64
-
65
- 4. **GraphQL Mutations** (Direct - NO Batch API):
66
- - Execute `createLocation` mutation directly for each location
67
- - **NO BPP (Batch Pre-Processing)** - Not applicable for direct GraphQL mutations
68
- - **NO Batch API** - Use direct GraphQL mutations with rate limiting instead
69
- - Use rate limiting (configurable mutations per second)
70
- - Add delay between mutations to avoid API throttling
71
- - Retry failed mutations with exponential backoff
72
- - Track successful vs failed mutations per file
73
-
74
- 5. **Job Tracking & State Management**:
75
- - **Use JobTracker** - Track job lifecycle (start, complete, fail) in KV store
76
- - Use StateService + VersoriKVAdapter for duplicate file prevention
77
- - Track processed files in KV store with metadata
78
- - Store error state with exponential backoff tracking
79
- - Support distributed state across workflow runs
80
- - Provide job status endpoint for monitoring
81
-
82
- 6. **Error Handling**:
83
- - File-level errors archived to `/errors/` subdirectory
84
- - Record-level errors tracked but don't stop file processing
85
- - Mapping errors logged with specific location context
86
- - Mutation errors retried with exponential backoff
87
- - Error state tracking with next retry timestamp
88
-
89
- 7. **Advanced Features**:
90
- - Configurable rate limiting (mutations per second)
91
- - Empty file detection and archival
92
- - Timestamp-based error tracking
93
- - Comprehensive monitoring and logging
94
- - Manual webhook trigger with job tracking
95
- - Job status query endpoint
96
-
97
- **Use SDK Components:**
98
-
99
- - `createClient()` - Universal client factory for Versori
100
- - `S3DataSource` - S3 operations with retry logic
101
- - `XMLParserService` - XML parsing with @ prefix attribute support
102
- - `GraphQLMutationMapper` - Field transformation with schema validation and nested object support
103
- - `StateService` + `VersoriKVAdapter` - Duplicate prevention with KV storage
104
- - Native Versori `log` - Structured logging
105
-
106
- **Configuration Variables** (from Versori activation):
107
-
108
- ```typescript
109
- {
110
- s3: {
111
- bucketName: string;
112
- region: string;
113
- accessKeyId: string;
114
- secretAccessKey: string;
115
- prefix: string; // e.g., 'locations/'
116
- archivePrefix: string; // e.g., 'processed/'
117
- errorPrefix: string; // e.g., 'errors/'
118
- filePattern: string; // e.g., '.xml'
119
- maxFilesToProcess: number;
120
- enableArchival: boolean;
121
- },
122
- fluent: {
123
- retailerId: string; // Optional: Only if mutation schema requires retailerId in input
124
- mutationRateLimit: number; // mutations per second (e.g., 5)
125
- }
126
- }
127
- ```
128
-
129
- **Architecture Pattern:**
130
-
131
- ```
132
- S3 Bucket → List Files → Download XML → Parse → Map → GraphQL Mutation → Archive/Move
133
- ↓ ↓ ↓ ↓ ↓ ↓ ↓
134
- Configure Filter by S3DataSource XML Universal Direct S3 moveFile()
135
- Connection Pattern + Retry Parser Mapper createLocation to processed/
136
- (@) (nested) + Rate Limit or errors/
137
- (deduplication)
138
- ```
139
-
140
- **Deliverables:**
141
-
142
- 1. Complete Versori workflow with package.json
143
- 2. Main workflow logic with S3 + XML + GraphQL patterns
144
- 3. Helper functions for rate limiting and retry
145
- 4. XML path resolution examples
146
- 5. Sample XML files with nested structures
147
- 6. Schema validation CLI commands
148
- 7. Testing and deployment instructions
149
- 8. Monitoring and troubleshooting guidance"
150
-
151
- ---
152
-
153
- # STEP 3: Complete Implementation
154
-
155
- ## Versori Scheduled: S3 XML → Location GraphQL
156
-
157
- **FC Connect SDK Use Case Guide**
158
-
159
- > **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
160
- > **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
161
-
162
- **Context**: Versori scheduled workflow that reads location XML files from S3 and creates/updates Fluent Commerce locations via GraphQL mutations with XML path resolution and rate limiting
163
-
164
- **Complexity**: Medium
165
-
166
- **Runtime**: Versori Platform
167
-
168
- **Estimated Lines**: ~850 lines (with comprehensive documentation)
169
-
170
- ---
171
-
172
- ## What You'll Build
173
-
174
- - Scheduled Versori workflow (daily location sync)
175
- - S3 file listing, download, and archival/move
176
- - XML parsing with @ prefix for attributes (XPath-style)
177
- - Array normalization (single element → array conversion)
178
- - GraphQLMutationMapper-based field transformations with nested objects
179
- - GraphQL mutations for location upserts with alias batching support
180
- - Retry logic with exponential backoff
181
- - StateService duplicate prevention (KV-backed)
182
- - Error state tracking and file error archival
183
- - Manual webhook trigger and job status endpoint
184
-
185
- ---
186
-
187
- ## When to Use GraphQL Mutations vs Batch API vs Event API
188
-
189
- ### ✅ Use GraphQL Mutations For:
190
-
191
- | Entity Type | Use Case | Why GraphQL |
192
- | -------------- | ---------------------------------------- | ------------------------------------- |
193
- | **Locations** | Store/warehouse master data (low volume) | Direct control, immediate validation |
194
- | **Controls** | System configuration, settings | Single operations, complex queries |
195
- | **Prices** | Price updates (moderate volume) | Immediate feedback, custom logic |
196
- | **Single Ops** | One-off creates/updates | Testing, debugging, direct API access |
197
-
198
- ### ❌ Use Event API Instead For:
199
-
200
- | Entity Type | Use Case | Why Event API |
201
- | ------------------- | -------------------------------------- | -------------------------------------------- |
202
- | **Products** | Product catalog sync, variant updates | Triggers workflows, validates business rules |
203
- | **Customers** | Customer registration, profile updates | Needs workflow for downstream systems |
204
- | **Orders** | Order creation, status updates | Event-driven fulfillment workflows |
205
- | **Custom Entities** | Any entity needing workflow triggers | Full Rubix workflow support |
206
-
207
- ### 🔄 Use Batch API For:
208
-
209
- | Entity Type | Use Case | Why Batch API |
210
- | ------------------ | ---------------------------------- | ----------------------------------------------- |
211
- | **Inventory ONLY** | Bulk inventory updates, daily sync | Optimized for high-volume, BPP change detection |
212
-
213
- ---
214
-
215
- ## XML File Format
216
-
217
- ### Sample: locations.xml
218
-
219
- ```xml
220
- <?xml version="1.0" encoding="UTF-8"?>
221
- <locations>
222
- <location ref="LOC-001" type="WAREHOUSE">
223
- <name>Downtown Warehouse</name>
224
- <address country="USA">
225
- <street1>123 Main St</street1>
226
- <street2>Building A</street2>
227
- <city>New York</city>
228
- <state>NY</state>
229
- <postalCode>10001</postalCode>
230
- </address>
231
- <coordinates lat="40.7128" lon="-74.0060"/>
232
- <timeZone>America/New_York</timeZone>
233
- <openingSchedule>
234
- <allHours>false</allHours>
235
- <monStart>800</monStart>
236
- <monEnd>1800</monEnd>
237
- <tueStart>800</tueStart>
238
- <tueEnd>1800</tueEnd>
239
- <wedStart>800</wedStart>
240
- <wedEnd>1800</wedEnd>
241
- <thuStart>800</thuStart>
242
- <thuEnd>1800</thuEnd>
243
- <friStart>800</friStart>
244
- <friEnd>1800</friEnd>
245
- <satStart>0</satStart>
246
- <satEnd>0</satEnd>
247
- <sunStart>0</sunStart>
248
- <sunEnd>0</sunEnd>
249
- </openingSchedule>
250
- </location>
251
-
252
- <location ref="LOC-002" type="DC">
253
- <name>Regional DC</name>
254
- <address country="USA">
255
- <street1>456 Industrial Pkwy</street1>
256
- <city>Los Angeles</city>
257
- <state>CA</state>
258
- <postalCode>90001</postalCode>
259
- </address>
260
- <coordinates lat="34.0522" lon="-118.2437"/>
261
- <timeZone>America/Los_Angeles</timeZone>
262
- <openingSchedule>
263
- <allHours>true</allHours>
264
- <monStart>0</monStart>
265
- <monEnd>0</monEnd>
266
- <tueStart>0</tueStart>
267
- <tueEnd>0</tueEnd>
268
- <wedStart>0</wedStart>
269
- <wedEnd>0</wedEnd>
270
- <thuStart>0</thuStart>
271
- <thuEnd>0</thuEnd>
272
- <friStart>0</friStart>
273
- <friEnd>0</friEnd>
274
- <satStart>0</satStart>
275
- <satEnd>0</satEnd>
276
- <sunStart>0</sunStart>
277
- <sunEnd>0</sunEnd>
278
- </openingSchedule>
279
- </location>
280
- </locations>
281
- ```
282
-
283
- **XML Path Syntax (with @ prefix for attributes):**
284
-
285
- - `location.@ref` → XML attribute `ref` on `<location>` element
286
- - `location.name` → Text content of `<name>` element
287
- - `location.address.street1` → Nested element path
288
- - `location.address.@country` → XML attribute on nested `<address>` element
289
- - `location.coordinates.@lat` → XML attribute for latitude
290
- - `location.openingSchedule.monStart` → Deeply nested element
291
-
292
- **Note**: The SDK's `XMLParserService` automatically handles XML attributes using `@` prefix notation.
293
-
294
- ---
295
-
296
- ## Versori Workflows Structure
297
-
298
- **Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
299
-
300
- **Trigger Types:**
301
- - **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
302
- - **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
303
- - **`workflow()`** → Durable workflows (advanced, rarely used)
304
-
305
- **Execution Steps (chained to triggers):**
306
- - **`http()`** → External API calls (chained from schedule/webhook)
307
- - **`fn()`** → Internal processing (chained from schedule/webhook)
308
-
309
- ### Recommended Project Structure
310
-
311
- ```
312
- s3-xml-location-graphql/
313
- ├── index.ts # Entry point - exports all workflows
314
- └── src/
315
- ├── workflows/
316
- │ ├── scheduled/
317
- │ │ └── daily-location-sync.ts # Scheduled: Daily location sync
318
- │ │
319
- │ └── webhook/
320
- │ ├── adhoc-location-sync.ts # Webhook: Manual trigger
321
- │ └── job-status-check.ts # Webhook: Status query
322
-
323
- ├── services/
324
- │ └── location-sync.service.ts # Shared orchestration logic (reusable)
325
-
326
- └── config/
327
- └── location-mapping.json # GraphQL mapping config
328
- ```
329
-
330
- ---
331
-
332
- ## Complete Versori Workflow
333
-
334
- ### Step 1: Package Configuration
335
-
336
- **File: package.json**
337
-
338
- ```json
339
- {
340
- "name": "versori-s3-xml-location-sync",
341
- "version": "1.0.0",
342
- "description": "Versori workflow: S3 XML location sync to Fluent GraphQL",
343
- "versori": {
344
- "workflows": "./index.ts"
345
- },
346
- "type": "module",
347
- "scripts": {
348
- "deploy": "versori deploy",
349
- "logs": "versori logs"
350
- },
351
- "dependencies": {
352
- "@fluentcommerce/fc-connect-sdk": "^0.1.39",
353
- "@versori/run": "latest"
354
- },
355
- "devDependencies": {
356
- "typescript": "^5.0.0",
357
- "@types/node": "^20.0.0"
358
- }
359
- }
360
- ```
361
-
362
- ### Step 2: Workflow Entry Point (`index.ts`)
363
-
364
- **Purpose**: Register all workflows with Versori platform
365
-
366
- ```typescript
367
- /**
368
- * Entry Point - Registers all workflows with Versori platform
369
- *
370
- * Versori automatically discovers and registers exported workflows
371
- *
372
- * File Structure:
373
- * - src/workflows/scheduled/ → Time-based triggers (cron)
374
- * - src/workflows/webhook/ → HTTP-based triggers (webhooks)
375
- */
376
-
377
- // Scheduled workflows
378
- export { dailyLocationSync } from './src/workflows/scheduled/daily-location-sync';
379
-
380
- // Webhook workflows
381
- export { adhocLocationSync } from './src/workflows/webhook/adhoc-location-sync';
382
- export { locationSyncJobStatus } from './src/workflows/webhook/job-status-check';
383
- ```
384
-
385
- **What Gets Exposed:**
386
- - ✅ `adhocLocationSync` → `https://{workspace}.versori.run/location-sync-adhoc`
387
- - ✅ `locationSyncJobStatus` → `https://{workspace}.versori.run/location-sync-job-status`
388
- - ❌ `dailyLocationSync` → NOT exposed (runs automatically on cron)
389
-
390
- ---
391
-
392
- ### Step 3: Workflow Files
393
-
394
- #### `src/workflows/scheduled/daily-location-sync.ts`
395
-
396
- **Purpose**: Automatic daily location sync
397
- **Trigger**: Cron schedule (`0 2 * * *`)
398
- **Exposed as Endpoint**: ❌ NO - Runs automatically
399
-
400
- ```typescript
401
- import { schedule, http } from '@versori/run';
402
- import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
403
- import { executeLocationSync } from '../../services/location-sync.service';
404
-
405
- /**
406
- * Scheduled Workflow: Daily Location Sync
407
- *
408
- * Runs automatically daily at 2 AM UTC
409
- * NOT exposed as HTTP endpoint - Versori executes on schedule
410
- *
411
- * Uses shared service: location-sync.service.ts
412
- */
413
- export const dailyLocationSync = schedule(
414
- 'location-sync-scheduled',
415
- '0 2 * * *' // Daily at 2 AM UTC
416
- ).then(
417
- http('run-location-sync', { connection: 'fluent_commerce' }, async ctx => {
418
- const startTime = Date.now();
419
- const { log, openKv } = ctx;
420
- const jobId = `location-sync-${Date.now()}`;
421
- const tracker = new JobTracker(openKv(':project:'), log);
422
-
423
- log.info('🚀 [START] Daily location sync initiated', { jobId, trigger: 'schedule' });
424
-
425
- await tracker.createJob(jobId, { triggeredBy: 'schedule', stage: 'initialization' });
426
- await tracker.updateJob(jobId, { status: 'processing' });
427
-
428
- try {
429
- log.info('⚙️ [PROCESSING] Starting location synchronization workflow', { jobId });
430
- const result = await executeLocationSync(ctx, jobId, tracker);
431
- await tracker.markCompleted(jobId, result);
432
-
433
- const duration = Date.now() - startTime;
434
- log.info('✅ [SUCCESS] Daily location sync completed', {
435
- jobId,
436
- duration: `${duration}ms`,
437
- processed: result.processed,
438
- totalRecords: result.totalRecords
439
- });
440
-
441
- return { success: true, jobId, duration, ...result };
442
- } catch (e: any) {
443
- await tracker.markFailed(jobId, e);
444
- const duration = Date.now() - startTime;
445
-
446
- log.error('❌ [FAILED] Daily location sync failed', {
447
- jobId,
448
- duration: `${duration}ms`,
449
- error: e?.message,
450
- errorType: e?.name
451
- });
452
-
453
- return { success: false, jobId, duration, error: e?.message };
454
- }
455
- })
456
- );
457
- ```
458
-
459
- ---
460
-
461
- #### `src/workflows/webhook/adhoc-location-sync.ts`
462
-
463
- **Purpose**: Manual location sync trigger (on-demand)
464
- **Trigger**: HTTP POST
465
- **Endpoint**: `POST https://{workspace}.versori.run/location-sync-adhoc`
466
- **Use Cases**: Testing, priority processing, ad-hoc runs
467
-
468
- ```typescript
469
- import { webhook, http } from '@versori/run';
470
- import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
471
- import { executeLocationSync } from '../../services/location-sync.service';
472
-
473
- /**
474
- * Webhook: Manual Location Sync Trigger
475
- *
476
- * Endpoint: POST https://{workspace}.versori.run/location-sync-adhoc
477
- * Request body (optional): { filePattern: "urgent_*.xml", maxFiles: 5 }
478
- *
479
- * Pattern: webhook().then(http()) - needs Fluent API access
480
- * Uses shared service: location-sync.service.ts
481
- *
482
- * SECURITY: Authentication handled via connection parameter
483
- * No manual API key validation needed - Versori manages this via connection auth
484
- */
485
- export const adhocLocationSync = webhook('location-sync-adhoc', {
486
- response: { mode: 'sync' },
487
- connection: 'location-sync-adhoc', // Versori validates API key
488
- }).then(
489
- http('run-location-sync-adhoc', { connection: 'fluent_commerce' }, async ctx => {
490
- const startTime = Date.now();
491
- const { log, openKv, data } = ctx;
492
- const jobId = `location-sync-adhoc-${Date.now()}`;
493
- const tracker = new JobTracker(openKv(':project:'), log);
494
-
495
- log.info('🚀 [START] Manual location sync triggered', { jobId, trigger: 'webhook', options: data });
496
-
497
- await tracker.createJob(jobId, {
498
- triggeredBy: 'manual',
499
- stage: 'initialization',
500
- options: data // Optional: filePattern, maxFiles, etc.
501
- });
502
- await tracker.updateJob(jobId, { status: 'processing' });
503
-
504
- try {
505
- log.info('⚙️ [PROCESSING] Starting manual location synchronization', { jobId });
506
- const result = await executeLocationSync(ctx, jobId, tracker);
507
- await tracker.markCompleted(jobId, result);
508
-
509
- const duration = Date.now() - startTime;
510
- log.info('✅ [SUCCESS] Manual location sync completed', {
511
- jobId,
512
- duration: `${duration}ms`,
513
- processed: result.processed,
514
- totalRecords: result.totalRecords
515
- });
516
-
517
- return { success: true, jobId, duration, ...result };
518
- } catch (e: any) {
519
- await tracker.markFailed(jobId, e);
520
- const duration = Date.now() - startTime;
521
-
522
- log.error('❌ [FAILED] Manual location sync failed', {
523
- jobId,
524
- duration: `${duration}ms`,
525
- error: e?.message,
526
- errorType: e?.name
527
- });
528
-
529
- return { success: false, jobId, duration, error: e?.message };
530
- }
531
- })
532
- );
533
- ```
534
-
535
- ---
536
-
537
- #### `src/workflows/webhook/job-status-check.ts`
538
-
539
- **Purpose**: Query job status
540
- **Trigger**: HTTP POST
541
- **Endpoint**: `POST https://{workspace}.versori.run/location-sync-job-status`
542
- **Request body**: `{ "jobId": "location-sync-1234567890" }`
543
-
544
- ```typescript
545
- import { webhook, fn } from '@versori/run';
546
- import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
547
-
548
- /**
549
- * Webhook: Job Status Check
550
- *
551
- * Endpoint: POST https://{workspace}.versori.run/location-sync-job-status
552
- * Request body: { "jobId": "location-sync-1234567890" }
553
- *
554
- * Pattern: webhook().then(fn()) - no external API needed, only KV storage
555
- * Lightweight: Only queries KV store, no Fluent API calls
556
- *
557
- * SECURITY: Authentication handled via connection parameter
558
- * No manual API key validation needed - Versori manages this via connection auth
559
- */
560
- export const locationSyncJobStatus = webhook('location-sync-job-status', {
561
- response: { mode: 'sync' },
562
- connection: 'location-sync-job-status',
563
- }).then(
564
- fn('status', async ctx => {
565
- const { data, log, openKv } = ctx;
566
- const jobId = data?.jobId as string;
567
-
568
- if (!jobId) {
569
- return { success: false, error: 'jobId required' };
570
- }
571
-
572
- const tracker = new JobTracker(openKv(':project:'), log);
573
- const status = await tracker.getJob(jobId);
574
-
575
- return status
576
- ? { success: true, jobId, ...status }
577
- : { success: false, error: 'Job not found', jobId };
578
- })
579
- );
580
- ```
581
-
582
- ---
583
-
584
- ### Step 4: Main Orchestration Service (`src/services/location-sync.service.ts`)
585
-
586
- **Note:** This service file should contain the `executeLocationSync` function (renamed from `runLocationXmlWorkflow`). The main workflow logic should be moved here.
587
-
588
- ```typescript
589
- /**
590
- * Main Orchestration Service: Location Sync
591
- *
592
- * This service contains the core business logic for location synchronization.
593
- *
594
- * Features:
595
- * - S3 file operations with archival-based deduplication (moveFile to processed/)
596
- * - XML parsing with @ prefix for attributes
597
- * - Single element → array normalization
598
- * - GraphQLMutationMapper for nested field transformations
599
- * - GraphQL mutations with alias batching support
600
- * - StateService for metadata tracking (secondary to archival)
601
- * - Error state tracking with exponential backoff
602
- *
603
- * Deduplication Strategy:
604
- * - PRIMARY: S3 archival via moveFile() - Files in processed/ won't be re-listed
605
- * - SECONDARY: StateService KV tracking - Provides metadata and processing history
606
- */
607
- import { Buffer } from 'node:buffer'; // Required for Deno/Versori runtime
608
- import {
609
- createClient,
610
- S3DataSource,
611
- XMLParserService,
612
- GraphQLMutationMapper,
613
- StateService,
614
- VersoriKVAdapter,
615
- JobTracker,
616
- FluentClient,
617
- } from '@fluentcommerce/fc-connect-sdk';
618
-
619
- // ============================================================================
620
- // Type Definitions
621
- // ============================================================================
622
-
623
- interface FileProcessingResult {
624
- success: boolean;
625
- locations: any[];
626
- errors: string[];
627
- }
628
-
629
- interface MutationResult {
630
- successful: number;
631
- failed: number;
632
- errors: string[];
633
- }
634
-
635
- interface MutationLogEntry {
636
- timestamp: string;
637
- fileName: string;
638
- locationRef: string;
639
- status: 'success' | 'failure';
640
- error?: string;
641
- }
642
-
643
- // ============================================================================
644
- // Utility Functions
645
- // ============================================================================
646
-
647
- /**
648
- * Retry utility with exponential backoff
649
- * (User-defined - not part of SDK public API)
650
- */
651
- async function retryWithBackoff<T>(
652
- operation: () => Promise<T>,
653
- maxRetries = 3,
654
- baseDelayMs = 1000
655
- ): Promise<T> {
656
- let lastError: any;
657
- for (let attempt = 0; attempt < maxRetries; attempt++) {
658
- try {
659
- return await operation();
660
- } catch (error) {
661
- lastError = error;
662
- if (attempt < maxRetries - 1) {
663
- const delay = baseDelayMs * Math.pow(2, attempt) + Math.floor(Math.random() * 200);
664
- await new Promise(resolve => setTimeout(resolve, delay));
665
- }
666
- }
667
- }
668
- throw lastError;
669
- }
670
-
671
- /**
672
- * Rate limiter for GraphQL mutations
673
- */
674
- async function rateLimitedMutation(operation: () => Promise<any>, delayMs: number): Promise<any> {
675
- const result = await operation();
676
- await new Promise(resolve => setTimeout(resolve, delayMs));
677
- return result;
678
- }
679
-
680
- // ============================================================================
681
- // Service Functions
682
- // ============================================================================
683
-
684
- /**
685
- * Service Function 1: Process File
686
- *
687
- * Downloads XML from S3, parses with XMLParserService, normalizes arrays,
688
- * and maps fields with GraphQLMutationMapper.
689
- *
690
- * @param s3 - S3DataSource instance
691
- * @param parser - XMLParserService instance
692
- * @param mapper - GraphQLMutationMapper instance
693
- * @param filePath - Full S3 path to file
694
- * @param fileName - File name only (for logging)
695
- * @param log - Logger instance
696
- * @returns FileProcessingResult with locations array and errors
697
- */
698
- async function processFile(
699
- s3: S3DataSource,
700
- parser: XMLParserService,
701
- mapper: GraphQLMutationMapper,
702
- filePath: string,
703
- fileName: string,
704
- log: any
705
- ): Promise<FileProcessingResult> {
706
- try {
707
- log.info('Processing file', { fileName });
708
-
709
- // Download with retry
710
- const content = await retryWithBackoff(
711
- () => s3.downloadFile(filePath, { encoding: 'utf8' }) as Promise<string>
712
- );
713
-
714
- // Parse XML
715
- const xmlData = await parser.parse(content);
716
-
717
- // Extract location array (handle both single and multiple locations)
718
- // CRITICAL: XML array normalization - single <location> becomes object, not array
719
- const locationsData = xmlData.locations?.location;
720
- const locationsArray = Array.isArray(locationsData) ? locationsData : [locationsData];
721
-
722
- if (!locationsArray || locationsArray.length === 0) {
723
- log.warn('Empty file (no locations)', { fileName });
724
- return {
725
- success: true,
726
- locations: [],
727
- errors: [],
728
- };
729
- }
730
-
731
- // Map each location using GraphQLMutationMapper
732
- const mappedLocations: Array<{ query: string; variables: any; input: any }> = [];
733
- const mappingErrors: string[] = [];
734
-
735
- // ✅ PRODUCTION ENHANCEMENT: Log transformation start
736
- log.info('Transforming locations to GraphQL mutations', {
737
- fileName,
738
- totalLocations: locationsArray.length,
739
- });
740
-
741
- for (let i = 0; i < locationsArray.length; i++) {
742
- const locationNumber = i + 1;
743
-
744
- // ✅ PRODUCTION ENHANCEMENT: Log progress every 50 locations
745
- if (locationNumber % 50 === 0) {
746
- log.info(`📤 Transforming location ${locationNumber}/${locationsArray.length}`, {
747
- fileName,
748
- locationNumber,
749
- totalLocations: locationsArray.length,
750
- validSoFar: mappedLocations.length,
751
- errorsSoFar: mappingErrors.length,
752
- progress: `${((locationNumber / locationsArray.length) * 100).toFixed(1)}%`,
753
- });
754
- }
755
-
756
- // Wrap location in context for mapping
757
- const record = { location: locationsArray[i] };
758
- try {
759
- // GraphQLMutationMapper returns { query, variables } directly
760
- const mappingResult = await mapper.map(record);
761
-
762
- mappedLocations.push({
763
- query: mappingResult.query,
764
- variables: mappingResult.variables,
765
- input: mappingResult.variables.input || mappingResult.variables,
766
- });
767
- } catch (error: unknown) {
768
- const errorMsg = error instanceof Error ? error.message : String(error);
769
- mappingErrors.push(`Location ${locationNumber}: ${errorMsg}`);
770
- log.warn('Mapping failed for location', { fileName, index: i, error: errorMsg });
771
- }
772
- }
773
-
774
- log.info('File processed', {
775
- fileName,
776
- total: locationsArray.length,
777
- mapped: mappedLocations.length,
778
- errors: mappingErrors.length,
779
- successRate: `${((mappedLocations.length / locationsArray.length) * 100).toFixed(1)}%`,
780
- });
781
-
782
- return {
783
- success: true,
784
- locations: mappedLocations,
785
- errors: mappingErrors,
786
- };
787
- } catch (error: any) {
788
- // ✅ Enhanced error logging: Extract all error details for visibility
789
- const errorDetails = {
790
- message: error?.message || 'Unknown error',
791
- stack: error?.stack,
792
- fileName: error?.fileName,
793
- lineNumber: error?.lineNumber,
794
- originalError: error?.context?.originalError?.message,
795
- errorType: error?.name || 'Error',
796
- };
797
- log.error('File processing failed', errorDetails, { fileName });
798
- return {
799
- success: false,
800
- locations: [],
801
- errors: [error.message || 'Unknown error'],
802
- };
803
- }
804
- }
805
-
806
- /**
807
- * Service Function 2: Execute Mutations
808
- *
809
- * Executes GraphQL createLocation mutations with alias batching support.
810
- *
811
- * @param client - FluentClient instance
812
- * @param mapper - GraphQLMutationMapper instance
813
- * @param locations - Array of mapped location objects with query and variables
814
- * @param log - Logger instance
815
- * @param retailerId - Fluent retailer ID
816
- * @param batchSize - Number of concurrent requests (default: 1)
817
- * @param mutationsPerAliasBatch - Optional: Number of mutations per aliased request (default: undefined)
818
- * @returns MutationResult with success/failure counts
819
- */
820
- async function executeMutations(
821
- client: FluentClient,
822
- mapper: GraphQLMutationMapper,
823
- locations: Array<{ query: string; variables: any; input: any }>,
824
- log: any,
825
- retailerId: string,
826
- batchSize: number = 1, // ✅ Default: 1 (sequential)
827
- mutationsPerAliasBatch?: number // ✅ NEW: Alias batching parameter (default: undefined = disabled)
828
- ): Promise<MutationResult> {
829
- // Determine mode: use aliases if mutationsPerAliasBatch is set and > 1
830
- const useAliases = mutationsPerAliasBatch && mutationsPerAliasBatch > 1;
831
-
832
- if (useAliases) {
833
- return await executeMutationsWithAliases(
834
- client,
835
- mapper,
836
- locations,
837
- log,
838
- retailerId,
839
- batchSize,
840
- mutationsPerAliasBatch!
841
- );
842
- } else {
843
- return await executeMutationsSeparate(
844
- client,
845
- locations,
846
- log,
847
- batchSize
848
- );
849
- }
850
- }
851
-
852
- /**
853
- * Execute mutations using separate concurrent requests (current mode)
854
- */
855
- async function executeMutationsSeparate(
856
- client: FluentClient,
857
- locations: Array<{ query: string; variables: any; input: any }>,
858
- log: any,
859
- batchSize: number
860
- ): Promise<MutationResult> {
861
- const result: MutationResult = {
862
- successful: 0,
863
- failed: 0,
864
- errors: [],
865
- };
866
-
867
- const safeConc = Math.max(1, Math.floor(batchSize));
868
-
869
- // Sequential mode
870
- if (safeConc === 1) {
871
- for (const location of locations) {
872
- try {
873
- await retryWithBackoff(() =>
874
- client.graphql({
875
- query: location.query,
876
- variables: location.variables,
877
- })
878
- );
879
- result.successful++;
880
- } catch (error: unknown) {
881
- result.failed++;
882
- const errorMsg = error instanceof Error ? error.message : String(error);
883
- result.errors.push(errorMsg);
884
- }
885
- }
886
- return result;
887
- }
888
-
889
- // Parallel mode
890
- for (let i = 0; i < locations.length; i += safeConc) {
891
- const chunk = locations.slice(i, i + safeConc);
892
- const results = await Promise.allSettled(
893
- chunk.map(loc =>
894
- retryWithBackoff(() =>
895
- client.graphql({
896
- query: loc.query,
897
- variables: loc.variables,
898
- })
899
- )
900
- )
901
- );
902
-
903
- results.forEach((settledResult, idx) => {
904
- if (settledResult.status === 'fulfilled') {
905
- result.successful++;
906
- } else {
907
- result.failed++;
908
- result.errors.push(
909
- settledResult.reason instanceof Error ? settledResult.reason.message : String(settledResult.reason)
910
- );
911
- }
912
- });
913
- }
914
-
915
- return result;
916
- }
917
-
918
- /**
919
- * ✅ NEW: Execute mutations using GraphQL aliases (batched requests)
920
- */
921
- async function executeMutationsWithAliases(
922
- client: FluentClient,
923
- mapper: GraphQLMutationMapper,
924
- locations: Array<{ query: string; variables: any; input: any }>,
925
- log: any,
926
- retailerId: string,
927
- maxParallel: number,
928
- mutationsPerAliasBatch: number
929
- ): Promise<MutationResult> {
930
- const result: MutationResult = { successful: 0, failed: 0, errors: [] };
931
-
932
- const mutationName = (mapper as any).config.mutation || 'createLocation';
933
- const aliasBatches: Array<Array<typeof locations[0]>> = [];
934
-
935
- for (let i = 0; i < locations.length; i += mutationsPerAliasBatch) {
936
- aliasBatches.push(locations.slice(i, i + mutationsPerAliasBatch));
937
- }
938
-
939
- // Process batches with concurrency control
940
- for (let i = 0; i < aliasBatches.length; i += maxParallel) {
941
- const concurrentBatches = aliasBatches.slice(i, i + maxParallel);
942
-
943
- const batchResults = await Promise.allSettled(
944
- concurrentBatches.map(async (batch) => {
945
- const { query, variables } = buildAliasedBatch(batch, mutationName, retailerId);
946
- const response = await retryWithBackoff(() => client.graphql({ query, variables }));
947
- return parseAliasResponse(response, batch, mutationName);
948
- })
949
- );
950
-
951
- batchResults.forEach((batchResult, idx) => {
952
- if (batchResult.status === 'fulfilled') {
953
- const batchRes = batchResult.value;
954
- result.successful += batchRes.executed;
955
- result.failed += batchRes.failed;
956
- result.errors.push(...batchRes.errors);
957
- } else {
958
- const batch = concurrentBatches[idx];
959
- const errorMsg = batchResult.reason instanceof Error ? batchResult.reason.message : String(batchResult.reason);
960
- batch.forEach(loc => {
961
- result.failed++;
962
- result.errors.push(`Batch execution failed: ${errorMsg}`);
963
- });
964
- }
965
- });
966
-
967
- if (i + maxParallel < aliasBatches.length) {
968
- await new Promise(resolve => setTimeout(resolve, 500));
969
- }
970
- }
971
-
972
- return result;
973
- }
974
-
975
- /**
976
- * ✅ NEW: Build aliased batch query and variables
977
- */
978
- function buildAliasedBatch(
979
- batch: Array<{ query: string; variables: any; input: any }>,
980
- mutationName: string,
981
- retailerId: string
982
- ): { query: string; variables: Record<string, any> } {
983
- const batchSize = batch.length;
984
- const inputTypeName = `${mutationName.charAt(0).toUpperCase() + mutationName.slice(1)}Input`;
985
-
986
- const variables = Array.from({ length: batchSize }, (_, i) =>
987
- `$input${i + 1}: ${inputTypeName}!`
988
- ).join(', ');
989
-
990
- const aliasedMutations = Array.from({ length: batchSize }, (_, i) => {
991
- const alias = `${mutationName}${i + 1}`;
992
- return ` ${alias}: ${mutationName}(input: $input${i + 1}) { id ref name }`;
993
- }).join('\n');
994
-
995
- const operationName = `Batch${mutationName.charAt(0).toUpperCase() + mutationName.slice(1)}s`;
996
- const query = `mutation ${operationName}(${variables}) {\n${aliasedMutations}\n}`;
997
-
998
- const variablesObj: Record<string, any> = {};
999
- batch.forEach((loc, index) => {
1000
- const input = loc.variables.input || loc.variables;
1001
- if (input && !input.retailer) {
1002
- input.retailer = { id: parseInt(retailerId) };
1003
- }
1004
- variablesObj[`input${index + 1}`] = input;
1005
- });
1006
-
1007
- return { query, variables: variablesObj };
1008
- }
1009
-
1010
- /**
1011
- * ✅ NEW: Parse aliased GraphQL response
1012
- */
1013
- function parseAliasResponse(
1014
- response: any,
1015
- batch: Array<{ query: string; variables: any; input: any }>,
1016
- mutationName: string
1017
- ): { executed: number; failed: number; errors: string[] } {
1018
- const result = { executed: 0, failed: 0, errors: [] as string[] };
1019
-
1020
- const data = response.data || {};
1021
- const errors = response.errors || [];
1022
-
1023
- batch.forEach((loc, index) => {
1024
- const alias = `${mutationName}${index + 1}`;
1025
- const aliasData = data[alias];
1026
- const aliasErrors = errors.filter((e: any) =>
1027
- e.path && Array.isArray(e.path) && e.path.includes(alias)
1028
- );
1029
-
1030
- if (aliasData && !aliasErrors.length) {
1031
- result.executed++;
1032
- } else {
1033
- result.failed++;
1034
- const errorMsg = aliasErrors[0]?.message || 'Mutation failed';
1035
- result.errors.push(`${loc.input?.ref || 'unknown'}: ${errorMsg}`);
1036
- }
1037
- });
1038
-
1039
- return result;
1040
- }
1041
-
1042
- /**
1043
- * Service Function 3: Write Mutation Log
1044
- *
1045
- * Writes mutation results to S3 as a JSON log file.
1046
- *
1047
- * @param s3 - S3DataSource instance
1048
- * @param logEntries - Array of mutation log entries
1049
- * @param fileName - Original file name (used to generate log path)
1050
- * @param logPrefix - S3 prefix for logs (e.g., 'logs/')
1051
- * @param log - Logger instance
1052
- */
1053
- async function writeMutationLog(
1054
- s3: S3DataSource,
1055
- logEntries: MutationLogEntry[],
1056
- fileName: string,
1057
- logPrefix: string,
1058
- log: any
1059
- ): Promise<void> {
1060
- try {
1061
- const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
1062
- const logFileName = `${logPrefix}${fileName.replace('.xml', '')}_${timestamp}.json`;
1063
-
1064
- const logContent = JSON.stringify(
1065
- {
1066
- fileName,
1067
- timestamp: new Date().toISOString(),
1068
- totalMutations: logEntries.length,
1069
- successful: logEntries.filter(e => e.status === 'success').length,
1070
- failed: logEntries.filter(e => e.status === 'failure').length,
1071
- entries: logEntries,
1072
- },
1073
- null,
1074
- 2
1075
- );
1076
-
1077
- // Write to S3 (uploadFile accepts string or Buffer)
1078
- await s3.uploadFile(logFileName, logContent);
1079
-
1080
- log.info('Mutation log written', { logFileName, entries: logEntries.length });
1081
- } catch (error: any) {
1082
- // ✅ Enhanced error logging: Extract all error details for visibility
1083
- const errorDetails = {
1084
- message: error?.message || 'Unknown error',
1085
- stack: error?.stack,
1086
- errorType: error?.name || 'Error',
1087
- };
1088
- log.error('Failed to write mutation log', errorDetails, { fileName });
1089
- // Don't throw - logging failure shouldn't stop workflow
1090
- }
1091
- }
1092
-
1093
- // ============================================================================
1094
- // Main Workflow Function
1095
- // ============================================================================
1096
-
1097
- /**
1098
- * Main Orchestration Function: Execute Location Sync
1099
- *
1100
- * This function orchestrates the location synchronization workflow.
1101
- *
1102
- * Architecture:
1103
- * 1. List files from S3
1104
- * 2. For each file:
1105
- * a. processFile() - Download, parse, map
1106
- * b. executeMutations() - Send GraphQL mutations with rate limiting
1107
- * c. writeMutationLog() - Log results to S3
1108
- * d. Archive file (primary deduplication)
1109
- * e. Mark processed in KV (metadata tracking)
1110
- *
1111
- * @param ctx - Versori context
1112
- * @param jobId - Job identifier
1113
- * @param tracker - JobTracker instance
1114
- * @returns Processing result
1115
- */
1116
- export async function executeLocationSync(ctx: any, jobId: string, tracker: JobTracker) {
1117
- const { log, activation } = ctx;
1118
-
1119
- log.info('📋 [INIT] Reading activation variables', { jobId });
1120
-
1121
- // Read activation variables
1122
- const s3Bucket = activation?.getVariable('s3BucketName');
1123
- const s3Region = activation?.getVariable('awsRegion') || 'us-east-1';
1124
- const s3AccessKeyId = activation?.getVariable('awsAccessKeyId');
1125
- const s3SecretAccessKey = activation?.getVariable('awsSecretAccessKey');
1126
- const s3Prefix = activation?.getVariable('s3Prefix') || 'locations/';
1127
- const archivePrefix = activation?.getVariable('archivePrefix') || 'processed/';
1128
- const errorPrefix = activation?.getVariable('errorPrefix') || 'errors/';
1129
- const logPrefix = activation?.getVariable('logPrefix') || 'logs/';
1130
- const filePattern = (activation?.getVariable('filePattern') || '.xml').toLowerCase();
1131
- const maxFiles = parseInt(activation?.getVariable('maxFilesToProcess') || '10', 10);
1132
- const retailerId = activation?.getVariable('retailerId'); // Optional: Only if mutation schema requires it
1133
- const enableArchival = activation?.getVariable('enableArchival') !== 'false';
1134
- const enableMutationLogs = activation?.getVariable('enableMutationLogs') !== 'false';
1135
- const enableFileTracking = activation?.getVariable('enableFileTracking') !== 'false';
1136
-
1137
- // ✅ Configuration with defaults
1138
- const mutationBatchSize = parseInt(
1139
- activation?.getVariable('mutationBatchSize') || '1', // ✅ Default: 1 (sequential)
1140
- 10
1141
- );
1142
-
1143
- const mutationsPerAliasBatch = activation?.getVariable('mutationsPerAliasBatch')
1144
- ? parseInt(activation?.getVariable('mutationsPerAliasBatch') || '1', 10)
1145
- : undefined; // ✅ Default: undefined (disabled, use separate requests)
1146
-
1147
- // Validate required variables
1148
- const missingVars: string[] = [];
1149
- if (!s3Bucket) missingVars.push('s3BucketName');
1150
- if (!s3AccessKeyId) missingVars.push('awsAccessKeyId');
1151
- if (!s3SecretAccessKey) missingVars.push('awsSecretAccessKey');
1152
- // Note: retailerId is optional - only needed if mutation schema requires it
1153
-
1154
- if (missingVars.length > 0) {
1155
- const errorMsg = `Missing required variables: ${missingVars.join(', ')}`;
1156
- log.error('❌ [VALIDATION] Missing required activation variables', {
1157
- missingVars,
1158
- recommendation: 'Add missing variables in Versori activation settings'
1159
- });
1160
- return { success: false, error: errorMsg, processed: 0 };
1161
- }
1162
-
1163
- log.info('✅ [VALIDATION] All required variables present', {
1164
- s3Bucket,
1165
- s3Region,
1166
- s3Prefix,
1167
- enableFileTracking,
1168
- mutationBatchSize
1169
- });
1170
-
1171
- try {
1172
- log.info('🔧 [INIT] Initializing Fluent Commerce client', { jobId });
1173
-
1174
- // Initialize services with validateConnection
1175
- const client = await createClient(ctx, { validateConnection: true });
1176
- if (!client) {
1177
- throw new Error('Failed to create Fluent Commerce client');
1178
- }
1179
-
1180
- log.info('✅ [INIT] Fluent Commerce client validated and ready', { jobId });
1181
-
1182
- // ✅ CORRECT: GraphQL mutations do NOT need client.setRetailerId()
1183
- // setRetailerId() is only for Job/Event API, NOT GraphQL
1184
- // Check your GraphQL schema to determine retailerId handling:
1185
- // - Mandatory retailerId → Must pass it in mutation input
1186
- // - Optional retailerId → Can pass it if needed
1187
- // - No retailerId field → Don't pass it
1188
- // See: fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md
1189
-
1190
- log.info('🗄️ [INIT] Initializing S3 data source', { s3Bucket, s3Region, s3Prefix });
1191
-
1192
- const s3 = new S3DataSource(
1193
- {
1194
- type: 'S3_XML',
1195
- connectionId: 's3-location-sync',
1196
- name: 'Source S3',
1197
- s3Config: {
1198
- bucket: s3Bucket,
1199
- region: s3Region,
1200
- accessKeyId: s3AccessKeyId,
1201
- secretAccessKey: s3SecretAccessKey,
1202
- },
1203
- },
1204
- log
1205
- );
1206
-
1207
- const parser = new XMLParserService();
1208
-
1209
- // Initialize state tracking (only if enabled)
1210
- let stateService: StateService | null = null;
1211
- if (enableFileTracking) {
1212
- log.info('🔄 [INIT] Enabling file tracking with StateService', { jobId });
1213
- const stateKV = new VersoriKVAdapter(ctx);
1214
- stateService = new StateService(stateKV);
1215
- } else {
1216
- log.info('⏭️ [INIT] File tracking disabled - relying on S3 archival only', { jobId });
1217
- }
1218
-
1219
- log.info('📝 [INIT] Loading mapping configuration', { jobId });
1220
-
1221
- // ✅ CRITICAL: Load mapping config from external JSON file
1222
- // Mapping config uses GraphQLMutationMapper structure (nested objects, not dot notation)
1223
- // File: src/config/location-mapping.json
1224
- const mappingConfigJson = await import('../config/location-mapping.json', { assert: { type: 'json' } });
1225
- const mappingConfig = mappingConfigJson.default;
1226
-
1227
- // Initialize GraphQLMutationMapper with client for schema introspection
1228
- const mapper = new GraphQLMutationMapper(mappingConfig, log, { fluentClient: client });
1229
-
1230
- log.info('📂 [S3] Listing files from S3', { s3Bucket, s3Prefix, filePattern });
1231
-
1232
- // List files (pattern filtering handled by listFiles)
1233
- const files = await s3.listFiles({
1234
- prefix: s3Prefix,
1235
- pattern: filePattern,
1236
- maxKeys: 1000
1237
- });
1238
-
1239
- // Newest-first ordering
1240
- const xmlFiles = files
1241
- .sort((a: any, b: any) => {
1242
- const aTime = a.lastModified ? new Date(a.lastModified).getTime() : 0;
1243
- const bTime = b.lastModified ? new Date(b.lastModified).getTime() : 0;
1244
- return bTime - aTime;
1245
- })
1246
- .slice(0, maxFiles);
1247
-
1248
- log.info('📊 [S3] File discovery complete', {
1249
- totalFiles: files.length,
1250
- xmlFiles: xmlFiles.length,
1251
- maxFiles,
1252
- selectedFiles: xmlFiles.map(f => f.name)
1253
- });
1254
-
1255
- const results = {
1256
- processed: 0,
1257
- skipped: 0,
1258
- failed: 0,
1259
- totalRecords: 0,
1260
- errors: [] as string[],
1261
- };
1262
-
1263
- log.info('🔄 [PROCESSING] Starting file processing loop', {
1264
- fileCount: xmlFiles.length,
1265
- jobId
1266
- });
1267
-
1268
- // Per-file processing loop
1269
- for (const file of xmlFiles) {
1270
- const filePath = file.path;
1271
- const fileName = file.name;
1272
-
1273
- log.info('📄 [FILE] Processing file', { fileName, filePath });
1274
-
1275
- // Duplicate prevention (secondary check - files in processed/ won't be listed)
1276
- // Primary deduplication: S3 archival (files moved to processed/ subdirectory)
1277
- if (enableFileTracking && stateService) {
1278
- const wasProcessed = await stateService.isFileProcessed(fileName);
1279
- if (wasProcessed) {
1280
- log.info('⏭️ [SKIP] File already processed (KV check)', { fileName });
1281
- results.skipped++;
1282
- continue;
1283
- }
1284
- }
1285
-
1286
- try {
1287
- // Step 1: Process file (download, parse, map)
1288
- log.info('📥 [DOWNLOAD] Downloading and parsing file', { fileName });
1289
- const processingResult = await processFile(s3, parser, mapper, filePath, fileName, log);
1290
-
1291
- if (!processingResult.success) {
1292
- throw new Error(`File processing failed: ${processingResult.errors.join(', ')}`);
1293
- }
1294
-
1295
- if (processingResult.locations.length === 0) {
1296
- log.warn('⚠️ [SKIP] Empty file detected, archiving', { fileName });
1297
- if (enableArchival) {
1298
- await s3.moveFile(filePath, `${archivePrefix}${fileName}`);
1299
- log.info('📦 [ARCHIVE] Empty file archived', { fileName, destination: `${archivePrefix}${fileName}` });
1300
- }
1301
- results.skipped++;
1302
- continue;
1303
- }
1304
-
1305
- log.info('✅ [PARSE] File parsed successfully', {
1306
- fileName,
1307
- locationCount: processingResult.locations.length,
1308
- mappingErrors: processingResult.errors.length
1309
- });
1310
-
1311
- // Step 2: Execute mutations
1312
- // Step 2: Execute mutations with alias batching support
1313
- // ? Enhanced: Extract context for progress logging
1314
- const sampleLocationRefs = processingResult.locations.slice(0, 5).map((loc: any) => loc.input?.ref || loc.ref || 'unknown');
1315
- const mutationType = mapper?.mutationName || 'createLocation';
1316
-
1317
- // ? Enhanced: Start logging with context
1318
- log.info(`[GraphQLMutations] Sending mutations for file "${fileName}"`, {
1319
- totalMutations: processingResult.locations.length,
1320
- mutationType,
1321
- batchSize: mutationBatchSize,
1322
- batchMode: mutationBatchSize === 1 ? 'sequential' : `parallel (${mutationBatchSize})`,
1323
- sampleLocationRefs: sampleLocationRefs.join(', '),
1324
- aliasBatching: mutationsPerAliasBatch ? `enabled (${mutationsPerAliasBatch} per alias)` : 'disabled'
1325
- });
1326
-
1327
- const mutationResult = await executeMutations(
1328
- client,
1329
- mapper,
1330
- processingResult.locations,
1331
- log,
1332
- retailerId, // Pass retailerId for mutations that require it in input
1333
- mutationBatchSize, // Concurrency control (default: 1)
1334
- mutationsPerAliasBatch // ✅ NEW: Alias batching (default: undefined)
1335
- );
1336
-
1337
- // ? Enhanced: Completion logging with summary
1338
- log.info(`[GraphQLMutations] Mutation submission completed for file "${fileName}"`, {
1339
- totalMutations: processingResult.locations.length,
1340
- successful: mutationResult.successful,
1341
- failed: mutationResult.failed,
1342
- successRate: processingResult.locations.length > 0 ? `${Math.round((mutationResult.successful / processingResult.locations.length) * 100)}%` : '0%',
1343
- mutationType
1344
- });
1345
-
1346
- // Step 3: Write mutation log (if enabled)
1347
- if (enableMutationLogs) {
1348
- const logEntries: MutationLogEntry[] = processingResult.locations.map(loc => {
1349
- const failed = mutationResult.errors.find(e => e.startsWith(loc.ref));
1350
- return {
1351
- timestamp: new Date().toISOString(),
1352
- fileName,
1353
- locationRef: loc.ref,
1354
- status: failed ? 'failure' : 'success',
1355
- error: failed,
1356
- };
1357
- });
1358
-
1359
- await writeMutationLog(s3, logEntries, fileName, logPrefix, log);
1360
- }
1361
-
1362
- // Step 4: Archive file (PRIMARY deduplication - file won't be listed again)
1363
- if (enableArchival) {
1364
- log.info('📦 [ARCHIVE] Moving file to processed directory', { fileName });
1365
- await s3.moveFile(filePath, `${archivePrefix}${fileName}`);
1366
- log.info('✅ [ARCHIVE] File archived successfully', {
1367
- fileName,
1368
- destination: `${archivePrefix}${fileName}`
1369
- });
1370
- }
1371
-
1372
- // Step 5: Mark processed in KV (SECONDARY - provides metadata/history)
1373
- if (enableFileTracking && stateService) {
1374
- log.info('💾 [STATE] Recording file processing metadata', { fileName });
1375
- await stateService.markFileProcessed(fileName, {
1376
- recordCount: processingResult.locations.length,
1377
- successful: mutationResult.successful,
1378
- failed: mutationResult.failed,
1379
- mappingErrors: processingResult.errors.length,
1380
- timestamp: new Date().toISOString(),
1381
- });
1382
- }
1383
-
1384
- results.processed++;
1385
- results.totalRecords += mutationResult.successful;
1386
-
1387
- log.info('✅ [COMPLETE] File processing complete', {
1388
- fileName,
1389
- locations: processingResult.locations.length,
1390
- successful: mutationResult.successful,
1391
- failed: mutationResult.failed,
1392
- mappingErrors: processingResult.errors.length,
1393
- });
1394
-
1395
- if (processingResult.errors.length > 0 || mutationResult.errors.length > 0) {
1396
- results.errors.push(
1397
- `${fileName}: ${processingResult.errors.length} mapping errors, ${mutationResult.errors.length} mutation errors`
1398
- );
1399
- }
1400
- } catch (error: any) {
1401
- // ✅ Enhanced error logging: Extract all error details for visibility
1402
- const errorDetails = {
1403
- message: error?.message || 'Unknown error',
1404
- stack: error?.stack,
1405
- fileName: error?.fileName,
1406
- lineNumber: error?.lineNumber,
1407
- originalError: error?.context?.originalError?.message,
1408
- errorType: error?.name || 'Error',
1409
- };
1410
-
1411
- log.error('❌ [ERROR] File processing failed', errorDetails, { fileName });
1412
-
1413
- // Provide error recommendations based on error type
1414
- const recommendation = getErrorRecommendation(error);
1415
- if (recommendation) {
1416
- log.warn('💡 [RECOMMENDATION]', { fileName, recommendation });
1417
- }
1418
-
1419
- results.failed++;
1420
- results.errors.push(`${fileName}: ${error.message}`);
1421
-
1422
- // Attempt to move to error directory, ignore failures
1423
- try {
1424
- log.info('🗂️ [ERROR] Moving failed file to error directory', { fileName });
1425
- await s3.moveFile(filePath, `${errorPrefix}${fileName}`);
1426
- log.info('✅ [ERROR] Failed file moved to error directory', {
1427
- fileName,
1428
- destination: `${errorPrefix}${fileName}`
1429
- });
1430
- } catch (moveError) {
1431
- log.warn('⚠️ [ERROR] Failed to move file to error directory', {
1432
- fileName,
1433
- moveError: moveError instanceof Error ? moveError.message : String(moveError)
1434
- });
1435
- }
1436
-
1437
- // Track error state with exponential backoff (only if file tracking enabled)
1438
- if (enableFileTracking && stateService) {
1439
- try {
1440
- const stateKV = new VersoriKVAdapter(ctx);
1441
- const key = ['error-state', fileName];
1442
- const prev = (await stateKV.get(key))?.value as any;
1443
- const attempts = (prev?.attemptCount || 0) + 1;
1444
- const backoffMinutes = Math.min(Math.pow(2, attempts) * 5, 24 * 60);
1445
- const nextRetryAt = new Date(Date.now() + backoffMinutes * 60000).toISOString();
1446
-
1447
- await stateKV.set(key, {
1448
- fileName,
1449
- attemptCount: attempts,
1450
- lastError: error?.message || 'unknown',
1451
- lastAttemptAt: new Date().toISOString(),
1452
- firstFailedAt: prev?.firstFailedAt || new Date().toISOString(),
1453
- nextRetryAt,
1454
- });
1455
-
1456
- log.info('💾 [ERROR] Error state tracked with exponential backoff', {
1457
- fileName,
1458
- attempts,
1459
- nextRetryAt
1460
- });
1461
- } catch (stateError) {
1462
- log.warn('⚠️ [ERROR] Failed to track error state', {
1463
- fileName,
1464
- stateError: stateError instanceof Error ? stateError.message : String(stateError)
1465
- });
1466
- }
1467
- }
1468
- }
1469
- }
1470
-
1471
- log.info('🏁 [COMPLETE] File processing loop finished', {
1472
- processed: results.processed,
1473
- skipped: results.skipped,
1474
- failed: results.failed,
1475
- totalRecords: results.totalRecords
1476
- });
1477
-
1478
- return results;
1479
- } catch (error: any) {
1480
- // ✅ Enhanced error logging: Extract all error details for visibility
1481
- const errorDetails = {
1482
- message: error?.message || 'Unknown error',
1483
- stack: error?.stack,
1484
- errorType: error?.name || 'Error',
1485
- };
1486
-
1487
- log.error('❌ [FATAL] Location sync failed', errorDetails);
1488
-
1489
- // Provide fatal error recommendations
1490
- const recommendation = getErrorRecommendation(error);
1491
- if (recommendation) {
1492
- log.warn('💡 [RECOMMENDATION]', { recommendation });
1493
- }
1494
-
1495
- return {
1496
- success: false,
1497
- error: error.message,
1498
- processed: 0,
1499
- timestamp: new Date().toISOString(),
1500
- };
1501
- }
1502
- }
1503
-
1504
- /**
1505
- * Get error recommendation based on error type
1506
- */
1507
- function getErrorRecommendation(error: any): string | null {
1508
- const message = error?.message?.toLowerCase() || '';
1509
-
1510
- if (message.includes('s3') || message.includes('access denied')) {
1511
- return 'Check S3 credentials and bucket permissions. Verify IAM policy includes s3:ListBucket, s3:GetObject, s3:PutObject, s3:DeleteObject.';
1512
- }
1513
-
1514
- if (message.includes('xml') || message.includes('parse')) {
1515
- return 'Verify XML file structure matches expected schema. Check for malformed XML or encoding issues.';
1516
- }
1517
-
1518
- if (message.includes('mapping') || message.includes('field')) {
1519
- return 'Review field mapping configuration. Ensure all required fields are present and source paths are correct.';
1520
- }
1521
-
1522
- if (message.includes('graphql') || message.includes('mutation')) {
1523
- return 'Check GraphQL schema and mutation input. Verify all required fields are provided and types match schema.';
1524
- }
1525
-
1526
- if (message.includes('auth') || message.includes('401') || message.includes('403')) {
1527
- return 'Verify Fluent Commerce credentials. Check OAuth2 client ID/secret and ensure connection is active.';
1528
- }
1529
-
1530
- if (message.includes('timeout') || message.includes('econnrefused')) {
1531
- return 'Check network connectivity. Verify API endpoints are accessible and not rate-limited.';
1532
- }
1533
-
1534
- return null;
1535
- }
1536
- ```
1537
-
1538
- **Note:** The `executeLocationSync` function should contain the full implementation of `runLocationXmlWorkflow` (renamed to `executeLocationSync`), including all the logic for processing files, executing mutations, and logging. The implementation details are shown in the service function code above.
1539
-
1540
- ---
1541
-
1542
- ### Step 5: TypeScript Configuration
1543
-
1544
- **File: tsconfig.json**
1545
-
1546
- ```json
1547
- {
1548
- "compilerOptions": {
1549
- "module": "ES2022",
1550
- "target": "ES2024",
1551
- "moduleResolution": "node"
1552
- }
1553
- }
1554
- ```
1555
-
1556
- ---
1557
-
1558
- ## Code Flow Explanation
1559
-
1560
- ### Initialization Phase
1561
-
1562
- 1. **Read activation variables** - S3 config, Fluent config, rate limiting, logging options
1563
- 2. **Validate required variables** - Fail fast if missing credentials
1564
- 3. **Initialize SDK services** - FluentClient, S3DataSource, XMLParserService, StateService
1565
- 4. **Create mapping configuration** - Define XML → GraphQL field mappings with nested objects
1566
- 5. **Calculate rate limit delay** - Convert mutations/second to delay in milliseconds
1567
-
1568
- ### File Discovery Phase
1569
-
1570
- 1. **List S3 files** - Use `s3.listFiles()` with prefix filter (excludes processed/ subdirectory)
1571
- 2. **Filter by pattern** - Match file extension (e.g., `.xml`)
1572
- 3. **Sort newest-first** - Process most recent files first
1573
- 4. **Apply max files limit** - Prevent overwhelming the workflow
1574
- 5. **Note**: Files in `processed/` subdirectory won't be listed (primary deduplication)
1575
-
1576
- ### Per-File Processing (Service Functions)
1577
-
1578
- **Step 1: processFile()** - Download, parse, map
1579
-
1580
- 1. **Download file** - Use S3DataSource with retry logic
1581
- 2. **Parse XML** - XMLParserService converts to JavaScript object
1582
- 3. **Normalize array** - Handle single vs multiple `<location>` elements (CRITICAL for XML)
1583
- 4. **Map locations** - Use UniversalMapper with nested field mapping
1584
- 5. **Collect errors** - Track mapping errors without stopping
1585
- 6. **Return result** - FileProcessingResult with locations array and errors
1586
-
1587
- **Step 2: executeMutations()** - GraphQL mutations with rate limiting
1588
-
1589
- 1. **Loop through locations** - Process each location individually
1590
- 2. **Build mutation input** - Extract nested fields (primaryAddress, openingSchedule)
1591
- 3. **Execute mutation** - Direct GraphQL `createLocation` with retry logic
1592
- 4. **Apply rate limiting** - Add configurable delay between mutations
1593
- 5. **Track results** - Count successful vs failed, collect error messages
1594
- 6. **Return result** - MutationResult with counts and errors
1595
-
1596
- **Step 3: writeMutationLog()** - S3 log file (optional)
1597
-
1598
- 1. **Create log entries** - Map locations to log entries with status
1599
- 2. **Build JSON log** - Include timestamp, summary, detailed entries
1600
- 3. **Write to S3** - Use Buffer.from() for Deno/Versori runtime compatibility
1601
- 4. **Non-blocking** - Logging failure doesn't stop workflow
1602
- 5. **Timestamped naming** - Unique log file per processed file
1603
-
1604
- ### Cleanup Phase
1605
-
1606
- 1. **Archive file FIRST** - Move to `processed/` or `errors/` (PRIMARY deduplication)
1607
- 2. **Mark processed in KV** - StateService tracks metadata and processing history
1608
- 3. **Error state tracking** - Store error info with exponential backoff timestamp
1609
- 4. **Return results** - Summary of processed, skipped, failed files
1610
-
1611
- **Note**: The order matters! Archive first (primary deduplication), then KV tracking (metadata/history).
1612
-
1613
- ### Service Function Benefits
1614
-
1615
- - **Composability**: Each function has single responsibility and can be reused
1616
- - **Testability**: Service functions can be unit tested independently
1617
- - **Clarity**: Main workflow shows high-level orchestration
1618
- - **Error Handling**: Isolated error handling per service function
1619
- - **Logging**: Detailed logging at each step with structured context
1620
-
1621
- ---
1622
-
1623
- ## S3 Archival Deduplication Pattern
1624
-
1625
- This template uses **S3 archival** as the primary deduplication mechanism, NOT VersoriFileTracker.
1626
-
1627
- ### How It Works
1628
-
1629
- **S3 Directory Structure:**
1630
-
1631
- ```
1632
- s3://my-bucket/
1633
- ├── locations/ ← listFiles() reads from here
1634
- │ ├── new-file-1.xml
1635
- │ └── new-file-2.xml
1636
- ├── processed/ ← Successfully processed files
1637
- │ ├── old-file-1.xml
1638
- │ └── old-file-2.xml
1639
- └── errors/ ← Failed files
1640
- └── bad-file.xml
1641
- ```
1642
-
1643
- **Deduplication Flow:**
1644
-
1645
- 1. **List files**: `s3.listFiles({ prefix: 'locations/' })` - Only lists `locations/` subdirectory
1646
- 2. **Process file**: Download, parse, transform, send mutations
1647
- 3. **Archive**: `s3.moveFile(filePath, 'processed/new-file-1.xml')` - Moves file out of `locations/`
1648
- 4. **Next run**: File is now in `processed/`, won't be listed again
1649
-
1650
- **Why This Works:**
1651
-
1652
- - Files in `processed/` subdirectory are **never listed** when prefix is `locations/`
1653
- - No need to track file state in KV store for deduplication
1654
- - S3 is the single source of truth for file status
1655
- - Simple, reliable, scales to millions of files
1656
-
1657
- **StateService Role (Secondary):**
1658
-
1659
- - Provides metadata and processing history
1660
- - Backup check in case archival fails mid-process
1661
- - Useful for monitoring and debugging
1662
- - NOT the primary deduplication mechanism
1663
-
1664
- **When to Use VersoriFileTracker:**
1665
-
1666
- - **NEVER for S3 sources** - Use archival pattern instead
1667
- - **Only for SFTP sources** - Where archival might not be possible
1668
- - See SFTP templates for VersoriFileTracker usage
1669
-
1670
- ---
1671
-
1672
- ## XML Path Resolution Patterns
1673
-
1674
- ### Pattern 1: XML Attribute Access with @ Prefix
1675
-
1676
- ```typescript
1677
- const mappingConfig = {
1678
- fields: {
1679
- // XML attribute access
1680
- ref: { source: 'location.@ref' }, // <location ref="LOC-001">
1681
- type: { source: 'location.@type' }, // <location type="WAREHOUSE">
1682
- country: { source: 'location.address.@country' }, // <address country="USA">
1683
-
1684
- // XML element text content
1685
- name: { source: 'location.name' }, // <name>Downtown</name>
1686
- city: { source: 'location.address.city' }, // <address><city>NYC</city></address>
1687
- },
1688
- };
1689
- ```
1690
-
1691
- ### Pattern 2: Handling Single vs Multiple Elements
1692
-
1693
- ```typescript
1694
- // XML can have single or multiple <location> elements
1695
- const locationsData = xmlData.locations?.location;
1696
-
1697
- // Normalize to array
1698
- const locations = Array.isArray(locationsData) ? locationsData : [locationsData];
1699
-
1700
- // Process each location
1701
- for (const loc of locations) {
1702
- const record = { location: loc }; // Wrap for mapping
1703
- const result = await mapper.map(record);
1704
- }
1705
- ```
1706
-
1707
- ### Pattern 3: Nested Object Mapping
1708
-
1709
- ```typescript
1710
- const mappingConfig = {
1711
- fields: {
1712
- // Root fields
1713
- ref: { source: 'location.@ref', required: true },
1714
- name: { source: 'location.name', required: true },
1715
- type: { source: 'location.@type', required: true },
1716
-
1717
- // Nested primaryAddress object
1718
- 'primaryAddress.ref': { source: 'location.@ref' },
1719
- 'primaryAddress.street': { source: 'location.address.street1' },
1720
- 'primaryAddress.city': { source: 'location.address.city' },
1721
- 'primaryAddress.latitude': { source: 'location.coordinates.@lat', resolver: 'sdk.parseFloat' },
1722
-
1723
- // Nested openingSchedule object
1724
- 'openingSchedule.allHours': {
1725
- source: 'location.openingSchedule.allHours',
1726
- resolver: 'sdk.boolean',
1727
- },
1728
- 'openingSchedule.monStart': {
1729
- source: 'location.openingSchedule.monStart',
1730
- resolver: 'sdk.parseInt',
1731
- },
1732
-
1733
- // Note: retailer.id not shown - standard createLocation does not have this field
1734
- // If your schema requires it, add: 'retailer.id': { value: parseInt(retailerId) }
1735
- },
1736
- };
1737
- ```
1738
-
1739
- ---
1740
-
1741
- ## Sample XML Files
1742
-
1743
- ### Minimal Test File
1744
-
1745
- **File: test-location.xml**
1746
-
1747
- ```xml
1748
- <?xml version="1.0" encoding="UTF-8"?>
1749
- <locations>
1750
- <location ref="TEST-001" type="WAREHOUSE">
1751
- <name>Test Warehouse</name>
1752
- <address country="USA">
1753
- <street1>123 Test St</street1>
1754
- <city>TestCity</city>
1755
- <state>TC</state>
1756
- <postalCode>12345</postalCode>
1757
- </address>
1758
- <coordinates lat="40.7128" lon="-74.0060"/>
1759
- <timeZone>America/New_York</timeZone>
1760
- <openingSchedule>
1761
- <allHours>false</allHours>
1762
- <monStart>800</monStart>
1763
- <monEnd>1800</monEnd>
1764
- <tueStart>800</tueStart>
1765
- <tueEnd>1800</tueEnd>
1766
- <wedStart>800</wedStart>
1767
- <wedEnd>1800</wedEnd>
1768
- <thuStart>800</thuStart>
1769
- <thuEnd>1800</thuEnd>
1770
- <friStart>800</friStart>
1771
- <friEnd>1800</friEnd>
1772
- <satStart>0</satStart>
1773
- <satEnd>0</satEnd>
1774
- <sunStart>0</sunStart>
1775
- <sunEnd>0</sunEnd>
1776
- </openingSchedule>
1777
- </location>
1778
- </locations>
1779
- ```
1780
-
1781
- ---
1782
-
1783
- ## Service Functions Deep Dive
1784
-
1785
- This template demonstrates service function composition for maintainable workflows.
1786
-
1787
- ### Function 1: processFile()
1788
-
1789
- **Purpose**: Download, parse, normalize, and map XML data
1790
-
1791
- **Inputs**:
1792
- - `s3: S3DataSource` - For file download
1793
- - `parser: XMLParserService` - For XML parsing
1794
- - `mapper: UniversalMapper` - For field mapping
1795
- - `filePath: string` - S3 path to file
1796
- - `fileName: string` - File name for logging
1797
- - `log: any` - Logger instance
1798
-
1799
- **Outputs**: `FileProcessingResult`
1800
- ```typescript
1801
- {
1802
- success: boolean; // Overall success
1803
- locations: any[]; // Mapped location objects
1804
- errors: string[]; // Mapping error messages
1805
- }
1806
- ```
1807
-
1808
- **Key Operations**:
1809
- 1. Downloads file with retry logic
1810
- 2. Parses XML using XMLParserService
1811
- 3. **Normalizes arrays** - Handles single vs multiple `<location>` elements
1812
- 4. Maps each location using UniversalMapper
1813
- 5. Collects errors without stopping processing
1814
- 6. Returns all mapped locations + errors
1815
-
1816
- **Why separate function?**
1817
- - Testable independently with mock data
1818
- - Reusable across workflows (scheduled, webhook, adhoc)
1819
- - Clear error boundaries - file-level errors vs record-level errors
1820
- - Easy to add XML validation or schema checks
1821
-
1822
- ### Function 2: executeMutations()
1823
-
1824
- **Purpose**: Execute GraphQL createLocation mutations with rate limiting
1825
-
1826
- **Inputs**:
1827
- - `client: FluentClient` - For GraphQL mutations
1828
- - `locations: any[]` - Mapped location data
1829
- - `mutationDelayMs: number` - Rate limit delay
1830
- - `fileName: string` - For logging context
1831
- - `log: any` - Logger instance
1832
-
1833
- **Outputs**: `MutationResult`
1834
- ```typescript
1835
- {
1836
- successful: number; // Count of successful mutations
1837
- failed: number; // Count of failed mutations
1838
- errors: string[]; // Error messages with location refs
1839
- }
1840
- ```
1841
-
1842
- **Key Operations**:
1843
- 1. Loops through locations
1844
- 2. Builds GraphQL mutation input
1845
- 3. Executes with retry logic + rate limiting
1846
- 4. Tracks success/failure per location
1847
- 5. Returns summary counts + errors
1848
-
1849
- **Why separate function?**
1850
- - Clear separation: mapping vs mutation execution
1851
- - Rate limiting logic isolated and configurable
1852
- - Easy to swap mutation types (create vs update)
1853
- - Testable with mock FluentClient
1854
- - Can parallelize in future (batch mutations)
1855
-
1856
- ### Function 3: writeMutationLog()
1857
-
1858
- **Purpose**: Write detailed mutation results to S3 as JSON log
1859
-
1860
- **Inputs**:
1861
- - `s3: S3DataSource` - For log upload
1862
- - `logEntries: MutationLogEntry[]` - Mutation status per location
1863
- - `fileName: string` - Original file name
1864
- - `logPrefix: string` - S3 prefix for logs (e.g., `logs/`)
1865
- - `log: any` - Logger instance
1866
-
1867
- **Outputs**: `void` (non-blocking - errors logged but not thrown)
1868
-
1869
- **Key Operations**:
1870
- 1. Creates timestamped log file name
1871
- 2. Builds JSON log with summary + entries
1872
- 3. **Uses Buffer.from()** - Required for Deno/Versori runtime
1873
- 4. Writes to S3 with `uploadFile()`
1874
- 5. Errors don't stop workflow (logging is non-critical)
1875
-
1876
- **Why separate function?**
1877
- - Optional feature - can be disabled via config
1878
- - Non-blocking - logging failure doesn't fail workflow
1879
- - Structured logging for audit trails
1880
- - Easy to change log format (JSON, CSV, XML)
1881
- - Can add log rotation/cleanup logic later
1882
-
1883
- ### Service Function Composition Benefits
1884
-
1885
- **1. Maintainability**
1886
- ```typescript
1887
- // Clear workflow orchestration
1888
- const processingResult = await processFile(...);
1889
- const mutationResult = await executeMutations(...);
1890
- await writeMutationLog(...);
1891
- ```
1892
-
1893
- **2. Testability**
1894
- ```typescript
1895
- // Unit test processFile() with mock XML
1896
- const mockS3 = { downloadFile: jest.fn() };
1897
- const result = await processFile(mockS3, parser, mapper, ...);
1898
- expect(result.locations).toHaveLength(5);
1899
- ```
1900
-
1901
- **3. Reusability**
1902
- ```typescript
1903
- // Use processFile() in different workflows
1904
- export const webhook = webhook('location-webhook').then(async ctx => {
1905
- const result = await processFile(s3, parser, mapper, filePath, fileName, log);
1906
- return { locations: result.locations };
1907
- });
1908
- ```
1909
-
1910
- **4. Error Isolation**
1911
- ```typescript
1912
- // Each function has clear error boundaries
1913
- try {
1914
- const processingResult = await processFile(...);
1915
- // File-level errors caught here
1916
- } catch (error) {
1917
- // Handle file processing failure
1918
- }
1919
-
1920
- // Mutation errors don't stop file processing
1921
- const mutationResult = await executeMutations(...);
1922
- // mutationResult.errors contains per-location failures
1923
- ```
1924
-
1925
- **5. Progressive Enhancement**
1926
- ```typescript
1927
- // Easy to add features without touching core logic
1928
- async function validateLocations(locations: any[]) {
1929
- // Add validation step
1930
- }
1931
-
1932
- const processingResult = await processFile(...);
1933
- await validateLocations(processingResult.locations); // New step
1934
- const mutationResult = await executeMutations(...);
1935
- ```
1936
-
1937
- ---
1938
-
1939
- ## Versori Environment Variables
1940
-
1941
- **Activation Variables:**
1942
-
1943
- ```bash
1944
- # ============================================================================
1945
- # Required Variables
1946
- # ============================================================================
1947
- s3BucketName=my-location-bucket
1948
- awsAccessKeyId=AKIAXXXXXXXXXXXX
1949
- awsSecretAccessKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
1950
-
1951
- # ============================================================================
1952
- # S3 Configuration (Optional - with defaults)
1953
- # ============================================================================
1954
- awsRegion=us-east-1
1955
- s3Prefix=locations/
1956
- archivePrefix=processed/
1957
- errorPrefix=errors/
1958
- logPrefix=logs/
1959
- filePattern=.xml
1960
- maxFilesToProcess=10
1961
-
1962
- # ============================================================================
1963
- # Feature Toggles (Optional - with defaults)
1964
- # ============================================================================
1965
- # Enable S3 archival (move files to processed/errors directories)
1966
- enableArchival=true
1967
-
1968
- # Enable mutation logs (write detailed mutation results to S3)
1969
- enableMutationLogs=true
1970
-
1971
- # Enable file tracking via StateService + KV store
1972
- # When disabled, relies on S3 archival only for deduplication
1973
- enableFileTracking=true
1974
-
1975
- # ============================================================================
1976
- # Mutation Configuration (Optional - with defaults)
1977
- # ============================================================================
1978
- # Mutation batch size (concurrent requests)
1979
- # - 1 = Sequential (default, safest)
1980
- # - 5 = Process 5 mutations in parallel
1981
- # - 10 = Process 10 mutations in parallel
1982
- mutationBatchSize=1
1983
-
1984
- # Alias batching (combine multiple mutations into single request)
1985
- # - undefined = Disabled (default, use separate requests)
1986
- # - 5 = Combine 5 mutations per aliased request
1987
- # - 10 = Combine 10 mutations per aliased request
1988
- mutationsPerAliasBatch=
1989
-
1990
- # ============================================================================
1991
- # Fluent Commerce Configuration (Optional)
1992
- # ============================================================================
1993
- # Retailer ID - Only if mutation schema requires retailerId in input
1994
- # Standard createLocation does NOT require this field
1995
- # Check your GraphQL schema to determine if needed
1996
- retailerId=my-retailer-id
1997
- ```
1998
-
1999
- **Notes:**
2000
- - Webhook security is handled by Versori's native connection authentication. No manual API key configuration needed.
2001
- - `retailerId` - Standard createLocation does not have this field. Only use if YOUR custom schema requires it.
2002
- - See `fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md` for details.
2003
-
2004
- ---
2005
-
2006
- ## Schema Validation CLI Commands
2007
-
2008
- Before deploying, validate your field mappings against the Fluent GraphQL schema:
2009
-
2010
- ### 1. Introspect Schema
2011
-
2012
- ```bash
2013
- # Generate schema.json from live Fluent API
2014
- npx fc-connect introspect-schema \
2015
- --url https://api.fluentcommerce.com/graphql \
2016
- --client-id your-client-id \
2017
- --client-secret your-client-secret \
2018
- --output schema.json
2019
- ```
2020
-
2021
- ### 2. Create Mapping Config File
2022
-
2023
- **File: location-mapping.json**
2024
-
2025
- ```json
2026
- {
2027
- "version": "1.0.0",
2028
- "mutation": "createLocation",
2029
- "sourceFormat": "xml",
2030
- "returnFields": ["id", "ref", "name", "type", "status"],
2031
- "description": "XML location to Fluent Commerce GraphQL mapping",
2032
- "fields": {
2033
- "ref": { "source": "location.@ref", "required": true, "resolver": "sdk.trim" },
2034
- "name": { "source": "location.name", "required": true, "resolver": "sdk.trim" },
2035
- "type": { "source": "location.@type", "required": true, "resolver": "sdk.uppercase" },
2036
- "primaryAddress.ref": { "source": "location.@ref", "required": true },
2037
- "primaryAddress.street": { "source": "location.address.street1" },
2038
- "primaryAddress.latitude": {
2039
- "source": "location.coordinates.@lat",
2040
- "resolver": "sdk.parseFloat"
2041
- },
2042
- "primaryAddress.longitude": {
2043
- "source": "location.coordinates.@lon",
2044
- "resolver": "sdk.parseFloat"
2045
- }
2046
- }
2047
- }
2048
- ```
2049
-
2050
- ### 3. Validate Mapping
2051
-
2052
- ```bash
2053
- # Validate that all target fields exist in schema
2054
- npx fc-connect validate-schema \
2055
- --mapping location-mapping.json \
2056
- --schema schema.json
2057
- ```
2058
-
2059
- ### 4. Analyze Coverage
2060
-
2061
- ```bash
2062
- # Check which Location fields are mapped vs available
2063
- npx fc-connect analyze-coverage \
2064
- --mapping location-mapping.json \
2065
- --schema schema.json \
2066
- --type CreateLocationInput
2067
- ```
2068
-
2069
- **Output:**
2070
-
2071
- ```
2072
- ✅ Mapped: 15/42 fields (35%)
2073
- ❌ Missing required: timezone (String!)
2074
- ⚠️ Optional unmapped: supportPhoneNumber, networkId, attributes
2075
- ```
2076
-
2077
- ---
2078
-
2079
- ## Testing Locally
2080
-
2081
- ### 1. Upload Test XML to S3
2082
-
2083
- ```bash
2084
- aws s3 cp test-location.xml s3://my-location-bucket/locations/test-location.xml
2085
- ```
2086
-
2087
- ### 2. Deploy to Versori
2088
-
2089
- ```bash
2090
- npm run deploy
2091
- ```
2092
-
2093
- ### 3. Manual Testing
2094
-
2095
- ```bash
2096
- # Trigger manual sync (auth handled by Versori connection)
2097
- curl -X POST https://your-workspace.versori.run/location-xml-adhoc
2098
-
2099
- # Check job status
2100
- curl -X POST https://your-workspace.versori.run/location-xml-job-status \
2101
- -H "Content-Type: application/json" \
2102
- -d '{"jobId": "location-xml-adhoc-1737525600000"}'
2103
- ```
2104
-
2105
- ### 4. Verify Processing
2106
-
2107
- - Upload a small XML to S3 (2-3 locations) and trigger `adhoc` webhook
2108
- - Verify GraphQL mutations are executed for each location
2109
- - Confirm file moved from `locations/` to `processed/` in S3
2110
- - Check KV state: errors and last processed metadata
2111
- - Monitor rate limiting: verify mutations respect configured rate
2112
-
2113
- ---
2114
-
2115
- ## Deployment
2116
-
2117
- ```bash
2118
- # Deploy to Versori
2119
- npm run deploy
2120
-
2121
- # View logs
2122
- npm run logs
2123
-
2124
- # Monitor execution
2125
- versori logs --follow
2126
- ```
2127
-
2128
- ---
2129
-
2130
- ## Monitoring
2131
-
2132
- ### Success Response
2133
-
2134
- ```json
2135
- {
2136
- "success": true,
2137
- "jobId": "location-xml-scheduled-1737525600000",
2138
- "processed": 3,
2139
- "skipped": 0,
2140
- "failed": 0,
2141
- "totalRecords": 12,
2142
- "errors": []
2143
- }
2144
- ```
2145
-
2146
- ### Partial Success Response
2147
-
2148
- ```json
2149
- {
2150
- "success": true,
2151
- "jobId": "location-xml-scheduled-1737525600000",
2152
- "processed": 3,
2153
- "skipped": 1,
2154
- "failed": 0,
2155
- "totalRecords": 10,
2156
- "errors": ["locations-003.xml: 2 mapping errors"]
2157
- }
2158
- ```
2159
-
2160
- ### Error Response
2161
-
2162
- ```json
2163
- {
2164
- "success": false,
2165
- "jobId": "location-xml-scheduled-1737525600000",
2166
- "processed": 0,
2167
- "skipped": 0,
2168
- "failed": 1,
2169
- "totalRecords": 0,
2170
- "errors": ["locations-001.xml: Invalid XML structure"]
2171
- }
2172
- ```
2173
-
2174
- ---
2175
-
2176
- ## Common Pitfalls and Solutions
2177
-
2178
- ### 1. XML Attribute Not Found
2179
-
2180
- **Symptoms**: Mapping errors like "field not found"
2181
-
2182
- **Solution**:
2183
-
2184
- ```typescript
2185
- // ❌ WRONG - Missing @ prefix for attribute
2186
- ref: {
2187
- source: 'location.ref';
2188
- }
2189
-
2190
- // ✅ CORRECT - Use @ prefix for XML attributes
2191
- ref: {
2192
- source: 'location.@ref';
2193
- }
2194
- ```
2195
-
2196
- ### 2. Single Element Not Array
2197
-
2198
- **Symptoms**: "Cannot read property forEach of undefined"
2199
-
2200
- **Solution**:
2201
-
2202
- ```typescript
2203
- // Always normalize to array
2204
- const locationsData = xmlData.locations?.location;
2205
- const locations = Array.isArray(locationsData) ? locationsData : [locationsData];
2206
- ```
2207
-
2208
- ### 3. Rate Limiting Too Aggressive
2209
-
2210
- **Symptoms**: Slow processing, mutations taking too long
2211
-
2212
- **Solution**:
2213
-
2214
- ```bash
2215
- # Increase rate limit (mutations per second)
2216
- mutationRateLimit=10 # Default is 5
2217
- ```
2218
-
2219
- ### 4. Empty Element vs Missing Element
2220
-
2221
- **Solution**:
2222
-
2223
- ```typescript
2224
- // Use required: false and defaultValue for optional fields
2225
- 'primaryAddress.street2': {
2226
- source: 'location.address.street2',
2227
- required: false,
2228
- defaultValue: ''
2229
- }
2230
- ```
2231
-
2232
- ### 5. S3 Access Denied
2233
-
2234
- **Symptoms**: S3 operations fail with 403 errors
2235
-
2236
- **Solution**: Validate IAM permissions
2237
-
2238
- **Required IAM Permissions:**
2239
-
2240
- ```json
2241
- {
2242
- "Version": "2012-10-17",
2243
- "Statement": [
2244
- {
2245
- "Effect": "Allow",
2246
- "Action": ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
2247
- "Resource": ["arn:aws:s3:::my-location-bucket", "arn:aws:s3:::my-location-bucket/*"]
2248
- }
2249
- ]
2250
- }
2251
- ```
2252
-
2253
- ### 6. GraphQL Schema Mismatch
2254
-
2255
- **Symptoms**: Mutation errors like "Unknown field", "Invalid input type"
2256
-
2257
- **Solution**: Use CLI tools to validate mappings
2258
-
2259
- ```bash
2260
- npx fc-connect validate-schema --mapping location-mapping.json --schema schema.json
2261
- ```
2262
-
2263
- ### 7. Nested Object Mapping Errors
2264
-
2265
- **Symptoms**: Flat structure instead of nested objects in mutation input
2266
-
2267
- **Solution**: Use dot notation in field mapping
2268
-
2269
- ```typescript
2270
- // ✅ CORRECT - Creates nested structure
2271
- 'primaryAddress.city': { source: 'location.address.city' }
2272
-
2273
- // ❌ WRONG - Creates flat structure
2274
- primaryAddress_city: { source: 'location.address.city' }
2275
- ```
2276
-
2277
- ### 8. retailerId Configuration Errors
2278
-
2279
- **Symptoms**: "retailerId is required" errors or confusion about when to use `setRetailerId()`
2280
-
2281
- **Solution**: Understand the correct pattern
2282
-
2283
- ```typescript
2284
- // ✅ CORRECT - GraphQL mutations don't need setRetailerId()
2285
- // Check your GraphQL schema to determine retailerId handling:
2286
- // - Mandatory retailerId → Must pass it in mutation input
2287
- // - Optional retailerId → Can pass it if needed
2288
- // - No retailerId field → Don't pass it
2289
- // Standard createLocation does not have retailerId field in schema
2290
-
2291
- // ✅ IF mutation schema requires retailerId (mandatory):
2292
- const { query, variables } = await mapper.map(location);
2293
- if (retailerId && variables.input) {
2294
- variables.input.retailer = { id: parseInt(retailerId) };
2295
- }
2296
- await client.graphql({ query, variables });
2297
- ```
2298
-
2299
- **Reference:** `fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md`
2300
-
2301
- ---
2302
-
2303
- ## Key Takeaways
2304
-
2305
- ### Architecture Patterns
2306
-
2307
- - **Service Function Composition**: Workflow broken into 3 service functions - `processFile()`, `executeMutations()`, `writeMutationLog()`
2308
- - **Per-File Processing**: Main workflow orchestrates service functions for each file
2309
- - **Clear Separation of Concerns**: Parse/map, execute mutations, logging are independent functions
2310
-
2311
- ### SDK Usage
2312
-
2313
- - **Buffer Import (CRITICAL)**: Always `import { Buffer } from 'node:buffer'` for Deno/Versori runtime
2314
- - **S3 Archival Deduplication**: Use `s3.moveFile()` to `processed/` subdirectory - PRIMARY deduplication mechanism
2315
- - **NO VersoriFileTracker for S3**: S3 archival is simpler and more reliable
2316
- - **StateService Role**: SECONDARY - provides metadata/history, not primary deduplication
2317
-
2318
- ### XML Processing
2319
-
2320
- - **XML @ Prefix**: Always use `@` prefix for XML attributes (`location.@ref`)
2321
- - **Array Normalization (CRITICAL)**: Handle single vs multiple elements with `Array.isArray()` check - single `<location>` becomes object, not array
2322
- - **Nested Mapping**: Use dot notation for nested objects (`primaryAddress.street`, `openingSchedule.monStart`)
2323
-
2324
- ### GraphQL Mutations
2325
-
2326
- - **NO setRetailerId() Required**: GraphQL mutations do NOT need `client.setRetailerId()` - only Job/Event API needs it
2327
- - **retailerId in Input**: Check your GraphQL schema to determine retailerId handling:
2328
- - **Mandatory retailerId** - Field exists and is required (`!`) → Must pass it
2329
- - **Optional retailerId** - Field exists and is optional → Can pass it if needed
2330
- - **No retailerId field** - Field doesn't exist → Don't pass it
2331
- - **Rate Limiting**: Implement configurable delays between mutations to avoid API throttling
2332
- - **Retry Logic**: Exponential backoff for failed mutations with `retryWithBackoff()`
2333
- - **Direct Mutations**: Use `client.graphql()` for location upserts (NOT Batch API)
2334
- - **GraphQL vs Batch API**: Use GraphQL for low-volume master data, Batch API for high-volume inventory
2335
-
2336
- ### Error Handling & Monitoring
2337
-
2338
- - **Error Recovery**: Exponential backoff for error state tracking with retry timestamps
2339
- - **Mutation Logging**: Optional S3 JSON logs with detailed per-location mutation status
2340
- - **Schema Validation**: Use CLI tools before deployment to catch mapping errors
2341
- - **Archival Order**: Archive FIRST (deduplication), then KV tracking (metadata)
2342
-
2343
- ---
2344
-
2345
- ## Related Documentation
2346
-
2347
- ### Core Guides
2348
-
2349
- - **GraphQL Mutation Mapping**: `fc-connect-sdk/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/readme.md`
2350
- - **Universal Mapping**: `fc-connect-sdk/docs/02-CORE-GUIDES/mapping/modules/readme.md`
2351
- - **XML Parser**: `fc-connect-sdk/docs/02-CORE-GUIDES/parsers/modules/05-xml-parser.md`
2352
- - **Data Sources**: `fc-connect-sdk/docs/02-CORE-GUIDES/data-sources/readme.md`
2353
- - **State Management**: `fc-connect-sdk/docs/03-PATTERN-GUIDES/file-operations/state-duplicate-prevention.md`
2354
-
2355
- ### Related Templates
2356
-
2357
- - **CSV Version**: `template-ingestion-s3-csv-location-graphql.md`
2358
- - **JSON Version**: `template-ingestion-s3-json-location-graphql.md`
2359
- - **SFTP XML Version**: `template-ingestion-sftp-xml-location-graphql.md`
2360
- - **Event API Pattern**: `../event-api/template-ingestion-s3-xml-product-event.md`
2361
- - **Batch API Pattern**: `../batch-api/template-ingestion-s3-xml-inventory-batch.md`
2362
-
2363
- ### CLI Tools
2364
-
2365
- - **Schema Introspection**: `fc-connect-sdk/bin/readme.md#introspect-schema`
2366
- - **Mapping Validation**: `fc-connect-sdk/bin/readme.md#validate-schema`
2367
- - **Coverage Analysis**: `fc-connect-sdk/bin/readme.md#analyze-coverage`
2368
-
2369
- ### Patterns
2370
-
2371
- - **Error Handling**: `fc-connect-sdk/docs/01-TEMPLATES/patterns/error-handling-retry.md`
2372
- - **Rate Limiting**: `fc-connect-sdk/docs/03-PATTERN-GUIDES/integration-patterns/rate-limiting.md`
2373
- - **XML Patterns**: `fc-connect-sdk/docs/01-TEMPLATES/versori/patterns/xml-response-patterns.md`
1
+ ---
2
+ template_id: tpl-ingest-s3-xml-to-location-graphql
3
+ canonical_filename: template-ingestion-s3-xml-location-graphql.md
4
+ version: 2.0.0
5
+ sdk_version: ^0.1.39
6
+ runtime: versori
7
+ direction: ingestion
8
+ source: s3-xml
9
+ destination: fluent-graphql
10
+ entity: location
11
+ format: xml
12
+ logging: versori
13
+ status: stable
14
+ compliance: gold-standard
15
+ features:
16
+ - graphql-mutation-mapper
17
+ - memory-management
18
+ - enhanced-logging
19
+ - attribute-transformation
20
+ ---
21
+
22
+ Template: Ingestion - S3 XML to Location GraphQL
23
+
24
+ **Template Version:** 2.0.0
25
+ **SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
26
+ **Last Updated:** 2025-01-24
27
+
28
+ **🆕 Version 2.0.0 Enhancements:**
29
+ - ✅ **GraphQL Mutation Mapper** - Direct field mapping to mutation variables
30
+ - ✅ **Memory Management** - Clear large arrays after processing batches
31
+ - ✅ **Enhanced Logging** - Track mutation execution with emoji indicators
32
+ - ✅ **Attribute Transformation** - Handle complex nested data structures
33
+
34
+ ---
35
+
36
+ ## Implementation Prompt
37
+
38
+ "Create a Versori scheduled workflow that reads location XML files from S3, transforms the data using GraphQLMutationMapper with nested object mapping, and creates/updates Fluent Commerce locations via direct GraphQL mutations with alias batching support.
39
+
40
+ **Requirements:**
41
+
42
+ 1. **S3 Source**:
43
+ - List and download XML files with configurable prefix and pattern
44
+ - Archive processed files to `processed/` directory (deduplication via archiving)
45
+ - Move failed files to `errors/` directory
46
+ - Use S3DataSource with retry logic and built-in archival
47
+ - Support configurable bucket, region, credentials
48
+
49
+ 2. **XML Parsing**:
50
+ - Use XMLParserService with @ prefix for attribute access
51
+ - Handle both single and multiple `<location>` elements (array normalization)
52
+ - Support nested XML paths (`location.address.street1`, `location.coordinates.@lat`)
53
+ - Parse complex structures (addresses, coordinates, opening schedules)
54
+
55
+ 3. **Field Mapping**:
56
+ - Map XML fields to Location GraphQL input type with nested objects:
57
+ - `ref`, `name`, `type` (root fields)
58
+ - `primaryAddress.*` (nested address object with coordinates)
59
+ - `openingSchedule.*` (nested schedule object with 7-day hours)
60
+ - Use SDK resolvers (trim, uppercase, parseFloat, parseInt, boolean)
61
+ - Support custom resolvers for complex transformations
62
+ - Validate required fields
63
+ - Note: Check your GraphQL schema to determine if `retailer.id` field exists and is mandatory/optional
64
+
65
+ 4. **GraphQL Mutations** (Direct - NO Batch API):
66
+ - Execute `createLocation` mutation directly for each location
67
+ - **NO BPP (Batch Pre-Processing)** - Not applicable for direct GraphQL mutations
68
+ - **NO Batch API** - Use direct GraphQL mutations with rate limiting instead
69
+ - Use rate limiting (configurable mutations per second)
70
+ - Add delay between mutations to avoid API throttling
71
+ - Retry failed mutations with exponential backoff
72
+ - Track successful vs failed mutations per file
73
+
74
+ 5. **Job Tracking & State Management**:
75
+ - **Use JobTracker** - Track job lifecycle (start, complete, fail) in KV store
76
+ - Use StateService + VersoriKVAdapter for duplicate file prevention
77
+ - Track processed files in KV store with metadata
78
+ - Store error state with exponential backoff tracking
79
+ - Support distributed state across workflow runs
80
+ - Provide job status endpoint for monitoring
81
+
82
+ 6. **Error Handling**:
83
+ - File-level errors archived to `/errors/` subdirectory
84
+ - Record-level errors tracked but don't stop file processing
85
+ - Mapping errors logged with specific location context
86
+ - Mutation errors retried with exponential backoff
87
+ - Error state tracking with next retry timestamp
88
+
89
+ 7. **Advanced Features**:
90
+ - Configurable rate limiting (mutations per second)
91
+ - Empty file detection and archival
92
+ - Timestamp-based error tracking
93
+ - Comprehensive monitoring and logging
94
+ - Manual webhook trigger with job tracking
95
+ - Job status query endpoint
96
+
97
+ **Use SDK Components:**
98
+
99
+ - `createClient()` - Universal client factory for Versori
100
+ - `S3DataSource` - S3 operations with retry logic
101
+ - `XMLParserService` - XML parsing with @ prefix attribute support
102
+ - `GraphQLMutationMapper` - Field transformation with schema validation and nested object support
103
+ - `StateService` + `VersoriKVAdapter` - Duplicate prevention with KV storage
104
+ - Native Versori `log` - Structured logging
105
+
106
+ **Configuration Variables** (from Versori activation):
107
+
108
+ ```typescript
109
+ {
110
+ s3: {
111
+ bucketName: string;
112
+ region: string;
113
+ accessKeyId: string;
114
+ secretAccessKey: string;
115
+ prefix: string; // e.g., 'locations/'
116
+ archivePrefix: string; // e.g., 'processed/'
117
+ errorPrefix: string; // e.g., 'errors/'
118
+ filePattern: string; // e.g., '.xml'
119
+ maxFilesToProcess: number;
120
+ enableArchival: boolean;
121
+ },
122
+ fluent: {
123
+ retailerId: string; // Optional: Only if mutation schema requires retailerId in input
124
+ mutationRateLimit: number; // mutations per second (e.g., 5)
125
+ }
126
+ }
127
+ ```
128
+
129
+ **Architecture Pattern:**
130
+
131
+ ```
132
+ S3 Bucket → List Files → Download XML → Parse → Map → GraphQL Mutation → Archive/Move
133
+ ↓ ↓ ↓ ↓ ↓ ↓ ↓
134
+ Configure Filter by S3DataSource XML Universal Direct S3 moveFile()
135
+ Connection Pattern + Retry Parser Mapper createLocation to processed/
136
+ (@) (nested) + Rate Limit or errors/
137
+ (deduplication)
138
+ ```
139
+
140
+ **Deliverables:**
141
+
142
+ 1. Complete Versori workflow with package.json
143
+ 2. Main workflow logic with S3 + XML + GraphQL patterns
144
+ 3. Helper functions for rate limiting and retry
145
+ 4. XML path resolution examples
146
+ 5. Sample XML files with nested structures
147
+ 6. Schema validation CLI commands
148
+ 7. Testing and deployment instructions
149
+ 8. Monitoring and troubleshooting guidance"
150
+
151
+ ---
152
+
153
+ # STEP 3: Complete Implementation
154
+
155
+ ## Versori Scheduled: S3 XML → Location GraphQL
156
+
157
+ **FC Connect SDK Use Case Guide**
158
+
159
+ > **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
160
+ > **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
161
+
162
+ **Context**: Versori scheduled workflow that reads location XML files from S3 and creates/updates Fluent Commerce locations via GraphQL mutations with XML path resolution and rate limiting
163
+
164
+ **Complexity**: Medium
165
+
166
+ **Runtime**: Versori Platform
167
+
168
+ **Estimated Lines**: ~850 lines (with comprehensive documentation)
169
+
170
+ ---
171
+
172
+ ## What You'll Build
173
+
174
+ - Scheduled Versori workflow (daily location sync)
175
+ - S3 file listing, download, and archival/move
176
+ - XML parsing with @ prefix for attributes (XPath-style)
177
+ - Array normalization (single element → array conversion)
178
+ - GraphQLMutationMapper-based field transformations with nested objects
179
+ - GraphQL mutations for location upserts with alias batching support
180
+ - Retry logic with exponential backoff
181
+ - StateService duplicate prevention (KV-backed)
182
+ - Error state tracking and file error archival
183
+ - Manual webhook trigger and job status endpoint
184
+
185
+ ---
186
+
187
+ ## When to Use GraphQL Mutations vs Batch API vs Event API
188
+
189
+ ### ✅ Use GraphQL Mutations For:
190
+
191
+ | Entity Type | Use Case | Why GraphQL |
192
+ | -------------- | ---------------------------------------- | ------------------------------------- |
193
+ | **Locations** | Store/warehouse master data (low volume) | Direct control, immediate validation |
194
+ | **Controls** | System configuration, settings | Single operations, complex queries |
195
+ | **Prices** | Price updates (moderate volume) | Immediate feedback, custom logic |
196
+ | **Single Ops** | One-off creates/updates | Testing, debugging, direct API access |
197
+
198
+ ### ❌ Use Event API Instead For:
199
+
200
+ | Entity Type | Use Case | Why Event API |
201
+ | ------------------- | -------------------------------------- | -------------------------------------------- |
202
+ | **Products** | Product catalog sync, variant updates | Triggers workflows, validates business rules |
203
+ | **Customers** | Customer registration, profile updates | Needs workflow for downstream systems |
204
+ | **Orders** | Order creation, status updates | Event-driven fulfillment workflows |
205
+ | **Custom Entities** | Any entity needing workflow triggers | Full Rubix workflow support |
206
+
207
+ ### 🔄 Use Batch API For:
208
+
209
+ | Entity Type | Use Case | Why Batch API |
210
+ | ------------------ | ---------------------------------- | ----------------------------------------------- |
211
+ | **Inventory ONLY** | Bulk inventory updates, daily sync | Optimized for high-volume, BPP change detection |
212
+
213
+ ---
214
+
215
+ ## XML File Format
216
+
217
+ ### Sample: locations.xml
218
+
219
+ ```xml
220
+ <?xml version="1.0" encoding="UTF-8"?>
221
+ <locations>
222
+ <location ref="LOC-001" type="WAREHOUSE">
223
+ <name>Downtown Warehouse</name>
224
+ <address country="USA">
225
+ <street1>123 Main St</street1>
226
+ <street2>Building A</street2>
227
+ <city>New York</city>
228
+ <state>NY</state>
229
+ <postalCode>10001</postalCode>
230
+ </address>
231
+ <coordinates lat="40.7128" lon="-74.0060"/>
232
+ <timeZone>America/New_York</timeZone>
233
+ <openingSchedule>
234
+ <allHours>false</allHours>
235
+ <monStart>800</monStart>
236
+ <monEnd>1800</monEnd>
237
+ <tueStart>800</tueStart>
238
+ <tueEnd>1800</tueEnd>
239
+ <wedStart>800</wedStart>
240
+ <wedEnd>1800</wedEnd>
241
+ <thuStart>800</thuStart>
242
+ <thuEnd>1800</thuEnd>
243
+ <friStart>800</friStart>
244
+ <friEnd>1800</friEnd>
245
+ <satStart>0</satStart>
246
+ <satEnd>0</satEnd>
247
+ <sunStart>0</sunStart>
248
+ <sunEnd>0</sunEnd>
249
+ </openingSchedule>
250
+ </location>
251
+
252
+ <location ref="LOC-002" type="DC">
253
+ <name>Regional DC</name>
254
+ <address country="USA">
255
+ <street1>456 Industrial Pkwy</street1>
256
+ <city>Los Angeles</city>
257
+ <state>CA</state>
258
+ <postalCode>90001</postalCode>
259
+ </address>
260
+ <coordinates lat="34.0522" lon="-118.2437"/>
261
+ <timeZone>America/Los_Angeles</timeZone>
262
+ <openingSchedule>
263
+ <allHours>true</allHours>
264
+ <monStart>0</monStart>
265
+ <monEnd>0</monEnd>
266
+ <tueStart>0</tueStart>
267
+ <tueEnd>0</tueEnd>
268
+ <wedStart>0</wedStart>
269
+ <wedEnd>0</wedEnd>
270
+ <thuStart>0</thuStart>
271
+ <thuEnd>0</thuEnd>
272
+ <friStart>0</friStart>
273
+ <friEnd>0</friEnd>
274
+ <satStart>0</satStart>
275
+ <satEnd>0</satEnd>
276
+ <sunStart>0</sunStart>
277
+ <sunEnd>0</sunEnd>
278
+ </openingSchedule>
279
+ </location>
280
+ </locations>
281
+ ```
282
+
283
+ **XML Path Syntax (with @ prefix for attributes):**
284
+
285
+ - `location.@ref` → XML attribute `ref` on `<location>` element
286
+ - `location.name` → Text content of `<name>` element
287
+ - `location.address.street1` → Nested element path
288
+ - `location.address.@country` → XML attribute on nested `<address>` element
289
+ - `location.coordinates.@lat` → XML attribute for latitude
290
+ - `location.openingSchedule.monStart` → Deeply nested element
291
+
292
+ **Note**: The SDK's `XMLParserService` automatically handles XML attributes using `@` prefix notation.
293
+
294
+ ---
295
+
296
+ ## Versori Workflows Structure
297
+
298
+ **Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
299
+
300
+ **Trigger Types:**
301
+ - **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
302
+ - **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
303
+ - **`workflow()`** → Durable workflows (advanced, rarely used)
304
+
305
+ **Execution Steps (chained to triggers):**
306
+ - **`http()`** → External API calls (chained from schedule/webhook)
307
+ - **`fn()`** → Internal processing (chained from schedule/webhook)
308
+
309
+ ### Recommended Project Structure
310
+
311
+ ```
312
+ s3-xml-location-graphql/
313
+ ├── index.ts # Entry point - exports all workflows
314
+ └── src/
315
+ ├── workflows/
316
+ │ ├── scheduled/
317
+ │ │ └── daily-location-sync.ts # Scheduled: Daily location sync
318
+ │ │
319
+ │ └── webhook/
320
+ │ ├── adhoc-location-sync.ts # Webhook: Manual trigger
321
+ │ └── job-status-check.ts # Webhook: Status query
322
+
323
+ ├── services/
324
+ │ └── location-sync.service.ts # Shared orchestration logic (reusable)
325
+
326
+ └── config/
327
+ └── location-mapping.json # GraphQL mapping config
328
+ ```
329
+
330
+ ---
331
+
332
+ ## Complete Versori Workflow
333
+
334
+ ### Step 1: Package Configuration
335
+
336
+ **File: package.json**
337
+
338
+ ```json
339
+ {
340
+ "name": "versori-s3-xml-location-sync",
341
+ "version": "1.0.0",
342
+ "description": "Versori workflow: S3 XML location sync to Fluent GraphQL",
343
+ "versori": {
344
+ "workflows": "./index.ts"
345
+ },
346
+ "type": "module",
347
+ "scripts": {
348
+ "deploy": "versori deploy",
349
+ "logs": "versori logs"
350
+ },
351
+ "dependencies": {
352
+ "@fluentcommerce/fc-connect-sdk": "^0.1.39",
353
+ "@versori/run": "latest"
354
+ },
355
+ "devDependencies": {
356
+ "typescript": "^5.0.0",
357
+ "@types/node": "^20.0.0"
358
+ }
359
+ }
360
+ ```
361
+
362
+ ### Step 2: Workflow Entry Point (`index.ts`)
363
+
364
+ **Purpose**: Register all workflows with Versori platform
365
+
366
+ ```typescript
367
+ /**
368
+ * Entry Point - Registers all workflows with Versori platform
369
+ *
370
+ * Versori automatically discovers and registers exported workflows
371
+ *
372
+ * File Structure:
373
+ * - src/workflows/scheduled/ → Time-based triggers (cron)
374
+ * - src/workflows/webhook/ → HTTP-based triggers (webhooks)
375
+ */
376
+
377
+ // Scheduled workflows
378
+ export { dailyLocationSync } from './src/workflows/scheduled/daily-location-sync';
379
+
380
+ // Webhook workflows
381
+ export { adhocLocationSync } from './src/workflows/webhook/adhoc-location-sync';
382
+ export { locationSyncJobStatus } from './src/workflows/webhook/job-status-check';
383
+ ```
384
+
385
+ **What Gets Exposed:**
386
+ - ✅ `adhocLocationSync` → `https://{workspace}.versori.run/location-sync-adhoc`
387
+ - ✅ `locationSyncJobStatus` → `https://{workspace}.versori.run/location-sync-job-status`
388
+ - ❌ `dailyLocationSync` → NOT exposed (runs automatically on cron)
389
+
390
+ ---
391
+
392
+ ### Step 3: Workflow Files
393
+
394
+ #### `src/workflows/scheduled/daily-location-sync.ts`
395
+
396
+ **Purpose**: Automatic daily location sync
397
+ **Trigger**: Cron schedule (`0 2 * * *`)
398
+ **Exposed as Endpoint**: ❌ NO - Runs automatically
399
+
400
+ ```typescript
401
+ import { schedule, http } from '@versori/run';
402
+ import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
403
+ import { executeLocationSync } from '../../services/location-sync.service';
404
+
405
+ /**
406
+ * Scheduled Workflow: Daily Location Sync
407
+ *
408
+ * Runs automatically daily at 2 AM UTC
409
+ * NOT exposed as HTTP endpoint - Versori executes on schedule
410
+ *
411
+ * Uses shared service: location-sync.service.ts
412
+ */
413
+ export const dailyLocationSync = schedule(
414
+ 'location-sync-scheduled',
415
+ '0 2 * * *' // Daily at 2 AM UTC
416
+ ).then(
417
+ http('run-location-sync', { connection: 'fluent_commerce' }, async ctx => {
418
+ const startTime = Date.now();
419
+ const { log, openKv } = ctx;
420
+ const jobId = `location-sync-${Date.now()}`;
421
+ const tracker = new JobTracker(openKv(':project:'), log);
422
+
423
+ log.info('🚀 [START] Daily location sync initiated', { jobId, trigger: 'schedule' });
424
+
425
+ await tracker.createJob(jobId, { triggeredBy: 'schedule', stage: 'initialization' });
426
+ await tracker.updateJob(jobId, { status: 'processing' });
427
+
428
+ try {
429
+ log.info('⚙️ [PROCESSING] Starting location synchronization workflow', { jobId });
430
+ const result = await executeLocationSync(ctx, jobId, tracker);
431
+ await tracker.markCompleted(jobId, result);
432
+
433
+ const duration = Date.now() - startTime;
434
+ log.info('✅ [SUCCESS] Daily location sync completed', {
435
+ jobId,
436
+ duration: `${duration}ms`,
437
+ processed: result.processed,
438
+ totalRecords: result.totalRecords
439
+ });
440
+
441
+ return { success: true, jobId, duration, ...result };
442
+ } catch (e: any) {
443
+ await tracker.markFailed(jobId, e);
444
+ const duration = Date.now() - startTime;
445
+
446
+ log.error('❌ [FAILED] Daily location sync failed', {
447
+ jobId,
448
+ duration: `${duration}ms`,
449
+ error: e?.message,
450
+ errorType: e?.name
451
+ });
452
+
453
+ return { success: false, jobId, duration, error: e?.message };
454
+ }
455
+ })
456
+ );
457
+ ```
458
+
459
+ ---
460
+
461
+ #### `src/workflows/webhook/adhoc-location-sync.ts`
462
+
463
+ **Purpose**: Manual location sync trigger (on-demand)
464
+ **Trigger**: HTTP POST
465
+ **Endpoint**: `POST https://{workspace}.versori.run/location-sync-adhoc`
466
+ **Use Cases**: Testing, priority processing, ad-hoc runs
467
+
468
+ ```typescript
469
+ import { webhook, http } from '@versori/run';
470
+ import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
471
+ import { executeLocationSync } from '../../services/location-sync.service';
472
+
473
+ /**
474
+ * Webhook: Manual Location Sync Trigger
475
+ *
476
+ * Endpoint: POST https://{workspace}.versori.run/location-sync-adhoc
477
+ * Request body (optional): { filePattern: "urgent_*.xml", maxFiles: 5 }
478
+ *
479
+ * Pattern: webhook().then(http()) - needs Fluent API access
480
+ * Uses shared service: location-sync.service.ts
481
+ *
482
+ * SECURITY: Authentication handled via connection parameter
483
+ * No manual API key validation needed - Versori manages this via connection auth
484
+ */
485
+ export const adhocLocationSync = webhook('location-sync-adhoc', {
486
+ response: { mode: 'sync' },
487
+ connection: 'location-sync-adhoc', // Versori validates API key
488
+ }).then(
489
+ http('run-location-sync-adhoc', { connection: 'fluent_commerce' }, async ctx => {
490
+ const startTime = Date.now();
491
+ const { log, openKv, data } = ctx;
492
+ const jobId = `location-sync-adhoc-${Date.now()}`;
493
+ const tracker = new JobTracker(openKv(':project:'), log);
494
+
495
+ log.info('🚀 [START] Manual location sync triggered', { jobId, trigger: 'webhook', options: data });
496
+
497
+ await tracker.createJob(jobId, {
498
+ triggeredBy: 'manual',
499
+ stage: 'initialization',
500
+ options: data // Optional: filePattern, maxFiles, etc.
501
+ });
502
+ await tracker.updateJob(jobId, { status: 'processing' });
503
+
504
+ try {
505
+ log.info('⚙️ [PROCESSING] Starting manual location synchronization', { jobId });
506
+ const result = await executeLocationSync(ctx, jobId, tracker);
507
+ await tracker.markCompleted(jobId, result);
508
+
509
+ const duration = Date.now() - startTime;
510
+ log.info('✅ [SUCCESS] Manual location sync completed', {
511
+ jobId,
512
+ duration: `${duration}ms`,
513
+ processed: result.processed,
514
+ totalRecords: result.totalRecords
515
+ });
516
+
517
+ return { success: true, jobId, duration, ...result };
518
+ } catch (e: any) {
519
+ await tracker.markFailed(jobId, e);
520
+ const duration = Date.now() - startTime;
521
+
522
+ log.error('❌ [FAILED] Manual location sync failed', {
523
+ jobId,
524
+ duration: `${duration}ms`,
525
+ error: e?.message,
526
+ errorType: e?.name
527
+ });
528
+
529
+ return { success: false, jobId, duration, error: e?.message };
530
+ }
531
+ })
532
+ );
533
+ ```
534
+
535
+ ---
536
+
537
+ #### `src/workflows/webhook/job-status-check.ts`
538
+
539
+ **Purpose**: Query job status
540
+ **Trigger**: HTTP POST
541
+ **Endpoint**: `POST https://{workspace}.versori.run/location-sync-job-status`
542
+ **Request body**: `{ "jobId": "location-sync-1234567890" }`
543
+
544
+ ```typescript
545
+ import { webhook, fn } from '@versori/run';
546
+ import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
547
+
548
+ /**
549
+ * Webhook: Job Status Check
550
+ *
551
+ * Endpoint: POST https://{workspace}.versori.run/location-sync-job-status
552
+ * Request body: { "jobId": "location-sync-1234567890" }
553
+ *
554
+ * Pattern: webhook().then(fn()) - no external API needed, only KV storage
555
+ * Lightweight: Only queries KV store, no Fluent API calls
556
+ *
557
+ * SECURITY: Authentication handled via connection parameter
558
+ * No manual API key validation needed - Versori manages this via connection auth
559
+ */
560
+ export const locationSyncJobStatus = webhook('location-sync-job-status', {
561
+ response: { mode: 'sync' },
562
+ connection: 'location-sync-job-status',
563
+ }).then(
564
+ fn('status', async ctx => {
565
+ const { data, log, openKv } = ctx;
566
+ const jobId = data?.jobId as string;
567
+
568
+ if (!jobId) {
569
+ return { success: false, error: 'jobId required' };
570
+ }
571
+
572
+ const tracker = new JobTracker(openKv(':project:'), log);
573
+ const status = await tracker.getJob(jobId);
574
+
575
+ return status
576
+ ? { success: true, jobId, ...status }
577
+ : { success: false, error: 'Job not found', jobId };
578
+ })
579
+ );
580
+ ```
581
+
582
+ ---
583
+
584
+ ### Step 4: Main Orchestration Service (`src/services/location-sync.service.ts`)
585
+
586
+ **Note:** This service file should contain the `executeLocationSync` function (renamed from `runLocationXmlWorkflow`). The main workflow logic should be moved here.
587
+
588
+ ```typescript
589
+ /**
590
+ * Main Orchestration Service: Location Sync
591
+ *
592
+ * This service contains the core business logic for location synchronization.
593
+ *
594
+ * Features:
595
+ * - S3 file operations with archival-based deduplication (moveFile to processed/)
596
+ * - XML parsing with @ prefix for attributes
597
+ * - Single element → array normalization
598
+ * - GraphQLMutationMapper for nested field transformations
599
+ * - GraphQL mutations with alias batching support
600
+ * - StateService for metadata tracking (secondary to archival)
601
+ * - Error state tracking with exponential backoff
602
+ *
603
+ * Deduplication Strategy:
604
+ * - PRIMARY: S3 archival via moveFile() - Files in processed/ won't be re-listed
605
+ * - SECONDARY: StateService KV tracking - Provides metadata and processing history
606
+ */
607
+ import { Buffer } from 'node:buffer'; // Required for Deno/Versori runtime
608
+ import {
609
+ createClient,
610
+ S3DataSource,
611
+ XMLParserService,
612
+ GraphQLMutationMapper,
613
+ StateService,
614
+ VersoriKVAdapter,
615
+ JobTracker,
616
+ FluentClient,
617
+ } from '@fluentcommerce/fc-connect-sdk';
618
+
619
+ // ============================================================================
620
+ // Type Definitions
621
+ // ============================================================================
622
+
623
+ interface FileProcessingResult {
624
+ success: boolean;
625
+ locations: any[];
626
+ errors: string[];
627
+ }
628
+
629
+ interface MutationResult {
630
+ successful: number;
631
+ failed: number;
632
+ errors: string[];
633
+ }
634
+
635
+ interface MutationLogEntry {
636
+ timestamp: string;
637
+ fileName: string;
638
+ locationRef: string;
639
+ status: 'success' | 'failure';
640
+ error?: string;
641
+ }
642
+
643
+ // ============================================================================
644
+ // Utility Functions
645
+ // ============================================================================
646
+
647
+ /**
648
+ * Retry utility with exponential backoff
649
+ * (User-defined - not part of SDK public API)
650
+ */
651
+ async function retryWithBackoff<T>(
652
+ operation: () => Promise<T>,
653
+ maxRetries = 3,
654
+ baseDelayMs = 1000
655
+ ): Promise<T> {
656
+ let lastError: any;
657
+ for (let attempt = 0; attempt < maxRetries; attempt++) {
658
+ try {
659
+ return await operation();
660
+ } catch (error) {
661
+ lastError = error;
662
+ if (attempt < maxRetries - 1) {
663
+ const delay = baseDelayMs * Math.pow(2, attempt) + Math.floor(Math.random() * 200);
664
+ await new Promise(resolve => setTimeout(resolve, delay));
665
+ }
666
+ }
667
+ }
668
+ throw lastError;
669
+ }
670
+
671
+ /**
672
+ * Rate limiter for GraphQL mutations
673
+ */
674
+ async function rateLimitedMutation(operation: () => Promise<any>, delayMs: number): Promise<any> {
675
+ const result = await operation();
676
+ await new Promise(resolve => setTimeout(resolve, delayMs));
677
+ return result;
678
+ }
679
+
680
+ // ============================================================================
681
+ // Service Functions
682
+ // ============================================================================
683
+
684
+ /**
685
+ * Service Function 1: Process File
686
+ *
687
+ * Downloads XML from S3, parses with XMLParserService, normalizes arrays,
688
+ * and maps fields with GraphQLMutationMapper.
689
+ *
690
+ * @param s3 - S3DataSource instance
691
+ * @param parser - XMLParserService instance
692
+ * @param mapper - GraphQLMutationMapper instance
693
+ * @param filePath - Full S3 path to file
694
+ * @param fileName - File name only (for logging)
695
+ * @param log - Logger instance
696
+ * @returns FileProcessingResult with locations array and errors
697
+ */
698
+ async function processFile(
699
+ s3: S3DataSource,
700
+ parser: XMLParserService,
701
+ mapper: GraphQLMutationMapper,
702
+ filePath: string,
703
+ fileName: string,
704
+ log: any
705
+ ): Promise<FileProcessingResult> {
706
+ try {
707
+ log.info('Processing file', { fileName });
708
+
709
+ // Download with retry
710
+ const content = await retryWithBackoff(
711
+ () => s3.downloadFile(filePath, { encoding: 'utf8' }) as Promise<string>
712
+ );
713
+
714
+ // Parse XML
715
+ const xmlData = await parser.parse(content);
716
+
717
+ // Extract location array (handle both single and multiple locations)
718
+ // CRITICAL: XML array normalization - single <location> becomes object, not array
719
+ const locationsData = xmlData.locations?.location;
720
+ const locationsArray = Array.isArray(locationsData) ? locationsData : [locationsData];
721
+
722
+ if (!locationsArray || locationsArray.length === 0) {
723
+ log.warn('Empty file (no locations)', { fileName });
724
+ return {
725
+ success: true,
726
+ locations: [],
727
+ errors: [],
728
+ };
729
+ }
730
+
731
+ // Map each location using GraphQLMutationMapper
732
+ const mappedLocations: Array<{ query: string; variables: any; input: any }> = [];
733
+ const mappingErrors: string[] = [];
734
+
735
+ // ✅ PRODUCTION ENHANCEMENT: Log transformation start
736
+ log.info('Transforming locations to GraphQL mutations', {
737
+ fileName,
738
+ totalLocations: locationsArray.length,
739
+ });
740
+
741
+ for (let i = 0; i < locationsArray.length; i++) {
742
+ const locationNumber = i + 1;
743
+
744
+ // ✅ PRODUCTION ENHANCEMENT: Log progress every 50 locations
745
+ if (locationNumber % 50 === 0) {
746
+ log.info(`📤 Transforming location ${locationNumber}/${locationsArray.length}`, {
747
+ fileName,
748
+ locationNumber,
749
+ totalLocations: locationsArray.length,
750
+ validSoFar: mappedLocations.length,
751
+ errorsSoFar: mappingErrors.length,
752
+ progress: `${((locationNumber / locationsArray.length) * 100).toFixed(1)}%`,
753
+ });
754
+ }
755
+
756
+ // Wrap location in context for mapping
757
+ const record = { location: locationsArray[i] };
758
+ try {
759
+ // GraphQLMutationMapper returns { query, variables } directly
760
+ const mappingResult = await mapper.map(record);
761
+
762
+ mappedLocations.push({
763
+ query: mappingResult.query,
764
+ variables: mappingResult.variables,
765
+ input: mappingResult.variables.input || mappingResult.variables,
766
+ });
767
+ } catch (error: unknown) {
768
+ const errorMsg = error instanceof Error ? error.message : String(error);
769
+ mappingErrors.push(`Location ${locationNumber}: ${errorMsg}`);
770
+ log.warn('Mapping failed for location', { fileName, index: i, error: errorMsg });
771
+ }
772
+ }
773
+
774
+ log.info('File processed', {
775
+ fileName,
776
+ total: locationsArray.length,
777
+ mapped: mappedLocations.length,
778
+ errors: mappingErrors.length,
779
+ successRate: `${((mappedLocations.length / locationsArray.length) * 100).toFixed(1)}%`,
780
+ });
781
+
782
+ return {
783
+ success: true,
784
+ locations: mappedLocations,
785
+ errors: mappingErrors,
786
+ };
787
+ } catch (error: any) {
788
+ // ✅ Enhanced error logging: Extract all error details for visibility
789
+ const errorDetails = {
790
+ message: error?.message || 'Unknown error',
791
+ stack: error?.stack,
792
+ fileName: error?.fileName,
793
+ lineNumber: error?.lineNumber,
794
+ originalError: error?.context?.originalError?.message,
795
+ errorType: error?.name || 'Error',
796
+ };
797
+ log.error('File processing failed', errorDetails, { fileName });
798
+ return {
799
+ success: false,
800
+ locations: [],
801
+ errors: [error.message || 'Unknown error'],
802
+ };
803
+ }
804
+ }
805
+
806
+ /**
807
+ * Service Function 2: Execute Mutations
808
+ *
809
+ * Executes GraphQL createLocation mutations with alias batching support.
810
+ *
811
+ * @param client - FluentClient instance
812
+ * @param mapper - GraphQLMutationMapper instance
813
+ * @param locations - Array of mapped location objects with query and variables
814
+ * @param log - Logger instance
815
+ * @param retailerId - Fluent retailer ID
816
+ * @param batchSize - Number of concurrent requests (default: 1)
817
+ * @param mutationsPerAliasBatch - Optional: Number of mutations per aliased request (default: undefined)
818
+ * @returns MutationResult with success/failure counts
819
+ */
820
+ async function executeMutations(
821
+ client: FluentClient,
822
+ mapper: GraphQLMutationMapper,
823
+ locations: Array<{ query: string; variables: any; input: any }>,
824
+ log: any,
825
+ retailerId: string,
826
+ batchSize: number = 1, // ✅ Default: 1 (sequential)
827
+ mutationsPerAliasBatch?: number // ✅ NEW: Alias batching parameter (default: undefined = disabled)
828
+ ): Promise<MutationResult> {
829
+ // Determine mode: use aliases if mutationsPerAliasBatch is set and > 1
830
+ const useAliases = mutationsPerAliasBatch && mutationsPerAliasBatch > 1;
831
+
832
+ if (useAliases) {
833
+ return await executeMutationsWithAliases(
834
+ client,
835
+ mapper,
836
+ locations,
837
+ log,
838
+ retailerId,
839
+ batchSize,
840
+ mutationsPerAliasBatch!
841
+ );
842
+ } else {
843
+ return await executeMutationsSeparate(
844
+ client,
845
+ locations,
846
+ log,
847
+ batchSize
848
+ );
849
+ }
850
+ }
851
+
852
+ /**
853
+ * Execute mutations using separate concurrent requests (current mode)
854
+ */
855
+ async function executeMutationsSeparate(
856
+ client: FluentClient,
857
+ locations: Array<{ query: string; variables: any; input: any }>,
858
+ log: any,
859
+ batchSize: number
860
+ ): Promise<MutationResult> {
861
+ const result: MutationResult = {
862
+ successful: 0,
863
+ failed: 0,
864
+ errors: [],
865
+ };
866
+
867
+ const safeConc = Math.max(1, Math.floor(batchSize));
868
+
869
+ // Sequential mode
870
+ if (safeConc === 1) {
871
+ for (const location of locations) {
872
+ try {
873
+ await retryWithBackoff(() =>
874
+ client.graphql({
875
+ query: location.query,
876
+ variables: location.variables,
877
+ })
878
+ );
879
+ result.successful++;
880
+ } catch (error: unknown) {
881
+ result.failed++;
882
+ const errorMsg = error instanceof Error ? error.message : String(error);
883
+ result.errors.push(errorMsg);
884
+ }
885
+ }
886
+ return result;
887
+ }
888
+
889
+ // Parallel mode
890
+ for (let i = 0; i < locations.length; i += safeConc) {
891
+ const chunk = locations.slice(i, i + safeConc);
892
+ const results = await Promise.allSettled(
893
+ chunk.map(loc =>
894
+ retryWithBackoff(() =>
895
+ client.graphql({
896
+ query: loc.query,
897
+ variables: loc.variables,
898
+ })
899
+ )
900
+ )
901
+ );
902
+
903
+ results.forEach((settledResult, idx) => {
904
+ if (settledResult.status === 'fulfilled') {
905
+ result.successful++;
906
+ } else {
907
+ result.failed++;
908
+ result.errors.push(
909
+ settledResult.reason instanceof Error ? settledResult.reason.message : String(settledResult.reason)
910
+ );
911
+ }
912
+ });
913
+ }
914
+
915
+ return result;
916
+ }
917
+
918
+ /**
919
+ * ✅ NEW: Execute mutations using GraphQL aliases (batched requests)
920
+ */
921
+ async function executeMutationsWithAliases(
922
+ client: FluentClient,
923
+ mapper: GraphQLMutationMapper,
924
+ locations: Array<{ query: string; variables: any; input: any }>,
925
+ log: any,
926
+ retailerId: string,
927
+ maxParallel: number,
928
+ mutationsPerAliasBatch: number
929
+ ): Promise<MutationResult> {
930
+ const result: MutationResult = { successful: 0, failed: 0, errors: [] };
931
+
932
+ const mutationName = (mapper as any).config.mutation || 'createLocation';
933
+ const aliasBatches: Array<Array<typeof locations[0]>> = [];
934
+
935
+ for (let i = 0; i < locations.length; i += mutationsPerAliasBatch) {
936
+ aliasBatches.push(locations.slice(i, i + mutationsPerAliasBatch));
937
+ }
938
+
939
+ // Process batches with concurrency control
940
+ for (let i = 0; i < aliasBatches.length; i += maxParallel) {
941
+ const concurrentBatches = aliasBatches.slice(i, i + maxParallel);
942
+
943
+ const batchResults = await Promise.allSettled(
944
+ concurrentBatches.map(async (batch) => {
945
+ const { query, variables } = buildAliasedBatch(batch, mutationName, retailerId);
946
+ const response = await retryWithBackoff(() => client.graphql({ query, variables }));
947
+ return parseAliasResponse(response, batch, mutationName);
948
+ })
949
+ );
950
+
951
+ batchResults.forEach((batchResult, idx) => {
952
+ if (batchResult.status === 'fulfilled') {
953
+ const batchRes = batchResult.value;
954
+ result.successful += batchRes.executed;
955
+ result.failed += batchRes.failed;
956
+ result.errors.push(...batchRes.errors);
957
+ } else {
958
+ const batch = concurrentBatches[idx];
959
+ const errorMsg = batchResult.reason instanceof Error ? batchResult.reason.message : String(batchResult.reason);
960
+ batch.forEach(loc => {
961
+ result.failed++;
962
+ result.errors.push(`Batch execution failed: ${errorMsg}`);
963
+ });
964
+ }
965
+ });
966
+
967
+ if (i + maxParallel < aliasBatches.length) {
968
+ await new Promise(resolve => setTimeout(resolve, 500));
969
+ }
970
+ }
971
+
972
+ return result;
973
+ }
974
+
975
+ /**
976
+ * ✅ NEW: Build aliased batch query and variables
977
+ */
978
+ function buildAliasedBatch(
979
+ batch: Array<{ query: string; variables: any; input: any }>,
980
+ mutationName: string,
981
+ retailerId: string
982
+ ): { query: string; variables: Record<string, any> } {
983
+ const batchSize = batch.length;
984
+ const inputTypeName = `${mutationName.charAt(0).toUpperCase() + mutationName.slice(1)}Input`;
985
+
986
+ const variables = Array.from({ length: batchSize }, (_, i) =>
987
+ `$input${i + 1}: ${inputTypeName}!`
988
+ ).join(', ');
989
+
990
+ const aliasedMutations = Array.from({ length: batchSize }, (_, i) => {
991
+ const alias = `${mutationName}${i + 1}`;
992
+ return ` ${alias}: ${mutationName}(input: $input${i + 1}) { id ref name }`;
993
+ }).join('\n');
994
+
995
+ const operationName = `Batch${mutationName.charAt(0).toUpperCase() + mutationName.slice(1)}s`;
996
+ const query = `mutation ${operationName}(${variables}) {\n${aliasedMutations}\n}`;
997
+
998
+ const variablesObj: Record<string, any> = {};
999
+ batch.forEach((loc, index) => {
1000
+ const input = loc.variables.input || loc.variables;
1001
+ if (input && !input.retailer) {
1002
+ input.retailer = { id: parseInt(retailerId) };
1003
+ }
1004
+ variablesObj[`input${index + 1}`] = input;
1005
+ });
1006
+
1007
+ return { query, variables: variablesObj };
1008
+ }
1009
+
1010
+ /**
1011
+ * ✅ NEW: Parse aliased GraphQL response
1012
+ */
1013
+ function parseAliasResponse(
1014
+ response: any,
1015
+ batch: Array<{ query: string; variables: any; input: any }>,
1016
+ mutationName: string
1017
+ ): { executed: number; failed: number; errors: string[] } {
1018
+ const result = { executed: 0, failed: 0, errors: [] as string[] };
1019
+
1020
+ const data = response.data || {};
1021
+ const errors = response.errors || [];
1022
+
1023
+ batch.forEach((loc, index) => {
1024
+ const alias = `${mutationName}${index + 1}`;
1025
+ const aliasData = data[alias];
1026
+ const aliasErrors = errors.filter((e: any) =>
1027
+ e.path && Array.isArray(e.path) && e.path.includes(alias)
1028
+ );
1029
+
1030
+ if (aliasData && !aliasErrors.length) {
1031
+ result.executed++;
1032
+ } else {
1033
+ result.failed++;
1034
+ const errorMsg = aliasErrors[0]?.message || 'Mutation failed';
1035
+ result.errors.push(`${loc.input?.ref || 'unknown'}: ${errorMsg}`);
1036
+ }
1037
+ });
1038
+
1039
+ return result;
1040
+ }
1041
+
1042
+ /**
1043
+ * Service Function 3: Write Mutation Log
1044
+ *
1045
+ * Writes mutation results to S3 as a JSON log file.
1046
+ *
1047
+ * @param s3 - S3DataSource instance
1048
+ * @param logEntries - Array of mutation log entries
1049
+ * @param fileName - Original file name (used to generate log path)
1050
+ * @param logPrefix - S3 prefix for logs (e.g., 'logs/')
1051
+ * @param log - Logger instance
1052
+ */
1053
+ async function writeMutationLog(
1054
+ s3: S3DataSource,
1055
+ logEntries: MutationLogEntry[],
1056
+ fileName: string,
1057
+ logPrefix: string,
1058
+ log: any
1059
+ ): Promise<void> {
1060
+ try {
1061
+ const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
1062
+ const logFileName = `${logPrefix}${fileName.replace('.xml', '')}_${timestamp}.json`;
1063
+
1064
+ const logContent = JSON.stringify(
1065
+ {
1066
+ fileName,
1067
+ timestamp: new Date().toISOString(),
1068
+ totalMutations: logEntries.length,
1069
+ successful: logEntries.filter(e => e.status === 'success').length,
1070
+ failed: logEntries.filter(e => e.status === 'failure').length,
1071
+ entries: logEntries,
1072
+ },
1073
+ null,
1074
+ 2
1075
+ );
1076
+
1077
+ // Write to S3 (uploadFile accepts string or Buffer)
1078
+ await s3.uploadFile(logFileName, logContent);
1079
+
1080
+ log.info('Mutation log written', { logFileName, entries: logEntries.length });
1081
+ } catch (error: any) {
1082
+ // ✅ Enhanced error logging: Extract all error details for visibility
1083
+ const errorDetails = {
1084
+ message: error?.message || 'Unknown error',
1085
+ stack: error?.stack,
1086
+ errorType: error?.name || 'Error',
1087
+ };
1088
+ log.error('Failed to write mutation log', errorDetails, { fileName });
1089
+ // Don't throw - logging failure shouldn't stop workflow
1090
+ }
1091
+ }
1092
+
1093
+ // ============================================================================
1094
+ // Main Workflow Function
1095
+ // ============================================================================
1096
+
1097
+ /**
1098
+ * Main Orchestration Function: Execute Location Sync
1099
+ *
1100
+ * This function orchestrates the location synchronization workflow.
1101
+ *
1102
+ * Architecture:
1103
+ * 1. List files from S3
1104
+ * 2. For each file:
1105
+ * a. processFile() - Download, parse, map
1106
+ * b. executeMutations() - Send GraphQL mutations with rate limiting
1107
+ * c. writeMutationLog() - Log results to S3
1108
+ * d. Archive file (primary deduplication)
1109
+ * e. Mark processed in KV (metadata tracking)
1110
+ *
1111
+ * @param ctx - Versori context
1112
+ * @param jobId - Job identifier
1113
+ * @param tracker - JobTracker instance
1114
+ * @returns Processing result
1115
+ */
1116
+ export async function executeLocationSync(ctx: any, jobId: string, tracker: JobTracker) {
1117
+ const { log, activation } = ctx;
1118
+
1119
+ log.info('📋 [INIT] Reading activation variables', { jobId });
1120
+
1121
+ // Read activation variables
1122
+ const s3Bucket = activation?.getVariable('s3BucketName');
1123
+ const s3Region = activation?.getVariable('awsRegion') || 'us-east-1';
1124
+ const s3AccessKeyId = activation?.getVariable('awsAccessKeyId');
1125
+ const s3SecretAccessKey = activation?.getVariable('awsSecretAccessKey');
1126
+ const s3Prefix = activation?.getVariable('s3Prefix') || 'locations/';
1127
+ const archivePrefix = activation?.getVariable('archivePrefix') || 'processed/';
1128
+ const errorPrefix = activation?.getVariable('errorPrefix') || 'errors/';
1129
+ const logPrefix = activation?.getVariable('logPrefix') || 'logs/';
1130
+ const filePattern = (activation?.getVariable('filePattern') || '.xml').toLowerCase();
1131
+ const maxFiles = parseInt(activation?.getVariable('maxFilesToProcess') || '10', 10);
1132
+ const retailerId = activation?.getVariable('retailerId'); // Optional: Only if mutation schema requires it
1133
+ const enableArchival = activation?.getVariable('enableArchival') !== 'false';
1134
+ const enableMutationLogs = activation?.getVariable('enableMutationLogs') !== 'false';
1135
+ const enableFileTracking = activation?.getVariable('enableFileTracking') !== 'false';
1136
+
1137
+ // ✅ Configuration with defaults
1138
+ const mutationBatchSize = parseInt(
1139
+ activation?.getVariable('mutationBatchSize') || '1', // ✅ Default: 1 (sequential)
1140
+ 10
1141
+ );
1142
+
1143
+ const mutationsPerAliasBatch = activation?.getVariable('mutationsPerAliasBatch')
1144
+ ? parseInt(activation?.getVariable('mutationsPerAliasBatch') || '1', 10)
1145
+ : undefined; // ✅ Default: undefined (disabled, use separate requests)
1146
+
1147
+ // Validate required variables
1148
+ const missingVars: string[] = [];
1149
+ if (!s3Bucket) missingVars.push('s3BucketName');
1150
+ if (!s3AccessKeyId) missingVars.push('awsAccessKeyId');
1151
+ if (!s3SecretAccessKey) missingVars.push('awsSecretAccessKey');
1152
+ // Note: retailerId is optional - only needed if mutation schema requires it
1153
+
1154
+ if (missingVars.length > 0) {
1155
+ const errorMsg = `Missing required variables: ${missingVars.join(', ')}`;
1156
+ log.error('❌ [VALIDATION] Missing required activation variables', {
1157
+ missingVars,
1158
+ recommendation: 'Add missing variables in Versori activation settings'
1159
+ });
1160
+ return { success: false, error: errorMsg, processed: 0 };
1161
+ }
1162
+
1163
+ log.info('✅ [VALIDATION] All required variables present', {
1164
+ s3Bucket,
1165
+ s3Region,
1166
+ s3Prefix,
1167
+ enableFileTracking,
1168
+ mutationBatchSize
1169
+ });
1170
+
1171
+ try {
1172
+ log.info('🔧 [INIT] Initializing Fluent Commerce client', { jobId });
1173
+
1174
+ // Initialize services with validateConnection
1175
+ const client = await createClient(ctx, { validateConnection: true });
1176
+ if (!client) {
1177
+ throw new Error('Failed to create Fluent Commerce client');
1178
+ }
1179
+
1180
+ log.info('✅ [INIT] Fluent Commerce client validated and ready', { jobId });
1181
+
1182
+ // ✅ CORRECT: GraphQL mutations do NOT need client.setRetailerId()
1183
+ // setRetailerId() is only for Job/Event API, NOT GraphQL
1184
+ // Check your GraphQL schema to determine retailerId handling:
1185
+ // - Mandatory retailerId → Must pass it in mutation input
1186
+ // - Optional retailerId → Can pass it if needed
1187
+ // - No retailerId field → Don't pass it
1188
+ // See: fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md
1189
+
1190
+ log.info('🗄️ [INIT] Initializing S3 data source', { s3Bucket, s3Region, s3Prefix });
1191
+
1192
+ const s3 = new S3DataSource(
1193
+ {
1194
+ type: 'S3_XML',
1195
+ connectionId: 's3-location-sync',
1196
+ name: 'Source S3',
1197
+ s3Config: {
1198
+ bucket: s3Bucket,
1199
+ region: s3Region,
1200
+ accessKeyId: s3AccessKeyId,
1201
+ secretAccessKey: s3SecretAccessKey,
1202
+ },
1203
+ },
1204
+ log
1205
+ );
1206
+
1207
+ const parser = new XMLParserService();
1208
+
1209
+ // Initialize state tracking (only if enabled)
1210
+ let stateService: StateService | null = null;
1211
+ if (enableFileTracking) {
1212
+ log.info('🔄 [INIT] Enabling file tracking with StateService', { jobId });
1213
+ const stateKV = new VersoriKVAdapter(ctx);
1214
+ stateService = new StateService(stateKV);
1215
+ } else {
1216
+ log.info('⏭️ [INIT] File tracking disabled - relying on S3 archival only', { jobId });
1217
+ }
1218
+
1219
+ log.info('📝 [INIT] Loading mapping configuration', { jobId });
1220
+
1221
+ // ✅ CRITICAL: Load mapping config from external JSON file
1222
+ // Mapping config uses GraphQLMutationMapper structure (nested objects, not dot notation)
1223
+ // File: src/config/location-mapping.json
1224
+ const mappingConfigJson = await import('../config/location-mapping.json', { assert: { type: 'json' } });
1225
+ const mappingConfig = mappingConfigJson.default;
1226
+
1227
+ // Initialize GraphQLMutationMapper with client for schema introspection
1228
+ const mapper = new GraphQLMutationMapper(mappingConfig, log, { fluentClient: client });
1229
+
1230
+ log.info('📂 [S3] Listing files from S3', { s3Bucket, s3Prefix, filePattern });
1231
+
1232
+ // List files (pattern filtering handled by listFiles)
1233
+ const files = await s3.listFiles({
1234
+ prefix: s3Prefix,
1235
+ pattern: filePattern,
1236
+ maxKeys: 1000
1237
+ });
1238
+
1239
+ // Newest-first ordering
1240
+ const xmlFiles = files
1241
+ .sort((a: any, b: any) => {
1242
+ const aTime = a.lastModified ? new Date(a.lastModified).getTime() : 0;
1243
+ const bTime = b.lastModified ? new Date(b.lastModified).getTime() : 0;
1244
+ return bTime - aTime;
1245
+ })
1246
+ .slice(0, maxFiles);
1247
+
1248
+ log.info('📊 [S3] File discovery complete', {
1249
+ totalFiles: files.length,
1250
+ xmlFiles: xmlFiles.length,
1251
+ maxFiles,
1252
+ selectedFiles: xmlFiles.map(f => f.name)
1253
+ });
1254
+
1255
+ const results = {
1256
+ processed: 0,
1257
+ skipped: 0,
1258
+ failed: 0,
1259
+ totalRecords: 0,
1260
+ errors: [] as string[],
1261
+ };
1262
+
1263
+ log.info('🔄 [PROCESSING] Starting file processing loop', {
1264
+ fileCount: xmlFiles.length,
1265
+ jobId
1266
+ });
1267
+
1268
+ // Per-file processing loop
1269
+ for (const file of xmlFiles) {
1270
+ const filePath = file.path;
1271
+ const fileName = file.name;
1272
+
1273
+ log.info('📄 [FILE] Processing file', { fileName, filePath });
1274
+
1275
+ // Duplicate prevention (secondary check - files in processed/ won't be listed)
1276
+ // Primary deduplication: S3 archival (files moved to processed/ subdirectory)
1277
+ if (enableFileTracking && stateService) {
1278
+ const wasProcessed = await stateService.isFileProcessed(fileName);
1279
+ if (wasProcessed) {
1280
+ log.info('⏭️ [SKIP] File already processed (KV check)', { fileName });
1281
+ results.skipped++;
1282
+ continue;
1283
+ }
1284
+ }
1285
+
1286
+ try {
1287
+ // Step 1: Process file (download, parse, map)
1288
+ log.info('📥 [DOWNLOAD] Downloading and parsing file', { fileName });
1289
+ const processingResult = await processFile(s3, parser, mapper, filePath, fileName, log);
1290
+
1291
+ if (!processingResult.success) {
1292
+ throw new Error(`File processing failed: ${processingResult.errors.join(', ')}`);
1293
+ }
1294
+
1295
+ if (processingResult.locations.length === 0) {
1296
+ log.warn('⚠️ [SKIP] Empty file detected, archiving', { fileName });
1297
+ if (enableArchival) {
1298
+ await s3.moveFile(filePath, `${archivePrefix}${fileName}`);
1299
+ log.info('📦 [ARCHIVE] Empty file archived', { fileName, destination: `${archivePrefix}${fileName}` });
1300
+ }
1301
+ results.skipped++;
1302
+ continue;
1303
+ }
1304
+
1305
+ log.info('✅ [PARSE] File parsed successfully', {
1306
+ fileName,
1307
+ locationCount: processingResult.locations.length,
1308
+ mappingErrors: processingResult.errors.length
1309
+ });
1310
+
1311
+ // Step 2: Execute mutations
1312
+ // Step 2: Execute mutations with alias batching support
1313
+ // ? Enhanced: Extract context for progress logging
1314
+ const sampleLocationRefs = processingResult.locations.slice(0, 5).map((loc: any) => loc.input?.ref || loc.ref || 'unknown');
1315
+ const mutationType = mapper?.mutationName || 'createLocation';
1316
+
1317
+ // ? Enhanced: Start logging with context
1318
+ log.info(`[GraphQLMutations] Sending mutations for file "${fileName}"`, {
1319
+ totalMutations: processingResult.locations.length,
1320
+ mutationType,
1321
+ batchSize: mutationBatchSize,
1322
+ batchMode: mutationBatchSize === 1 ? 'sequential' : `parallel (${mutationBatchSize})`,
1323
+ sampleLocationRefs: sampleLocationRefs.join(', '),
1324
+ aliasBatching: mutationsPerAliasBatch ? `enabled (${mutationsPerAliasBatch} per alias)` : 'disabled'
1325
+ });
1326
+
1327
+ const mutationResult = await executeMutations(
1328
+ client,
1329
+ mapper,
1330
+ processingResult.locations,
1331
+ log,
1332
+ retailerId, // Pass retailerId for mutations that require it in input
1333
+ mutationBatchSize, // Concurrency control (default: 1)
1334
+ mutationsPerAliasBatch // ✅ NEW: Alias batching (default: undefined)
1335
+ );
1336
+
1337
+ // ? Enhanced: Completion logging with summary
1338
+ log.info(`[GraphQLMutations] Mutation submission completed for file "${fileName}"`, {
1339
+ totalMutations: processingResult.locations.length,
1340
+ successful: mutationResult.successful,
1341
+ failed: mutationResult.failed,
1342
+ successRate: processingResult.locations.length > 0 ? `${Math.round((mutationResult.successful / processingResult.locations.length) * 100)}%` : '0%',
1343
+ mutationType
1344
+ });
1345
+
1346
+ // Step 3: Write mutation log (if enabled)
1347
+ if (enableMutationLogs) {
1348
+ const logEntries: MutationLogEntry[] = processingResult.locations.map(loc => {
1349
+ const failed = mutationResult.errors.find(e => e.startsWith(loc.ref));
1350
+ return {
1351
+ timestamp: new Date().toISOString(),
1352
+ fileName,
1353
+ locationRef: loc.ref,
1354
+ status: failed ? 'failure' : 'success',
1355
+ error: failed,
1356
+ };
1357
+ });
1358
+
1359
+ await writeMutationLog(s3, logEntries, fileName, logPrefix, log);
1360
+ }
1361
+
1362
+ // Step 4: Archive file (PRIMARY deduplication - file won't be listed again)
1363
+ if (enableArchival) {
1364
+ log.info('📦 [ARCHIVE] Moving file to processed directory', { fileName });
1365
+ await s3.moveFile(filePath, `${archivePrefix}${fileName}`);
1366
+ log.info('✅ [ARCHIVE] File archived successfully', {
1367
+ fileName,
1368
+ destination: `${archivePrefix}${fileName}`
1369
+ });
1370
+ }
1371
+
1372
+ // Step 5: Mark processed in KV (SECONDARY - provides metadata/history)
1373
+ if (enableFileTracking && stateService) {
1374
+ log.info('💾 [STATE] Recording file processing metadata', { fileName });
1375
+ await stateService.markFileProcessed(fileName, {
1376
+ recordCount: processingResult.locations.length,
1377
+ successful: mutationResult.successful,
1378
+ failed: mutationResult.failed,
1379
+ mappingErrors: processingResult.errors.length,
1380
+ timestamp: new Date().toISOString(),
1381
+ });
1382
+ }
1383
+
1384
+ results.processed++;
1385
+ results.totalRecords += mutationResult.successful;
1386
+
1387
+ log.info('✅ [COMPLETE] File processing complete', {
1388
+ fileName,
1389
+ locations: processingResult.locations.length,
1390
+ successful: mutationResult.successful,
1391
+ failed: mutationResult.failed,
1392
+ mappingErrors: processingResult.errors.length,
1393
+ });
1394
+
1395
+ if (processingResult.errors.length > 0 || mutationResult.errors.length > 0) {
1396
+ results.errors.push(
1397
+ `${fileName}: ${processingResult.errors.length} mapping errors, ${mutationResult.errors.length} mutation errors`
1398
+ );
1399
+ }
1400
+ } catch (error: any) {
1401
+ // ✅ Enhanced error logging: Extract all error details for visibility
1402
+ const errorDetails = {
1403
+ message: error?.message || 'Unknown error',
1404
+ stack: error?.stack,
1405
+ fileName: error?.fileName,
1406
+ lineNumber: error?.lineNumber,
1407
+ originalError: error?.context?.originalError?.message,
1408
+ errorType: error?.name || 'Error',
1409
+ };
1410
+
1411
+ log.error('❌ [ERROR] File processing failed', errorDetails, { fileName });
1412
+
1413
+ // Provide error recommendations based on error type
1414
+ const recommendation = getErrorRecommendation(error);
1415
+ if (recommendation) {
1416
+ log.warn('💡 [RECOMMENDATION]', { fileName, recommendation });
1417
+ }
1418
+
1419
+ results.failed++;
1420
+ results.errors.push(`${fileName}: ${error.message}`);
1421
+
1422
+ // Attempt to move to error directory, ignore failures
1423
+ try {
1424
+ log.info('🗂️ [ERROR] Moving failed file to error directory', { fileName });
1425
+ await s3.moveFile(filePath, `${errorPrefix}${fileName}`);
1426
+ log.info('✅ [ERROR] Failed file moved to error directory', {
1427
+ fileName,
1428
+ destination: `${errorPrefix}${fileName}`
1429
+ });
1430
+ } catch (moveError) {
1431
+ log.warn('⚠️ [ERROR] Failed to move file to error directory', {
1432
+ fileName,
1433
+ moveError: moveError instanceof Error ? moveError.message : String(moveError)
1434
+ });
1435
+ }
1436
+
1437
+ // Track error state with exponential backoff (only if file tracking enabled)
1438
+ if (enableFileTracking && stateService) {
1439
+ try {
1440
+ const stateKV = new VersoriKVAdapter(ctx);
1441
+ const key = ['error-state', fileName];
1442
+ const prev = (await stateKV.get(key))?.value as any;
1443
+ const attempts = (prev?.attemptCount || 0) + 1;
1444
+ const backoffMinutes = Math.min(Math.pow(2, attempts) * 5, 24 * 60);
1445
+ const nextRetryAt = new Date(Date.now() + backoffMinutes * 60000).toISOString();
1446
+
1447
+ await stateKV.set(key, {
1448
+ fileName,
1449
+ attemptCount: attempts,
1450
+ lastError: error?.message || 'unknown',
1451
+ lastAttemptAt: new Date().toISOString(),
1452
+ firstFailedAt: prev?.firstFailedAt || new Date().toISOString(),
1453
+ nextRetryAt,
1454
+ });
1455
+
1456
+ log.info('💾 [ERROR] Error state tracked with exponential backoff', {
1457
+ fileName,
1458
+ attempts,
1459
+ nextRetryAt
1460
+ });
1461
+ } catch (stateError) {
1462
+ log.warn('⚠️ [ERROR] Failed to track error state', {
1463
+ fileName,
1464
+ stateError: stateError instanceof Error ? stateError.message : String(stateError)
1465
+ });
1466
+ }
1467
+ }
1468
+ }
1469
+ }
1470
+
1471
+ log.info('🏁 [COMPLETE] File processing loop finished', {
1472
+ processed: results.processed,
1473
+ skipped: results.skipped,
1474
+ failed: results.failed,
1475
+ totalRecords: results.totalRecords
1476
+ });
1477
+
1478
+ return results;
1479
+ } catch (error: any) {
1480
+ // ✅ Enhanced error logging: Extract all error details for visibility
1481
+ const errorDetails = {
1482
+ message: error?.message || 'Unknown error',
1483
+ stack: error?.stack,
1484
+ errorType: error?.name || 'Error',
1485
+ };
1486
+
1487
+ log.error('❌ [FATAL] Location sync failed', errorDetails);
1488
+
1489
+ // Provide fatal error recommendations
1490
+ const recommendation = getErrorRecommendation(error);
1491
+ if (recommendation) {
1492
+ log.warn('💡 [RECOMMENDATION]', { recommendation });
1493
+ }
1494
+
1495
+ return {
1496
+ success: false,
1497
+ error: error.message,
1498
+ processed: 0,
1499
+ timestamp: new Date().toISOString(),
1500
+ };
1501
+ }
1502
+ }
1503
+
1504
+ /**
1505
+ * Get error recommendation based on error type
1506
+ */
1507
+ function getErrorRecommendation(error: any): string | null {
1508
+ const message = error?.message?.toLowerCase() || '';
1509
+
1510
+ if (message.includes('s3') || message.includes('access denied')) {
1511
+ return 'Check S3 credentials and bucket permissions. Verify IAM policy includes s3:ListBucket, s3:GetObject, s3:PutObject, s3:DeleteObject.';
1512
+ }
1513
+
1514
+ if (message.includes('xml') || message.includes('parse')) {
1515
+ return 'Verify XML file structure matches expected schema. Check for malformed XML or encoding issues.';
1516
+ }
1517
+
1518
+ if (message.includes('mapping') || message.includes('field')) {
1519
+ return 'Review field mapping configuration. Ensure all required fields are present and source paths are correct.';
1520
+ }
1521
+
1522
+ if (message.includes('graphql') || message.includes('mutation')) {
1523
+ return 'Check GraphQL schema and mutation input. Verify all required fields are provided and types match schema.';
1524
+ }
1525
+
1526
+ if (message.includes('auth') || message.includes('401') || message.includes('403')) {
1527
+ return 'Verify Fluent Commerce credentials. Check OAuth2 client ID/secret and ensure connection is active.';
1528
+ }
1529
+
1530
+ if (message.includes('timeout') || message.includes('econnrefused')) {
1531
+ return 'Check network connectivity. Verify API endpoints are accessible and not rate-limited.';
1532
+ }
1533
+
1534
+ return null;
1535
+ }
1536
+ ```
1537
+
1538
+ **Note:** The `executeLocationSync` function should contain the full implementation of `runLocationXmlWorkflow` (renamed to `executeLocationSync`), including all the logic for processing files, executing mutations, and logging. The implementation details are shown in the service function code above.
1539
+
1540
+ ---
1541
+
1542
+ ### Step 5: TypeScript Configuration
1543
+
1544
+ **File: tsconfig.json**
1545
+
1546
+ ```json
1547
+ {
1548
+ "compilerOptions": {
1549
+ "module": "ES2022",
1550
+ "target": "ES2024",
1551
+ "moduleResolution": "node"
1552
+ }
1553
+ }
1554
+ ```
1555
+
1556
+ ---
1557
+
1558
+ ## Code Flow Explanation
1559
+
1560
+ ### Initialization Phase
1561
+
1562
+ 1. **Read activation variables** - S3 config, Fluent config, rate limiting, logging options
1563
+ 2. **Validate required variables** - Fail fast if missing credentials
1564
+ 3. **Initialize SDK services** - FluentClient, S3DataSource, XMLParserService, StateService
1565
+ 4. **Create mapping configuration** - Define XML → GraphQL field mappings with nested objects
1566
+ 5. **Calculate rate limit delay** - Convert mutations/second to delay in milliseconds
1567
+
1568
+ ### File Discovery Phase
1569
+
1570
+ 1. **List S3 files** - Use `s3.listFiles()` with prefix filter (excludes processed/ subdirectory)
1571
+ 2. **Filter by pattern** - Match file extension (e.g., `.xml`)
1572
+ 3. **Sort newest-first** - Process most recent files first
1573
+ 4. **Apply max files limit** - Prevent overwhelming the workflow
1574
+ 5. **Note**: Files in `processed/` subdirectory won't be listed (primary deduplication)
1575
+
1576
+ ### Per-File Processing (Service Functions)
1577
+
1578
+ **Step 1: processFile()** - Download, parse, map
1579
+
1580
+ 1. **Download file** - Use S3DataSource with retry logic
1581
+ 2. **Parse XML** - XMLParserService converts to JavaScript object
1582
+ 3. **Normalize array** - Handle single vs multiple `<location>` elements (CRITICAL for XML)
1583
+ 4. **Map locations** - Use UniversalMapper with nested field mapping
1584
+ 5. **Collect errors** - Track mapping errors without stopping
1585
+ 6. **Return result** - FileProcessingResult with locations array and errors
1586
+
1587
+ **Step 2: executeMutations()** - GraphQL mutations with rate limiting
1588
+
1589
+ 1. **Loop through locations** - Process each location individually
1590
+ 2. **Build mutation input** - Extract nested fields (primaryAddress, openingSchedule)
1591
+ 3. **Execute mutation** - Direct GraphQL `createLocation` with retry logic
1592
+ 4. **Apply rate limiting** - Add configurable delay between mutations
1593
+ 5. **Track results** - Count successful vs failed, collect error messages
1594
+ 6. **Return result** - MutationResult with counts and errors
1595
+
1596
+ **Step 3: writeMutationLog()** - S3 log file (optional)
1597
+
1598
+ 1. **Create log entries** - Map locations to log entries with status
1599
+ 2. **Build JSON log** - Include timestamp, summary, detailed entries
1600
+ 3. **Write to S3** - Use Buffer.from() for Deno/Versori runtime compatibility
1601
+ 4. **Non-blocking** - Logging failure doesn't stop workflow
1602
+ 5. **Timestamped naming** - Unique log file per processed file
1603
+
1604
+ ### Cleanup Phase
1605
+
1606
+ 1. **Archive file FIRST** - Move to `processed/` or `errors/` (PRIMARY deduplication)
1607
+ 2. **Mark processed in KV** - StateService tracks metadata and processing history
1608
+ 3. **Error state tracking** - Store error info with exponential backoff timestamp
1609
+ 4. **Return results** - Summary of processed, skipped, failed files
1610
+
1611
+ **Note**: The order matters! Archive first (primary deduplication), then KV tracking (metadata/history).
1612
+
1613
+ ### Service Function Benefits
1614
+
1615
+ - **Composability**: Each function has single responsibility and can be reused
1616
+ - **Testability**: Service functions can be unit tested independently
1617
+ - **Clarity**: Main workflow shows high-level orchestration
1618
+ - **Error Handling**: Isolated error handling per service function
1619
+ - **Logging**: Detailed logging at each step with structured context
1620
+
1621
+ ---
1622
+
1623
+ ## S3 Archival Deduplication Pattern
1624
+
1625
+ This template uses **S3 archival** as the primary deduplication mechanism, NOT VersoriFileTracker.
1626
+
1627
+ ### How It Works
1628
+
1629
+ **S3 Directory Structure:**
1630
+
1631
+ ```
1632
+ s3://my-bucket/
1633
+ ├── locations/ ← listFiles() reads from here
1634
+ │ ├── new-file-1.xml
1635
+ │ └── new-file-2.xml
1636
+ ├── processed/ ← Successfully processed files
1637
+ │ ├── old-file-1.xml
1638
+ │ └── old-file-2.xml
1639
+ └── errors/ ← Failed files
1640
+ └── bad-file.xml
1641
+ ```
1642
+
1643
+ **Deduplication Flow:**
1644
+
1645
+ 1. **List files**: `s3.listFiles({ prefix: 'locations/' })` - Only lists `locations/` subdirectory
1646
+ 2. **Process file**: Download, parse, transform, send mutations
1647
+ 3. **Archive**: `s3.moveFile(filePath, 'processed/new-file-1.xml')` - Moves file out of `locations/`
1648
+ 4. **Next run**: File is now in `processed/`, won't be listed again
1649
+
1650
+ **Why This Works:**
1651
+
1652
+ - Files in `processed/` subdirectory are **never listed** when prefix is `locations/`
1653
+ - No need to track file state in KV store for deduplication
1654
+ - S3 is the single source of truth for file status
1655
+ - Simple, reliable, scales to millions of files
1656
+
1657
+ **StateService Role (Secondary):**
1658
+
1659
+ - Provides metadata and processing history
1660
+ - Backup check in case archival fails mid-process
1661
+ - Useful for monitoring and debugging
1662
+ - NOT the primary deduplication mechanism
1663
+
1664
+ **When to Use VersoriFileTracker:**
1665
+
1666
+ - **NEVER for S3 sources** - Use archival pattern instead
1667
+ - **Only for SFTP sources** - Where archival might not be possible
1668
+ - See SFTP templates for VersoriFileTracker usage
1669
+
1670
+ ---
1671
+
1672
+ ## XML Path Resolution Patterns
1673
+
1674
+ ### Pattern 1: XML Attribute Access with @ Prefix
1675
+
1676
+ ```typescript
1677
+ const mappingConfig = {
1678
+ fields: {
1679
+ // XML attribute access
1680
+ ref: { source: 'location.@ref' }, // <location ref="LOC-001">
1681
+ type: { source: 'location.@type' }, // <location type="WAREHOUSE">
1682
+ country: { source: 'location.address.@country' }, // <address country="USA">
1683
+
1684
+ // XML element text content
1685
+ name: { source: 'location.name' }, // <name>Downtown</name>
1686
+ city: { source: 'location.address.city' }, // <address><city>NYC</city></address>
1687
+ },
1688
+ };
1689
+ ```
1690
+
1691
+ ### Pattern 2: Handling Single vs Multiple Elements
1692
+
1693
+ ```typescript
1694
+ // XML can have single or multiple <location> elements
1695
+ const locationsData = xmlData.locations?.location;
1696
+
1697
+ // Normalize to array
1698
+ const locations = Array.isArray(locationsData) ? locationsData : [locationsData];
1699
+
1700
+ // Process each location
1701
+ for (const loc of locations) {
1702
+ const record = { location: loc }; // Wrap for mapping
1703
+ const result = await mapper.map(record);
1704
+ }
1705
+ ```
1706
+
1707
+ ### Pattern 3: Nested Object Mapping
1708
+
1709
+ ```typescript
1710
+ const mappingConfig = {
1711
+ fields: {
1712
+ // Root fields
1713
+ ref: { source: 'location.@ref', required: true },
1714
+ name: { source: 'location.name', required: true },
1715
+ type: { source: 'location.@type', required: true },
1716
+
1717
+ // Nested primaryAddress object
1718
+ 'primaryAddress.ref': { source: 'location.@ref' },
1719
+ 'primaryAddress.street': { source: 'location.address.street1' },
1720
+ 'primaryAddress.city': { source: 'location.address.city' },
1721
+ 'primaryAddress.latitude': { source: 'location.coordinates.@lat', resolver: 'sdk.parseFloat' },
1722
+
1723
+ // Nested openingSchedule object
1724
+ 'openingSchedule.allHours': {
1725
+ source: 'location.openingSchedule.allHours',
1726
+ resolver: 'sdk.boolean',
1727
+ },
1728
+ 'openingSchedule.monStart': {
1729
+ source: 'location.openingSchedule.monStart',
1730
+ resolver: 'sdk.parseInt',
1731
+ },
1732
+
1733
+ // Note: retailer.id not shown - standard createLocation does not have this field
1734
+ // If your schema requires it, add: 'retailer.id': { value: parseInt(retailerId) }
1735
+ },
1736
+ };
1737
+ ```
1738
+
1739
+ ---
1740
+
1741
+ ## Sample XML Files
1742
+
1743
+ ### Minimal Test File
1744
+
1745
+ **File: test-location.xml**
1746
+
1747
+ ```xml
1748
+ <?xml version="1.0" encoding="UTF-8"?>
1749
+ <locations>
1750
+ <location ref="TEST-001" type="WAREHOUSE">
1751
+ <name>Test Warehouse</name>
1752
+ <address country="USA">
1753
+ <street1>123 Test St</street1>
1754
+ <city>TestCity</city>
1755
+ <state>TC</state>
1756
+ <postalCode>12345</postalCode>
1757
+ </address>
1758
+ <coordinates lat="40.7128" lon="-74.0060"/>
1759
+ <timeZone>America/New_York</timeZone>
1760
+ <openingSchedule>
1761
+ <allHours>false</allHours>
1762
+ <monStart>800</monStart>
1763
+ <monEnd>1800</monEnd>
1764
+ <tueStart>800</tueStart>
1765
+ <tueEnd>1800</tueEnd>
1766
+ <wedStart>800</wedStart>
1767
+ <wedEnd>1800</wedEnd>
1768
+ <thuStart>800</thuStart>
1769
+ <thuEnd>1800</thuEnd>
1770
+ <friStart>800</friStart>
1771
+ <friEnd>1800</friEnd>
1772
+ <satStart>0</satStart>
1773
+ <satEnd>0</satEnd>
1774
+ <sunStart>0</sunStart>
1775
+ <sunEnd>0</sunEnd>
1776
+ </openingSchedule>
1777
+ </location>
1778
+ </locations>
1779
+ ```
1780
+
1781
+ ---
1782
+
1783
+ ## Service Functions Deep Dive
1784
+
1785
+ This template demonstrates service function composition for maintainable workflows.
1786
+
1787
+ ### Function 1: processFile()
1788
+
1789
+ **Purpose**: Download, parse, normalize, and map XML data
1790
+
1791
+ **Inputs**:
1792
+ - `s3: S3DataSource` - For file download
1793
+ - `parser: XMLParserService` - For XML parsing
1794
+ - `mapper: UniversalMapper` - For field mapping
1795
+ - `filePath: string` - S3 path to file
1796
+ - `fileName: string` - File name for logging
1797
+ - `log: any` - Logger instance
1798
+
1799
+ **Outputs**: `FileProcessingResult`
1800
+ ```typescript
1801
+ {
1802
+ success: boolean; // Overall success
1803
+ locations: any[]; // Mapped location objects
1804
+ errors: string[]; // Mapping error messages
1805
+ }
1806
+ ```
1807
+
1808
+ **Key Operations**:
1809
+ 1. Downloads file with retry logic
1810
+ 2. Parses XML using XMLParserService
1811
+ 3. **Normalizes arrays** - Handles single vs multiple `<location>` elements
1812
+ 4. Maps each location using UniversalMapper
1813
+ 5. Collects errors without stopping processing
1814
+ 6. Returns all mapped locations + errors
1815
+
1816
+ **Why separate function?**
1817
+ - Testable independently with mock data
1818
+ - Reusable across workflows (scheduled, webhook, adhoc)
1819
+ - Clear error boundaries - file-level errors vs record-level errors
1820
+ - Easy to add XML validation or schema checks
1821
+
1822
+ ### Function 2: executeMutations()
1823
+
1824
+ **Purpose**: Execute GraphQL createLocation mutations with rate limiting
1825
+
1826
+ **Inputs**:
1827
+ - `client: FluentClient` - For GraphQL mutations
1828
+ - `locations: any[]` - Mapped location data
1829
+ - `mutationDelayMs: number` - Rate limit delay
1830
+ - `fileName: string` - For logging context
1831
+ - `log: any` - Logger instance
1832
+
1833
+ **Outputs**: `MutationResult`
1834
+ ```typescript
1835
+ {
1836
+ successful: number; // Count of successful mutations
1837
+ failed: number; // Count of failed mutations
1838
+ errors: string[]; // Error messages with location refs
1839
+ }
1840
+ ```
1841
+
1842
+ **Key Operations**:
1843
+ 1. Loops through locations
1844
+ 2. Builds GraphQL mutation input
1845
+ 3. Executes with retry logic + rate limiting
1846
+ 4. Tracks success/failure per location
1847
+ 5. Returns summary counts + errors
1848
+
1849
+ **Why separate function?**
1850
+ - Clear separation: mapping vs mutation execution
1851
+ - Rate limiting logic isolated and configurable
1852
+ - Easy to swap mutation types (create vs update)
1853
+ - Testable with mock FluentClient
1854
+ - Can parallelize in future (batch mutations)
1855
+
1856
+ ### Function 3: writeMutationLog()
1857
+
1858
+ **Purpose**: Write detailed mutation results to S3 as JSON log
1859
+
1860
+ **Inputs**:
1861
+ - `s3: S3DataSource` - For log upload
1862
+ - `logEntries: MutationLogEntry[]` - Mutation status per location
1863
+ - `fileName: string` - Original file name
1864
+ - `logPrefix: string` - S3 prefix for logs (e.g., `logs/`)
1865
+ - `log: any` - Logger instance
1866
+
1867
+ **Outputs**: `void` (non-blocking - errors logged but not thrown)
1868
+
1869
+ **Key Operations**:
1870
+ 1. Creates timestamped log file name
1871
+ 2. Builds JSON log with summary + entries
1872
+ 3. **Uses Buffer.from()** - Required for Deno/Versori runtime
1873
+ 4. Writes to S3 with `uploadFile()`
1874
+ 5. Errors don't stop workflow (logging is non-critical)
1875
+
1876
+ **Why separate function?**
1877
+ - Optional feature - can be disabled via config
1878
+ - Non-blocking - logging failure doesn't fail workflow
1879
+ - Structured logging for audit trails
1880
+ - Easy to change log format (JSON, CSV, XML)
1881
+ - Can add log rotation/cleanup logic later
1882
+
1883
+ ### Service Function Composition Benefits
1884
+
1885
+ **1. Maintainability**
1886
+ ```typescript
1887
+ // Clear workflow orchestration
1888
+ const processingResult = await processFile(...);
1889
+ const mutationResult = await executeMutations(...);
1890
+ await writeMutationLog(...);
1891
+ ```
1892
+
1893
+ **2. Testability**
1894
+ ```typescript
1895
+ // Unit test processFile() with mock XML
1896
+ const mockS3 = { downloadFile: jest.fn() };
1897
+ const result = await processFile(mockS3, parser, mapper, ...);
1898
+ expect(result.locations).toHaveLength(5);
1899
+ ```
1900
+
1901
+ **3. Reusability**
1902
+ ```typescript
1903
+ // Use processFile() in different workflows
1904
+ export const webhook = webhook('location-webhook').then(async ctx => {
1905
+ const result = await processFile(s3, parser, mapper, filePath, fileName, log);
1906
+ return { locations: result.locations };
1907
+ });
1908
+ ```
1909
+
1910
+ **4. Error Isolation**
1911
+ ```typescript
1912
+ // Each function has clear error boundaries
1913
+ try {
1914
+ const processingResult = await processFile(...);
1915
+ // File-level errors caught here
1916
+ } catch (error) {
1917
+ // Handle file processing failure
1918
+ }
1919
+
1920
+ // Mutation errors don't stop file processing
1921
+ const mutationResult = await executeMutations(...);
1922
+ // mutationResult.errors contains per-location failures
1923
+ ```
1924
+
1925
+ **5. Progressive Enhancement**
1926
+ ```typescript
1927
+ // Easy to add features without touching core logic
1928
+ async function validateLocations(locations: any[]) {
1929
+ // Add validation step
1930
+ }
1931
+
1932
+ const processingResult = await processFile(...);
1933
+ await validateLocations(processingResult.locations); // New step
1934
+ const mutationResult = await executeMutations(...);
1935
+ ```
1936
+
1937
+ ---
1938
+
1939
+ ## Versori Environment Variables
1940
+
1941
+ **Activation Variables:**
1942
+
1943
+ ```bash
1944
+ # ============================================================================
1945
+ # Required Variables
1946
+ # ============================================================================
1947
+ s3BucketName=my-location-bucket
1948
+ awsAccessKeyId=AKIAXXXXXXXXXXXX
1949
+ awsSecretAccessKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
1950
+
1951
+ # ============================================================================
1952
+ # S3 Configuration (Optional - with defaults)
1953
+ # ============================================================================
1954
+ awsRegion=us-east-1
1955
+ s3Prefix=locations/
1956
+ archivePrefix=processed/
1957
+ errorPrefix=errors/
1958
+ logPrefix=logs/
1959
+ filePattern=.xml
1960
+ maxFilesToProcess=10
1961
+
1962
+ # ============================================================================
1963
+ # Feature Toggles (Optional - with defaults)
1964
+ # ============================================================================
1965
+ # Enable S3 archival (move files to processed/errors directories)
1966
+ enableArchival=true
1967
+
1968
+ # Enable mutation logs (write detailed mutation results to S3)
1969
+ enableMutationLogs=true
1970
+
1971
+ # Enable file tracking via StateService + KV store
1972
+ # When disabled, relies on S3 archival only for deduplication
1973
+ enableFileTracking=true
1974
+
1975
+ # ============================================================================
1976
+ # Mutation Configuration (Optional - with defaults)
1977
+ # ============================================================================
1978
+ # Mutation batch size (concurrent requests)
1979
+ # - 1 = Sequential (default, safest)
1980
+ # - 5 = Process 5 mutations in parallel
1981
+ # - 10 = Process 10 mutations in parallel
1982
+ mutationBatchSize=1
1983
+
1984
+ # Alias batching (combine multiple mutations into single request)
1985
+ # - undefined = Disabled (default, use separate requests)
1986
+ # - 5 = Combine 5 mutations per aliased request
1987
+ # - 10 = Combine 10 mutations per aliased request
1988
+ mutationsPerAliasBatch=
1989
+
1990
+ # ============================================================================
1991
+ # Fluent Commerce Configuration (Optional)
1992
+ # ============================================================================
1993
+ # Retailer ID - Only if mutation schema requires retailerId in input
1994
+ # Standard createLocation does NOT require this field
1995
+ # Check your GraphQL schema to determine if needed
1996
+ retailerId=my-retailer-id
1997
+ ```
1998
+
1999
+ **Notes:**
2000
+ - Webhook security is handled by Versori's native connection authentication. No manual API key configuration needed.
2001
+ - `retailerId` - Standard createLocation does not have this field. Only use if YOUR custom schema requires it.
2002
+ - See `fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md` for details.
2003
+
2004
+ ---
2005
+
2006
+ ## Schema Validation CLI Commands
2007
+
2008
+ Before deploying, validate your field mappings against the Fluent GraphQL schema:
2009
+
2010
+ ### 1. Introspect Schema
2011
+
2012
+ ```bash
2013
+ # Generate schema.json from live Fluent API
2014
+ npx fc-connect introspect-schema \
2015
+ --url https://api.fluentcommerce.com/graphql \
2016
+ --client-id your-client-id \
2017
+ --client-secret your-client-secret \
2018
+ --output schema.json
2019
+ ```
2020
+
2021
+ ### 2. Create Mapping Config File
2022
+
2023
+ **File: location-mapping.json**
2024
+
2025
+ ```json
2026
+ {
2027
+ "version": "1.0.0",
2028
+ "mutation": "createLocation",
2029
+ "sourceFormat": "xml",
2030
+ "returnFields": ["id", "ref", "name", "type", "status"],
2031
+ "description": "XML location to Fluent Commerce GraphQL mapping",
2032
+ "fields": {
2033
+ "ref": { "source": "location.@ref", "required": true, "resolver": "sdk.trim" },
2034
+ "name": { "source": "location.name", "required": true, "resolver": "sdk.trim" },
2035
+ "type": { "source": "location.@type", "required": true, "resolver": "sdk.uppercase" },
2036
+ "primaryAddress.ref": { "source": "location.@ref", "required": true },
2037
+ "primaryAddress.street": { "source": "location.address.street1" },
2038
+ "primaryAddress.latitude": {
2039
+ "source": "location.coordinates.@lat",
2040
+ "resolver": "sdk.parseFloat"
2041
+ },
2042
+ "primaryAddress.longitude": {
2043
+ "source": "location.coordinates.@lon",
2044
+ "resolver": "sdk.parseFloat"
2045
+ }
2046
+ }
2047
+ }
2048
+ ```
2049
+
2050
+ ### 3. Validate Mapping
2051
+
2052
+ ```bash
2053
+ # Validate that all target fields exist in schema
2054
+ npx fc-connect validate-schema \
2055
+ --mapping location-mapping.json \
2056
+ --schema schema.json
2057
+ ```
2058
+
2059
+ ### 4. Analyze Coverage
2060
+
2061
+ ```bash
2062
+ # Check which Location fields are mapped vs available
2063
+ npx fc-connect analyze-coverage \
2064
+ --mapping location-mapping.json \
2065
+ --schema schema.json \
2066
+ --type CreateLocationInput
2067
+ ```
2068
+
2069
+ **Output:**
2070
+
2071
+ ```
2072
+ ✅ Mapped: 15/42 fields (35%)
2073
+ ❌ Missing required: timezone (String!)
2074
+ ⚠️ Optional unmapped: supportPhoneNumber, networkId, attributes
2075
+ ```
2076
+
2077
+ ---
2078
+
2079
+ ## Testing Locally
2080
+
2081
+ ### 1. Upload Test XML to S3
2082
+
2083
+ ```bash
2084
+ aws s3 cp test-location.xml s3://my-location-bucket/locations/test-location.xml
2085
+ ```
2086
+
2087
+ ### 2. Deploy to Versori
2088
+
2089
+ ```bash
2090
+ npm run deploy
2091
+ ```
2092
+
2093
+ ### 3. Manual Testing
2094
+
2095
+ ```bash
2096
+ # Trigger manual sync (auth handled by Versori connection)
2097
+ curl -X POST https://your-workspace.versori.run/location-xml-adhoc
2098
+
2099
+ # Check job status
2100
+ curl -X POST https://your-workspace.versori.run/location-xml-job-status \
2101
+ -H "Content-Type: application/json" \
2102
+ -d '{"jobId": "location-xml-adhoc-1737525600000"}'
2103
+ ```
2104
+
2105
+ ### 4. Verify Processing
2106
+
2107
+ - Upload a small XML to S3 (2-3 locations) and trigger `adhoc` webhook
2108
+ - Verify GraphQL mutations are executed for each location
2109
+ - Confirm file moved from `locations/` to `processed/` in S3
2110
+ - Check KV state: errors and last processed metadata
2111
+ - Monitor rate limiting: verify mutations respect configured rate
2112
+
2113
+ ---
2114
+
2115
+ ## Deployment
2116
+
2117
+ ```bash
2118
+ # Deploy to Versori
2119
+ npm run deploy
2120
+
2121
+ # View logs
2122
+ npm run logs
2123
+
2124
+ # Monitor execution
2125
+ versori logs --follow
2126
+ ```
2127
+
2128
+ ---
2129
+
2130
+ ## Monitoring
2131
+
2132
+ ### Success Response
2133
+
2134
+ ```json
2135
+ {
2136
+ "success": true,
2137
+ "jobId": "location-xml-scheduled-1737525600000",
2138
+ "processed": 3,
2139
+ "skipped": 0,
2140
+ "failed": 0,
2141
+ "totalRecords": 12,
2142
+ "errors": []
2143
+ }
2144
+ ```
2145
+
2146
+ ### Partial Success Response
2147
+
2148
+ ```json
2149
+ {
2150
+ "success": true,
2151
+ "jobId": "location-xml-scheduled-1737525600000",
2152
+ "processed": 3,
2153
+ "skipped": 1,
2154
+ "failed": 0,
2155
+ "totalRecords": 10,
2156
+ "errors": ["locations-003.xml: 2 mapping errors"]
2157
+ }
2158
+ ```
2159
+
2160
+ ### Error Response
2161
+
2162
+ ```json
2163
+ {
2164
+ "success": false,
2165
+ "jobId": "location-xml-scheduled-1737525600000",
2166
+ "processed": 0,
2167
+ "skipped": 0,
2168
+ "failed": 1,
2169
+ "totalRecords": 0,
2170
+ "errors": ["locations-001.xml: Invalid XML structure"]
2171
+ }
2172
+ ```
2173
+
2174
+ ---
2175
+
2176
+ ## Common Pitfalls and Solutions
2177
+
2178
+ ### 1. XML Attribute Not Found
2179
+
2180
+ **Symptoms**: Mapping errors like "field not found"
2181
+
2182
+ **Solution**:
2183
+
2184
+ ```typescript
2185
+ // ❌ WRONG - Missing @ prefix for attribute
2186
+ ref: {
2187
+ source: 'location.ref';
2188
+ }
2189
+
2190
+ // ✅ CORRECT - Use @ prefix for XML attributes
2191
+ ref: {
2192
+ source: 'location.@ref';
2193
+ }
2194
+ ```
2195
+
2196
+ ### 2. Single Element Not Array
2197
+
2198
+ **Symptoms**: "Cannot read property forEach of undefined"
2199
+
2200
+ **Solution**:
2201
+
2202
+ ```typescript
2203
+ // Always normalize to array
2204
+ const locationsData = xmlData.locations?.location;
2205
+ const locations = Array.isArray(locationsData) ? locationsData : [locationsData];
2206
+ ```
2207
+
2208
+ ### 3. Rate Limiting Too Aggressive
2209
+
2210
+ **Symptoms**: Slow processing, mutations taking too long
2211
+
2212
+ **Solution**:
2213
+
2214
+ ```bash
2215
+ # Increase rate limit (mutations per second)
2216
+ mutationRateLimit=10 # Default is 5
2217
+ ```
2218
+
2219
+ ### 4. Empty Element vs Missing Element
2220
+
2221
+ **Solution**:
2222
+
2223
+ ```typescript
2224
+ // Use required: false and defaultValue for optional fields
2225
+ 'primaryAddress.street2': {
2226
+ source: 'location.address.street2',
2227
+ required: false,
2228
+ defaultValue: ''
2229
+ }
2230
+ ```
2231
+
2232
+ ### 5. S3 Access Denied
2233
+
2234
+ **Symptoms**: S3 operations fail with 403 errors
2235
+
2236
+ **Solution**: Validate IAM permissions
2237
+
2238
+ **Required IAM Permissions:**
2239
+
2240
+ ```json
2241
+ {
2242
+ "Version": "2012-10-17",
2243
+ "Statement": [
2244
+ {
2245
+ "Effect": "Allow",
2246
+ "Action": ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
2247
+ "Resource": ["arn:aws:s3:::my-location-bucket", "arn:aws:s3:::my-location-bucket/*"]
2248
+ }
2249
+ ]
2250
+ }
2251
+ ```
2252
+
2253
+ ### 6. GraphQL Schema Mismatch
2254
+
2255
+ **Symptoms**: Mutation errors like "Unknown field", "Invalid input type"
2256
+
2257
+ **Solution**: Use CLI tools to validate mappings
2258
+
2259
+ ```bash
2260
+ npx fc-connect validate-schema --mapping location-mapping.json --schema schema.json
2261
+ ```
2262
+
2263
+ ### 7. Nested Object Mapping Errors
2264
+
2265
+ **Symptoms**: Flat structure instead of nested objects in mutation input
2266
+
2267
+ **Solution**: Use dot notation in field mapping
2268
+
2269
+ ```typescript
2270
+ // ✅ CORRECT - Creates nested structure
2271
+ 'primaryAddress.city': { source: 'location.address.city' }
2272
+
2273
+ // ❌ WRONG - Creates flat structure
2274
+ primaryAddress_city: { source: 'location.address.city' }
2275
+ ```
2276
+
2277
+ ### 8. retailerId Configuration Errors
2278
+
2279
+ **Symptoms**: "retailerId is required" errors or confusion about when to use `setRetailerId()`
2280
+
2281
+ **Solution**: Understand the correct pattern
2282
+
2283
+ ```typescript
2284
+ // ✅ CORRECT - GraphQL mutations don't need setRetailerId()
2285
+ // Check your GraphQL schema to determine retailerId handling:
2286
+ // - Mandatory retailerId → Must pass it in mutation input
2287
+ // - Optional retailerId → Can pass it if needed
2288
+ // - No retailerId field → Don't pass it
2289
+ // Standard createLocation does not have retailerId field in schema
2290
+
2291
+ // ✅ IF mutation schema requires retailerId (mandatory):
2292
+ const { query, variables } = await mapper.map(location);
2293
+ if (retailerId && variables.input) {
2294
+ variables.input.retailer = { id: parseInt(retailerId) };
2295
+ }
2296
+ await client.graphql({ query, variables });
2297
+ ```
2298
+
2299
+ **Reference:** `fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md`
2300
+
2301
+ ---
2302
+
2303
+ ## Key Takeaways
2304
+
2305
+ ### Architecture Patterns
2306
+
2307
+ - **Service Function Composition**: Workflow broken into 3 service functions - `processFile()`, `executeMutations()`, `writeMutationLog()`
2308
+ - **Per-File Processing**: Main workflow orchestrates service functions for each file
2309
+ - **Clear Separation of Concerns**: Parse/map, execute mutations, logging are independent functions
2310
+
2311
+ ### SDK Usage
2312
+
2313
+ - **Buffer Import (CRITICAL)**: Always `import { Buffer } from 'node:buffer'` for Deno/Versori runtime
2314
+ - **S3 Archival Deduplication**: Use `s3.moveFile()` to `processed/` subdirectory - PRIMARY deduplication mechanism
2315
+ - **NO VersoriFileTracker for S3**: S3 archival is simpler and more reliable
2316
+ - **StateService Role**: SECONDARY - provides metadata/history, not primary deduplication
2317
+
2318
+ ### XML Processing
2319
+
2320
+ - **XML @ Prefix**: Always use `@` prefix for XML attributes (`location.@ref`)
2321
+ - **Array Normalization (CRITICAL)**: Handle single vs multiple elements with `Array.isArray()` check - single `<location>` becomes object, not array
2322
+ - **Nested Mapping**: Use dot notation for nested objects (`primaryAddress.street`, `openingSchedule.monStart`)
2323
+
2324
+ ### GraphQL Mutations
2325
+
2326
+ - **NO setRetailerId() Required**: GraphQL mutations do NOT need `client.setRetailerId()` - only Job/Event API needs it
2327
+ - **retailerId in Input**: Check your GraphQL schema to determine retailerId handling:
2328
+ - **Mandatory retailerId** - Field exists and is required (`!`) → Must pass it
2329
+ - **Optional retailerId** - Field exists and is optional → Can pass it if needed
2330
+ - **No retailerId field** - Field doesn't exist → Don't pass it
2331
+ - **Rate Limiting**: Implement configurable delays between mutations to avoid API throttling
2332
+ - **Retry Logic**: Exponential backoff for failed mutations with `retryWithBackoff()`
2333
+ - **Direct Mutations**: Use `client.graphql()` for location upserts (NOT Batch API)
2334
+ - **GraphQL vs Batch API**: Use GraphQL for low-volume master data, Batch API for high-volume inventory
2335
+
2336
+ ### Error Handling & Monitoring
2337
+
2338
+ - **Error Recovery**: Exponential backoff for error state tracking with retry timestamps
2339
+ - **Mutation Logging**: Optional S3 JSON logs with detailed per-location mutation status
2340
+ - **Schema Validation**: Use CLI tools before deployment to catch mapping errors
2341
+ - **Archival Order**: Archive FIRST (deduplication), then KV tracking (metadata)
2342
+
2343
+ ---
2344
+
2345
+ ## Related Documentation
2346
+
2347
+ ### Core Guides
2348
+
2349
+ - **GraphQL Mutation Mapping**: `fc-connect-sdk/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/readme.md`
2350
+ - **Universal Mapping**: `fc-connect-sdk/docs/02-CORE-GUIDES/mapping/modules/readme.md`
2351
+ - **XML Parser**: `fc-connect-sdk/docs/02-CORE-GUIDES/parsers/modules/05-xml-parser.md`
2352
+ - **Data Sources**: `fc-connect-sdk/docs/02-CORE-GUIDES/data-sources/readme.md`
2353
+ - **State Management**: `fc-connect-sdk/docs/03-PATTERN-GUIDES/file-operations/state-duplicate-prevention.md`
2354
+
2355
+ ### Related Templates
2356
+
2357
+ - **CSV Version**: `template-ingestion-s3-csv-location-graphql.md`
2358
+ - **JSON Version**: `template-ingestion-s3-json-location-graphql.md`
2359
+ - **SFTP XML Version**: `template-ingestion-sftp-xml-location-graphql.md`
2360
+ - **Event API Pattern**: `../event-api/template-ingestion-s3-xml-product-event.md`
2361
+ - **Batch API Pattern**: `../batch-api/template-ingestion-s3-xml-inventory-batch.md`
2362
+
2363
+ ### CLI Tools
2364
+
2365
+ - **Schema Introspection**: `fc-connect-sdk/bin/readme.md#introspect-schema`
2366
+ - **Mapping Validation**: `fc-connect-sdk/bin/readme.md#validate-schema`
2367
+ - **Coverage Analysis**: `fc-connect-sdk/bin/readme.md#analyze-coverage`
2368
+
2369
+ ### Patterns
2370
+
2371
+ - **Error Handling**: `fc-connect-sdk/docs/01-TEMPLATES/patterns/error-handling-retry.md`
2372
+ - **Rate Limiting**: `fc-connect-sdk/docs/03-PATTERN-GUIDES/integration-patterns/rate-limiting.md`
2373
+ - **XML Patterns**: `fc-connect-sdk/docs/01-TEMPLATES/versori/patterns/xml-response-patterns.md`