@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (495) hide show
  1. package/CHANGELOG.md +30 -2
  2. package/README.md +39 -0
  3. package/dist/cjs/auth/index.d.ts +3 -0
  4. package/dist/cjs/auth/index.js +13 -0
  5. package/dist/cjs/auth/profile-loader.d.ts +18 -0
  6. package/dist/cjs/auth/profile-loader.js +208 -0
  7. package/dist/cjs/client-factory.d.ts +4 -0
  8. package/dist/cjs/client-factory.js +10 -0
  9. package/dist/cjs/clients/fluent-client.js +13 -6
  10. package/dist/cjs/index.d.ts +3 -1
  11. package/dist/cjs/index.js +8 -2
  12. package/dist/cjs/utils/pagination-helpers.js +38 -2
  13. package/dist/cjs/versori/fluent-versori-client.js +11 -5
  14. package/dist/esm/auth/index.d.ts +3 -0
  15. package/dist/esm/auth/index.js +2 -0
  16. package/dist/esm/auth/profile-loader.d.ts +18 -0
  17. package/dist/esm/auth/profile-loader.js +169 -0
  18. package/dist/esm/client-factory.d.ts +4 -0
  19. package/dist/esm/client-factory.js +9 -0
  20. package/dist/esm/clients/fluent-client.js +13 -6
  21. package/dist/esm/index.d.ts +3 -1
  22. package/dist/esm/index.js +2 -1
  23. package/dist/esm/utils/pagination-helpers.js +38 -2
  24. package/dist/esm/versori/fluent-versori-client.js +11 -5
  25. package/dist/tsconfig.esm.tsbuildinfo +1 -1
  26. package/dist/tsconfig.tsbuildinfo +1 -1
  27. package/dist/tsconfig.types.tsbuildinfo +1 -1
  28. package/dist/types/auth/index.d.ts +3 -0
  29. package/dist/types/auth/profile-loader.d.ts +18 -0
  30. package/dist/types/client-factory.d.ts +4 -0
  31. package/dist/types/index.d.ts +3 -1
  32. package/docs/00-START-HERE/EXPORT-VALIDATION.md +158 -158
  33. package/docs/00-START-HERE/cli-analyze-source-structure-guide.md +655 -655
  34. package/docs/00-START-HERE/cli-documentation-index.md +202 -202
  35. package/docs/00-START-HERE/cli-quick-reference.md +252 -252
  36. package/docs/00-START-HERE/decision-tree.md +552 -552
  37. package/docs/00-START-HERE/getting-started.md +1070 -1070
  38. package/docs/00-START-HERE/mapper-quick-decision-guide.md +235 -235
  39. package/docs/00-START-HERE/readme.md +237 -237
  40. package/docs/00-START-HERE/retailerid-configuration.md +404 -404
  41. package/docs/00-START-HERE/sdk-philosophy.md +794 -794
  42. package/docs/00-START-HERE/troubleshooting-quick-reference.md +1086 -1086
  43. package/docs/01-TEMPLATES/faq.md +686 -686
  44. package/docs/01-TEMPLATES/patterns/pattern-templates-guide.md +68 -68
  45. package/docs/01-TEMPLATES/patterns/patterns-csv-schema-validation-and-rejection-report.md +233 -233
  46. package/docs/01-TEMPLATES/patterns/patterns-custom-resolvers.md +407 -407
  47. package/docs/01-TEMPLATES/patterns/patterns-error-handling-retry.md +511 -511
  48. package/docs/01-TEMPLATES/patterns/patterns-field-mapping-universal.md +701 -701
  49. package/docs/01-TEMPLATES/patterns/patterns-large-file-splitting.md +1430 -1430
  50. package/docs/01-TEMPLATES/patterns/patterns-master-data-etl.md +2399 -2399
  51. package/docs/01-TEMPLATES/patterns/patterns-pagination-streaming.md +447 -447
  52. package/docs/01-TEMPLATES/patterns/patterns-state-duplicate-prevention.md +385 -385
  53. package/docs/01-TEMPLATES/readme.md +957 -957
  54. package/docs/01-TEMPLATES/standalone/standalone-asn-inbound-processing.md +1209 -1209
  55. package/docs/01-TEMPLATES/standalone/standalone-graphql-query-export.md +1140 -1140
  56. package/docs/01-TEMPLATES/standalone/standalone-graphql-to-parquet-partitioned-s3.md +432 -432
  57. package/docs/01-TEMPLATES/standalone/standalone-multi-channel-inventory-sync.md +1185 -1185
  58. package/docs/01-TEMPLATES/standalone/standalone-multi-source-aggregation.md +1462 -1462
  59. package/docs/01-TEMPLATES/standalone/standalone-s3-csv-batch-api.md +1390 -1390
  60. package/docs/01-TEMPLATES/standalone/standalone-s3-csv-inventory-to-batch.md +330 -330
  61. package/docs/01-TEMPLATES/standalone/standalone-scripts-guide.md +87 -87
  62. package/docs/01-TEMPLATES/standalone/standalone-sftp-xml-graphql.md +1444 -1444
  63. package/docs/01-TEMPLATES/standalone/standalone-webhook-payload-processing.md +688 -688
  64. package/docs/01-TEMPLATES/versori/business-examples/business-examples-dropship-order-routing.md +193 -193
  65. package/docs/01-TEMPLATES/versori/business-examples/business-examples-graphql-parquet-extraction.md +518 -518
  66. package/docs/01-TEMPLATES/versori/business-examples/business-examples-inter-location-transfers.md +2162 -2162
  67. package/docs/01-TEMPLATES/versori/business-examples/business-examples-pre-order-allocation.md +2226 -2226
  68. package/docs/01-TEMPLATES/versori/business-examples/business-scenarios-guide.md +87 -87
  69. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-connection-validation-pattern.md +656 -656
  70. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-dual-workflow-connector.md +835 -835
  71. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-guide.md +108 -108
  72. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-kv-state-management.md +1533 -1533
  73. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-xml-response-patterns.md +1160 -1160
  74. package/docs/01-TEMPLATES/versori/versori-platform-guide.md +201 -201
  75. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-asn-purchase-order.md +1906 -1906
  76. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-dropship-routing.md +1074 -1074
  77. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-flash-sale-reserve.md +1395 -1395
  78. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-generic-xml-order.md +888 -888
  79. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-payment-gateway-integration.md +2478 -2478
  80. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-rma-returns-comprehensive.md +2240 -2240
  81. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-xml-order-ingestion.md +2029 -2029
  82. package/docs/01-TEMPLATES/versori/webhooks/webhook-templates-guide.md +140 -140
  83. package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/inventory-mapping.json +20 -20
  84. package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/products_2025-01-22.csv +11 -11
  85. package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/sample-data-guide.md +34 -34
  86. package/docs/01-TEMPLATES/versori/workflows/_examples/workflow-examples-guide.md +36 -36
  87. package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-modes-guide.md +1038 -1038
  88. package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-workflows-guide.md +138 -138
  89. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/graphql-extraction-guide.md +63 -63
  90. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-csv.md +2062 -2062
  91. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-xml.md +2294 -2294
  92. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-s3-csv.md +2461 -2461
  93. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-sftp-xml.md +2529 -2529
  94. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-csv.md +2464 -2464
  95. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-json.md +1959 -1959
  96. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-s3-csv.md +1953 -1953
  97. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-sftp-xml.md +2541 -2541
  98. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-s3-json.md +2384 -2384
  99. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-sftp-xml.md +2445 -2445
  100. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-csv.md +2355 -2355
  101. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-json.md +2042 -2042
  102. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-sftp-xml.md +2726 -2726
  103. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/batch-api-guide.md +206 -206
  104. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-cycle-count-reconciliation.md +2030 -2030
  105. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-multi-channel-inventory-sync.md +1882 -1882
  106. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-csv-inventory-batch.md +2827 -2827
  107. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-json-inventory-batch.md +1952 -1952
  108. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-xml-inventory-batch.md +3289 -3289
  109. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-csv-inventory-batch.md +3064 -3064
  110. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-json-inventory-batch.md +3238 -3238
  111. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-xml-inventory-batch.md +2977 -2977
  112. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/event-api-guide.md +321 -321
  113. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-json-order-cancel-event.md +959 -959
  114. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-xml-order-cancel-event.md +1170 -1170
  115. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-csv-product-event.md +2312 -2312
  116. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-json-product-event.md +2999 -2999
  117. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-parquet-product-event.md +2836 -2836
  118. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-xml-product-event.md +2395 -2395
  119. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-csv-product-event.md +2295 -2295
  120. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-json-product-event.md +2602 -2602
  121. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-parquet-product-event.md +2589 -2589
  122. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-xml-product-event.md +3578 -3578
  123. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/graphql-mutations-guide.md +93 -93
  124. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-json-order-update-graphql.md +1260 -1260
  125. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-xml-order-update-graphql.md +1472 -1472
  126. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-control-graphql.md +2417 -2417
  127. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-location-graphql.md +2811 -2811
  128. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-price-graphql.md +2619 -2619
  129. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-json-location-graphql.md +2807 -2807
  130. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-xml-location-graphql.md +2373 -2373
  131. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-control-graphql.md +2740 -2740
  132. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-location-graphql.md +2760 -2760
  133. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-json-location-graphql.md +1710 -1710
  134. package/docs/01-TEMPLATES/versori/workflows/ingestion/ingestion-workflows-guide.md +136 -136
  135. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/rubix-webhooks-guide.md +520 -520
  136. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-inline.md +1418 -1418
  137. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-universal-mapper.md +1785 -1785
  138. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-order-attribute-update.md +824 -824
  139. package/docs/01-TEMPLATES/versori/workflows/workflows-overview-guide.md +646 -646
  140. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-batch-archival.md +724 -724
  141. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-job-tracker.md +627 -627
  142. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-partial-batch-recovery.md +561 -561
  143. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-quick-reference.md +367 -367
  144. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-readme.md +407 -407
  145. package/docs/02-CORE-GUIDES/advanced-services/readme.md +49 -49
  146. package/docs/02-CORE-GUIDES/api-reference/api-reference-quick-reference.md +548 -548
  147. package/docs/02-CORE-GUIDES/api-reference/event-api-input-output-reference.md +702 -1171
  148. package/docs/02-CORE-GUIDES/api-reference/examples/client-initialization.ts +286 -286
  149. package/docs/02-CORE-GUIDES/api-reference/graphql-error-classification.md +337 -337
  150. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-01-client-api.md +399 -482
  151. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-03-authentication.md +199 -199
  152. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-04-graphql-mapping.md +925 -925
  153. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-05-services.md +1198 -1198
  154. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-06-data-sources.md +1083 -1083
  155. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-07-parsers.md +1097 -1097
  156. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-pagination.md +513 -513
  157. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-types.md +545 -597
  158. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-error-handling.md +527 -527
  159. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-webhook-validation.md +514 -514
  160. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-extraction.md +557 -557
  161. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-utilities.md +412 -412
  162. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-cli-tools.md +423 -423
  163. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-error-handling.md +716 -716
  164. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-analyze-source-structure.md +518 -518
  165. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-partial-responses.md +212 -212
  166. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-testing.md +300 -300
  167. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-13-resolver-builder.md +322 -322
  168. package/docs/02-CORE-GUIDES/api-reference/readme.md +279 -279
  169. package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-quick-reference.md +351 -351
  170. package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-readme.md +277 -277
  171. package/docs/02-CORE-GUIDES/auto-pagination/examples/auto-pagination-readme.md +178 -178
  172. package/docs/02-CORE-GUIDES/auto-pagination/examples/common-patterns.ts +351 -351
  173. package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-products.ts +384 -384
  174. package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-virtual-positions.ts +308 -308
  175. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-01-foundations.md +470 -470
  176. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-02-quick-start.md +713 -713
  177. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-03-configuration.md +754 -754
  178. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-04-advanced-patterns.md +732 -732
  179. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-05-sdk-integration.md +847 -847
  180. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-06-troubleshooting.md +359 -359
  181. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-07-api-reference.md +462 -462
  182. package/docs/02-CORE-GUIDES/auto-pagination/readme.md +54 -54
  183. package/docs/02-CORE-GUIDES/data-sources/data-sources-file-operations-error-handling.md +1487 -1487
  184. package/docs/02-CORE-GUIDES/data-sources/data-sources-quick-reference.md +836 -836
  185. package/docs/02-CORE-GUIDES/data-sources/data-sources-readme.md +276 -276
  186. package/docs/02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md +553 -553
  187. package/docs/02-CORE-GUIDES/data-sources/examples/common-patterns.ts +409 -409
  188. package/docs/02-CORE-GUIDES/data-sources/examples/data-sources-readme.md +178 -178
  189. package/docs/02-CORE-GUIDES/data-sources/examples/s3-operations.ts +308 -308
  190. package/docs/02-CORE-GUIDES/data-sources/examples/sftp-operations.ts +371 -371
  191. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-01-foundations.md +735 -735
  192. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-02-s3-operations.md +1302 -1302
  193. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-03-sftp-operations.md +1379 -1379
  194. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-04-file-patterns.md +941 -941
  195. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-05-advanced-topics.md +813 -813
  196. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-06-integration-patterns.md +486 -486
  197. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-07-troubleshooting.md +387 -387
  198. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-08-api-reference.md +417 -417
  199. package/docs/02-CORE-GUIDES/data-sources/readme.md +77 -77
  200. package/docs/02-CORE-GUIDES/error-handling-guide.md +936 -936
  201. package/docs/02-CORE-GUIDES/extraction/examples/02-core-guides-extraction-readme.md +116 -116
  202. package/docs/02-CORE-GUIDES/extraction/examples/common-patterns.ts +428 -428
  203. package/docs/02-CORE-GUIDES/extraction/examples/extract-inventory-basic.ts +187 -187
  204. package/docs/02-CORE-GUIDES/extraction/extraction-quick-reference.md +596 -596
  205. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-01-foundations.md +514 -514
  206. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-02-basic-extraction.md +823 -823
  207. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-03-parquet-processing.md +507 -507
  208. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-04-data-enrichment.md +546 -546
  209. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-05-transformation.md +494 -494
  210. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-export-formats.md +458 -458
  211. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-performance.md +138 -138
  212. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-api-reference.md +148 -148
  213. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-optimization.md +692 -692
  214. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-08-extraction-orchestrator.md +1008 -1008
  215. package/docs/02-CORE-GUIDES/extraction/readme.md +151 -151
  216. package/docs/02-CORE-GUIDES/ingestion/examples/_simple-kv-store.ts +40 -40
  217. package/docs/02-CORE-GUIDES/ingestion/examples/error-recovery.ts +728 -728
  218. package/docs/02-CORE-GUIDES/ingestion/examples/event-driven.ts +501 -501
  219. package/docs/02-CORE-GUIDES/ingestion/examples/local-file-ingestion.ts +88 -88
  220. package/docs/02-CORE-GUIDES/ingestion/examples/parquet-ingestion.ts +117 -117
  221. package/docs/02-CORE-GUIDES/ingestion/examples/performance-optimized.ts +647 -647
  222. package/docs/02-CORE-GUIDES/ingestion/examples/s3-csv-ingestion.ts +169 -169
  223. package/docs/02-CORE-GUIDES/ingestion/examples/sftp-csv-ingestion.ts +134 -134
  224. package/docs/02-CORE-GUIDES/ingestion/ingestion-quick-reference.md +546 -546
  225. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-01-introduction.md +626 -626
  226. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-02-quick-start.md +658 -658
  227. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-03-data-sources.md +1052 -1052
  228. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-04-field-mapping.md +763 -763
  229. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-05-advanced-parsers.md +676 -676
  230. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-06-batch-api.md +1295 -1295
  231. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-api-reference.md +138 -138
  232. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-state-management.md +1037 -1037
  233. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-08-performance-optimization.md +1349 -1349
  234. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-09-best-practices.md +1893 -1893
  235. package/docs/02-CORE-GUIDES/ingestion/readme.md +160 -160
  236. package/docs/02-CORE-GUIDES/logging-guide.md +585 -585
  237. package/docs/02-CORE-GUIDES/mapping/error-handling-patterns.md +401 -401
  238. package/docs/02-CORE-GUIDES/mapping/examples/02-core-guides-mapping-readme.md +128 -128
  239. package/docs/02-CORE-GUIDES/mapping/examples/common-patterns.ts +273 -273
  240. package/docs/02-CORE-GUIDES/mapping/examples/csv-location-ingestion.json +36 -36
  241. package/docs/02-CORE-GUIDES/mapping/examples/csv-mapping.ts +242 -242
  242. package/docs/02-CORE-GUIDES/mapping/examples/graphql-to-parquet-extraction.json +36 -36
  243. package/docs/02-CORE-GUIDES/mapping/examples/json-mapping.ts +213 -213
  244. package/docs/02-CORE-GUIDES/mapping/examples/json-product-to-mutation.json +48 -48
  245. package/docs/02-CORE-GUIDES/mapping/examples/xml-mapping.ts +291 -291
  246. package/docs/02-CORE-GUIDES/mapping/examples/xml-order-to-mutation.json +45 -45
  247. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-quick-reference.md +463 -463
  248. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-readme.md +227 -227
  249. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-01-introduction.md +222 -222
  250. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-02-quick-start.md +351 -351
  251. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-03-schema-validation.md +569 -569
  252. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-04-mapping-patterns.md +471 -471
  253. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-05-configuration-reference.md +611 -611
  254. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-advanced-xpath.md +148 -148
  255. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-path-syntax.md +464 -464
  256. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-api-reference.md +94 -94
  257. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-array-handling.md +307 -307
  258. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-08-custom-resolvers.md +544 -544
  259. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-09-advanced-patterns.md +427 -427
  260. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-10-hooks-and-variables.md +336 -336
  261. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-11-error-handling.md +488 -488
  262. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-12-arguments-vs-nodes.md +383 -383
  263. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-13-best-practices.md +477 -477
  264. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/readme.md +62 -62
  265. package/docs/02-CORE-GUIDES/mapping/mapping-format-decision-tree.md +480 -480
  266. package/docs/02-CORE-GUIDES/mapping/mapping-graphql-alias-batching-guide.md +820 -820
  267. package/docs/02-CORE-GUIDES/mapping/mapping-javascript-objects.md +2369 -2369
  268. package/docs/02-CORE-GUIDES/mapping/mapping-mapper-comparison-guide.md +682 -682
  269. package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-07-api-reference.md +1327 -1327
  270. package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-08-error-handling.md +1142 -1142
  271. package/docs/02-CORE-GUIDES/mapping/modules/mapping-04-use-cases.md +891 -891
  272. package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-helpers-resolvers.md +1126 -1126
  273. package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-sdk-resolvers.md +199 -199
  274. package/docs/02-CORE-GUIDES/mapping/modules/mapping-07-api-reference.md +1319 -1319
  275. package/docs/02-CORE-GUIDES/mapping/readme.md +178 -178
  276. package/docs/02-CORE-GUIDES/mapping/resolver-registration.md +410 -410
  277. package/docs/02-CORE-GUIDES/mapping/resolvers/examples/common-patterns.ts +226 -226
  278. package/docs/02-CORE-GUIDES/mapping/resolvers/examples/custom-resolvers.ts +227 -227
  279. package/docs/02-CORE-GUIDES/mapping/resolvers/examples/sdk-resolvers-usage.ts +203 -203
  280. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-readme.md +274 -274
  281. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-api-reference.md +679 -679
  282. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-cookbook.md +826 -826
  283. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-guide.md +1330 -1330
  284. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-helpers-reference.md +1437 -1437
  285. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-parameters-reference.md +553 -553
  286. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-troubleshooting.md +854 -854
  287. package/docs/02-CORE-GUIDES/mapping/resolvers/readme.md +75 -75
  288. package/docs/02-CORE-GUIDES/parsers/examples/02-core-guides-parsers-readme.md +161 -161
  289. package/docs/02-CORE-GUIDES/parsers/examples/csv-parser-examples.ts +110 -110
  290. package/docs/02-CORE-GUIDES/parsers/examples/json-parser-examples.ts +33 -33
  291. package/docs/02-CORE-GUIDES/parsers/examples/parquet-parser-examples.ts +47 -47
  292. package/docs/02-CORE-GUIDES/parsers/examples/xml-parser-examples.ts +38 -38
  293. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-01-foundations.md +355 -355
  294. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-02-csv-parser.md +772 -772
  295. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-03-json-parser.md +789 -789
  296. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-04-xml-parser.md +857 -857
  297. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-05-parquet-parser.md +603 -603
  298. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-integration-patterns.md +702 -702
  299. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-streaming.md +121 -121
  300. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-api-reference.md +89 -89
  301. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-troubleshooting.md +727 -727
  302. package/docs/02-CORE-GUIDES/parsers/parsers-quick-reference.md +482 -482
  303. package/docs/02-CORE-GUIDES/parsers/parsers-readme.md +258 -258
  304. package/docs/02-CORE-GUIDES/parsers/readme.md +65 -65
  305. package/docs/02-CORE-GUIDES/readme.md +194 -194
  306. package/docs/02-CORE-GUIDES/webhook-validation/examples/basic-validation.ts +108 -108
  307. package/docs/02-CORE-GUIDES/webhook-validation/examples/common-patterns.ts +316 -316
  308. package/docs/02-CORE-GUIDES/webhook-validation/examples/webhook-validation-readme.md +61 -61
  309. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-01-foundations.md +440 -440
  310. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-02-quick-start.md +525 -525
  311. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-03-versori-integration.md +741 -741
  312. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-04-platform-integration.md +629 -629
  313. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-05-configuration.md +535 -535
  314. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-error-handling.md +611 -611
  315. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-troubleshooting.md +124 -124
  316. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-07-api-reference.md +511 -511
  317. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-08-rubix-webhooks.md +590 -590
  318. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-09-rubix-event-vs-http-call.md +432 -432
  319. package/docs/02-CORE-GUIDES/webhook-validation/readme.md +239 -239
  320. package/docs/02-CORE-GUIDES/webhook-validation/webhook-validation-quick-reference.md +392 -392
  321. package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-quick-reference.md +498 -498
  322. package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-readme.md +313 -313
  323. package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/common-patterns.ts +612 -612
  324. package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/connector-scenarios-readme.md +253 -253
  325. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-01-foundations.md +452 -452
  326. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-02-simple-scenarios.md +681 -681
  327. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-03-intermediate-scenarios.md +637 -637
  328. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-04-advanced-scenarios.md +650 -650
  329. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-05-bidirectional-sync.md +233 -233
  330. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-06-production-patterns.md +442 -442
  331. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-07-reference.md +445 -445
  332. package/docs/03-PATTERN-GUIDES/connector-scenarios/readme.md +31 -31
  333. package/docs/03-PATTERN-GUIDES/enterprise-integration-patterns.md +1528 -1528
  334. package/docs/03-PATTERN-GUIDES/error-handling/comprehensive-error-handling-guide.md +1437 -1437
  335. package/docs/03-PATTERN-GUIDES/error-handling/error-handling-quick-reference.md +390 -390
  336. package/docs/03-PATTERN-GUIDES/error-handling/examples/common-patterns.ts +438 -438
  337. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-01-foundations.md +362 -362
  338. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-02-error-types.md +850 -850
  339. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-03-utf8-handling.md +456 -456
  340. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-04-error-scenarios.md +658 -658
  341. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-05-calling-patterns.md +671 -671
  342. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-06-retry-strategies.md +1034 -1034
  343. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-07-monitoring.md +653 -653
  344. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-08-api-reference.md +847 -847
  345. package/docs/03-PATTERN-GUIDES/error-handling/readme.md +36 -36
  346. package/docs/03-PATTERN-GUIDES/examples/__tests__/readme.md +40 -40
  347. package/docs/03-PATTERN-GUIDES/examples/__tests__/resolver-examples.test.js +282 -282
  348. package/docs/03-PATTERN-GUIDES/examples/test-data/03-pattern-guides-readme.md +110 -110
  349. package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-inventory.json +123 -123
  350. package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-order.json +171 -171
  351. package/docs/03-PATTERN-GUIDES/examples/test-data/readme.md +28 -28
  352. package/docs/03-PATTERN-GUIDES/extraction/extraction-readme.md +15 -15
  353. package/docs/03-PATTERN-GUIDES/extraction/readme.md +25 -25
  354. package/docs/03-PATTERN-GUIDES/file-operations/examples/common-patterns.ts +407 -407
  355. package/docs/03-PATTERN-GUIDES/file-operations/examples/file-operations-readme.md +142 -142
  356. package/docs/03-PATTERN-GUIDES/file-operations/file-operations-quick-reference.md +462 -462
  357. package/docs/03-PATTERN-GUIDES/file-operations/file-operations-readme.md +379 -379
  358. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-01-foundations.md +430 -430
  359. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-02-quick-start.md +484 -484
  360. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-03-s3-operations.md +507 -507
  361. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-04-sftp-operations.md +963 -963
  362. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-05-streaming-performance.md +503 -503
  363. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-archive-patterns.md +386 -386
  364. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-error-handling.md +117 -117
  365. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-api-reference.md +78 -78
  366. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-testing-troubleshooting.md +567 -567
  367. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-08-api-reference.md +1055 -1055
  368. package/docs/03-PATTERN-GUIDES/file-operations/readme.md +32 -32
  369. package/docs/03-PATTERN-GUIDES/ingestion/ingestion-readme.md +15 -15
  370. package/docs/03-PATTERN-GUIDES/ingestion/readme.md +25 -25
  371. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/batch-processing.ts +130 -130
  372. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/common-patterns.ts +360 -360
  373. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/delta-sync.ts +130 -130
  374. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/integration-patterns-readme.md +100 -100
  375. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/real-time-webhook.ts +398 -398
  376. package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-quick-reference.md +962 -962
  377. package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-readme.md +134 -134
  378. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-01-real-time-processing.md +991 -991
  379. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-02-batch-processing.md +1547 -1547
  380. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-03-delta-sync.md +1108 -1108
  381. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-04-webhook-patterns.md +1181 -1181
  382. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-05-error-handling.md +1061 -1061
  383. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-advanced-integration-services.md +1547 -1547
  384. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-performance.md +109 -109
  385. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-07-api-reference.md +34 -34
  386. package/docs/03-PATTERN-GUIDES/integration-patterns/readme.md +30 -30
  387. package/docs/03-PATTERN-GUIDES/logging-minimal-mode.md +128 -128
  388. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/common-patterns.ts +380 -380
  389. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/multiple-connections-readme.md +139 -139
  390. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/parallel-root-connections.ts +149 -149
  391. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/real-world-scenarios.ts +405 -405
  392. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-01-foundations.md +378 -378
  393. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-02-quick-start.md +566 -566
  394. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-03-targeting-connections.md +659 -659
  395. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-04-parallel-queries.md +656 -656
  396. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-05-best-practices.md +624 -624
  397. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-api-reference.md +824 -824
  398. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-versori.md +119 -119
  399. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-07-api-reference.md +87 -87
  400. package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-quick-reference.md +353 -353
  401. package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-readme.md +270 -270
  402. package/docs/03-PATTERN-GUIDES/multiple-connections/readme.md +30 -30
  403. package/docs/03-PATTERN-GUIDES/pagination/pagination-readme.md +14 -14
  404. package/docs/03-PATTERN-GUIDES/pagination/readme.md +24 -24
  405. package/docs/03-PATTERN-GUIDES/parquet/examples/common-patterns.ts +180 -180
  406. package/docs/03-PATTERN-GUIDES/parquet/examples/read-parquet.ts +48 -48
  407. package/docs/03-PATTERN-GUIDES/parquet/examples/write-parquet.ts +65 -65
  408. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-01-introduction.md +393 -393
  409. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-02-quick-start.md +572 -572
  410. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-03-reading-parquet.md +525 -525
  411. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-04-writing-parquet.md +554 -554
  412. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-05-graphql-extraction.md +405 -405
  413. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-performance.md +104 -104
  414. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-s3-integration.md +511 -511
  415. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-api-reference.md +90 -90
  416. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-performance-optimization.md +525 -525
  417. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-08-best-practices.md +712 -712
  418. package/docs/03-PATTERN-GUIDES/parquet/parquet-quick-reference.md +683 -683
  419. package/docs/03-PATTERN-GUIDES/parquet/parquet-readme.md +248 -248
  420. package/docs/03-PATTERN-GUIDES/parquet/readme.md +32 -32
  421. package/docs/03-PATTERN-GUIDES/parsers/parsers-readme.md +12 -12
  422. package/docs/03-PATTERN-GUIDES/parsers/readme.md +24 -24
  423. package/docs/03-PATTERN-GUIDES/readme.md +159 -159
  424. package/docs/03-PATTERN-GUIDES/webhooks/readme.md +24 -24
  425. package/docs/03-PATTERN-GUIDES/webhooks/webhooks-readme.md +8 -8
  426. package/docs/04-REFERENCE/architecture/architecture-01-overview.md +427 -427
  427. package/docs/04-REFERENCE/architecture/architecture-02-client-architecture.md +424 -424
  428. package/docs/04-REFERENCE/architecture/architecture-03-data-flow.md +690 -690
  429. package/docs/04-REFERENCE/architecture/architecture-04-service-layer.md +834 -834
  430. package/docs/04-REFERENCE/architecture/architecture-05-integration-architecture.md +655 -655
  431. package/docs/04-REFERENCE/architecture/architecture-06-state-management.md +653 -653
  432. package/docs/04-REFERENCE/architecture/architecture-adding-new-data-sources.md +686 -686
  433. package/docs/04-REFERENCE/architecture/readme.md +279 -279
  434. package/docs/04-REFERENCE/platforms/deno/readme.md +117 -117
  435. package/docs/04-REFERENCE/platforms/nodejs/readme.md +146 -146
  436. package/docs/04-REFERENCE/platforms/readme.md +135 -135
  437. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-01-introduction.md +398 -398
  438. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-02-quick-start.md +560 -560
  439. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-03-authentication.md +757 -757
  440. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-04-workflows.md +2476 -2476
  441. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-05-connections.md +1167 -1167
  442. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-kv-storage.md +990 -990
  443. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-state-management.md +121 -121
  444. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-api-reference.md +68 -68
  445. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-deployment.md +731 -731
  446. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-08-best-practices.md +1111 -1111
  447. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-09-signature-reference.md +766 -766
  448. package/docs/04-REFERENCE/platforms/versori/platforms-versori-readme.md +299 -299
  449. package/docs/04-REFERENCE/platforms/versori/platforms-versori-s3-sftp-configuration-guide.md +1425 -1425
  450. package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-api-key-security.md +816 -816
  451. package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-connection-security.md +681 -681
  452. package/docs/04-REFERENCE/platforms/versori/platforms-versori-workflow-task-types.md +708 -708
  453. package/docs/04-REFERENCE/platforms/versori/readme.md +108 -108
  454. package/docs/04-REFERENCE/readme.md +148 -148
  455. package/docs/04-REFERENCE/resolver-signature/examples/advanced-resolvers.ts +482 -482
  456. package/docs/04-REFERENCE/resolver-signature/examples/async-resolvers.ts +496 -496
  457. package/docs/04-REFERENCE/resolver-signature/examples/basic-resolvers.ts +343 -343
  458. package/docs/04-REFERENCE/resolver-signature/examples/resolver-signature-readme.md +188 -188
  459. package/docs/04-REFERENCE/resolver-signature/examples/testing-resolvers.ts +463 -463
  460. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-01-foundations.md +286 -286
  461. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-02-parameter-reference.md +643 -643
  462. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-03-basic-examples.md +521 -521
  463. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-04-advanced-patterns.md +739 -739
  464. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-05-sdk-resolvers.md +531 -531
  465. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-migration-guide.md +650 -650
  466. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-testing.md +125 -125
  467. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-07-api-reference.md +794 -794
  468. package/docs/04-REFERENCE/resolver-signature/readme.md +64 -64
  469. package/docs/04-REFERENCE/resolver-signature/resolver-signature-quick-reference.md +270 -270
  470. package/docs/04-REFERENCE/resolver-signature/resolver-signature-readme.md +351 -351
  471. package/docs/04-REFERENCE/schema/fluent-commerce-schema.json +764 -764
  472. package/docs/04-REFERENCE/schema/readme.md +141 -141
  473. package/docs/04-REFERENCE/testing/examples/04-reference-testing-readme.md +158 -158
  474. package/docs/04-REFERENCE/testing/examples/fluent-testing.ts +62 -62
  475. package/docs/04-REFERENCE/testing/examples/health-check.ts +155 -155
  476. package/docs/04-REFERENCE/testing/examples/integration-test.ts +119 -119
  477. package/docs/04-REFERENCE/testing/examples/performance-test.ts +183 -183
  478. package/docs/04-REFERENCE/testing/examples/s3-testing.ts +127 -127
  479. package/docs/04-REFERENCE/testing/modules/04-reference-testing-01-foundations.md +267 -267
  480. package/docs/04-REFERENCE/testing/modules/04-reference-testing-02-s3-testing.md +599 -599
  481. package/docs/04-REFERENCE/testing/modules/04-reference-testing-03-fluent-testing.md +589 -589
  482. package/docs/04-REFERENCE/testing/modules/04-reference-testing-04-integration-testing.md +699 -699
  483. package/docs/04-REFERENCE/testing/modules/04-reference-testing-05-debugging.md +478 -478
  484. package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-cicd-integration.md +463 -463
  485. package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-preflight-validation.md +131 -131
  486. package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-best-practices.md +499 -499
  487. package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-coverage-ci.md +165 -165
  488. package/docs/04-REFERENCE/testing/modules/04-reference-testing-08-api-reference.md +634 -634
  489. package/docs/04-REFERENCE/testing/readme.md +86 -86
  490. package/docs/04-REFERENCE/testing/testing-quick-reference.md +667 -667
  491. package/docs/04-REFERENCE/testing/testing-readme.md +286 -286
  492. package/docs/04-REFERENCE/troubleshooting/readme.md +144 -144
  493. package/docs/04-REFERENCE/troubleshooting/troubleshooting-deno-sftp-compatibility.md +392 -392
  494. package/docs/template-loading-matrix.md +242 -242
  495. package/package.json +5 -3
@@ -1,1710 +1,1710 @@
1
- ---
2
- template_id: tpl-ingest-sftp-json-to-location-graphql
3
- canonical_filename: template-ingestion-sftp-json-location-graphql.md
4
- version: 2.0.0
5
- sdk_version: ^0.1.39
6
- runtime: versori
7
- direction: ingestion
8
- source: sftp-json
9
- destination: fluent-graphql
10
- entity: location
11
- format: json
12
- logging: versori
13
- status: stable
14
- compliance: gold-standard
15
- features:
16
- - graphql-mutation-mapper
17
- - memory-management
18
- - enhanced-logging
19
- - attribute-transformation
20
- - dispose-finally
21
- ---
22
-
23
- # Template: Ingestion - SFTP JSON to Location GraphQL
24
-
25
- **Deployment Target:** Versori Platform
26
- **Compliance Status:** ✅ GOLD STANDARD APPROVED
27
-
28
- ---
29
-
30
- ## 📋 Implementation Prompt
31
-
32
- ```
33
- I need a Versori scheduled ingestion that:
34
-
35
- 1) Discovers JSON files on SFTP with file tracking to skip duplicates
36
- 2) Downloads and parses JSON (locations array) with JSONParserService
37
- 3) Transforms records with GraphQLMutationMapper (NOT UniversalMapper)
38
- 4) Executes GraphQL createLocation mutations with rate limiting
39
- 5) Archives files to processed/ or errors/ and writes error reports
40
- 6) Tracks progress with JobTracker and exposes job status webhook
41
- 7) Uses native Versori log and follows modular service architecture
42
-
43
- CRITICAL: This template uses GraphQLMutationMapper for GraphQL mutations,
44
- NOT UniversalMapper (which is for Batch API). Do NOT use setRetailerId()
45
- for GraphQL mutations - it's only needed for Job/Event API.
46
-
47
- Use the loaded docs to fill in SDK specifics and best practices.
48
- ```
49
-
50
- ---
51
-
52
- ## 📋 Template Overview
53
-
54
- This connector runs on the Versori platform with **gold standard compliance**. It reads location data from SFTP JSON files, transforms it with GraphQLMutationMapper, and creates/updates locations via GraphQL mutations.
55
-
56
- ### What This Template Does
57
-
58
- ```
59
- ┌─────────────────────────────────────────────────────────────────┐
60
- │ INGESTION WORKFLOW (GraphQL Mutations Pattern) │
61
- └─────────────────────────────────────────────────────────────────┘
62
-
63
- 1. TRIGGER
64
- ├─ Scheduled (Cron): Daily at 2 AM
65
- ├─ Ad hoc (Webhook): Manual trigger
66
- └─ Status Query (Webhook): Check progress
67
-
68
- 2. DISCOVER FILES (SftpDataSource)
69
- ├─ List files matching pattern
70
- ├─ Check VersoriFileTracker (skip processed)
71
- └─ Sort by oldest first
72
-
73
- 3. DOWNLOAD & PARSE (LocationFileProcessorService)
74
- ├─ Download from SFTP
75
- ├─ Parse JSON with JSONParserService
76
- ├─ Validate structure with type guards
77
- └─ Return LocationRecord[]
78
-
79
- 4. TRANSFORM (GraphQLMutationMapper)
80
- ├─ Map to GraphQL CreateLocationInput
81
- ├─ Handle nested objects (primaryAddress, openingSchedule)
82
- ├─ Validate against GraphQL schema
83
- └─ Generate mutation query
84
-
85
- 5. EXECUTE MUTATIONS (MutationSenderService)
86
- ├─ Execute createLocation mutations
87
- ├─ Rate limiting (configurable concurrency)
88
- ├─ Per-record error handling
89
- └─ Track success/failure per mutation
90
-
91
- 6. ARCHIVE & LOG
92
- ├─ Move to processed/ or errors/
93
- ├─ Write error reports
94
- └─ Track state in VersoriFileTracker
95
-
96
- 7. JOB TRACKING
97
- ├─ Update job status at each step
98
- └─ Enable status queries via webhook
99
- ```
100
-
101
- ### Key Features
102
-
103
- - ✅ **Gold Standard Compliant** - Modular architecture, proper patterns
104
- - ✅ **GraphQLMutationMapper** - Correct mapper for GraphQL mutations
105
- - ✅ **retailerId Configuration** - CRITICAL for GraphQL API calls
106
- - ✅ **Type-Safe** - Comprehensive TypeScript interfaces
107
- - ✅ **Modular Services** - 4 services + utils (testable, reusable)
108
- - ✅ **External Config** - Mapping in JSON file
109
- - ✅ **Error Handling** - Per-record failures don't block others
110
- - ✅ **File Tracking** - Prevents duplicate processing
111
- - ✅ **Job Tracking** - Status queries via webhook
112
-
113
- ---
114
-
115
- ## 📦 SDK Imports
116
-
117
- ```typescript
118
- import { Buffer } from 'node:buffer'; // Required for Versori/Deno
119
-
120
- import {
121
- createClient,
122
- SftpDataSource,
123
- JSONParserService,
124
- GraphQLMutationMapper, // ✅ CRITICAL: Use this for GraphQL (NOT UniversalMapper)
125
- VersoriFileTracker,
126
- JobTracker,
127
- } from '@fluentcommerce/fc-connect-sdk';
128
-
129
- import type {
130
- FluentClient,
131
- FileMetadata,
132
- Logger,
133
- } from '@fluentcommerce/fc-connect-sdk';
134
-
135
- import { schedule, webhook, http, fn } from '@versori/run';
136
- ```
137
-
138
- **✅ Type-Only Imports:** Separating value and type imports improves tree-shaking and bundle size.
139
-
140
- ---
141
-
142
- ---
143
-
144
- ## SFTP Connection Setup (Recommended)
145
-
146
- **🔒 BEST PRACTICE:** Store SFTP credentials in a Versori connection object with Basic Auth:
147
-
148
- ### Connection Configuration
149
-
150
- 1. In Versori platform, create a connection named `SFTP`
151
- 2. Set **Authentication Type**: `Basic Auth`
152
- 3. Enter **Username**: Your SFTP username
153
- 4. Enter **Password**: Your SFTP password
154
- 5. The SDK will automatically retrieve and decode credentials
155
-
156
- ### Code Implementation
157
-
158
- The workflow retrieves credentials programmatically from the Versori connection:
159
-
160
- ```typescript
161
- import { Buffer } from 'node:buffer';
162
-
163
- // Retrieve SFTP credentials from connection configuration
164
- log.info('Retrieving SFTP credentials from connection configuration');
165
-
166
- let sftpUsername: string;
167
- let sftpPassword: string;
168
-
169
- try {
170
- // Retrieve credentials from the 'SFTP' connection
171
- const sftpCred = await ctx.credentials().getAccessToken('SFTP');
172
-
173
- if (!sftpCred?.accessToken) {
174
- throw new Error('No SFTP credentials found in connection configuration');
175
- }
176
-
177
- // Decode base64 accessToken to get "username:password"
178
- const rawBasicAuth = Buffer.from(sftpCred.accessToken, 'base64').toString('utf-8');
179
-
180
- // Split on ':' to extract username and password
181
- const parts = rawBasicAuth.split(':');
182
-
183
- if (parts.length !== 2) {
184
- throw new Error('Invalid SFTP credential format - expected username:password');
185
- }
186
-
187
- sftpUsername = parts[0];
188
- sftpPassword = parts[1];
189
-
190
- log.info('SFTP credentials retrieved successfully', {
191
- hasUsername: !!sftpUsername,
192
- hasPassword: !!sftpPassword,
193
- usernameLength: sftpUsername.length,
194
- passwordLength: sftpPassword.length,
195
- });
196
- } catch (error: any) {
197
- log.error('Failed to retrieve SFTP credentials', {
198
- message: error instanceof Error ? error.message : String(error),
199
- stack: error instanceof Error ? error.stack : undefined,
200
- errorType: error instanceof Error ? error.constructor.name : 'Error'
201
- });
202
-
203
- return {
204
- success: false,
205
- error: 'Failed to retrieve SFTP credentials from connection configuration',
206
- details: error?.message,
207
- recommendation: 'Please ensure the SFTP connection is configured in the Connections section with Basic Authentication (username and password)',
208
- };
209
- }
210
-
211
- // Use credentials in SftpDataSource
212
- const sftp = new SftpDataSource({
213
- type: 'SFTP_JSON',
214
- connectionId: 'sftp-location-sync',
215
- name: 'Location Sync SFTP',
216
- settings: {
217
- host: activation.getVariable('sftpHost'),
218
- port: parseInt(activation.getVariable('sftpPort') || '22', 10),
219
- username: sftpUsername, // From connection
220
- password: sftpPassword, // From connection
221
- remotePath: activation.getVariable('sftpRemotePath'),
222
- filePattern: activation.getVariable('filePattern'),
223
- encoding: 'utf8',
224
- },
225
- }, log);
226
- ```
227
-
228
- ### Why Use Connections Instead of Activation Variables?
229
-
230
- | Aspect | Activation Variables ❌ | Versori Connections ✅ |
231
- |--------|------------------------|------------------------|
232
- | **Security** | Plain text in config UI | Encrypted in Versori vault |
233
- | **Visibility** | Visible to all users | Controlled access |
234
- | **Rotation** | Manual update per workflow | Update once, affects all |
235
- | **Best Practice** | Legacy approach | **Recommended approach** |
236
- | **Audit Trail** | Limited | Full connection audit logs |
237
-
238
- **Benefits:**
239
- - ✅ Credentials stored securely in Versori vault
240
- - ✅ Connection can be reused across workflows
241
- - ✅ No sensitive data in activation variables
242
- - ✅ Easier credential rotation (one place to update)
243
- - ✅ Better security posture (credentials never exposed in UI)
244
-
245
- ---
246
-
247
- ## ⚙️ Activation Variables
248
-
249
- **Configuration is driven by activation variables - modify these instead of code:**
250
-
251
- > **Note:** `sftpUsername` and `sftpPassword` are fetched from the `SFTP` Basic Auth connection (see SFTP Connection Setup above).
252
-
253
- ```bash
254
- # SFTP Configuration
255
- SFTP_HOST=sftp.partner.com
256
- SFTP_PORT=22
257
-
258
- # SFTP Paths
259
- SFTP_REMOTE_PATH=/locations/incoming
260
- SFTP_ARCHIVE_PATH=/locations/processed
261
- SFTP_ERROR_PATH=/locations/errors
262
-
263
- # File Processing
264
- FILE_PATTERN=locations_*.json
265
- MAX_FILES_PER_RUN=10
266
-
267
- # GraphQL Configuration
268
- FLUENT_RETAILER_ID=1 # Optional: Only if mutation schema requires retailerId in input
269
- MAX_PARALLEL_MUTATIONS=10 # Concurrent mutation limit (1=sequential, 3-10=parallel)
270
-
271
- # Feature Toggles
272
- REQUIRE_ABSOLUTE_PATHS=true # "true" for AWS Transfer Family, "false" for standard OpenSSH
273
- VALIDATE_CONNECTION=true # Validate SFTP connection on startup (default: true)
274
- ENABLE_FILE_TRACKING=true # Enable/disable file tracking (default: true)
275
- ```
276
-
277
- **Security:**
278
- - **SFTP Authentication:** Credentials stored in Versori connection vault (not in activation variables). See [SFTP Connection Setup](#sftp-connection-setup-recommended) section above.
279
- - **Webhook Authentication:** Enforced by Versori connection configuration. Configure your webhook connection with API key authentication in the Versori Dashboard, then reference it in `webhook({ connection: 'webhook-auth' })`.
280
-
281
- ---
282
-
283
- ## 📄 Mapping Configuration
284
-
285
- **File:** `src/config/location-mapping.json`
286
-
287
- **✅ PRODUCTION STANDARD:** External JSON file with GraphQLMutationMapper structure
288
-
289
- ```json
290
- {
291
- "mutation": "createLocation",
292
- "sourceFormat": "json",
293
- "version": "1.0.0",
294
- "returnFields": ["id", "ref", "name", "type", "status"],
295
- "arguments": {
296
- "input": {
297
- "ref": {
298
- "source": "locationRef",
299
- "required": true,
300
- "resolver": "trim"
301
- },
302
- "name": {
303
- "source": "locationName",
304
- "required": true,
305
- "resolver": "trim"
306
- },
307
- "type": {
308
- "source": "locationType",
309
- "required": true,
310
- "resolver": "toUpperCase"
311
- },
312
- "status": {
313
- "value": "ACTIVE",
314
- "required": true
315
- },
316
- "primaryAddress": {
317
- "ref": {
318
- "source": "locationRef",
319
- "required": true,
320
- "resolver": "trim"
321
- },
322
- "street": {
323
- "source": "address.street1",
324
- "resolver": "trim"
325
- },
326
- "city": {
327
- "source": "address.city",
328
- "resolver": "trim"
329
- },
330
- "state": {
331
- "source": "address.state",
332
- "resolver": "toUpperCase"
333
- },
334
- "postcode": {
335
- "source": "address.postalCode",
336
- "resolver": "trim"
337
- },
338
- "country": {
339
- "source": "address.country",
340
- "resolver": "toUpperCase"
341
- },
342
- "latitude": {
343
- "source": "geo.latitude",
344
- "required": true,
345
- "resolver": "parseFloat"
346
- },
347
- "longitude": {
348
- "source": "geo.longitude",
349
- "required": true,
350
- "resolver": "parseFloat"
351
- },
352
- "timeZone": {
353
- "source": "timeZone",
354
- "resolver": "trim"
355
- }
356
- },
357
- "openingSchedule": {
358
- "allHours": {
359
- "source": "openingSchedule.allHours",
360
- "required": true,
361
- "resolver": "toBoolean"
362
- },
363
- "monStart": {
364
- "source": "openingSchedule.monStart",
365
- "required": true,
366
- "resolver": "parseInt"
367
- },
368
- "monEnd": {
369
- "source": "openingSchedule.monEnd",
370
- "required": true,
371
- "resolver": "parseInt"
372
- }
373
- }
374
- }
375
- }
376
- }
377
- ```
378
-
379
- **Key Differences from UniversalMapper:**
380
- - ✅ `mutation` property (not `mutationName`)
381
- - ✅ `arguments.input` structure (matches GraphQL schema)
382
- - ✅ Nested objects (not dotted paths like `primaryAddress.street`)
383
- - ✅ Resolver names without `sdk.` prefix (`trim`, `toUpperCase`, `parseInt`, `parseFloat`, `toBoolean`)
384
- - ✅ Used with `GraphQLMutationMapper.map()` method (not `UniversalMapper.map()`)
385
-
386
- ---
387
-
388
- ## Expected JSON Format
389
-
390
- ```json
391
- {
392
- "locations": [
393
- {
394
- "locationRef": "LOC-001",
395
- "locationName": "Main Warehouse",
396
- "locationType": "WAREHOUSE",
397
- "address": {
398
- "street1": "123 Main St",
399
- "city": "New York",
400
- "state": "NY",
401
- "postalCode": "10001",
402
- "country": "US"
403
- },
404
- "geo": {
405
- "latitude": 40.7128,
406
- "longitude": -74.0060
407
- },
408
- "timeZone": "America/New_York",
409
- "openingSchedule": {
410
- "allHours": false,
411
- "monStart": 480,
412
- "monEnd": 1020
413
- }
414
- }
415
- ]
416
- }
417
- ```
418
-
419
- **Alternative:** Root array `[{...}]` is also supported.
420
-
421
- ---
422
-
423
- ## 🔧 Production Code Structure
424
-
425
- ### Modular Architecture (Gold Standard)
426
-
427
- ```
428
- sftp-json-location-graphql/
429
- ├── package.json
430
- ├── tsconfig.json
431
- ├── index.ts # Entry point
432
- └── src/
433
- ├── workflows/
434
- │ └── location-ingestion.ts # 3 workflows
435
- ├── services/
436
- │ ├── location-file-processor.service.ts # Download, parse, transform
437
- │ ├── mutation-sender.service.ts # Execute GraphQL mutations
438
- │ ├── mutation-logger.service.ts # Write error logs
439
- │ └── location-ingestion.service.ts # Main orchestration
440
- ├── types/
441
- │ └── location-ingestion.types.ts # TypeScript interfaces
442
- ├── utils/
443
- │ ├── sftp-path.utils.ts # Path helpers
444
- │ ├── retry.utils.ts # Retry logic
445
- │ └── job-id-generator.ts # Job IDs
446
- └── config/
447
- └── location-mapping.json # GraphQL mapping config
448
- ```
449
-
450
- ---
451
-
452
- ## Critical Implementation Patterns
453
-
454
- ### ✅ MUST HAVE: retailerId Configuration ⚠️ VERIFICATION REQUIRED
455
-
456
- ```typescript
457
- // In main orchestration service (location-ingestion.service.ts)
458
-
459
- export async function executeLocationIngestion(ctx: VersoriContext, params: LocationIngestionParams) {
460
- const { log, activation } = ctx;
461
-
462
- // Step 1: Create client
463
- const client = await createClient(ctx);
464
- if (!client) {
465
- throw new Error('Failed to create Fluent Commerce client');
466
- }
467
-
468
- // ✅ CORRECT: GraphQL mutations don't need setRetailerId()
469
- // Check your GraphQL schema to determine retailerId handling:
470
- // - Mandatory retailerId → Must pass it in mutation input
471
- // - Optional retailerId → Can pass it if needed
472
- // - No retailerId field → Don't pass it
473
- // See: fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md
474
-
475
- // Step 2: Initialize GraphQLMutationMapper with client
476
- import mappingConfig from '../config/location-mapping.json' with { type: 'json' };
477
- const mapper = new GraphQLMutationMapper(mappingConfig, log, { fluentClient: client });
478
-
479
- // Step 3: Process files and execute mutations
480
- // ... rest of workflow
481
-
482
- // ✅ Configuration with defaults
483
- const mutationBatchSize = parseInt(
484
- activation?.getVariable('mutationBatchSize') || '1', // ✅ Default: 1 (sequential)
485
- 10
486
- );
487
-
488
- const mutationsPerAliasBatch = activation?.getVariable('mutationsPerAliasBatch')
489
- ? parseInt(activation?.getVariable('mutationsPerAliasBatch') || '1', 10)
490
- : undefined; // ✅ Default: undefined (disabled, use separate requests)
491
-
492
- // ? Enhanced: Extract context for progress logging
493
- const sampleLocationRefs = locations.slice(0, 5).map((loc: any) => loc.ref || loc.input?.ref || 'unknown');
494
- const mutationType = mapper?.mutationName || 'createLocation';
495
-
496
- // ? Enhanced: Start logging with context
497
- log.info(`[GraphQLMutations] Sending mutations for locations`, {
498
- totalMutations: locations.length,
499
- mutationType,
500
- batchSize: mutationBatchSize,
501
- batchMode: mutationBatchSize === 1 ? 'sequential' : `parallel (${mutationBatchSize})`,
502
- sampleLocationRefs: sampleLocationRefs.join(', '),
503
- aliasBatching: mutationsPerAliasBatch ? `enabled (${mutationsPerAliasBatch} per alias)` : 'disabled'
504
- });
505
-
506
- // Use in executeMutations:
507
- const mutationResults = await mutationSender.executeMutations(
508
- locations,
509
- mutationBatchSize, // Concurrency control (default: 1)
510
- mutationsPerAliasBatch // ✅ NEW: Alias batching (default: undefined)
511
- );
512
-
513
- // ? Enhanced: Completion logging with summary
514
- log.info(`[GraphQLMutations] Mutation submission completed`, {
515
- totalMutations: locations.length,
516
- mutationsExecuted: mutationResults.mutationsExecuted,
517
- mutationsFailed: mutationResults.mutationsFailed,
518
- successRate: locations.length > 0 ? `${Math.round((mutationResults.mutationsExecuted / locations.length) * 100)}%` : '0%',
519
- mutationType
520
- });
521
- }
522
- ```
523
-
524
- **✅ VERIFIED Pattern:**
525
- - GraphQL mutations do NOT need `client.setRetailerId()`
526
- - `setRetailerId()` is only for Job API and Event API
527
- - Pass retailerId in mutation input variables if schema requires it
528
-
529
- **Example (For mutations that require retailerId in schema):**
530
- ```typescript
531
- // Only needed if mutation schema explicitly requires retailerId
532
- const fluentRetailerId = activation.getVariable('fluentRetailerId');
533
- const { query, variables } = await mapper.map(location);
534
-
535
- // Add to input if mutation schema requires it
536
- if (fluentRetailerId && variables.input) {
537
- variables.input.retailer = { id: parseInt(fluentRetailerId) };
538
- }
539
-
540
- await client.graphql({ query, variables });
541
- ```
542
-
543
- **Note:** Check your GraphQL schema (via introspection) to determine retailerId handling:
544
- 1. Does the mutation have `retailerId` or `retailer.id` field?
545
- 2. Is it mandatory (`!`) or optional?
546
- 3. If field doesn't exist → don't pass it
547
-
548
- ### ✅ MUST HAVE: Double-Finally SFTP Disposal
549
-
550
- **What:** Defensive resource cleanup with nested finally blocks (matches Event API gold standard)
551
-
552
- **Gold Standard Pattern:**
553
-
554
- ```typescript
555
- // ✅ CRITICAL: Declare SFTP outside try block for safe disposal
556
- let sftp: SftpDataSource | undefined;
557
-
558
- try {
559
- // 📊 Execution boundary: Start workflow
560
- const startTime = Date.now();
561
- log.info('🚀 Starting location ingestion workflow', {
562
- stage: 'initialization',
563
- jobId,
564
- triggeredBy: params.triggeredBy
565
- });
566
-
567
- // Initialize SFTP
568
- sftp = new SftpDataSource(...);
569
-
570
- // ✅ Validate connection (if enabled)
571
- const validateConnection = activation?.getVariable('validateConnection') === 'true';
572
- if (validateConnection) {
573
- log.info('🔌 Validating SFTP connection', { stage: 'connection_validation' });
574
- await sftp.validateConnection();
575
- log.info('✅ SFTP connection validated successfully');
576
- }
577
-
578
- try {
579
- // Use SFTP
580
- await sftp.listFiles(...);
581
- await sftp.downloadFile(...);
582
- // ... rest of workflow
583
-
584
- // 📊 Execution boundary: End workflow
585
- const duration = Date.now() - startTime;
586
- log.info('✅ Location ingestion workflow completed', {
587
- stage: 'completion',
588
- jobId,
589
- duration,
590
- filesProcessed,
591
- mutationsExecuted
592
- });
593
- } finally {
594
- // Inner finally - normal cleanup
595
- if (sftp) {
596
- await sftp.dispose();
597
- log.info('🔌 SFTP connection disposed');
598
- }
599
- }
600
- } catch (error: unknown) {
601
- // ✅ Enhanced error logging: Extract all error details + recommendations
602
- const errorMessage = error instanceof Error ? error.message : String(error);
603
- const errorDetails = {
604
- message: errorMessage,
605
- stack: error instanceof Error ? error.stack : undefined,
606
- errorType: error instanceof Error ? error.constructor.name : 'Error',
607
- stage: 'workflow_execution',
608
- jobId
609
- };
610
-
611
- // ✅ Add error recommendations
612
- const recommendations: string[] = [];
613
- if (errorMessage.includes('ECONNREFUSED')) {
614
- recommendations.push('Check SFTP host and port configuration');
615
- recommendations.push('Verify network connectivity to SFTP server');
616
- } else if (errorMessage.includes('Authentication failed')) {
617
- recommendations.push('Verify SFTP username and password');
618
- recommendations.push('Check if account is locked or expired');
619
- } else if (errorMessage.includes('No such file')) {
620
- recommendations.push('Verify incoming path exists on SFTP server');
621
- recommendations.push('Check file pattern matches existing files');
622
- }
623
-
624
- log.error('❌ Workflow failed', { ...errorDetails, recommendations });
625
-
626
- // Handle error (mark job failed, etc.)
627
- } finally {
628
- // Outer finally - defensive cleanup
629
- // ⚠️ CRITICAL: Ensures disposal even if error occurs after SFTP creation
630
- // but before inner try block (e.g., during connection validation)
631
- if (sftp) {
632
- try {
633
- await sftp.dispose();
634
- log.info('🔌 SFTP connection disposed (outer finally)');
635
- } catch (disposeError: unknown) {
636
- const disposeErrorMessage = disposeError instanceof Error
637
- ? disposeError.message
638
- : String(disposeError);
639
- log.warn('⚠️ Error disposing SFTP in outer finally', { error: disposeErrorMessage });
640
- }
641
- }
642
- }
643
- ```
644
-
645
- **Why This is Critical:**
646
- - Ensures disposal even if error occurs after SFTP creation but before inner try block
647
- - Defensive cleanup catches edge cases
648
- - Prevents resource leaks in error scenarios
649
- - Matches Event API gold standard pattern
650
-
651
- ---
652
-
653
- ## Processing Modes
654
-
655
- **⚠️ IMPORTANT:** Choose ONE processing mode per connector. These are alternative patterns, not features to use together.
656
-
657
- The SDK supports two processing modes for handling multiple files. **Select the mode that best fits your use case** and configure your connector accordingly:
658
-
659
- ### Mode 1: Per-File Processing (Recommended Default) ✅ IMPLEMENTED
660
-
661
- **When to use:**
662
- - Multiple large files that shouldn't be in memory together
663
- - Need file-level consistency (file 3 fails → files 1-2 already archived)
664
- - Clear error isolation per file
665
-
666
- **How it works:**
667
- 1. Process file 1 completely (download → parse → transform → execute mutations → archive)
668
- 2. Move to file 2
669
- 3. Continue sequentially
670
-
671
- **Benefits:**
672
- - ✅ Low memory usage (one file at a time)
673
- - ✅ Clear error isolation (failed file doesn't block others)
674
- - ✅ Incremental progress (files archived as completed)
675
-
676
- **Example implementation (see code section below)**
677
-
678
- ### Mode 2: Chunked Per-File Processing ⚠️ OPTIONAL - Choose if needed
679
-
680
- **When to use:**
681
- - Many small files that can be processed in parallel
682
- - Need better throughput for many files
683
- - Files are independent (no cross-file dependencies)
684
-
685
- **How it works:**
686
- 1. Group files into chunks (e.g., 5 files per chunk)
687
- 2. Process chunk 1 files in parallel
688
- 3. Wait for chunk completion, then move to chunk 2
689
-
690
- **Benefits:**
691
- - ✅ Better throughput (parallel processing)
692
- - ✅ Still maintains file-level isolation
693
- - ✅ Configurable chunk size
694
-
695
- **Note:** GraphQL mutations are typically one-off operations, so per-file processing is usually sufficient. Chunked mode is optional for high-volume scenarios.
696
-
697
- ---
698
-
699
- ## Key Service Patterns
700
-
701
- ### Service 1: Location File Processor (Complete Implementation)
702
-
703
- **File:** `src/services/location-file-processor.service.ts`
704
-
705
- ```typescript
706
- /**
707
- * Location File Processor Service
708
- *
709
- * Downloads JSON files from SFTP, parses content, and transforms with GraphQLMutationMapper.
710
- * Reusable service for location file processing workflows.
711
- */
712
-
713
- import {
714
- SftpDataSource,
715
- JSONParserService,
716
- GraphQLMutationMapper,
717
- } from '@fluentcommerce/fc-connect-sdk';
718
- import type {
719
- ProcessFileResult,
720
- LocationRecord,
721
- } from '../types/location-ingestion.types';
722
- import { extractFileName } from '../utils/sftp-path.utils';
723
-
724
- /**
725
- * Location file processing service
726
- * Coordinates SFTP download, JSON parsing (via SDK), and GraphQL mutation generation
727
- */
728
- export class LocationFileProcessorService {
729
- constructor(
730
- private sftp: SftpDataSource,
731
- private jsonParser: JSONParserService,
732
- private mapper: GraphQLMutationMapper
733
- ) {} // ✅ No logger param - workflow handles logging
734
-
735
- /**
736
- * Download JSON file from SFTP, parse, and generate GraphQL mutations
737
- */
738
- async downloadParseAndTransform(remoteFilePath: string): Promise<ProcessFileResult> {
739
- // ✅ Extract filename for logging
740
- const fileName = extractFileName(remoteFilePath);
741
-
742
- try {
743
- // Download JSON content
744
- const content = (await this.sftp.downloadFile(remoteFilePath, {
745
- encoding: 'utf8',
746
- })) as string;
747
-
748
- // Parse JSON with type safety
749
- let parsed: unknown;
750
- try {
751
- parsed = await this.jsonParser.parse(content);
752
- } catch (parseError: unknown) {
753
- const errorMessage = parseError instanceof Error ? parseError.message : String(parseError);
754
- return {
755
- success: false,
756
- locations: [],
757
- error: `JSON parse error: ${errorMessage}`,
758
- fileName,
759
- };
760
- }
761
-
762
- // Normalize locations array (handle both { locations: [...] } and root array formats)
763
- const jsonData = parsed as Record<string, unknown>;
764
- const locations = jsonData.locations
765
- ? (Array.isArray(jsonData.locations) ? jsonData.locations : [jsonData.locations])
766
- : (Array.isArray(jsonData) ? jsonData : []);
767
-
768
- if (locations.length === 0) {
769
- return {
770
- success: false,
771
- locations: [],
772
- error: 'No locations found in JSON',
773
- fileName,
774
- };
775
- }
776
-
777
- // Validate with type guard
778
- const validLocations = locations.filter(isLocationRecord);
779
-
780
- if (validLocations.length === 0) {
781
- return {
782
- success: false,
783
- locations: [],
784
- error: 'No valid locations found after validation',
785
- fileName,
786
- };
787
- }
788
-
789
- // Transform with GraphQLMutationMapper
790
- const mapped = await Promise.all(
791
- validLocations.map(loc => this.mapper.map(loc))
792
- );
793
-
794
- return {
795
- success: true,
796
- locations: mapped.map(m => m.variables.input),
797
- fileName,
798
- };
799
- } catch (error: unknown) {
800
- const errorMessage = error instanceof Error ? error.message : String(error);
801
- return {
802
- success: false,
803
- locations: [],
804
- error: errorMessage,
805
- fileName,
806
- };
807
- }
808
- }
809
- }
810
-
811
- /**
812
- * Type guard: Check if parsed value is a valid LocationRecord
813
- */
814
- function isLocationRecord(obj: unknown): obj is LocationRecord {
815
- if (typeof obj !== 'object' || obj === null) return false;
816
- const loc = obj as Record<string, unknown>;
817
-
818
- return (
819
- typeof loc.locationRef === 'string' &&
820
- typeof loc.locationName === 'string' &&
821
- typeof loc.locationType === 'string' &&
822
- typeof loc.address === 'object' &&
823
- loc.address !== null &&
824
- typeof loc.geo === 'object' &&
825
- loc.geo !== null
826
- );
827
- }
828
- ```
829
-
830
- ### Service 2: Mutation Sender (Complete Implementation)
831
-
832
- **File:** `src/services/mutation-sender.service.ts`
833
-
834
- ```typescript
835
- /**
836
- * Mutation Sender Service
837
- *
838
- * Executes GraphQL mutations with per-record error handling.
839
- * Continues processing on individual failures (GraphQL mutation pattern).
840
- */
841
-
842
- import { GraphQLMutationMapper } from '@fluentcommerce/fc-connect-sdk';
843
- import type { FluentClient } from '@fluentcommerce/fc-connect-sdk';
844
- import type { MutationResult, MappedLocation } from '../types/location-ingestion.types';
845
-
846
- /**
847
- * Service for executing GraphQL mutations
848
- */
849
- export class MutationSenderService {
850
- constructor(
851
- private client: FluentClient,
852
- private mapper: GraphQLMutationMapper
853
- ) {}
854
-
855
- /**
856
- * Execute GraphQL mutations with configurable concurrency and alias batching
857
- *
858
- * **Performance Characteristics:**
859
- * - `maxParallel: 1` → Sequential processing (safe default, ~1 mutation/sec)
860
- * - `maxParallel: 3-5` → Balanced throughput (~3-5 mutations/sec)
861
- * - `maxParallel: 10` → High-volume processing (~10 mutations/sec, 100+ locations)
862
- *
863
- * **Implementation Strategy:**
864
- * - Sequential (1): Optimized loop (no Promise.allSettled overhead)
865
- * - Parallel (>1): Chunked processing with bounded concurrency
866
- * - Alias batching: Groups mutations into aliased GraphQL requests (reduces network overhead)
867
- * - Both modes: Per-record error tracking (failures don't block others)
868
- *
869
- * **Alias Batching:**
870
- * - When `mutationsPerAliasBatch > 1`, groups mutations into aliased requests
871
- * - Example: 5 mutations → 1 HTTP request with aliases (createLocation1, createLocation2, ...)
872
- * - Reduces network overhead by ~80% for high-volume scenarios
873
- *
874
- * @param locations - Array of mapped locations to create/update
875
- * @param maxParallel - Number of concurrent mutations or alias batches (default: 1, min: 1)
876
- * @param mutationsPerAliasBatch - Optional: Number of mutations per aliased request (default: undefined = disabled)
877
- * @returns MutationResult with counts (mutationsExecuted/mutationsFailed) and error details
878
- */
879
- async executeMutations(
880
- locations: MappedLocation[],
881
- maxParallel: number = 1, // ✅ Default: 1 (sequential)
882
- mutationsPerAliasBatch?: number // ✅ NEW: Alias batching parameter (default: undefined = disabled)
883
- ): Promise<MutationResult> {
884
- // Determine mode: use aliases if mutationsPerAliasBatch is set and > 1
885
- const useAliases = mutationsPerAliasBatch && mutationsPerAliasBatch > 1;
886
-
887
- if (useAliases) {
888
- return await this.executeMutationsWithAliases(
889
- locations,
890
- maxParallel,
891
- mutationsPerAliasBatch!
892
- );
893
- } else {
894
- return await this.executeMutationsSeparate(locations, maxParallel);
895
- }
896
- }
897
-
898
- /**
899
- * Execute mutations using separate concurrent requests (current mode)
900
- */
901
- private async executeMutationsSeparate(
902
- locations: MappedLocation[],
903
- maxParallel: number
904
- ): Promise<MutationResult> {
905
- // Validate concurrency (guard against invalid values)
906
- const safeConc = Math.max(1, Math.floor(maxParallel));
907
-
908
- // Result accumulators
909
- let mutationsExecuted = 0;
910
- let mutationsFailed = 0;
911
- const errors: Array<{ locationRef: string; error: string }> = [];
912
-
913
- // ============================================================================
914
- // SEQUENTIAL MODE (maxParallel === 1)
915
- // ============================================================================
916
- if (safeConc === 1) {
917
- for (const location of locations) {
918
- try {
919
- const { query, variables } = await this.mapper.map(location);
920
- await this.client.graphql({ query, variables });
921
- mutationsExecuted++;
922
- } catch (err: unknown) {
923
- mutationsFailed++;
924
- const errorMsg = err instanceof Error ? err.message : String(err);
925
- errors.push({
926
- locationRef: location.ref || 'unknown',
927
- error: errorMsg,
928
- });
929
- // Continue processing (failure doesn't block other locations)
930
- }
931
- }
932
- return { mutationsExecuted, mutationsFailed, errors };
933
- }
934
-
935
- // ============================================================================
936
- // PARALLEL MODE (maxParallel > 1)
937
- // ============================================================================
938
- // Chunked processing with bounded concurrency
939
- for (let i = 0; i < locations.length; i += safeConc) {
940
- const chunk = locations.slice(i, i + safeConc);
941
-
942
- // Fire all mutations in chunk concurrently
943
- const results = await Promise.allSettled(
944
- chunk.map(async (location) => {
945
- const { query, variables } = await this.mapper.map(location);
946
- return await this.client.graphql({ query, variables });
947
- })
948
- );
949
-
950
- // Aggregate chunk results
951
- results.forEach((result, idx) => {
952
- if (result.status === 'fulfilled') {
953
- mutationsExecuted++;
954
- } else {
955
- mutationsFailed++;
956
- const error = result.reason;
957
- errors.push({
958
- locationRef: chunk[idx]?.ref || 'unknown',
959
- error: error?.message || String(error) || 'Unknown error',
960
- });
961
- }
962
- });
963
-
964
- // Small delay between batches to avoid rate limiting
965
- if (i + safeConc < locations.length) {
966
- await new Promise(resolve => setTimeout(resolve, 100));
967
- }
968
- }
969
-
970
- return { mutationsExecuted, mutationsFailed, errors };
971
- }
972
-
973
- /**
974
- * ✅ NEW: Execute mutations using GraphQL aliases (batched requests)
975
- *
976
- * Groups mutations into alias batches and executes them with concurrency control.
977
- * Each aliased request contains multiple mutations identified by alias names.
978
- *
979
- * @param locations - Array of mapped locations
980
- * @param maxParallel - Number of concurrent alias batch requests
981
- * @param mutationsPerAliasBatch - Number of mutations per aliased request
982
- * @returns Mutation execution results
983
- */
984
- private async executeMutationsWithAliases(
985
- locations: MappedLocation[],
986
- maxParallel: number,
987
- mutationsPerAliasBatch: number
988
- ): Promise<MutationResult> {
989
- const results: MutationResult = { mutationsExecuted: 0, mutationsFailed: 0, errors: [] };
990
-
991
- // Extract mutation name from mapper config
992
- const mutationName = (this.mapper as any).config.mutation || 'createLocation';
993
-
994
- // Group locations into alias batches
995
- const aliasBatches: Array<MappedLocation[]> = [];
996
- for (let i = 0; i < locations.length; i += mutationsPerAliasBatch) {
997
- aliasBatches.push(locations.slice(i, i + mutationsPerAliasBatch));
998
- }
999
-
1000
- // Process batches with concurrency control
1001
- for (let i = 0; i < aliasBatches.length; i += maxParallel) {
1002
- const concurrentBatches = aliasBatches.slice(i, i + maxParallel);
1003
-
1004
- const batchResults = await Promise.allSettled(
1005
- concurrentBatches.map(async (batch) => {
1006
- // Build aliased query and variables
1007
- const { query, variables } = await this.buildAliasedBatch(batch, mutationName);
1008
-
1009
- // Execute aliased mutation
1010
- const response = await this.client.graphql({ query, variables });
1011
-
1012
- // Parse results and errors
1013
- return this.parseAliasResponse(response, batch, mutationName);
1014
- })
1015
- );
1016
-
1017
- // Aggregate results
1018
- batchResults.forEach((result, idx) => {
1019
- if (result.status === 'fulfilled') {
1020
- const batchResult = result.value;
1021
- results.mutationsExecuted += batchResult.executed;
1022
- results.mutationsFailed += batchResult.failed;
1023
- results.errors.push(...batchResult.errors);
1024
- } else {
1025
- // Entire batch failed
1026
- const batch = concurrentBatches[idx];
1027
- const errorMsg = result.reason instanceof Error ? result.reason.message : String(result.reason);
1028
- batch.forEach(loc => {
1029
- results.mutationsFailed++;
1030
- results.errors.push({
1031
- locationRef: loc.ref || 'unknown',
1032
- error: `Batch execution failed: ${errorMsg}`
1033
- });
1034
- });
1035
- }
1036
- });
1037
-
1038
- // Add small delay between concurrent batches to respect rate limits
1039
- if (i + maxParallel < aliasBatches.length) {
1040
- await new Promise(resolve => setTimeout(resolve, 500));
1041
- }
1042
- }
1043
-
1044
- return results;
1045
- }
1046
-
1047
- /**
1048
- * ✅ NEW: Build aliased batch query and variables
1049
- *
1050
- * @param batch - Array of location objects
1051
- * @param mutationName - GraphQL mutation name (e.g., 'createLocation')
1052
- * @returns GraphQL query and variables for aliased batch
1053
- */
1054
- private async buildAliasedBatch(
1055
- batch: MappedLocation[],
1056
- mutationName: string
1057
- ): Promise<{ query: string; variables: Record<string, any> }> {
1058
- const batchSize = batch.length;
1059
- const inputTypeName = `${mutationName.charAt(0).toUpperCase() + mutationName.slice(1)}Input`;
1060
-
1061
- // Build aliased query (simple naming: mutationName + index)
1062
- const variables = Array.from({ length: batchSize }, (_, i) =>
1063
- `$input${i + 1}: ${inputTypeName}!`
1064
- ).join(', ');
1065
-
1066
- const aliasedMutations = Array.from({ length: batchSize }, (_, i) => {
1067
- const alias = `${mutationName}${i + 1}`; // Simple: createLocation1, createLocation2, etc.
1068
- return ` ${alias}: ${mutationName}(input: $input${i + 1}) { id ref name }`;
1069
- }).join('\n');
1070
-
1071
- const operationName = `Batch${mutationName.charAt(0).toUpperCase() + mutationName.slice(1)}s`;
1072
- const query = `mutation ${operationName}(${variables}) {\n${aliasedMutations}\n}`;
1073
-
1074
- // Build variables from mapped locations (await all mappings)
1075
- const mappedItems = await Promise.all(
1076
- batch.map(loc => this.mapper.map(loc))
1077
- );
1078
-
1079
- const variablesObj: Record<string, unknown> = {};
1080
- mappedItems.forEach((mapped, index) => {
1081
- const input = mapped.variables.input || mapped.variables;
1082
- variablesObj[`input${index + 1}`] = input;
1083
- });
1084
-
1085
- return { query, variables: variablesObj };
1086
- }
1087
-
1088
- /**
1089
- * ✅ NEW: Parse aliased GraphQL response and extract individual mutation results
1090
- *
1091
- * @param response - GraphQL response from aliased mutation
1092
- * @param batch - Original batch of location objects
1093
- * @param mutationName - GraphQL mutation name (e.g., 'createLocation')
1094
- * @returns Parsed results with executed/failed counts and errors
1095
- */
1096
- private parseAliasResponse(
1097
- response: { data?: Record<string, unknown>; errors?: Array<{ path?: string[]; message: string }> },
1098
- batch: MappedLocation[],
1099
- mutationName: string
1100
- ): { executed: number; failed: number; errors: Array<{ locationRef: string; error: string }> } {
1101
- const result = { executed: 0, failed: 0, errors: [] as Array<{ locationRef: string; error: string }> };
1102
-
1103
- const data = response.data || {};
1104
- const errors = response.errors || [];
1105
-
1106
- // Process each alias result
1107
- batch.forEach((loc, index) => {
1108
- const alias = `${mutationName}${index + 1}`; // Simple: createLocation1, createLocation2, etc.
1109
- const aliasData = data[alias];
1110
- const aliasErrors = errors.filter((e) =>
1111
- e.path && Array.isArray(e.path) && e.path.includes(alias)
1112
- );
1113
-
1114
- if (aliasData && !aliasErrors.length) {
1115
- result.executed++;
1116
- } else {
1117
- result.failed++;
1118
- const errorMsg = aliasErrors[0]?.message || 'Mutation failed';
1119
- const locationRef = loc.ref || 'unknown';
1120
- result.errors.push({
1121
- locationRef,
1122
- error: errorMsg,
1123
- });
1124
- }
1125
- });
1126
-
1127
- return result;
1128
- }
1129
- }
1130
- ```
1131
-
1132
- ### Service 3: Mutation Logger (Complete Implementation)
1133
-
1134
- **File:** `src/services/mutation-logger.service.ts`
1135
-
1136
- ```typescript
1137
- /**
1138
- * Mutation Logger Service
1139
- *
1140
- * Writes mutation processing logs to SFTP for audit purposes.
1141
- * Log writing failures don't stop the workflow (non-critical).
1142
- */
1143
-
1144
- import { Buffer } from 'node:buffer'; // Required for Versori/Deno
1145
- import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
1146
- import type { MutationResult } from '../types/location-ingestion.types';
1147
-
1148
- /**
1149
- * Service for writing mutation logs to SFTP
1150
- */
1151
- export class MutationLoggerService {
1152
- constructor(private sftp: SftpDataSource) {}
1153
-
1154
- /**
1155
- * Write mutation processing log to SFTP
1156
- *
1157
- * Creates JSON log file with mutation results.
1158
- * Only write logs when there are failures (optional).
1159
- *
1160
- * Returns log file path on success, or throws on critical errors.
1161
- */
1162
- async writeMutationLog(
1163
- fileName: string,
1164
- result: MutationResult,
1165
- remotePath: string,
1166
- requireAbsolutePaths: boolean
1167
- ): Promise<string> {
1168
- const timestamp = new Date().toISOString();
1169
- const logContent = JSON.stringify(
1170
- {
1171
- fileName,
1172
- timestamp,
1173
- mutationsExecuted: result.mutationsExecuted,
1174
- mutationsFailed: result.mutationsFailed,
1175
- errors: result.errors,
1176
- },
1177
- null,
1178
- 2
1179
- );
1180
-
1181
- const logFileName = `${fileName.replace(/\.json$/i, '')}-mutations-${timestamp.replace(/[:.]/g, '-')}.json`;
1182
-
1183
- const logPath =
1184
- requireAbsolutePaths && !remotePath.startsWith('/')
1185
- ? `/${remotePath}/${logFileName}`.replace(/\/+/g, '/')
1186
- : `${remotePath}/${logFileName}`.replace(/\/+/g, '/');
1187
-
1188
- await this.sftp.uploadFile(logPath, logContent);
1189
-
1190
- return logPath; // Return path for workflow logging
1191
- }
1192
- }
1193
- ```
1194
-
1195
- ---
1196
-
1197
- ## Type Definitions
1198
-
1199
- **File:** `src/types/location-ingestion.types.ts`
1200
-
1201
- ```typescript
1202
- export interface LocationRecord {
1203
- locationRef: string;
1204
- locationName: string;
1205
- locationType: string;
1206
- address: {
1207
- street1: string;
1208
- city: string;
1209
- state: string;
1210
- postalCode: string;
1211
- country: string;
1212
- };
1213
- geo: {
1214
- latitude: number;
1215
- longitude: number;
1216
- };
1217
- timeZone: string;
1218
- openingSchedule: {
1219
- allHours: boolean;
1220
- monStart: number;
1221
- monEnd: number;
1222
- // ... other schedule fields
1223
- };
1224
- }
1225
-
1226
- export interface MappedLocation {
1227
- ref: string;
1228
- name: string;
1229
- type: string;
1230
- status: string;
1231
- primaryAddress: {
1232
- ref: string;
1233
- street?: string;
1234
- city?: string;
1235
- latitude: number;
1236
- longitude: number;
1237
- };
1238
- openingSchedule: {
1239
- allHours: boolean;
1240
- monStart: number;
1241
- monEnd: number;
1242
- };
1243
- }
1244
-
1245
- export interface MutationResult {
1246
- mutationsExecuted: number;
1247
- mutationsFailed: number;
1248
- errors: Array<{ locationRef: string; error: string }>;
1249
- }
1250
-
1251
- export interface ProcessFileResult {
1252
- success: boolean;
1253
- locations: Array<{ query: string; variables: Record<string, unknown> }>;
1254
- error?: string;
1255
- fileName: string;
1256
- }
1257
-
1258
- // Type guard for validation
1259
- export function isLocationRecord(obj: unknown): obj is LocationRecord {
1260
- if (typeof obj !== 'object' || obj === null) return false;
1261
- const loc = obj as Record<string, unknown>;
1262
-
1263
- return (
1264
- typeof loc.locationRef === 'string' &&
1265
- typeof loc.locationName === 'string' &&
1266
- typeof loc.locationType === 'string' &&
1267
- typeof loc.address === 'object' &&
1268
- typeof loc.geo === 'object'
1269
- );
1270
- }
1271
- ```
1272
-
1273
- ---
1274
-
1275
- ---
1276
-
1277
- ## Versori Workflows Structure
1278
-
1279
- **Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
1280
-
1281
- **Trigger Types:**
1282
- - **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
1283
- - **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
1284
- - **`workflow()`** → Durable workflows (advanced, rarely used)
1285
-
1286
- **Execution Steps (chained to triggers):**
1287
- - **`http()`** → External API calls (chained from schedule/webhook)
1288
- - **`fn()`** → Internal processing (chained from schedule/webhook)
1289
-
1290
- ### Recommended Project Structure
1291
-
1292
- ```
1293
- sftp-json-location-graphql/
1294
- ├── index.ts # Entry point - exports all workflows
1295
- └── src/
1296
- ├── workflows/
1297
- │ ├── scheduled/
1298
- │ │ └── daily-location-sync.ts # Scheduled: Daily location sync
1299
- │ │
1300
- │ └── webhook/
1301
- │ ├── adhoc-location-sync.ts # Webhook: Manual trigger
1302
- │ └── job-status-check.ts # Webhook: Status query
1303
-
1304
- ├── services/
1305
- │ └── location-sync.service.ts # Shared orchestration logic (reusable)
1306
-
1307
- └── config/
1308
- └── location-mapping.json # GraphQL mapping config
1309
- ```
1310
-
1311
- ---
1312
-
1313
- ## Workflow Files
1314
-
1315
- ### 1. Scheduled Workflows (`src/workflows/scheduled/`)
1316
-
1317
- All time-based triggers that run automatically on cron schedules.
1318
-
1319
- #### `src/workflows/scheduled/daily-location-sync.ts`
1320
-
1321
- **Purpose**: Automatic daily location sync
1322
- **Trigger**: Cron schedule (`0 2 * * *`)
1323
- **Exposed as Endpoint**: ❌ NO - Runs automatically
1324
-
1325
- ```typescript
1326
- import { schedule, http } from '@versori/run';
1327
- import { executeLocationIngestion } from '../../services/location-sync.service';
1328
- import { generateJobId } from '../../utils/job-id-generator';
1329
-
1330
- /**
1331
- * Scheduled Workflow: Daily Location Sync
1332
- *
1333
- * Runs automatically daily at 2 AM UTC
1334
- * NOT exposed as HTTP endpoint - Versori executes on schedule
1335
- *
1336
- * Uses shared service: location-sync.service.ts
1337
- */
1338
- export const dailyLocationSync = schedule(
1339
- 'location-sync-scheduled',
1340
- '0 2 * * *' // Daily at 2 AM UTC
1341
- ).then(
1342
- http('run-location-sync', { connection: 'fluent_commerce' }, async ctx => {
1343
- const jobId = generateJobId('SCHEDULED', 'LOC');
1344
- return await executeLocationIngestion(ctx, { jobId, triggeredBy: 'schedule' });
1345
- })
1346
- );
1347
- ```
1348
-
1349
- ---
1350
-
1351
- ### 2. Webhook Workflows (`src/workflows/webhook/`)
1352
-
1353
- All HTTP-based triggers that create webhook endpoints.
1354
-
1355
- #### `src/workflows/webhook/adhoc-location-sync.ts`
1356
-
1357
- **Purpose**: Manual location sync trigger (on-demand)
1358
- **Trigger**: HTTP POST
1359
- **Endpoint**: `POST https://{workspace}.versori.run/location-sync-adhoc`
1360
- **Use Cases**: Testing, priority processing, ad-hoc runs
1361
-
1362
- ```typescript
1363
- import { webhook, http } from '@versori/run';
1364
- import { executeLocationIngestion } from '../../services/location-sync.service';
1365
- import { generateJobId } from '../../utils/job-id-generator';
1366
-
1367
- /**
1368
- * Webhook: Manual Location Sync Trigger
1369
- *
1370
- * Endpoint: POST https://{workspace}.versori.run/location-sync-adhoc
1371
- * Request body (optional): { filePattern: "urgent_*.json", maxFiles: 5, forceReprocess: true }
1372
- *
1373
- * Pattern: webhook().then(http()) - needs Fluent API access
1374
- * Uses shared service: location-sync.service.ts
1375
- *
1376
- * SECURITY: Authentication handled via connection parameter
1377
- * No manual API key validation needed - Versori manages this via connection auth
1378
- */
1379
- export const adhocLocationSync = webhook('location-sync-adhoc', {
1380
- response: { mode: 'sync' },
1381
- connection: 'location-sync-adhoc', // Versori validates API key
1382
- }).then(
1383
- http('run-location-sync-adhoc', { connection: 'fluent_commerce' }, async ctx => {
1384
- const jobId = generateJobId('ADHOC', 'LOC');
1385
- const { filePattern, maxFiles, forceReprocess } = ctx.data;
1386
- return await executeLocationIngestion(ctx, {
1387
- jobId,
1388
- triggeredBy: 'manual',
1389
- filePattern,
1390
- maxFiles,
1391
- forceReprocess,
1392
- });
1393
- })
1394
- );
1395
- ```
1396
-
1397
- ---
1398
-
1399
- #### `src/workflows/webhook/job-status-check.ts`
1400
-
1401
- **Purpose**: Query job status and progress
1402
- **Trigger**: HTTP POST
1403
- **Endpoint**: `POST https://{workspace}.versori.run/location-sync-job-status`
1404
- **Request Body**: `{ "jobId": "SCHEDULED_LOC_20250124_183045_abc123" }`
1405
-
1406
- ```typescript
1407
- import { webhook, fn } from '@versori/run';
1408
- import { getJobStatus } from '../../services/location-sync.service';
1409
-
1410
- /**
1411
- * Webhook: Job Status Check
1412
- *
1413
- * Endpoint: POST https://{workspace}.versori.run/location-sync-job-status
1414
- * Request body: { "jobId": "SCHEDULED_LOC_20250124_183045_abc123" }
1415
- *
1416
- * Pattern: webhook().then(fn()) - no external API needed, only KV storage
1417
- * Lightweight: Only queries KV store, no Fluent API calls
1418
- *
1419
- * SECURITY: Authentication handled via connection parameter
1420
- * No manual API key validation needed - Versori manages this via connection auth
1421
- */
1422
- export const locationSyncJobStatus = webhook('location-sync-job-status', {
1423
- response: { mode: 'sync' },
1424
- connection: 'location-sync-job-status',
1425
- }).then(
1426
- fn('query-job-status', async ctx => {
1427
- const { jobId } = ctx.data;
1428
- const status = await getJobStatus(ctx.openKv(':project:'), jobId, ctx.log);
1429
- return { success: !!status, jobId, ...status };
1430
- })
1431
- );
1432
- ```
1433
-
1434
- ---
1435
-
1436
- ### 3. Entry Point (`index.ts`)
1437
-
1438
- **Purpose**: Register all workflows with Versori platform
1439
-
1440
- ```typescript
1441
- /**
1442
- * Entry Point - Registers all workflows with Versori platform
1443
- *
1444
- * Versori automatically discovers and registers exported workflows
1445
- *
1446
- * File Structure:
1447
- * - src/workflows/scheduled/ → Time-based triggers (cron)
1448
- * - src/workflows/webhook/ → HTTP-based triggers (webhooks)
1449
- */
1450
-
1451
- import { MemoryInterpreter } from '@versori/run';
1452
-
1453
- // Import scheduled workflows
1454
- import { dailyLocationSync } from './src/workflows/scheduled/daily-location-sync';
1455
-
1456
- // Import webhook workflows
1457
- import { adhocLocationSync } from './src/workflows/webhook/adhoc-location-sync';
1458
- import { locationSyncJobStatus } from './src/workflows/webhook/job-status-check';
1459
-
1460
- // Register all workflows
1461
- export {
1462
- // Scheduled (time-based triggers)
1463
- dailyLocationSync,
1464
-
1465
- // Webhooks (HTTP-based triggers)
1466
- adhocLocationSync,
1467
- locationSyncJobStatus,
1468
- };
1469
-
1470
- // ✅ Memory interpreter for local development
1471
- export const interpreter = new MemoryInterpreter({
1472
- workflows: [dailyLocationSync, adhocLocationSync, locationSyncJobStatus],
1473
- });
1474
- ```
1475
-
1476
- **What Gets Exposed:**
1477
- - ✅ `adhocLocationSync` → `https://{workspace}.versori.run/location-sync-adhoc`
1478
- - ✅ `locationSyncJobStatus` → `https://{workspace}.versori.run/location-sync-job-status`
1479
- - ❌ `dailyLocationSync` → NOT exposed (runs automatically on cron)
1480
-
1481
- ---
1482
-
1483
- ## Package Configuration
1484
-
1485
- **File:** `package.json`
1486
-
1487
- ```json
1488
- {
1489
- "name": "sftp-json-location-graphql",
1490
- "type": "module",
1491
- "main": "index.ts",
1492
- "dependencies": {
1493
- "@fluentcommerce/fc-connect-sdk": "^0.1.39",
1494
- "@versori/run": "latest"
1495
- },
1496
- "devDependencies": {
1497
- "@types/node": "^20.0.0",
1498
- "typescript": "^5.0.0"
1499
- }
1500
- }
1501
- ```
1502
-
1503
- **File:** `tsconfig.json`
1504
-
1505
- ```json
1506
- {
1507
- "compilerOptions": {
1508
- "module": "ES2022",
1509
- "target": "ES2024",
1510
- "moduleResolution": "node"
1511
- }
1512
- }
1513
- ```
1514
-
1515
- ---
1516
-
1517
- ## Deployment
1518
-
1519
- ```bash
1520
- # 1. Install dependencies
1521
- npm install
1522
-
1523
- # 2. Deploy to Versori
1524
- npm run deploy
1525
-
1526
- # 3. Configure activation variables in Versori Dashboard
1527
- # CRITICAL: Set fluentRetailerId
1528
-
1529
- # 4. Test with sample file
1530
- curl -X POST https://your-workspace.versori.run/location-ingestion-adhoc \
1531
- -H "Content-Type: application/json" \
1532
- -d '{"forceReprocess": false}'
1533
- ```
1534
-
1535
- ---
1536
-
1537
- ## Testing
1538
-
1539
- ### Test Scheduled Run
1540
- Upload test JSON to SFTP and wait for scheduled execution (2 AM daily).
1541
-
1542
- ### Test Ad hoc Trigger
1543
- ```bash
1544
- curl -X POST https://workspace.versori.run/location-ingestion-adhoc \
1545
- -d '{"filePattern": "test_*.json"}'
1546
- ```
1547
-
1548
- ### Test Job Status
1549
- ```bash
1550
- curl -X POST https://workspace.versori.run/location-ingestion-job-status \
1551
- -d '{"jobId": "ADHOC_LOC_20251101_183045_abc123"}'
1552
- ```
1553
-
1554
- ---
1555
-
1556
- ## Monitoring
1557
-
1558
- ### Success Response
1559
-
1560
- ```json
1561
- {
1562
- "success": true,
1563
- "filesProcessed": 1,
1564
- "filesSkipped": 0,
1565
- "filesFailed": 0,
1566
- "totalRecords": 50,
1567
- "mutationsExecuted": 50,
1568
- "mutationsFailed": 0,
1569
- "results": [
1570
- {
1571
- "file": "locations_2025-01-22.json",
1572
- "success": true,
1573
- "recordsProcessed": 50,
1574
- "mutationsExecuted": 50,
1575
- "mutationsFailed": 0
1576
- }
1577
- ],
1578
- "duration": 12345
1579
- }
1580
- ```
1581
-
1582
- ### Partial Success Response
1583
-
1584
- ```json
1585
- {
1586
- "success": true,
1587
- "filesProcessed": 1,
1588
- "filesSkipped": 0,
1589
- "filesFailed": 0,
1590
- "totalRecords": 50,
1591
- "mutationsExecuted": 45,
1592
- "mutationsFailed": 5,
1593
- "results": [
1594
- {
1595
- "file": "locations_2025-01-22.json",
1596
- "success": true,
1597
- "recordsProcessed": 50,
1598
- "mutationsExecuted": 45,
1599
- "mutationsFailed": 5,
1600
- "errors": ["LOC-001: Invalid location ref", "LOC-002: Missing required field"]
1601
- }
1602
- ],
1603
- "duration": 12345
1604
- }
1605
- ```
1606
-
1607
- ### Error Response
1608
-
1609
- ```json
1610
- {
1611
- "success": false,
1612
- "filesProcessed": 0,
1613
- "filesFailed": 1,
1614
- "totalRecords": 0,
1615
- "mutationsExecuted": 0,
1616
- "mutationsFailed": 0,
1617
- "results": [
1618
- {
1619
- "file": "locations_2025-01-22.json",
1620
- "success": false,
1621
- "error": "JSON parse error: Invalid structure"
1622
- }
1623
- ],
1624
- "duration": 876
1625
- }
1626
- ```
1627
-
1628
- ### Monitoring Metrics
1629
-
1630
- Monitor these metrics via Versori logs filtered by `jobId` or `stage`:
1631
-
1632
- - **Files Processed** - Total files successfully processed
1633
- - **Mutations Executed** - Total GraphQL mutations executed successfully
1634
- - **Mutations Failed** - Mutations that failed (check error logs)
1635
- - **Processing Duration** - Time taken for complete workflow
1636
- - **Rate Limiting** - Watch for 429 errors indicating GraphQL throttling
1637
-
1638
- Use the status webhook for dashboards and automated monitoring.
1639
-
1640
- ---
1641
-
1642
- Before deployment, verify:
1643
-
1644
- - [ ] ✅ `GraphQLMutationMapper` imported (NOT `UniversalMapper`)
1645
- - [ ] ✅ NO `client.setRetailerId()` call (not needed for GraphQL mutations)
1646
- - [ ] ✅ `fluentRetailerId` activation variable configured (only if mutation schema requires it)
1647
- - [ ] ✅ Mapping config in external JSON file
1648
- - [ ] ✅ Mapping config uses correct structure (`mutation` property, `arguments.input`, resolver names without `sdk.` prefix)
1649
- - [ ] ✅ Uses `mapper.map()` method (returns `{ query, variables }`)
1650
- - [ ] ✅ Type definitions include `LocationRecord`, `MappedLocation`
1651
- - [ ] ✅ Services follow modular pattern (no logger params in business logic)
1652
- - [ ] ✅ Three workflows implemented (scheduled, ad hoc, status)
1653
- - [ ] ✅ Buffer imported from `node:buffer`
1654
- - [ ] ✅ **Double-finally SFTP disposal** pattern implemented (inner + outer finally)
1655
- - [ ] ✅ Complete service implementations (processor, sender, logger)
1656
- - [ ] ✅ Error handling with per-record tracking
1657
- - [ ] ✅ Processing mode documented (per-file vs chunked)
1658
-
1659
- ---
1660
-
1661
- ## Troubleshooting
1662
-
1663
- ### GraphQL Mutations Failing
1664
- **Problem:** Mutations return auth errors
1665
- **Solution:**
1666
- 1. Verify OAuth2 credentials are correct in Versori connection
1667
- 2. Check if mutation schema requires retailerId in input variables
1668
- 3. Do NOT use `client.setRetailerId()` - it's only for Job/Event API
1669
-
1670
- ### Wrong Mapper Error
1671
- **Problem:** "UniversalMapper not suitable for GraphQL"
1672
- **Solution:** Replace `UniversalMapper` with `GraphQLMutationMapper`
1673
-
1674
- ### Mapping Config Errors
1675
- **Problem:** "Invalid mapping structure"
1676
- **Solution:** Ensure config has `mutation` property (not `mutationName`) and `arguments.input` structure. Resolver names should be without `sdk.` prefix (`trim`, `toUpperCase`, `parseInt`, `parseFloat`, `toBoolean`)
1677
-
1678
- ### Wrong Method Name
1679
- **Problem:** "generateMutation is not a function"
1680
- **Solution:** Use `mapper.map()` method, not `generateMutation()`. Returns `{ query, variables }`
1681
-
1682
- ---
1683
-
1684
- ## Key Takeaways
1685
-
1686
- - ✅ **Use GraphQLMutationMapper** for GraphQL mutations (NOT UniversalMapper)
1687
- - ⚠️ **Set retailerId** on client before mutations (VERIFICATION REQUIRED - may need mutation input instead)
1688
- - ✅ **Double-finally SFTP disposal** for resource safety (matches Event API gold standard)
1689
- - ✅ **External JSON config** for production (not inline)
1690
- - ✅ **Modular services** for testability and reusability
1691
- - ✅ **Complete error handling** with per-record tracking
1692
- - ✅ **Processing modes** documented (per-file recommended, chunked optional)
1693
- - ✅ **Type-safe** with comprehensive interfaces
1694
- - ✅ **Native Versori logging** (LoggingService removed - use native log)
1695
- - ✅ **GraphQL alias batching** support for high-volume scenarios (mutationsPerAliasBatch parameter)
1696
-
1697
- ---
1698
-
1699
- ## References
1700
-
1701
- - **Gold Standard Guide:** `internal/GOLD-STANDARD-APPLICATION-GUIDE.md`
1702
- - **GraphQL Mutation Mapping:** `fc-connect-sdk/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/`
1703
- - **GraphQL Alias Batching:** `fc-connect-sdk/docs/02-CORE-GUIDES/mapping/graphql-alias-batching-guide.md`
1704
- - **retailerId Configuration:** `fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md`
1705
- - **Event API Template:** `template-ingestion-sftp-xml-product-event.md` (architectural reference)
1706
-
1707
- ---
1708
-
1709
- **Status:** ✅ Gold Standard Compliant
1710
- **Compliance:** All patterns verified against SDK and Fluent GraphQL schema
1
+ ---
2
+ template_id: tpl-ingest-sftp-json-to-location-graphql
3
+ canonical_filename: template-ingestion-sftp-json-location-graphql.md
4
+ version: 2.0.0
5
+ sdk_version: ^0.1.39
6
+ runtime: versori
7
+ direction: ingestion
8
+ source: sftp-json
9
+ destination: fluent-graphql
10
+ entity: location
11
+ format: json
12
+ logging: versori
13
+ status: stable
14
+ compliance: gold-standard
15
+ features:
16
+ - graphql-mutation-mapper
17
+ - memory-management
18
+ - enhanced-logging
19
+ - attribute-transformation
20
+ - dispose-finally
21
+ ---
22
+
23
+ # Template: Ingestion - SFTP JSON to Location GraphQL
24
+
25
+ **Deployment Target:** Versori Platform
26
+ **Compliance Status:** ✅ GOLD STANDARD APPROVED
27
+
28
+ ---
29
+
30
+ ## 📋 Implementation Prompt
31
+
32
+ ```
33
+ I need a Versori scheduled ingestion that:
34
+
35
+ 1) Discovers JSON files on SFTP with file tracking to skip duplicates
36
+ 2) Downloads and parses JSON (locations array) with JSONParserService
37
+ 3) Transforms records with GraphQLMutationMapper (NOT UniversalMapper)
38
+ 4) Executes GraphQL createLocation mutations with rate limiting
39
+ 5) Archives files to processed/ or errors/ and writes error reports
40
+ 6) Tracks progress with JobTracker and exposes job status webhook
41
+ 7) Uses native Versori log and follows modular service architecture
42
+
43
+ CRITICAL: This template uses GraphQLMutationMapper for GraphQL mutations,
44
+ NOT UniversalMapper (which is for Batch API). Do NOT use setRetailerId()
45
+ for GraphQL mutations - it's only needed for Job/Event API.
46
+
47
+ Use the loaded docs to fill in SDK specifics and best practices.
48
+ ```
49
+
50
+ ---
51
+
52
+ ## 📋 Template Overview
53
+
54
+ This connector runs on the Versori platform with **gold standard compliance**. It reads location data from SFTP JSON files, transforms it with GraphQLMutationMapper, and creates/updates locations via GraphQL mutations.
55
+
56
+ ### What This Template Does
57
+
58
+ ```
59
+ ┌─────────────────────────────────────────────────────────────────┐
60
+ │ INGESTION WORKFLOW (GraphQL Mutations Pattern) │
61
+ └─────────────────────────────────────────────────────────────────┘
62
+
63
+ 1. TRIGGER
64
+ ├─ Scheduled (Cron): Daily at 2 AM
65
+ ├─ Ad hoc (Webhook): Manual trigger
66
+ └─ Status Query (Webhook): Check progress
67
+
68
+ 2. DISCOVER FILES (SftpDataSource)
69
+ ├─ List files matching pattern
70
+ ├─ Check VersoriFileTracker (skip processed)
71
+ └─ Sort by oldest first
72
+
73
+ 3. DOWNLOAD & PARSE (LocationFileProcessorService)
74
+ ├─ Download from SFTP
75
+ ├─ Parse JSON with JSONParserService
76
+ ├─ Validate structure with type guards
77
+ └─ Return LocationRecord[]
78
+
79
+ 4. TRANSFORM (GraphQLMutationMapper)
80
+ ├─ Map to GraphQL CreateLocationInput
81
+ ├─ Handle nested objects (primaryAddress, openingSchedule)
82
+ ├─ Validate against GraphQL schema
83
+ └─ Generate mutation query
84
+
85
+ 5. EXECUTE MUTATIONS (MutationSenderService)
86
+ ├─ Execute createLocation mutations
87
+ ├─ Rate limiting (configurable concurrency)
88
+ ├─ Per-record error handling
89
+ └─ Track success/failure per mutation
90
+
91
+ 6. ARCHIVE & LOG
92
+ ├─ Move to processed/ or errors/
93
+ ├─ Write error reports
94
+ └─ Track state in VersoriFileTracker
95
+
96
+ 7. JOB TRACKING
97
+ ├─ Update job status at each step
98
+ └─ Enable status queries via webhook
99
+ ```
100
+
101
+ ### Key Features
102
+
103
+ - ✅ **Gold Standard Compliant** - Modular architecture, proper patterns
104
+ - ✅ **GraphQLMutationMapper** - Correct mapper for GraphQL mutations
105
+ - ✅ **retailerId Configuration** - CRITICAL for GraphQL API calls
106
+ - ✅ **Type-Safe** - Comprehensive TypeScript interfaces
107
+ - ✅ **Modular Services** - 4 services + utils (testable, reusable)
108
+ - ✅ **External Config** - Mapping in JSON file
109
+ - ✅ **Error Handling** - Per-record failures don't block others
110
+ - ✅ **File Tracking** - Prevents duplicate processing
111
+ - ✅ **Job Tracking** - Status queries via webhook
112
+
113
+ ---
114
+
115
+ ## 📦 SDK Imports
116
+
117
+ ```typescript
118
+ import { Buffer } from 'node:buffer'; // Required for Versori/Deno
119
+
120
+ import {
121
+ createClient,
122
+ SftpDataSource,
123
+ JSONParserService,
124
+ GraphQLMutationMapper, // ✅ CRITICAL: Use this for GraphQL (NOT UniversalMapper)
125
+ VersoriFileTracker,
126
+ JobTracker,
127
+ } from '@fluentcommerce/fc-connect-sdk';
128
+
129
+ import type {
130
+ FluentClient,
131
+ FileMetadata,
132
+ Logger,
133
+ } from '@fluentcommerce/fc-connect-sdk';
134
+
135
+ import { schedule, webhook, http, fn } from '@versori/run';
136
+ ```
137
+
138
+ **✅ Type-Only Imports:** Separating value and type imports improves tree-shaking and bundle size.
139
+
140
+ ---
141
+
142
+ ---
143
+
144
+ ## SFTP Connection Setup (Recommended)
145
+
146
+ **🔒 BEST PRACTICE:** Store SFTP credentials in a Versori connection object with Basic Auth:
147
+
148
+ ### Connection Configuration
149
+
150
+ 1. In Versori platform, create a connection named `SFTP`
151
+ 2. Set **Authentication Type**: `Basic Auth`
152
+ 3. Enter **Username**: Your SFTP username
153
+ 4. Enter **Password**: Your SFTP password
154
+ 5. The SDK will automatically retrieve and decode credentials
155
+
156
+ ### Code Implementation
157
+
158
+ The workflow retrieves credentials programmatically from the Versori connection:
159
+
160
+ ```typescript
161
+ import { Buffer } from 'node:buffer';
162
+
163
+ // Retrieve SFTP credentials from connection configuration
164
+ log.info('Retrieving SFTP credentials from connection configuration');
165
+
166
+ let sftpUsername: string;
167
+ let sftpPassword: string;
168
+
169
+ try {
170
+ // Retrieve credentials from the 'SFTP' connection
171
+ const sftpCred = await ctx.credentials().getAccessToken('SFTP');
172
+
173
+ if (!sftpCred?.accessToken) {
174
+ throw new Error('No SFTP credentials found in connection configuration');
175
+ }
176
+
177
+ // Decode base64 accessToken to get "username:password"
178
+ const rawBasicAuth = Buffer.from(sftpCred.accessToken, 'base64').toString('utf-8');
179
+
180
+ // Split on ':' to extract username and password
181
+ const parts = rawBasicAuth.split(':');
182
+
183
+ if (parts.length !== 2) {
184
+ throw new Error('Invalid SFTP credential format - expected username:password');
185
+ }
186
+
187
+ sftpUsername = parts[0];
188
+ sftpPassword = parts[1];
189
+
190
+ log.info('SFTP credentials retrieved successfully', {
191
+ hasUsername: !!sftpUsername,
192
+ hasPassword: !!sftpPassword,
193
+ usernameLength: sftpUsername.length,
194
+ passwordLength: sftpPassword.length,
195
+ });
196
+ } catch (error: any) {
197
+ log.error('Failed to retrieve SFTP credentials', {
198
+ message: error instanceof Error ? error.message : String(error),
199
+ stack: error instanceof Error ? error.stack : undefined,
200
+ errorType: error instanceof Error ? error.constructor.name : 'Error'
201
+ });
202
+
203
+ return {
204
+ success: false,
205
+ error: 'Failed to retrieve SFTP credentials from connection configuration',
206
+ details: error?.message,
207
+ recommendation: 'Please ensure the SFTP connection is configured in the Connections section with Basic Authentication (username and password)',
208
+ };
209
+ }
210
+
211
+ // Use credentials in SftpDataSource
212
+ const sftp = new SftpDataSource({
213
+ type: 'SFTP_JSON',
214
+ connectionId: 'sftp-location-sync',
215
+ name: 'Location Sync SFTP',
216
+ settings: {
217
+ host: activation.getVariable('sftpHost'),
218
+ port: parseInt(activation.getVariable('sftpPort') || '22', 10),
219
+ username: sftpUsername, // From connection
220
+ password: sftpPassword, // From connection
221
+ remotePath: activation.getVariable('sftpRemotePath'),
222
+ filePattern: activation.getVariable('filePattern'),
223
+ encoding: 'utf8',
224
+ },
225
+ }, log);
226
+ ```
227
+
228
+ ### Why Use Connections Instead of Activation Variables?
229
+
230
+ | Aspect | Activation Variables ❌ | Versori Connections ✅ |
231
+ |--------|------------------------|------------------------|
232
+ | **Security** | Plain text in config UI | Encrypted in Versori vault |
233
+ | **Visibility** | Visible to all users | Controlled access |
234
+ | **Rotation** | Manual update per workflow | Update once, affects all |
235
+ | **Best Practice** | Legacy approach | **Recommended approach** |
236
+ | **Audit Trail** | Limited | Full connection audit logs |
237
+
238
+ **Benefits:**
239
+ - ✅ Credentials stored securely in Versori vault
240
+ - ✅ Connection can be reused across workflows
241
+ - ✅ No sensitive data in activation variables
242
+ - ✅ Easier credential rotation (one place to update)
243
+ - ✅ Better security posture (credentials never exposed in UI)
244
+
245
+ ---
246
+
247
+ ## ⚙️ Activation Variables
248
+
249
+ **Configuration is driven by activation variables - modify these instead of code:**
250
+
251
+ > **Note:** `sftpUsername` and `sftpPassword` are fetched from the `SFTP` Basic Auth connection (see SFTP Connection Setup above).
252
+
253
+ ```bash
254
+ # SFTP Configuration
255
+ SFTP_HOST=sftp.partner.com
256
+ SFTP_PORT=22
257
+
258
+ # SFTP Paths
259
+ SFTP_REMOTE_PATH=/locations/incoming
260
+ SFTP_ARCHIVE_PATH=/locations/processed
261
+ SFTP_ERROR_PATH=/locations/errors
262
+
263
+ # File Processing
264
+ FILE_PATTERN=locations_*.json
265
+ MAX_FILES_PER_RUN=10
266
+
267
+ # GraphQL Configuration
268
+ FLUENT_RETAILER_ID=1 # Optional: Only if mutation schema requires retailerId in input
269
+ MAX_PARALLEL_MUTATIONS=10 # Concurrent mutation limit (1=sequential, 3-10=parallel)
270
+
271
+ # Feature Toggles
272
+ REQUIRE_ABSOLUTE_PATHS=true # "true" for AWS Transfer Family, "false" for standard OpenSSH
273
+ VALIDATE_CONNECTION=true # Validate SFTP connection on startup (default: true)
274
+ ENABLE_FILE_TRACKING=true # Enable/disable file tracking (default: true)
275
+ ```
276
+
277
+ **Security:**
278
+ - **SFTP Authentication:** Credentials stored in Versori connection vault (not in activation variables). See [SFTP Connection Setup](#sftp-connection-setup-recommended) section above.
279
+ - **Webhook Authentication:** Enforced by Versori connection configuration. Configure your webhook connection with API key authentication in the Versori Dashboard, then reference it in `webhook({ connection: 'webhook-auth' })`.
280
+
281
+ ---
282
+
283
+ ## 📄 Mapping Configuration
284
+
285
+ **File:** `src/config/location-mapping.json`
286
+
287
+ **✅ PRODUCTION STANDARD:** External JSON file with GraphQLMutationMapper structure
288
+
289
+ ```json
290
+ {
291
+ "mutation": "createLocation",
292
+ "sourceFormat": "json",
293
+ "version": "1.0.0",
294
+ "returnFields": ["id", "ref", "name", "type", "status"],
295
+ "arguments": {
296
+ "input": {
297
+ "ref": {
298
+ "source": "locationRef",
299
+ "required": true,
300
+ "resolver": "trim"
301
+ },
302
+ "name": {
303
+ "source": "locationName",
304
+ "required": true,
305
+ "resolver": "trim"
306
+ },
307
+ "type": {
308
+ "source": "locationType",
309
+ "required": true,
310
+ "resolver": "toUpperCase"
311
+ },
312
+ "status": {
313
+ "value": "ACTIVE",
314
+ "required": true
315
+ },
316
+ "primaryAddress": {
317
+ "ref": {
318
+ "source": "locationRef",
319
+ "required": true,
320
+ "resolver": "trim"
321
+ },
322
+ "street": {
323
+ "source": "address.street1",
324
+ "resolver": "trim"
325
+ },
326
+ "city": {
327
+ "source": "address.city",
328
+ "resolver": "trim"
329
+ },
330
+ "state": {
331
+ "source": "address.state",
332
+ "resolver": "toUpperCase"
333
+ },
334
+ "postcode": {
335
+ "source": "address.postalCode",
336
+ "resolver": "trim"
337
+ },
338
+ "country": {
339
+ "source": "address.country",
340
+ "resolver": "toUpperCase"
341
+ },
342
+ "latitude": {
343
+ "source": "geo.latitude",
344
+ "required": true,
345
+ "resolver": "parseFloat"
346
+ },
347
+ "longitude": {
348
+ "source": "geo.longitude",
349
+ "required": true,
350
+ "resolver": "parseFloat"
351
+ },
352
+ "timeZone": {
353
+ "source": "timeZone",
354
+ "resolver": "trim"
355
+ }
356
+ },
357
+ "openingSchedule": {
358
+ "allHours": {
359
+ "source": "openingSchedule.allHours",
360
+ "required": true,
361
+ "resolver": "toBoolean"
362
+ },
363
+ "monStart": {
364
+ "source": "openingSchedule.monStart",
365
+ "required": true,
366
+ "resolver": "parseInt"
367
+ },
368
+ "monEnd": {
369
+ "source": "openingSchedule.monEnd",
370
+ "required": true,
371
+ "resolver": "parseInt"
372
+ }
373
+ }
374
+ }
375
+ }
376
+ }
377
+ ```
378
+
379
+ **Key Differences from UniversalMapper:**
380
+ - ✅ `mutation` property (not `mutationName`)
381
+ - ✅ `arguments.input` structure (matches GraphQL schema)
382
+ - ✅ Nested objects (not dotted paths like `primaryAddress.street`)
383
+ - ✅ Resolver names without `sdk.` prefix (`trim`, `toUpperCase`, `parseInt`, `parseFloat`, `toBoolean`)
384
+ - ✅ Used with `GraphQLMutationMapper.map()` method (not `UniversalMapper.map()`)
385
+
386
+ ---
387
+
388
+ ## Expected JSON Format
389
+
390
+ ```json
391
+ {
392
+ "locations": [
393
+ {
394
+ "locationRef": "LOC-001",
395
+ "locationName": "Main Warehouse",
396
+ "locationType": "WAREHOUSE",
397
+ "address": {
398
+ "street1": "123 Main St",
399
+ "city": "New York",
400
+ "state": "NY",
401
+ "postalCode": "10001",
402
+ "country": "US"
403
+ },
404
+ "geo": {
405
+ "latitude": 40.7128,
406
+ "longitude": -74.0060
407
+ },
408
+ "timeZone": "America/New_York",
409
+ "openingSchedule": {
410
+ "allHours": false,
411
+ "monStart": 480,
412
+ "monEnd": 1020
413
+ }
414
+ }
415
+ ]
416
+ }
417
+ ```
418
+
419
+ **Alternative:** Root array `[{...}]` is also supported.
420
+
421
+ ---
422
+
423
+ ## 🔧 Production Code Structure
424
+
425
+ ### Modular Architecture (Gold Standard)
426
+
427
+ ```
428
+ sftp-json-location-graphql/
429
+ ├── package.json
430
+ ├── tsconfig.json
431
+ ├── index.ts # Entry point
432
+ └── src/
433
+ ├── workflows/
434
+ │ └── location-ingestion.ts # 3 workflows
435
+ ├── services/
436
+ │ ├── location-file-processor.service.ts # Download, parse, transform
437
+ │ ├── mutation-sender.service.ts # Execute GraphQL mutations
438
+ │ ├── mutation-logger.service.ts # Write error logs
439
+ │ └── location-ingestion.service.ts # Main orchestration
440
+ ├── types/
441
+ │ └── location-ingestion.types.ts # TypeScript interfaces
442
+ ├── utils/
443
+ │ ├── sftp-path.utils.ts # Path helpers
444
+ │ ├── retry.utils.ts # Retry logic
445
+ │ └── job-id-generator.ts # Job IDs
446
+ └── config/
447
+ └── location-mapping.json # GraphQL mapping config
448
+ ```
449
+
450
+ ---
451
+
452
+ ## Critical Implementation Patterns
453
+
454
+ ### ✅ MUST HAVE: retailerId Configuration ⚠️ VERIFICATION REQUIRED
455
+
456
+ ```typescript
457
+ // In main orchestration service (location-ingestion.service.ts)
458
+
459
+ export async function executeLocationIngestion(ctx: VersoriContext, params: LocationIngestionParams) {
460
+ const { log, activation } = ctx;
461
+
462
+ // Step 1: Create client
463
+ const client = await createClient(ctx);
464
+ if (!client) {
465
+ throw new Error('Failed to create Fluent Commerce client');
466
+ }
467
+
468
+ // ✅ CORRECT: GraphQL mutations don't need setRetailerId()
469
+ // Check your GraphQL schema to determine retailerId handling:
470
+ // - Mandatory retailerId → Must pass it in mutation input
471
+ // - Optional retailerId → Can pass it if needed
472
+ // - No retailerId field → Don't pass it
473
+ // See: fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md
474
+
475
+ // Step 2: Initialize GraphQLMutationMapper with client
476
+ import mappingConfig from '../config/location-mapping.json' with { type: 'json' };
477
+ const mapper = new GraphQLMutationMapper(mappingConfig, log, { fluentClient: client });
478
+
479
+ // Step 3: Process files and execute mutations
480
+ // ... rest of workflow
481
+
482
+ // ✅ Configuration with defaults
483
+ const mutationBatchSize = parseInt(
484
+ activation?.getVariable('mutationBatchSize') || '1', // ✅ Default: 1 (sequential)
485
+ 10
486
+ );
487
+
488
+ const mutationsPerAliasBatch = activation?.getVariable('mutationsPerAliasBatch')
489
+ ? parseInt(activation?.getVariable('mutationsPerAliasBatch') || '1', 10)
490
+ : undefined; // ✅ Default: undefined (disabled, use separate requests)
491
+
492
+ // ? Enhanced: Extract context for progress logging
493
+ const sampleLocationRefs = locations.slice(0, 5).map((loc: any) => loc.ref || loc.input?.ref || 'unknown');
494
+ const mutationType = mapper?.mutationName || 'createLocation';
495
+
496
+ // ? Enhanced: Start logging with context
497
+ log.info(`[GraphQLMutations] Sending mutations for locations`, {
498
+ totalMutations: locations.length,
499
+ mutationType,
500
+ batchSize: mutationBatchSize,
501
+ batchMode: mutationBatchSize === 1 ? 'sequential' : `parallel (${mutationBatchSize})`,
502
+ sampleLocationRefs: sampleLocationRefs.join(', '),
503
+ aliasBatching: mutationsPerAliasBatch ? `enabled (${mutationsPerAliasBatch} per alias)` : 'disabled'
504
+ });
505
+
506
+ // Use in executeMutations:
507
+ const mutationResults = await mutationSender.executeMutations(
508
+ locations,
509
+ mutationBatchSize, // Concurrency control (default: 1)
510
+ mutationsPerAliasBatch // ✅ NEW: Alias batching (default: undefined)
511
+ );
512
+
513
+ // ? Enhanced: Completion logging with summary
514
+ log.info(`[GraphQLMutations] Mutation submission completed`, {
515
+ totalMutations: locations.length,
516
+ mutationsExecuted: mutationResults.mutationsExecuted,
517
+ mutationsFailed: mutationResults.mutationsFailed,
518
+ successRate: locations.length > 0 ? `${Math.round((mutationResults.mutationsExecuted / locations.length) * 100)}%` : '0%',
519
+ mutationType
520
+ });
521
+ }
522
+ ```
523
+
524
+ **✅ VERIFIED Pattern:**
525
+ - GraphQL mutations do NOT need `client.setRetailerId()`
526
+ - `setRetailerId()` is only for Job API and Event API
527
+ - Pass retailerId in mutation input variables if schema requires it
528
+
529
+ **Example (For mutations that require retailerId in schema):**
530
+ ```typescript
531
+ // Only needed if mutation schema explicitly requires retailerId
532
+ const fluentRetailerId = activation.getVariable('fluentRetailerId');
533
+ const { query, variables } = await mapper.map(location);
534
+
535
+ // Add to input if mutation schema requires it
536
+ if (fluentRetailerId && variables.input) {
537
+ variables.input.retailer = { id: parseInt(fluentRetailerId) };
538
+ }
539
+
540
+ await client.graphql({ query, variables });
541
+ ```
542
+
543
+ **Note:** Check your GraphQL schema (via introspection) to determine retailerId handling:
544
+ 1. Does the mutation have `retailerId` or `retailer.id` field?
545
+ 2. Is it mandatory (`!`) or optional?
546
+ 3. If field doesn't exist → don't pass it
547
+
548
+ ### ✅ MUST HAVE: Double-Finally SFTP Disposal
549
+
550
+ **What:** Defensive resource cleanup with nested finally blocks (matches Event API gold standard)
551
+
552
+ **Gold Standard Pattern:**
553
+
554
+ ```typescript
555
+ // ✅ CRITICAL: Declare SFTP outside try block for safe disposal
556
+ let sftp: SftpDataSource | undefined;
557
+
558
+ try {
559
+ // 📊 Execution boundary: Start workflow
560
+ const startTime = Date.now();
561
+ log.info('🚀 Starting location ingestion workflow', {
562
+ stage: 'initialization',
563
+ jobId,
564
+ triggeredBy: params.triggeredBy
565
+ });
566
+
567
+ // Initialize SFTP
568
+ sftp = new SftpDataSource(...);
569
+
570
+ // ✅ Validate connection (if enabled)
571
+ const validateConnection = activation?.getVariable('validateConnection') === 'true';
572
+ if (validateConnection) {
573
+ log.info('🔌 Validating SFTP connection', { stage: 'connection_validation' });
574
+ await sftp.validateConnection();
575
+ log.info('✅ SFTP connection validated successfully');
576
+ }
577
+
578
+ try {
579
+ // Use SFTP
580
+ await sftp.listFiles(...);
581
+ await sftp.downloadFile(...);
582
+ // ... rest of workflow
583
+
584
+ // 📊 Execution boundary: End workflow
585
+ const duration = Date.now() - startTime;
586
+ log.info('✅ Location ingestion workflow completed', {
587
+ stage: 'completion',
588
+ jobId,
589
+ duration,
590
+ filesProcessed,
591
+ mutationsExecuted
592
+ });
593
+ } finally {
594
+ // Inner finally - normal cleanup
595
+ if (sftp) {
596
+ await sftp.dispose();
597
+ log.info('🔌 SFTP connection disposed');
598
+ }
599
+ }
600
+ } catch (error: unknown) {
601
+ // ✅ Enhanced error logging: Extract all error details + recommendations
602
+ const errorMessage = error instanceof Error ? error.message : String(error);
603
+ const errorDetails = {
604
+ message: errorMessage,
605
+ stack: error instanceof Error ? error.stack : undefined,
606
+ errorType: error instanceof Error ? error.constructor.name : 'Error',
607
+ stage: 'workflow_execution',
608
+ jobId
609
+ };
610
+
611
+ // ✅ Add error recommendations
612
+ const recommendations: string[] = [];
613
+ if (errorMessage.includes('ECONNREFUSED')) {
614
+ recommendations.push('Check SFTP host and port configuration');
615
+ recommendations.push('Verify network connectivity to SFTP server');
616
+ } else if (errorMessage.includes('Authentication failed')) {
617
+ recommendations.push('Verify SFTP username and password');
618
+ recommendations.push('Check if account is locked or expired');
619
+ } else if (errorMessage.includes('No such file')) {
620
+ recommendations.push('Verify incoming path exists on SFTP server');
621
+ recommendations.push('Check file pattern matches existing files');
622
+ }
623
+
624
+ log.error('❌ Workflow failed', { ...errorDetails, recommendations });
625
+
626
+ // Handle error (mark job failed, etc.)
627
+ } finally {
628
+ // Outer finally - defensive cleanup
629
+ // ⚠️ CRITICAL: Ensures disposal even if error occurs after SFTP creation
630
+ // but before inner try block (e.g., during connection validation)
631
+ if (sftp) {
632
+ try {
633
+ await sftp.dispose();
634
+ log.info('🔌 SFTP connection disposed (outer finally)');
635
+ } catch (disposeError: unknown) {
636
+ const disposeErrorMessage = disposeError instanceof Error
637
+ ? disposeError.message
638
+ : String(disposeError);
639
+ log.warn('⚠️ Error disposing SFTP in outer finally', { error: disposeErrorMessage });
640
+ }
641
+ }
642
+ }
643
+ ```
644
+
645
+ **Why This is Critical:**
646
+ - Ensures disposal even if error occurs after SFTP creation but before inner try block
647
+ - Defensive cleanup catches edge cases
648
+ - Prevents resource leaks in error scenarios
649
+ - Matches Event API gold standard pattern
650
+
651
+ ---
652
+
653
+ ## Processing Modes
654
+
655
+ **⚠️ IMPORTANT:** Choose ONE processing mode per connector. These are alternative patterns, not features to use together.
656
+
657
+ The SDK supports two processing modes for handling multiple files. **Select the mode that best fits your use case** and configure your connector accordingly:
658
+
659
+ ### Mode 1: Per-File Processing (Recommended Default) ✅ IMPLEMENTED
660
+
661
+ **When to use:**
662
+ - Multiple large files that shouldn't be in memory together
663
+ - Need file-level consistency (file 3 fails → files 1-2 already archived)
664
+ - Clear error isolation per file
665
+
666
+ **How it works:**
667
+ 1. Process file 1 completely (download → parse → transform → execute mutations → archive)
668
+ 2. Move to file 2
669
+ 3. Continue sequentially
670
+
671
+ **Benefits:**
672
+ - ✅ Low memory usage (one file at a time)
673
+ - ✅ Clear error isolation (failed file doesn't block others)
674
+ - ✅ Incremental progress (files archived as completed)
675
+
676
+ **Example implementation (see code section below)**
677
+
678
+ ### Mode 2: Chunked Per-File Processing ⚠️ OPTIONAL - Choose if needed
679
+
680
+ **When to use:**
681
+ - Many small files that can be processed in parallel
682
+ - Need better throughput for many files
683
+ - Files are independent (no cross-file dependencies)
684
+
685
+ **How it works:**
686
+ 1. Group files into chunks (e.g., 5 files per chunk)
687
+ 2. Process chunk 1 files in parallel
688
+ 3. Wait for chunk completion, then move to chunk 2
689
+
690
+ **Benefits:**
691
+ - ✅ Better throughput (parallel processing)
692
+ - ✅ Still maintains file-level isolation
693
+ - ✅ Configurable chunk size
694
+
695
+ **Note:** GraphQL mutations are typically one-off operations, so per-file processing is usually sufficient. Chunked mode is optional for high-volume scenarios.
696
+
697
+ ---
698
+
699
+ ## Key Service Patterns
700
+
701
+ ### Service 1: Location File Processor (Complete Implementation)
702
+
703
+ **File:** `src/services/location-file-processor.service.ts`
704
+
705
+ ```typescript
706
+ /**
707
+ * Location File Processor Service
708
+ *
709
+ * Downloads JSON files from SFTP, parses content, and transforms with GraphQLMutationMapper.
710
+ * Reusable service for location file processing workflows.
711
+ */
712
+
713
+ import {
714
+ SftpDataSource,
715
+ JSONParserService,
716
+ GraphQLMutationMapper,
717
+ } from '@fluentcommerce/fc-connect-sdk';
718
+ import type {
719
+ ProcessFileResult,
720
+ LocationRecord,
721
+ } from '../types/location-ingestion.types';
722
+ import { extractFileName } from '../utils/sftp-path.utils';
723
+
724
+ /**
725
+ * Location file processing service
726
+ * Coordinates SFTP download, JSON parsing (via SDK), and GraphQL mutation generation
727
+ */
728
+ export class LocationFileProcessorService {
729
+ constructor(
730
+ private sftp: SftpDataSource,
731
+ private jsonParser: JSONParserService,
732
+ private mapper: GraphQLMutationMapper
733
+ ) {} // ✅ No logger param - workflow handles logging
734
+
735
+ /**
736
+ * Download JSON file from SFTP, parse, and generate GraphQL mutations
737
+ */
738
+ async downloadParseAndTransform(remoteFilePath: string): Promise<ProcessFileResult> {
739
+ // ✅ Extract filename for logging
740
+ const fileName = extractFileName(remoteFilePath);
741
+
742
+ try {
743
+ // Download JSON content
744
+ const content = (await this.sftp.downloadFile(remoteFilePath, {
745
+ encoding: 'utf8',
746
+ })) as string;
747
+
748
+ // Parse JSON with type safety
749
+ let parsed: unknown;
750
+ try {
751
+ parsed = await this.jsonParser.parse(content);
752
+ } catch (parseError: unknown) {
753
+ const errorMessage = parseError instanceof Error ? parseError.message : String(parseError);
754
+ return {
755
+ success: false,
756
+ locations: [],
757
+ error: `JSON parse error: ${errorMessage}`,
758
+ fileName,
759
+ };
760
+ }
761
+
762
+ // Normalize locations array (handle both { locations: [...] } and root array formats)
763
+ const jsonData = parsed as Record<string, unknown>;
764
+ const locations = jsonData.locations
765
+ ? (Array.isArray(jsonData.locations) ? jsonData.locations : [jsonData.locations])
766
+ : (Array.isArray(jsonData) ? jsonData : []);
767
+
768
+ if (locations.length === 0) {
769
+ return {
770
+ success: false,
771
+ locations: [],
772
+ error: 'No locations found in JSON',
773
+ fileName,
774
+ };
775
+ }
776
+
777
+ // Validate with type guard
778
+ const validLocations = locations.filter(isLocationRecord);
779
+
780
+ if (validLocations.length === 0) {
781
+ return {
782
+ success: false,
783
+ locations: [],
784
+ error: 'No valid locations found after validation',
785
+ fileName,
786
+ };
787
+ }
788
+
789
+ // Transform with GraphQLMutationMapper
790
+ const mapped = await Promise.all(
791
+ validLocations.map(loc => this.mapper.map(loc))
792
+ );
793
+
794
+ return {
795
+ success: true,
796
+ locations: mapped.map(m => m.variables.input),
797
+ fileName,
798
+ };
799
+ } catch (error: unknown) {
800
+ const errorMessage = error instanceof Error ? error.message : String(error);
801
+ return {
802
+ success: false,
803
+ locations: [],
804
+ error: errorMessage,
805
+ fileName,
806
+ };
807
+ }
808
+ }
809
+ }
810
+
811
+ /**
812
+ * Type guard: Check if parsed value is a valid LocationRecord
813
+ */
814
+ function isLocationRecord(obj: unknown): obj is LocationRecord {
815
+ if (typeof obj !== 'object' || obj === null) return false;
816
+ const loc = obj as Record<string, unknown>;
817
+
818
+ return (
819
+ typeof loc.locationRef === 'string' &&
820
+ typeof loc.locationName === 'string' &&
821
+ typeof loc.locationType === 'string' &&
822
+ typeof loc.address === 'object' &&
823
+ loc.address !== null &&
824
+ typeof loc.geo === 'object' &&
825
+ loc.geo !== null
826
+ );
827
+ }
828
+ ```
829
+
830
+ ### Service 2: Mutation Sender (Complete Implementation)
831
+
832
+ **File:** `src/services/mutation-sender.service.ts`
833
+
834
+ ```typescript
835
+ /**
836
+ * Mutation Sender Service
837
+ *
838
+ * Executes GraphQL mutations with per-record error handling.
839
+ * Continues processing on individual failures (GraphQL mutation pattern).
840
+ */
841
+
842
+ import { GraphQLMutationMapper } from '@fluentcommerce/fc-connect-sdk';
843
+ import type { FluentClient } from '@fluentcommerce/fc-connect-sdk';
844
+ import type { MutationResult, MappedLocation } from '../types/location-ingestion.types';
845
+
846
+ /**
847
+ * Service for executing GraphQL mutations
848
+ */
849
+ export class MutationSenderService {
850
+ constructor(
851
+ private client: FluentClient,
852
+ private mapper: GraphQLMutationMapper
853
+ ) {}
854
+
855
+ /**
856
+ * Execute GraphQL mutations with configurable concurrency and alias batching
857
+ *
858
+ * **Performance Characteristics:**
859
+ * - `maxParallel: 1` → Sequential processing (safe default, ~1 mutation/sec)
860
+ * - `maxParallel: 3-5` → Balanced throughput (~3-5 mutations/sec)
861
+ * - `maxParallel: 10` → High-volume processing (~10 mutations/sec, 100+ locations)
862
+ *
863
+ * **Implementation Strategy:**
864
+ * - Sequential (1): Optimized loop (no Promise.allSettled overhead)
865
+ * - Parallel (>1): Chunked processing with bounded concurrency
866
+ * - Alias batching: Groups mutations into aliased GraphQL requests (reduces network overhead)
867
+ * - Both modes: Per-record error tracking (failures don't block others)
868
+ *
869
+ * **Alias Batching:**
870
+ * - When `mutationsPerAliasBatch > 1`, groups mutations into aliased requests
871
+ * - Example: 5 mutations → 1 HTTP request with aliases (createLocation1, createLocation2, ...)
872
+ * - Reduces network overhead by ~80% for high-volume scenarios
873
+ *
874
+ * @param locations - Array of mapped locations to create/update
875
+ * @param maxParallel - Number of concurrent mutations or alias batches (default: 1, min: 1)
876
+ * @param mutationsPerAliasBatch - Optional: Number of mutations per aliased request (default: undefined = disabled)
877
+ * @returns MutationResult with counts (mutationsExecuted/mutationsFailed) and error details
878
+ */
879
+ async executeMutations(
880
+ locations: MappedLocation[],
881
+ maxParallel: number = 1, // ✅ Default: 1 (sequential)
882
+ mutationsPerAliasBatch?: number // ✅ NEW: Alias batching parameter (default: undefined = disabled)
883
+ ): Promise<MutationResult> {
884
+ // Determine mode: use aliases if mutationsPerAliasBatch is set and > 1
885
+ const useAliases = mutationsPerAliasBatch && mutationsPerAliasBatch > 1;
886
+
887
+ if (useAliases) {
888
+ return await this.executeMutationsWithAliases(
889
+ locations,
890
+ maxParallel,
891
+ mutationsPerAliasBatch!
892
+ );
893
+ } else {
894
+ return await this.executeMutationsSeparate(locations, maxParallel);
895
+ }
896
+ }
897
+
898
+ /**
899
+ * Execute mutations using separate concurrent requests (current mode)
900
+ */
901
+ private async executeMutationsSeparate(
902
+ locations: MappedLocation[],
903
+ maxParallel: number
904
+ ): Promise<MutationResult> {
905
+ // Validate concurrency (guard against invalid values)
906
+ const safeConc = Math.max(1, Math.floor(maxParallel));
907
+
908
+ // Result accumulators
909
+ let mutationsExecuted = 0;
910
+ let mutationsFailed = 0;
911
+ const errors: Array<{ locationRef: string; error: string }> = [];
912
+
913
+ // ============================================================================
914
+ // SEQUENTIAL MODE (maxParallel === 1)
915
+ // ============================================================================
916
+ if (safeConc === 1) {
917
+ for (const location of locations) {
918
+ try {
919
+ const { query, variables } = await this.mapper.map(location);
920
+ await this.client.graphql({ query, variables });
921
+ mutationsExecuted++;
922
+ } catch (err: unknown) {
923
+ mutationsFailed++;
924
+ const errorMsg = err instanceof Error ? err.message : String(err);
925
+ errors.push({
926
+ locationRef: location.ref || 'unknown',
927
+ error: errorMsg,
928
+ });
929
+ // Continue processing (failure doesn't block other locations)
930
+ }
931
+ }
932
+ return { mutationsExecuted, mutationsFailed, errors };
933
+ }
934
+
935
+ // ============================================================================
936
+ // PARALLEL MODE (maxParallel > 1)
937
+ // ============================================================================
938
+ // Chunked processing with bounded concurrency
939
+ for (let i = 0; i < locations.length; i += safeConc) {
940
+ const chunk = locations.slice(i, i + safeConc);
941
+
942
+ // Fire all mutations in chunk concurrently
943
+ const results = await Promise.allSettled(
944
+ chunk.map(async (location) => {
945
+ const { query, variables } = await this.mapper.map(location);
946
+ return await this.client.graphql({ query, variables });
947
+ })
948
+ );
949
+
950
+ // Aggregate chunk results
951
+ results.forEach((result, idx) => {
952
+ if (result.status === 'fulfilled') {
953
+ mutationsExecuted++;
954
+ } else {
955
+ mutationsFailed++;
956
+ const error = result.reason;
957
+ errors.push({
958
+ locationRef: chunk[idx]?.ref || 'unknown',
959
+ error: error?.message || String(error) || 'Unknown error',
960
+ });
961
+ }
962
+ });
963
+
964
+ // Small delay between batches to avoid rate limiting
965
+ if (i + safeConc < locations.length) {
966
+ await new Promise(resolve => setTimeout(resolve, 100));
967
+ }
968
+ }
969
+
970
+ return { mutationsExecuted, mutationsFailed, errors };
971
+ }
972
+
973
+ /**
974
+ * ✅ NEW: Execute mutations using GraphQL aliases (batched requests)
975
+ *
976
+ * Groups mutations into alias batches and executes them with concurrency control.
977
+ * Each aliased request contains multiple mutations identified by alias names.
978
+ *
979
+ * @param locations - Array of mapped locations
980
+ * @param maxParallel - Number of concurrent alias batch requests
981
+ * @param mutationsPerAliasBatch - Number of mutations per aliased request
982
+ * @returns Mutation execution results
983
+ */
984
+ private async executeMutationsWithAliases(
985
+ locations: MappedLocation[],
986
+ maxParallel: number,
987
+ mutationsPerAliasBatch: number
988
+ ): Promise<MutationResult> {
989
+ const results: MutationResult = { mutationsExecuted: 0, mutationsFailed: 0, errors: [] };
990
+
991
+ // Extract mutation name from mapper config
992
+ const mutationName = (this.mapper as any).config.mutation || 'createLocation';
993
+
994
+ // Group locations into alias batches
995
+ const aliasBatches: Array<MappedLocation[]> = [];
996
+ for (let i = 0; i < locations.length; i += mutationsPerAliasBatch) {
997
+ aliasBatches.push(locations.slice(i, i + mutationsPerAliasBatch));
998
+ }
999
+
1000
+ // Process batches with concurrency control
1001
+ for (let i = 0; i < aliasBatches.length; i += maxParallel) {
1002
+ const concurrentBatches = aliasBatches.slice(i, i + maxParallel);
1003
+
1004
+ const batchResults = await Promise.allSettled(
1005
+ concurrentBatches.map(async (batch) => {
1006
+ // Build aliased query and variables
1007
+ const { query, variables } = await this.buildAliasedBatch(batch, mutationName);
1008
+
1009
+ // Execute aliased mutation
1010
+ const response = await this.client.graphql({ query, variables });
1011
+
1012
+ // Parse results and errors
1013
+ return this.parseAliasResponse(response, batch, mutationName);
1014
+ })
1015
+ );
1016
+
1017
+ // Aggregate results
1018
+ batchResults.forEach((result, idx) => {
1019
+ if (result.status === 'fulfilled') {
1020
+ const batchResult = result.value;
1021
+ results.mutationsExecuted += batchResult.executed;
1022
+ results.mutationsFailed += batchResult.failed;
1023
+ results.errors.push(...batchResult.errors);
1024
+ } else {
1025
+ // Entire batch failed
1026
+ const batch = concurrentBatches[idx];
1027
+ const errorMsg = result.reason instanceof Error ? result.reason.message : String(result.reason);
1028
+ batch.forEach(loc => {
1029
+ results.mutationsFailed++;
1030
+ results.errors.push({
1031
+ locationRef: loc.ref || 'unknown',
1032
+ error: `Batch execution failed: ${errorMsg}`
1033
+ });
1034
+ });
1035
+ }
1036
+ });
1037
+
1038
+ // Add small delay between concurrent batches to respect rate limits
1039
+ if (i + maxParallel < aliasBatches.length) {
1040
+ await new Promise(resolve => setTimeout(resolve, 500));
1041
+ }
1042
+ }
1043
+
1044
+ return results;
1045
+ }
1046
+
1047
+ /**
1048
+ * ✅ NEW: Build aliased batch query and variables
1049
+ *
1050
+ * @param batch - Array of location objects
1051
+ * @param mutationName - GraphQL mutation name (e.g., 'createLocation')
1052
+ * @returns GraphQL query and variables for aliased batch
1053
+ */
1054
+ private async buildAliasedBatch(
1055
+ batch: MappedLocation[],
1056
+ mutationName: string
1057
+ ): Promise<{ query: string; variables: Record<string, any> }> {
1058
+ const batchSize = batch.length;
1059
+ const inputTypeName = `${mutationName.charAt(0).toUpperCase() + mutationName.slice(1)}Input`;
1060
+
1061
+ // Build aliased query (simple naming: mutationName + index)
1062
+ const variables = Array.from({ length: batchSize }, (_, i) =>
1063
+ `$input${i + 1}: ${inputTypeName}!`
1064
+ ).join(', ');
1065
+
1066
+ const aliasedMutations = Array.from({ length: batchSize }, (_, i) => {
1067
+ const alias = `${mutationName}${i + 1}`; // Simple: createLocation1, createLocation2, etc.
1068
+ return ` ${alias}: ${mutationName}(input: $input${i + 1}) { id ref name }`;
1069
+ }).join('\n');
1070
+
1071
+ const operationName = `Batch${mutationName.charAt(0).toUpperCase() + mutationName.slice(1)}s`;
1072
+ const query = `mutation ${operationName}(${variables}) {\n${aliasedMutations}\n}`;
1073
+
1074
+ // Build variables from mapped locations (await all mappings)
1075
+ const mappedItems = await Promise.all(
1076
+ batch.map(loc => this.mapper.map(loc))
1077
+ );
1078
+
1079
+ const variablesObj: Record<string, unknown> = {};
1080
+ mappedItems.forEach((mapped, index) => {
1081
+ const input = mapped.variables.input || mapped.variables;
1082
+ variablesObj[`input${index + 1}`] = input;
1083
+ });
1084
+
1085
+ return { query, variables: variablesObj };
1086
+ }
1087
+
1088
+ /**
1089
+ * ✅ NEW: Parse aliased GraphQL response and extract individual mutation results
1090
+ *
1091
+ * @param response - GraphQL response from aliased mutation
1092
+ * @param batch - Original batch of location objects
1093
+ * @param mutationName - GraphQL mutation name (e.g., 'createLocation')
1094
+ * @returns Parsed results with executed/failed counts and errors
1095
+ */
1096
+ private parseAliasResponse(
1097
+ response: { data?: Record<string, unknown>; errors?: Array<{ path?: string[]; message: string }> },
1098
+ batch: MappedLocation[],
1099
+ mutationName: string
1100
+ ): { executed: number; failed: number; errors: Array<{ locationRef: string; error: string }> } {
1101
+ const result = { executed: 0, failed: 0, errors: [] as Array<{ locationRef: string; error: string }> };
1102
+
1103
+ const data = response.data || {};
1104
+ const errors = response.errors || [];
1105
+
1106
+ // Process each alias result
1107
+ batch.forEach((loc, index) => {
1108
+ const alias = `${mutationName}${index + 1}`; // Simple: createLocation1, createLocation2, etc.
1109
+ const aliasData = data[alias];
1110
+ const aliasErrors = errors.filter((e) =>
1111
+ e.path && Array.isArray(e.path) && e.path.includes(alias)
1112
+ );
1113
+
1114
+ if (aliasData && !aliasErrors.length) {
1115
+ result.executed++;
1116
+ } else {
1117
+ result.failed++;
1118
+ const errorMsg = aliasErrors[0]?.message || 'Mutation failed';
1119
+ const locationRef = loc.ref || 'unknown';
1120
+ result.errors.push({
1121
+ locationRef,
1122
+ error: errorMsg,
1123
+ });
1124
+ }
1125
+ });
1126
+
1127
+ return result;
1128
+ }
1129
+ }
1130
+ ```
1131
+
1132
+ ### Service 3: Mutation Logger (Complete Implementation)
1133
+
1134
+ **File:** `src/services/mutation-logger.service.ts`
1135
+
1136
+ ```typescript
1137
+ /**
1138
+ * Mutation Logger Service
1139
+ *
1140
+ * Writes mutation processing logs to SFTP for audit purposes.
1141
+ * Log writing failures don't stop the workflow (non-critical).
1142
+ */
1143
+
1144
+ import { Buffer } from 'node:buffer'; // Required for Versori/Deno
1145
+ import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
1146
+ import type { MutationResult } from '../types/location-ingestion.types';
1147
+
1148
+ /**
1149
+ * Service for writing mutation logs to SFTP
1150
+ */
1151
+ export class MutationLoggerService {
1152
+ constructor(private sftp: SftpDataSource) {}
1153
+
1154
+ /**
1155
+ * Write mutation processing log to SFTP
1156
+ *
1157
+ * Creates JSON log file with mutation results.
1158
+ * Only write logs when there are failures (optional).
1159
+ *
1160
+ * Returns log file path on success, or throws on critical errors.
1161
+ */
1162
+ async writeMutationLog(
1163
+ fileName: string,
1164
+ result: MutationResult,
1165
+ remotePath: string,
1166
+ requireAbsolutePaths: boolean
1167
+ ): Promise<string> {
1168
+ const timestamp = new Date().toISOString();
1169
+ const logContent = JSON.stringify(
1170
+ {
1171
+ fileName,
1172
+ timestamp,
1173
+ mutationsExecuted: result.mutationsExecuted,
1174
+ mutationsFailed: result.mutationsFailed,
1175
+ errors: result.errors,
1176
+ },
1177
+ null,
1178
+ 2
1179
+ );
1180
+
1181
+ const logFileName = `${fileName.replace(/\.json$/i, '')}-mutations-${timestamp.replace(/[:.]/g, '-')}.json`;
1182
+
1183
+ const logPath =
1184
+ requireAbsolutePaths && !remotePath.startsWith('/')
1185
+ ? `/${remotePath}/${logFileName}`.replace(/\/+/g, '/')
1186
+ : `${remotePath}/${logFileName}`.replace(/\/+/g, '/');
1187
+
1188
+ await this.sftp.uploadFile(logPath, logContent);
1189
+
1190
+ return logPath; // Return path for workflow logging
1191
+ }
1192
+ }
1193
+ ```
1194
+
1195
+ ---
1196
+
1197
+ ## Type Definitions
1198
+
1199
+ **File:** `src/types/location-ingestion.types.ts`
1200
+
1201
+ ```typescript
1202
+ export interface LocationRecord {
1203
+ locationRef: string;
1204
+ locationName: string;
1205
+ locationType: string;
1206
+ address: {
1207
+ street1: string;
1208
+ city: string;
1209
+ state: string;
1210
+ postalCode: string;
1211
+ country: string;
1212
+ };
1213
+ geo: {
1214
+ latitude: number;
1215
+ longitude: number;
1216
+ };
1217
+ timeZone: string;
1218
+ openingSchedule: {
1219
+ allHours: boolean;
1220
+ monStart: number;
1221
+ monEnd: number;
1222
+ // ... other schedule fields
1223
+ };
1224
+ }
1225
+
1226
+ export interface MappedLocation {
1227
+ ref: string;
1228
+ name: string;
1229
+ type: string;
1230
+ status: string;
1231
+ primaryAddress: {
1232
+ ref: string;
1233
+ street?: string;
1234
+ city?: string;
1235
+ latitude: number;
1236
+ longitude: number;
1237
+ };
1238
+ openingSchedule: {
1239
+ allHours: boolean;
1240
+ monStart: number;
1241
+ monEnd: number;
1242
+ };
1243
+ }
1244
+
1245
+ export interface MutationResult {
1246
+ mutationsExecuted: number;
1247
+ mutationsFailed: number;
1248
+ errors: Array<{ locationRef: string; error: string }>;
1249
+ }
1250
+
1251
+ export interface ProcessFileResult {
1252
+ success: boolean;
1253
+ locations: Array<{ query: string; variables: Record<string, unknown> }>;
1254
+ error?: string;
1255
+ fileName: string;
1256
+ }
1257
+
1258
+ // Type guard for validation
1259
+ export function isLocationRecord(obj: unknown): obj is LocationRecord {
1260
+ if (typeof obj !== 'object' || obj === null) return false;
1261
+ const loc = obj as Record<string, unknown>;
1262
+
1263
+ return (
1264
+ typeof loc.locationRef === 'string' &&
1265
+ typeof loc.locationName === 'string' &&
1266
+ typeof loc.locationType === 'string' &&
1267
+ typeof loc.address === 'object' &&
1268
+ typeof loc.geo === 'object'
1269
+ );
1270
+ }
1271
+ ```
1272
+
1273
+ ---
1274
+
1275
+ ---
1276
+
1277
+ ## Versori Workflows Structure
1278
+
1279
+ **Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
1280
+
1281
+ **Trigger Types:**
1282
+ - **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
1283
+ - **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
1284
+ - **`workflow()`** → Durable workflows (advanced, rarely used)
1285
+
1286
+ **Execution Steps (chained to triggers):**
1287
+ - **`http()`** → External API calls (chained from schedule/webhook)
1288
+ - **`fn()`** → Internal processing (chained from schedule/webhook)
1289
+
1290
+ ### Recommended Project Structure
1291
+
1292
+ ```
1293
+ sftp-json-location-graphql/
1294
+ ├── index.ts # Entry point - exports all workflows
1295
+ └── src/
1296
+ ├── workflows/
1297
+ │ ├── scheduled/
1298
+ │ │ └── daily-location-sync.ts # Scheduled: Daily location sync
1299
+ │ │
1300
+ │ └── webhook/
1301
+ │ ├── adhoc-location-sync.ts # Webhook: Manual trigger
1302
+ │ └── job-status-check.ts # Webhook: Status query
1303
+
1304
+ ├── services/
1305
+ │ └── location-sync.service.ts # Shared orchestration logic (reusable)
1306
+
1307
+ └── config/
1308
+ └── location-mapping.json # GraphQL mapping config
1309
+ ```
1310
+
1311
+ ---
1312
+
1313
+ ## Workflow Files
1314
+
1315
+ ### 1. Scheduled Workflows (`src/workflows/scheduled/`)
1316
+
1317
+ All time-based triggers that run automatically on cron schedules.
1318
+
1319
+ #### `src/workflows/scheduled/daily-location-sync.ts`
1320
+
1321
+ **Purpose**: Automatic daily location sync
1322
+ **Trigger**: Cron schedule (`0 2 * * *`)
1323
+ **Exposed as Endpoint**: ❌ NO - Runs automatically
1324
+
1325
+ ```typescript
1326
+ import { schedule, http } from '@versori/run';
1327
+ import { executeLocationIngestion } from '../../services/location-sync.service';
1328
+ import { generateJobId } from '../../utils/job-id-generator';
1329
+
1330
+ /**
1331
+ * Scheduled Workflow: Daily Location Sync
1332
+ *
1333
+ * Runs automatically daily at 2 AM UTC
1334
+ * NOT exposed as HTTP endpoint - Versori executes on schedule
1335
+ *
1336
+ * Uses shared service: location-sync.service.ts
1337
+ */
1338
+ export const dailyLocationSync = schedule(
1339
+ 'location-sync-scheduled',
1340
+ '0 2 * * *' // Daily at 2 AM UTC
1341
+ ).then(
1342
+ http('run-location-sync', { connection: 'fluent_commerce' }, async ctx => {
1343
+ const jobId = generateJobId('SCHEDULED', 'LOC');
1344
+ return await executeLocationIngestion(ctx, { jobId, triggeredBy: 'schedule' });
1345
+ })
1346
+ );
1347
+ ```
1348
+
1349
+ ---
1350
+
1351
+ ### 2. Webhook Workflows (`src/workflows/webhook/`)
1352
+
1353
+ All HTTP-based triggers that create webhook endpoints.
1354
+
1355
+ #### `src/workflows/webhook/adhoc-location-sync.ts`
1356
+
1357
+ **Purpose**: Manual location sync trigger (on-demand)
1358
+ **Trigger**: HTTP POST
1359
+ **Endpoint**: `POST https://{workspace}.versori.run/location-sync-adhoc`
1360
+ **Use Cases**: Testing, priority processing, ad-hoc runs
1361
+
1362
+ ```typescript
1363
+ import { webhook, http } from '@versori/run';
1364
+ import { executeLocationIngestion } from '../../services/location-sync.service';
1365
+ import { generateJobId } from '../../utils/job-id-generator';
1366
+
1367
+ /**
1368
+ * Webhook: Manual Location Sync Trigger
1369
+ *
1370
+ * Endpoint: POST https://{workspace}.versori.run/location-sync-adhoc
1371
+ * Request body (optional): { filePattern: "urgent_*.json", maxFiles: 5, forceReprocess: true }
1372
+ *
1373
+ * Pattern: webhook().then(http()) - needs Fluent API access
1374
+ * Uses shared service: location-sync.service.ts
1375
+ *
1376
+ * SECURITY: Authentication handled via connection parameter
1377
+ * No manual API key validation needed - Versori manages this via connection auth
1378
+ */
1379
+ export const adhocLocationSync = webhook('location-sync-adhoc', {
1380
+ response: { mode: 'sync' },
1381
+ connection: 'location-sync-adhoc', // Versori validates API key
1382
+ }).then(
1383
+ http('run-location-sync-adhoc', { connection: 'fluent_commerce' }, async ctx => {
1384
+ const jobId = generateJobId('ADHOC', 'LOC');
1385
+ const { filePattern, maxFiles, forceReprocess } = ctx.data;
1386
+ return await executeLocationIngestion(ctx, {
1387
+ jobId,
1388
+ triggeredBy: 'manual',
1389
+ filePattern,
1390
+ maxFiles,
1391
+ forceReprocess,
1392
+ });
1393
+ })
1394
+ );
1395
+ ```
1396
+
1397
+ ---
1398
+
1399
+ #### `src/workflows/webhook/job-status-check.ts`
1400
+
1401
+ **Purpose**: Query job status and progress
1402
+ **Trigger**: HTTP POST
1403
+ **Endpoint**: `POST https://{workspace}.versori.run/location-sync-job-status`
1404
+ **Request Body**: `{ "jobId": "SCHEDULED_LOC_20250124_183045_abc123" }`
1405
+
1406
+ ```typescript
1407
+ import { webhook, fn } from '@versori/run';
1408
+ import { getJobStatus } from '../../services/location-sync.service';
1409
+
1410
+ /**
1411
+ * Webhook: Job Status Check
1412
+ *
1413
+ * Endpoint: POST https://{workspace}.versori.run/location-sync-job-status
1414
+ * Request body: { "jobId": "SCHEDULED_LOC_20250124_183045_abc123" }
1415
+ *
1416
+ * Pattern: webhook().then(fn()) - no external API needed, only KV storage
1417
+ * Lightweight: Only queries KV store, no Fluent API calls
1418
+ *
1419
+ * SECURITY: Authentication handled via connection parameter
1420
+ * No manual API key validation needed - Versori manages this via connection auth
1421
+ */
1422
+ export const locationSyncJobStatus = webhook('location-sync-job-status', {
1423
+ response: { mode: 'sync' },
1424
+ connection: 'location-sync-job-status',
1425
+ }).then(
1426
+ fn('query-job-status', async ctx => {
1427
+ const { jobId } = ctx.data;
1428
+ const status = await getJobStatus(ctx.openKv(':project:'), jobId, ctx.log);
1429
+ return { success: !!status, jobId, ...status };
1430
+ })
1431
+ );
1432
+ ```
1433
+
1434
+ ---
1435
+
1436
+ ### 3. Entry Point (`index.ts`)
1437
+
1438
+ **Purpose**: Register all workflows with Versori platform
1439
+
1440
+ ```typescript
1441
+ /**
1442
+ * Entry Point - Registers all workflows with Versori platform
1443
+ *
1444
+ * Versori automatically discovers and registers exported workflows
1445
+ *
1446
+ * File Structure:
1447
+ * - src/workflows/scheduled/ → Time-based triggers (cron)
1448
+ * - src/workflows/webhook/ → HTTP-based triggers (webhooks)
1449
+ */
1450
+
1451
+ import { MemoryInterpreter } from '@versori/run';
1452
+
1453
+ // Import scheduled workflows
1454
+ import { dailyLocationSync } from './src/workflows/scheduled/daily-location-sync';
1455
+
1456
+ // Import webhook workflows
1457
+ import { adhocLocationSync } from './src/workflows/webhook/adhoc-location-sync';
1458
+ import { locationSyncJobStatus } from './src/workflows/webhook/job-status-check';
1459
+
1460
+ // Register all workflows
1461
+ export {
1462
+ // Scheduled (time-based triggers)
1463
+ dailyLocationSync,
1464
+
1465
+ // Webhooks (HTTP-based triggers)
1466
+ adhocLocationSync,
1467
+ locationSyncJobStatus,
1468
+ };
1469
+
1470
+ // ✅ Memory interpreter for local development
1471
+ export const interpreter = new MemoryInterpreter({
1472
+ workflows: [dailyLocationSync, adhocLocationSync, locationSyncJobStatus],
1473
+ });
1474
+ ```
1475
+
1476
+ **What Gets Exposed:**
1477
+ - ✅ `adhocLocationSync` → `https://{workspace}.versori.run/location-sync-adhoc`
1478
+ - ✅ `locationSyncJobStatus` → `https://{workspace}.versori.run/location-sync-job-status`
1479
+ - ❌ `dailyLocationSync` → NOT exposed (runs automatically on cron)
1480
+
1481
+ ---
1482
+
1483
+ ## Package Configuration
1484
+
1485
+ **File:** `package.json`
1486
+
1487
+ ```json
1488
+ {
1489
+ "name": "sftp-json-location-graphql",
1490
+ "type": "module",
1491
+ "main": "index.ts",
1492
+ "dependencies": {
1493
+ "@fluentcommerce/fc-connect-sdk": "^0.1.39",
1494
+ "@versori/run": "latest"
1495
+ },
1496
+ "devDependencies": {
1497
+ "@types/node": "^20.0.0",
1498
+ "typescript": "^5.0.0"
1499
+ }
1500
+ }
1501
+ ```
1502
+
1503
+ **File:** `tsconfig.json`
1504
+
1505
+ ```json
1506
+ {
1507
+ "compilerOptions": {
1508
+ "module": "ES2022",
1509
+ "target": "ES2024",
1510
+ "moduleResolution": "node"
1511
+ }
1512
+ }
1513
+ ```
1514
+
1515
+ ---
1516
+
1517
+ ## Deployment
1518
+
1519
+ ```bash
1520
+ # 1. Install dependencies
1521
+ npm install
1522
+
1523
+ # 2. Deploy to Versori
1524
+ npm run deploy
1525
+
1526
+ # 3. Configure activation variables in Versori Dashboard
1527
+ # CRITICAL: Set fluentRetailerId
1528
+
1529
+ # 4. Test with sample file
1530
+ curl -X POST https://your-workspace.versori.run/location-ingestion-adhoc \
1531
+ -H "Content-Type: application/json" \
1532
+ -d '{"forceReprocess": false}'
1533
+ ```
1534
+
1535
+ ---
1536
+
1537
+ ## Testing
1538
+
1539
+ ### Test Scheduled Run
1540
+ Upload test JSON to SFTP and wait for scheduled execution (2 AM daily).
1541
+
1542
+ ### Test Ad hoc Trigger
1543
+ ```bash
1544
+ curl -X POST https://workspace.versori.run/location-ingestion-adhoc \
1545
+ -d '{"filePattern": "test_*.json"}'
1546
+ ```
1547
+
1548
+ ### Test Job Status
1549
+ ```bash
1550
+ curl -X POST https://workspace.versori.run/location-ingestion-job-status \
1551
+ -d '{"jobId": "ADHOC_LOC_20251101_183045_abc123"}'
1552
+ ```
1553
+
1554
+ ---
1555
+
1556
+ ## Monitoring
1557
+
1558
+ ### Success Response
1559
+
1560
+ ```json
1561
+ {
1562
+ "success": true,
1563
+ "filesProcessed": 1,
1564
+ "filesSkipped": 0,
1565
+ "filesFailed": 0,
1566
+ "totalRecords": 50,
1567
+ "mutationsExecuted": 50,
1568
+ "mutationsFailed": 0,
1569
+ "results": [
1570
+ {
1571
+ "file": "locations_2025-01-22.json",
1572
+ "success": true,
1573
+ "recordsProcessed": 50,
1574
+ "mutationsExecuted": 50,
1575
+ "mutationsFailed": 0
1576
+ }
1577
+ ],
1578
+ "duration": 12345
1579
+ }
1580
+ ```
1581
+
1582
+ ### Partial Success Response
1583
+
1584
+ ```json
1585
+ {
1586
+ "success": true,
1587
+ "filesProcessed": 1,
1588
+ "filesSkipped": 0,
1589
+ "filesFailed": 0,
1590
+ "totalRecords": 50,
1591
+ "mutationsExecuted": 45,
1592
+ "mutationsFailed": 5,
1593
+ "results": [
1594
+ {
1595
+ "file": "locations_2025-01-22.json",
1596
+ "success": true,
1597
+ "recordsProcessed": 50,
1598
+ "mutationsExecuted": 45,
1599
+ "mutationsFailed": 5,
1600
+ "errors": ["LOC-001: Invalid location ref", "LOC-002: Missing required field"]
1601
+ }
1602
+ ],
1603
+ "duration": 12345
1604
+ }
1605
+ ```
1606
+
1607
+ ### Error Response
1608
+
1609
+ ```json
1610
+ {
1611
+ "success": false,
1612
+ "filesProcessed": 0,
1613
+ "filesFailed": 1,
1614
+ "totalRecords": 0,
1615
+ "mutationsExecuted": 0,
1616
+ "mutationsFailed": 0,
1617
+ "results": [
1618
+ {
1619
+ "file": "locations_2025-01-22.json",
1620
+ "success": false,
1621
+ "error": "JSON parse error: Invalid structure"
1622
+ }
1623
+ ],
1624
+ "duration": 876
1625
+ }
1626
+ ```
1627
+
1628
+ ### Monitoring Metrics
1629
+
1630
+ Monitor these metrics via Versori logs filtered by `jobId` or `stage`:
1631
+
1632
+ - **Files Processed** - Total files successfully processed
1633
+ - **Mutations Executed** - Total GraphQL mutations executed successfully
1634
+ - **Mutations Failed** - Mutations that failed (check error logs)
1635
+ - **Processing Duration** - Time taken for complete workflow
1636
+ - **Rate Limiting** - Watch for 429 errors indicating GraphQL throttling
1637
+
1638
+ Use the status webhook for dashboards and automated monitoring.
1639
+
1640
+ ---
1641
+
1642
+ Before deployment, verify:
1643
+
1644
+ - [ ] ✅ `GraphQLMutationMapper` imported (NOT `UniversalMapper`)
1645
+ - [ ] ✅ NO `client.setRetailerId()` call (not needed for GraphQL mutations)
1646
+ - [ ] ✅ `fluentRetailerId` activation variable configured (only if mutation schema requires it)
1647
+ - [ ] ✅ Mapping config in external JSON file
1648
+ - [ ] ✅ Mapping config uses correct structure (`mutation` property, `arguments.input`, resolver names without `sdk.` prefix)
1649
+ - [ ] ✅ Uses `mapper.map()` method (returns `{ query, variables }`)
1650
+ - [ ] ✅ Type definitions include `LocationRecord`, `MappedLocation`
1651
+ - [ ] ✅ Services follow modular pattern (no logger params in business logic)
1652
+ - [ ] ✅ Three workflows implemented (scheduled, ad hoc, status)
1653
+ - [ ] ✅ Buffer imported from `node:buffer`
1654
+ - [ ] ✅ **Double-finally SFTP disposal** pattern implemented (inner + outer finally)
1655
+ - [ ] ✅ Complete service implementations (processor, sender, logger)
1656
+ - [ ] ✅ Error handling with per-record tracking
1657
+ - [ ] ✅ Processing mode documented (per-file vs chunked)
1658
+
1659
+ ---
1660
+
1661
+ ## Troubleshooting
1662
+
1663
+ ### GraphQL Mutations Failing
1664
+ **Problem:** Mutations return auth errors
1665
+ **Solution:**
1666
+ 1. Verify OAuth2 credentials are correct in Versori connection
1667
+ 2. Check if mutation schema requires retailerId in input variables
1668
+ 3. Do NOT use `client.setRetailerId()` - it's only for Job/Event API
1669
+
1670
+ ### Wrong Mapper Error
1671
+ **Problem:** "UniversalMapper not suitable for GraphQL"
1672
+ **Solution:** Replace `UniversalMapper` with `GraphQLMutationMapper`
1673
+
1674
+ ### Mapping Config Errors
1675
+ **Problem:** "Invalid mapping structure"
1676
+ **Solution:** Ensure config has `mutation` property (not `mutationName`) and `arguments.input` structure. Resolver names should be without `sdk.` prefix (`trim`, `toUpperCase`, `parseInt`, `parseFloat`, `toBoolean`)
1677
+
1678
+ ### Wrong Method Name
1679
+ **Problem:** "generateMutation is not a function"
1680
+ **Solution:** Use `mapper.map()` method, not `generateMutation()`. Returns `{ query, variables }`
1681
+
1682
+ ---
1683
+
1684
+ ## Key Takeaways
1685
+
1686
+ - ✅ **Use GraphQLMutationMapper** for GraphQL mutations (NOT UniversalMapper)
1687
+ - ⚠️ **Set retailerId** on client before mutations (VERIFICATION REQUIRED - may need mutation input instead)
1688
+ - ✅ **Double-finally SFTP disposal** for resource safety (matches Event API gold standard)
1689
+ - ✅ **External JSON config** for production (not inline)
1690
+ - ✅ **Modular services** for testability and reusability
1691
+ - ✅ **Complete error handling** with per-record tracking
1692
+ - ✅ **Processing modes** documented (per-file recommended, chunked optional)
1693
+ - ✅ **Type-safe** with comprehensive interfaces
1694
+ - ✅ **Native Versori logging** (LoggingService removed - use native log)
1695
+ - ✅ **GraphQL alias batching** support for high-volume scenarios (mutationsPerAliasBatch parameter)
1696
+
1697
+ ---
1698
+
1699
+ ## References
1700
+
1701
+ - **Gold Standard Guide:** `internal/GOLD-STANDARD-APPLICATION-GUIDE.md`
1702
+ - **GraphQL Mutation Mapping:** `fc-connect-sdk/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/`
1703
+ - **GraphQL Alias Batching:** `fc-connect-sdk/docs/02-CORE-GUIDES/mapping/graphql-alias-batching-guide.md`
1704
+ - **retailerId Configuration:** `fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md`
1705
+ - **Event API Template:** `template-ingestion-sftp-xml-product-event.md` (architectural reference)
1706
+
1707
+ ---
1708
+
1709
+ **Status:** ✅ Gold Standard Compliant
1710
+ **Compliance:** All patterns verified against SDK and Fluent GraphQL schema