@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (495) hide show
  1. package/CHANGELOG.md +30 -2
  2. package/README.md +39 -0
  3. package/dist/cjs/auth/index.d.ts +3 -0
  4. package/dist/cjs/auth/index.js +13 -0
  5. package/dist/cjs/auth/profile-loader.d.ts +18 -0
  6. package/dist/cjs/auth/profile-loader.js +208 -0
  7. package/dist/cjs/client-factory.d.ts +4 -0
  8. package/dist/cjs/client-factory.js +10 -0
  9. package/dist/cjs/clients/fluent-client.js +13 -6
  10. package/dist/cjs/index.d.ts +3 -1
  11. package/dist/cjs/index.js +8 -2
  12. package/dist/cjs/utils/pagination-helpers.js +38 -2
  13. package/dist/cjs/versori/fluent-versori-client.js +11 -5
  14. package/dist/esm/auth/index.d.ts +3 -0
  15. package/dist/esm/auth/index.js +2 -0
  16. package/dist/esm/auth/profile-loader.d.ts +18 -0
  17. package/dist/esm/auth/profile-loader.js +169 -0
  18. package/dist/esm/client-factory.d.ts +4 -0
  19. package/dist/esm/client-factory.js +9 -0
  20. package/dist/esm/clients/fluent-client.js +13 -6
  21. package/dist/esm/index.d.ts +3 -1
  22. package/dist/esm/index.js +2 -1
  23. package/dist/esm/utils/pagination-helpers.js +38 -2
  24. package/dist/esm/versori/fluent-versori-client.js +11 -5
  25. package/dist/tsconfig.esm.tsbuildinfo +1 -1
  26. package/dist/tsconfig.tsbuildinfo +1 -1
  27. package/dist/tsconfig.types.tsbuildinfo +1 -1
  28. package/dist/types/auth/index.d.ts +3 -0
  29. package/dist/types/auth/profile-loader.d.ts +18 -0
  30. package/dist/types/client-factory.d.ts +4 -0
  31. package/dist/types/index.d.ts +3 -1
  32. package/docs/00-START-HERE/EXPORT-VALIDATION.md +158 -158
  33. package/docs/00-START-HERE/cli-analyze-source-structure-guide.md +655 -655
  34. package/docs/00-START-HERE/cli-documentation-index.md +202 -202
  35. package/docs/00-START-HERE/cli-quick-reference.md +252 -252
  36. package/docs/00-START-HERE/decision-tree.md +552 -552
  37. package/docs/00-START-HERE/getting-started.md +1070 -1070
  38. package/docs/00-START-HERE/mapper-quick-decision-guide.md +235 -235
  39. package/docs/00-START-HERE/readme.md +237 -237
  40. package/docs/00-START-HERE/retailerid-configuration.md +404 -404
  41. package/docs/00-START-HERE/sdk-philosophy.md +794 -794
  42. package/docs/00-START-HERE/troubleshooting-quick-reference.md +1086 -1086
  43. package/docs/01-TEMPLATES/faq.md +686 -686
  44. package/docs/01-TEMPLATES/patterns/pattern-templates-guide.md +68 -68
  45. package/docs/01-TEMPLATES/patterns/patterns-csv-schema-validation-and-rejection-report.md +233 -233
  46. package/docs/01-TEMPLATES/patterns/patterns-custom-resolvers.md +407 -407
  47. package/docs/01-TEMPLATES/patterns/patterns-error-handling-retry.md +511 -511
  48. package/docs/01-TEMPLATES/patterns/patterns-field-mapping-universal.md +701 -701
  49. package/docs/01-TEMPLATES/patterns/patterns-large-file-splitting.md +1430 -1430
  50. package/docs/01-TEMPLATES/patterns/patterns-master-data-etl.md +2399 -2399
  51. package/docs/01-TEMPLATES/patterns/patterns-pagination-streaming.md +447 -447
  52. package/docs/01-TEMPLATES/patterns/patterns-state-duplicate-prevention.md +385 -385
  53. package/docs/01-TEMPLATES/readme.md +957 -957
  54. package/docs/01-TEMPLATES/standalone/standalone-asn-inbound-processing.md +1209 -1209
  55. package/docs/01-TEMPLATES/standalone/standalone-graphql-query-export.md +1140 -1140
  56. package/docs/01-TEMPLATES/standalone/standalone-graphql-to-parquet-partitioned-s3.md +432 -432
  57. package/docs/01-TEMPLATES/standalone/standalone-multi-channel-inventory-sync.md +1185 -1185
  58. package/docs/01-TEMPLATES/standalone/standalone-multi-source-aggregation.md +1462 -1462
  59. package/docs/01-TEMPLATES/standalone/standalone-s3-csv-batch-api.md +1390 -1390
  60. package/docs/01-TEMPLATES/standalone/standalone-s3-csv-inventory-to-batch.md +330 -330
  61. package/docs/01-TEMPLATES/standalone/standalone-scripts-guide.md +87 -87
  62. package/docs/01-TEMPLATES/standalone/standalone-sftp-xml-graphql.md +1444 -1444
  63. package/docs/01-TEMPLATES/standalone/standalone-webhook-payload-processing.md +688 -688
  64. package/docs/01-TEMPLATES/versori/business-examples/business-examples-dropship-order-routing.md +193 -193
  65. package/docs/01-TEMPLATES/versori/business-examples/business-examples-graphql-parquet-extraction.md +518 -518
  66. package/docs/01-TEMPLATES/versori/business-examples/business-examples-inter-location-transfers.md +2162 -2162
  67. package/docs/01-TEMPLATES/versori/business-examples/business-examples-pre-order-allocation.md +2226 -2226
  68. package/docs/01-TEMPLATES/versori/business-examples/business-scenarios-guide.md +87 -87
  69. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-connection-validation-pattern.md +656 -656
  70. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-dual-workflow-connector.md +835 -835
  71. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-guide.md +108 -108
  72. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-kv-state-management.md +1533 -1533
  73. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-xml-response-patterns.md +1160 -1160
  74. package/docs/01-TEMPLATES/versori/versori-platform-guide.md +201 -201
  75. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-asn-purchase-order.md +1906 -1906
  76. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-dropship-routing.md +1074 -1074
  77. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-flash-sale-reserve.md +1395 -1395
  78. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-generic-xml-order.md +888 -888
  79. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-payment-gateway-integration.md +2478 -2478
  80. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-rma-returns-comprehensive.md +2240 -2240
  81. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-xml-order-ingestion.md +2029 -2029
  82. package/docs/01-TEMPLATES/versori/webhooks/webhook-templates-guide.md +140 -140
  83. package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/inventory-mapping.json +20 -20
  84. package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/products_2025-01-22.csv +11 -11
  85. package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/sample-data-guide.md +34 -34
  86. package/docs/01-TEMPLATES/versori/workflows/_examples/workflow-examples-guide.md +36 -36
  87. package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-modes-guide.md +1038 -1038
  88. package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-workflows-guide.md +138 -138
  89. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/graphql-extraction-guide.md +63 -63
  90. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-csv.md +2062 -2062
  91. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-xml.md +2294 -2294
  92. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-s3-csv.md +2461 -2461
  93. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-sftp-xml.md +2529 -2529
  94. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-csv.md +2464 -2464
  95. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-json.md +1959 -1959
  96. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-s3-csv.md +1953 -1953
  97. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-sftp-xml.md +2541 -2541
  98. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-s3-json.md +2384 -2384
  99. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-sftp-xml.md +2445 -2445
  100. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-csv.md +2355 -2355
  101. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-json.md +2042 -2042
  102. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-sftp-xml.md +2726 -2726
  103. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/batch-api-guide.md +206 -206
  104. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-cycle-count-reconciliation.md +2030 -2030
  105. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-multi-channel-inventory-sync.md +1882 -1882
  106. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-csv-inventory-batch.md +2827 -2827
  107. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-json-inventory-batch.md +1952 -1952
  108. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-xml-inventory-batch.md +3289 -3289
  109. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-csv-inventory-batch.md +3064 -3064
  110. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-json-inventory-batch.md +3238 -3238
  111. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-xml-inventory-batch.md +2977 -2977
  112. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/event-api-guide.md +321 -321
  113. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-json-order-cancel-event.md +959 -959
  114. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-xml-order-cancel-event.md +1170 -1170
  115. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-csv-product-event.md +2312 -2312
  116. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-json-product-event.md +2999 -2999
  117. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-parquet-product-event.md +2836 -2836
  118. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-xml-product-event.md +2395 -2395
  119. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-csv-product-event.md +2295 -2295
  120. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-json-product-event.md +2602 -2602
  121. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-parquet-product-event.md +2589 -2589
  122. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-xml-product-event.md +3578 -3578
  123. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/graphql-mutations-guide.md +93 -93
  124. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-json-order-update-graphql.md +1260 -1260
  125. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-xml-order-update-graphql.md +1472 -1472
  126. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-control-graphql.md +2417 -2417
  127. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-location-graphql.md +2811 -2811
  128. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-price-graphql.md +2619 -2619
  129. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-json-location-graphql.md +2807 -2807
  130. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-xml-location-graphql.md +2373 -2373
  131. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-control-graphql.md +2740 -2740
  132. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-location-graphql.md +2760 -2760
  133. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-json-location-graphql.md +1710 -1710
  134. package/docs/01-TEMPLATES/versori/workflows/ingestion/ingestion-workflows-guide.md +136 -136
  135. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/rubix-webhooks-guide.md +520 -520
  136. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-inline.md +1418 -1418
  137. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-universal-mapper.md +1785 -1785
  138. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-order-attribute-update.md +824 -824
  139. package/docs/01-TEMPLATES/versori/workflows/workflows-overview-guide.md +646 -646
  140. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-batch-archival.md +724 -724
  141. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-job-tracker.md +627 -627
  142. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-partial-batch-recovery.md +561 -561
  143. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-quick-reference.md +367 -367
  144. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-readme.md +407 -407
  145. package/docs/02-CORE-GUIDES/advanced-services/readme.md +49 -49
  146. package/docs/02-CORE-GUIDES/api-reference/api-reference-quick-reference.md +548 -548
  147. package/docs/02-CORE-GUIDES/api-reference/event-api-input-output-reference.md +702 -1171
  148. package/docs/02-CORE-GUIDES/api-reference/examples/client-initialization.ts +286 -286
  149. package/docs/02-CORE-GUIDES/api-reference/graphql-error-classification.md +337 -337
  150. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-01-client-api.md +399 -482
  151. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-03-authentication.md +199 -199
  152. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-04-graphql-mapping.md +925 -925
  153. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-05-services.md +1198 -1198
  154. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-06-data-sources.md +1083 -1083
  155. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-07-parsers.md +1097 -1097
  156. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-pagination.md +513 -513
  157. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-types.md +545 -597
  158. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-error-handling.md +527 -527
  159. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-webhook-validation.md +514 -514
  160. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-extraction.md +557 -557
  161. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-utilities.md +412 -412
  162. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-cli-tools.md +423 -423
  163. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-error-handling.md +716 -716
  164. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-analyze-source-structure.md +518 -518
  165. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-partial-responses.md +212 -212
  166. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-testing.md +300 -300
  167. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-13-resolver-builder.md +322 -322
  168. package/docs/02-CORE-GUIDES/api-reference/readme.md +279 -279
  169. package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-quick-reference.md +351 -351
  170. package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-readme.md +277 -277
  171. package/docs/02-CORE-GUIDES/auto-pagination/examples/auto-pagination-readme.md +178 -178
  172. package/docs/02-CORE-GUIDES/auto-pagination/examples/common-patterns.ts +351 -351
  173. package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-products.ts +384 -384
  174. package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-virtual-positions.ts +308 -308
  175. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-01-foundations.md +470 -470
  176. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-02-quick-start.md +713 -713
  177. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-03-configuration.md +754 -754
  178. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-04-advanced-patterns.md +732 -732
  179. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-05-sdk-integration.md +847 -847
  180. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-06-troubleshooting.md +359 -359
  181. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-07-api-reference.md +462 -462
  182. package/docs/02-CORE-GUIDES/auto-pagination/readme.md +54 -54
  183. package/docs/02-CORE-GUIDES/data-sources/data-sources-file-operations-error-handling.md +1487 -1487
  184. package/docs/02-CORE-GUIDES/data-sources/data-sources-quick-reference.md +836 -836
  185. package/docs/02-CORE-GUIDES/data-sources/data-sources-readme.md +276 -276
  186. package/docs/02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md +553 -553
  187. package/docs/02-CORE-GUIDES/data-sources/examples/common-patterns.ts +409 -409
  188. package/docs/02-CORE-GUIDES/data-sources/examples/data-sources-readme.md +178 -178
  189. package/docs/02-CORE-GUIDES/data-sources/examples/s3-operations.ts +308 -308
  190. package/docs/02-CORE-GUIDES/data-sources/examples/sftp-operations.ts +371 -371
  191. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-01-foundations.md +735 -735
  192. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-02-s3-operations.md +1302 -1302
  193. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-03-sftp-operations.md +1379 -1379
  194. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-04-file-patterns.md +941 -941
  195. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-05-advanced-topics.md +813 -813
  196. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-06-integration-patterns.md +486 -486
  197. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-07-troubleshooting.md +387 -387
  198. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-08-api-reference.md +417 -417
  199. package/docs/02-CORE-GUIDES/data-sources/readme.md +77 -77
  200. package/docs/02-CORE-GUIDES/error-handling-guide.md +936 -936
  201. package/docs/02-CORE-GUIDES/extraction/examples/02-core-guides-extraction-readme.md +116 -116
  202. package/docs/02-CORE-GUIDES/extraction/examples/common-patterns.ts +428 -428
  203. package/docs/02-CORE-GUIDES/extraction/examples/extract-inventory-basic.ts +187 -187
  204. package/docs/02-CORE-GUIDES/extraction/extraction-quick-reference.md +596 -596
  205. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-01-foundations.md +514 -514
  206. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-02-basic-extraction.md +823 -823
  207. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-03-parquet-processing.md +507 -507
  208. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-04-data-enrichment.md +546 -546
  209. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-05-transformation.md +494 -494
  210. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-export-formats.md +458 -458
  211. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-performance.md +138 -138
  212. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-api-reference.md +148 -148
  213. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-optimization.md +692 -692
  214. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-08-extraction-orchestrator.md +1008 -1008
  215. package/docs/02-CORE-GUIDES/extraction/readme.md +151 -151
  216. package/docs/02-CORE-GUIDES/ingestion/examples/_simple-kv-store.ts +40 -40
  217. package/docs/02-CORE-GUIDES/ingestion/examples/error-recovery.ts +728 -728
  218. package/docs/02-CORE-GUIDES/ingestion/examples/event-driven.ts +501 -501
  219. package/docs/02-CORE-GUIDES/ingestion/examples/local-file-ingestion.ts +88 -88
  220. package/docs/02-CORE-GUIDES/ingestion/examples/parquet-ingestion.ts +117 -117
  221. package/docs/02-CORE-GUIDES/ingestion/examples/performance-optimized.ts +647 -647
  222. package/docs/02-CORE-GUIDES/ingestion/examples/s3-csv-ingestion.ts +169 -169
  223. package/docs/02-CORE-GUIDES/ingestion/examples/sftp-csv-ingestion.ts +134 -134
  224. package/docs/02-CORE-GUIDES/ingestion/ingestion-quick-reference.md +546 -546
  225. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-01-introduction.md +626 -626
  226. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-02-quick-start.md +658 -658
  227. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-03-data-sources.md +1052 -1052
  228. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-04-field-mapping.md +763 -763
  229. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-05-advanced-parsers.md +676 -676
  230. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-06-batch-api.md +1295 -1295
  231. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-api-reference.md +138 -138
  232. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-state-management.md +1037 -1037
  233. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-08-performance-optimization.md +1349 -1349
  234. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-09-best-practices.md +1893 -1893
  235. package/docs/02-CORE-GUIDES/ingestion/readme.md +160 -160
  236. package/docs/02-CORE-GUIDES/logging-guide.md +585 -585
  237. package/docs/02-CORE-GUIDES/mapping/error-handling-patterns.md +401 -401
  238. package/docs/02-CORE-GUIDES/mapping/examples/02-core-guides-mapping-readme.md +128 -128
  239. package/docs/02-CORE-GUIDES/mapping/examples/common-patterns.ts +273 -273
  240. package/docs/02-CORE-GUIDES/mapping/examples/csv-location-ingestion.json +36 -36
  241. package/docs/02-CORE-GUIDES/mapping/examples/csv-mapping.ts +242 -242
  242. package/docs/02-CORE-GUIDES/mapping/examples/graphql-to-parquet-extraction.json +36 -36
  243. package/docs/02-CORE-GUIDES/mapping/examples/json-mapping.ts +213 -213
  244. package/docs/02-CORE-GUIDES/mapping/examples/json-product-to-mutation.json +48 -48
  245. package/docs/02-CORE-GUIDES/mapping/examples/xml-mapping.ts +291 -291
  246. package/docs/02-CORE-GUIDES/mapping/examples/xml-order-to-mutation.json +45 -45
  247. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-quick-reference.md +463 -463
  248. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-readme.md +227 -227
  249. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-01-introduction.md +222 -222
  250. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-02-quick-start.md +351 -351
  251. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-03-schema-validation.md +569 -569
  252. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-04-mapping-patterns.md +471 -471
  253. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-05-configuration-reference.md +611 -611
  254. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-advanced-xpath.md +148 -148
  255. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-path-syntax.md +464 -464
  256. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-api-reference.md +94 -94
  257. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-array-handling.md +307 -307
  258. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-08-custom-resolvers.md +544 -544
  259. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-09-advanced-patterns.md +427 -427
  260. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-10-hooks-and-variables.md +336 -336
  261. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-11-error-handling.md +488 -488
  262. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-12-arguments-vs-nodes.md +383 -383
  263. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-13-best-practices.md +477 -477
  264. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/readme.md +62 -62
  265. package/docs/02-CORE-GUIDES/mapping/mapping-format-decision-tree.md +480 -480
  266. package/docs/02-CORE-GUIDES/mapping/mapping-graphql-alias-batching-guide.md +820 -820
  267. package/docs/02-CORE-GUIDES/mapping/mapping-javascript-objects.md +2369 -2369
  268. package/docs/02-CORE-GUIDES/mapping/mapping-mapper-comparison-guide.md +682 -682
  269. package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-07-api-reference.md +1327 -1327
  270. package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-08-error-handling.md +1142 -1142
  271. package/docs/02-CORE-GUIDES/mapping/modules/mapping-04-use-cases.md +891 -891
  272. package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-helpers-resolvers.md +1126 -1126
  273. package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-sdk-resolvers.md +199 -199
  274. package/docs/02-CORE-GUIDES/mapping/modules/mapping-07-api-reference.md +1319 -1319
  275. package/docs/02-CORE-GUIDES/mapping/readme.md +178 -178
  276. package/docs/02-CORE-GUIDES/mapping/resolver-registration.md +410 -410
  277. package/docs/02-CORE-GUIDES/mapping/resolvers/examples/common-patterns.ts +226 -226
  278. package/docs/02-CORE-GUIDES/mapping/resolvers/examples/custom-resolvers.ts +227 -227
  279. package/docs/02-CORE-GUIDES/mapping/resolvers/examples/sdk-resolvers-usage.ts +203 -203
  280. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-readme.md +274 -274
  281. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-api-reference.md +679 -679
  282. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-cookbook.md +826 -826
  283. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-guide.md +1330 -1330
  284. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-helpers-reference.md +1437 -1437
  285. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-parameters-reference.md +553 -553
  286. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-troubleshooting.md +854 -854
  287. package/docs/02-CORE-GUIDES/mapping/resolvers/readme.md +75 -75
  288. package/docs/02-CORE-GUIDES/parsers/examples/02-core-guides-parsers-readme.md +161 -161
  289. package/docs/02-CORE-GUIDES/parsers/examples/csv-parser-examples.ts +110 -110
  290. package/docs/02-CORE-GUIDES/parsers/examples/json-parser-examples.ts +33 -33
  291. package/docs/02-CORE-GUIDES/parsers/examples/parquet-parser-examples.ts +47 -47
  292. package/docs/02-CORE-GUIDES/parsers/examples/xml-parser-examples.ts +38 -38
  293. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-01-foundations.md +355 -355
  294. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-02-csv-parser.md +772 -772
  295. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-03-json-parser.md +789 -789
  296. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-04-xml-parser.md +857 -857
  297. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-05-parquet-parser.md +603 -603
  298. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-integration-patterns.md +702 -702
  299. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-streaming.md +121 -121
  300. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-api-reference.md +89 -89
  301. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-troubleshooting.md +727 -727
  302. package/docs/02-CORE-GUIDES/parsers/parsers-quick-reference.md +482 -482
  303. package/docs/02-CORE-GUIDES/parsers/parsers-readme.md +258 -258
  304. package/docs/02-CORE-GUIDES/parsers/readme.md +65 -65
  305. package/docs/02-CORE-GUIDES/readme.md +194 -194
  306. package/docs/02-CORE-GUIDES/webhook-validation/examples/basic-validation.ts +108 -108
  307. package/docs/02-CORE-GUIDES/webhook-validation/examples/common-patterns.ts +316 -316
  308. package/docs/02-CORE-GUIDES/webhook-validation/examples/webhook-validation-readme.md +61 -61
  309. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-01-foundations.md +440 -440
  310. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-02-quick-start.md +525 -525
  311. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-03-versori-integration.md +741 -741
  312. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-04-platform-integration.md +629 -629
  313. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-05-configuration.md +535 -535
  314. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-error-handling.md +611 -611
  315. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-troubleshooting.md +124 -124
  316. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-07-api-reference.md +511 -511
  317. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-08-rubix-webhooks.md +590 -590
  318. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-09-rubix-event-vs-http-call.md +432 -432
  319. package/docs/02-CORE-GUIDES/webhook-validation/readme.md +239 -239
  320. package/docs/02-CORE-GUIDES/webhook-validation/webhook-validation-quick-reference.md +392 -392
  321. package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-quick-reference.md +498 -498
  322. package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-readme.md +313 -313
  323. package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/common-patterns.ts +612 -612
  324. package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/connector-scenarios-readme.md +253 -253
  325. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-01-foundations.md +452 -452
  326. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-02-simple-scenarios.md +681 -681
  327. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-03-intermediate-scenarios.md +637 -637
  328. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-04-advanced-scenarios.md +650 -650
  329. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-05-bidirectional-sync.md +233 -233
  330. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-06-production-patterns.md +442 -442
  331. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-07-reference.md +445 -445
  332. package/docs/03-PATTERN-GUIDES/connector-scenarios/readme.md +31 -31
  333. package/docs/03-PATTERN-GUIDES/enterprise-integration-patterns.md +1528 -1528
  334. package/docs/03-PATTERN-GUIDES/error-handling/comprehensive-error-handling-guide.md +1437 -1437
  335. package/docs/03-PATTERN-GUIDES/error-handling/error-handling-quick-reference.md +390 -390
  336. package/docs/03-PATTERN-GUIDES/error-handling/examples/common-patterns.ts +438 -438
  337. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-01-foundations.md +362 -362
  338. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-02-error-types.md +850 -850
  339. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-03-utf8-handling.md +456 -456
  340. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-04-error-scenarios.md +658 -658
  341. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-05-calling-patterns.md +671 -671
  342. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-06-retry-strategies.md +1034 -1034
  343. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-07-monitoring.md +653 -653
  344. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-08-api-reference.md +847 -847
  345. package/docs/03-PATTERN-GUIDES/error-handling/readme.md +36 -36
  346. package/docs/03-PATTERN-GUIDES/examples/__tests__/readme.md +40 -40
  347. package/docs/03-PATTERN-GUIDES/examples/__tests__/resolver-examples.test.js +282 -282
  348. package/docs/03-PATTERN-GUIDES/examples/test-data/03-pattern-guides-readme.md +110 -110
  349. package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-inventory.json +123 -123
  350. package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-order.json +171 -171
  351. package/docs/03-PATTERN-GUIDES/examples/test-data/readme.md +28 -28
  352. package/docs/03-PATTERN-GUIDES/extraction/extraction-readme.md +15 -15
  353. package/docs/03-PATTERN-GUIDES/extraction/readme.md +25 -25
  354. package/docs/03-PATTERN-GUIDES/file-operations/examples/common-patterns.ts +407 -407
  355. package/docs/03-PATTERN-GUIDES/file-operations/examples/file-operations-readme.md +142 -142
  356. package/docs/03-PATTERN-GUIDES/file-operations/file-operations-quick-reference.md +462 -462
  357. package/docs/03-PATTERN-GUIDES/file-operations/file-operations-readme.md +379 -379
  358. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-01-foundations.md +430 -430
  359. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-02-quick-start.md +484 -484
  360. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-03-s3-operations.md +507 -507
  361. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-04-sftp-operations.md +963 -963
  362. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-05-streaming-performance.md +503 -503
  363. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-archive-patterns.md +386 -386
  364. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-error-handling.md +117 -117
  365. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-api-reference.md +78 -78
  366. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-testing-troubleshooting.md +567 -567
  367. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-08-api-reference.md +1055 -1055
  368. package/docs/03-PATTERN-GUIDES/file-operations/readme.md +32 -32
  369. package/docs/03-PATTERN-GUIDES/ingestion/ingestion-readme.md +15 -15
  370. package/docs/03-PATTERN-GUIDES/ingestion/readme.md +25 -25
  371. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/batch-processing.ts +130 -130
  372. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/common-patterns.ts +360 -360
  373. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/delta-sync.ts +130 -130
  374. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/integration-patterns-readme.md +100 -100
  375. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/real-time-webhook.ts +398 -398
  376. package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-quick-reference.md +962 -962
  377. package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-readme.md +134 -134
  378. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-01-real-time-processing.md +991 -991
  379. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-02-batch-processing.md +1547 -1547
  380. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-03-delta-sync.md +1108 -1108
  381. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-04-webhook-patterns.md +1181 -1181
  382. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-05-error-handling.md +1061 -1061
  383. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-advanced-integration-services.md +1547 -1547
  384. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-performance.md +109 -109
  385. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-07-api-reference.md +34 -34
  386. package/docs/03-PATTERN-GUIDES/integration-patterns/readme.md +30 -30
  387. package/docs/03-PATTERN-GUIDES/logging-minimal-mode.md +128 -128
  388. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/common-patterns.ts +380 -380
  389. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/multiple-connections-readme.md +139 -139
  390. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/parallel-root-connections.ts +149 -149
  391. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/real-world-scenarios.ts +405 -405
  392. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-01-foundations.md +378 -378
  393. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-02-quick-start.md +566 -566
  394. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-03-targeting-connections.md +659 -659
  395. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-04-parallel-queries.md +656 -656
  396. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-05-best-practices.md +624 -624
  397. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-api-reference.md +824 -824
  398. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-versori.md +119 -119
  399. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-07-api-reference.md +87 -87
  400. package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-quick-reference.md +353 -353
  401. package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-readme.md +270 -270
  402. package/docs/03-PATTERN-GUIDES/multiple-connections/readme.md +30 -30
  403. package/docs/03-PATTERN-GUIDES/pagination/pagination-readme.md +14 -14
  404. package/docs/03-PATTERN-GUIDES/pagination/readme.md +24 -24
  405. package/docs/03-PATTERN-GUIDES/parquet/examples/common-patterns.ts +180 -180
  406. package/docs/03-PATTERN-GUIDES/parquet/examples/read-parquet.ts +48 -48
  407. package/docs/03-PATTERN-GUIDES/parquet/examples/write-parquet.ts +65 -65
  408. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-01-introduction.md +393 -393
  409. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-02-quick-start.md +572 -572
  410. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-03-reading-parquet.md +525 -525
  411. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-04-writing-parquet.md +554 -554
  412. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-05-graphql-extraction.md +405 -405
  413. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-performance.md +104 -104
  414. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-s3-integration.md +511 -511
  415. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-api-reference.md +90 -90
  416. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-performance-optimization.md +525 -525
  417. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-08-best-practices.md +712 -712
  418. package/docs/03-PATTERN-GUIDES/parquet/parquet-quick-reference.md +683 -683
  419. package/docs/03-PATTERN-GUIDES/parquet/parquet-readme.md +248 -248
  420. package/docs/03-PATTERN-GUIDES/parquet/readme.md +32 -32
  421. package/docs/03-PATTERN-GUIDES/parsers/parsers-readme.md +12 -12
  422. package/docs/03-PATTERN-GUIDES/parsers/readme.md +24 -24
  423. package/docs/03-PATTERN-GUIDES/readme.md +159 -159
  424. package/docs/03-PATTERN-GUIDES/webhooks/readme.md +24 -24
  425. package/docs/03-PATTERN-GUIDES/webhooks/webhooks-readme.md +8 -8
  426. package/docs/04-REFERENCE/architecture/architecture-01-overview.md +427 -427
  427. package/docs/04-REFERENCE/architecture/architecture-02-client-architecture.md +424 -424
  428. package/docs/04-REFERENCE/architecture/architecture-03-data-flow.md +690 -690
  429. package/docs/04-REFERENCE/architecture/architecture-04-service-layer.md +834 -834
  430. package/docs/04-REFERENCE/architecture/architecture-05-integration-architecture.md +655 -655
  431. package/docs/04-REFERENCE/architecture/architecture-06-state-management.md +653 -653
  432. package/docs/04-REFERENCE/architecture/architecture-adding-new-data-sources.md +686 -686
  433. package/docs/04-REFERENCE/architecture/readme.md +279 -279
  434. package/docs/04-REFERENCE/platforms/deno/readme.md +117 -117
  435. package/docs/04-REFERENCE/platforms/nodejs/readme.md +146 -146
  436. package/docs/04-REFERENCE/platforms/readme.md +135 -135
  437. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-01-introduction.md +398 -398
  438. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-02-quick-start.md +560 -560
  439. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-03-authentication.md +757 -757
  440. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-04-workflows.md +2476 -2476
  441. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-05-connections.md +1167 -1167
  442. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-kv-storage.md +990 -990
  443. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-state-management.md +121 -121
  444. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-api-reference.md +68 -68
  445. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-deployment.md +731 -731
  446. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-08-best-practices.md +1111 -1111
  447. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-09-signature-reference.md +766 -766
  448. package/docs/04-REFERENCE/platforms/versori/platforms-versori-readme.md +299 -299
  449. package/docs/04-REFERENCE/platforms/versori/platforms-versori-s3-sftp-configuration-guide.md +1425 -1425
  450. package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-api-key-security.md +816 -816
  451. package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-connection-security.md +681 -681
  452. package/docs/04-REFERENCE/platforms/versori/platforms-versori-workflow-task-types.md +708 -708
  453. package/docs/04-REFERENCE/platforms/versori/readme.md +108 -108
  454. package/docs/04-REFERENCE/readme.md +148 -148
  455. package/docs/04-REFERENCE/resolver-signature/examples/advanced-resolvers.ts +482 -482
  456. package/docs/04-REFERENCE/resolver-signature/examples/async-resolvers.ts +496 -496
  457. package/docs/04-REFERENCE/resolver-signature/examples/basic-resolvers.ts +343 -343
  458. package/docs/04-REFERENCE/resolver-signature/examples/resolver-signature-readme.md +188 -188
  459. package/docs/04-REFERENCE/resolver-signature/examples/testing-resolvers.ts +463 -463
  460. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-01-foundations.md +286 -286
  461. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-02-parameter-reference.md +643 -643
  462. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-03-basic-examples.md +521 -521
  463. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-04-advanced-patterns.md +739 -739
  464. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-05-sdk-resolvers.md +531 -531
  465. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-migration-guide.md +650 -650
  466. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-testing.md +125 -125
  467. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-07-api-reference.md +794 -794
  468. package/docs/04-REFERENCE/resolver-signature/readme.md +64 -64
  469. package/docs/04-REFERENCE/resolver-signature/resolver-signature-quick-reference.md +270 -270
  470. package/docs/04-REFERENCE/resolver-signature/resolver-signature-readme.md +351 -351
  471. package/docs/04-REFERENCE/schema/fluent-commerce-schema.json +764 -764
  472. package/docs/04-REFERENCE/schema/readme.md +141 -141
  473. package/docs/04-REFERENCE/testing/examples/04-reference-testing-readme.md +158 -158
  474. package/docs/04-REFERENCE/testing/examples/fluent-testing.ts +62 -62
  475. package/docs/04-REFERENCE/testing/examples/health-check.ts +155 -155
  476. package/docs/04-REFERENCE/testing/examples/integration-test.ts +119 -119
  477. package/docs/04-REFERENCE/testing/examples/performance-test.ts +183 -183
  478. package/docs/04-REFERENCE/testing/examples/s3-testing.ts +127 -127
  479. package/docs/04-REFERENCE/testing/modules/04-reference-testing-01-foundations.md +267 -267
  480. package/docs/04-REFERENCE/testing/modules/04-reference-testing-02-s3-testing.md +599 -599
  481. package/docs/04-REFERENCE/testing/modules/04-reference-testing-03-fluent-testing.md +589 -589
  482. package/docs/04-REFERENCE/testing/modules/04-reference-testing-04-integration-testing.md +699 -699
  483. package/docs/04-REFERENCE/testing/modules/04-reference-testing-05-debugging.md +478 -478
  484. package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-cicd-integration.md +463 -463
  485. package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-preflight-validation.md +131 -131
  486. package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-best-practices.md +499 -499
  487. package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-coverage-ci.md +165 -165
  488. package/docs/04-REFERENCE/testing/modules/04-reference-testing-08-api-reference.md +634 -634
  489. package/docs/04-REFERENCE/testing/readme.md +86 -86
  490. package/docs/04-REFERENCE/testing/testing-quick-reference.md +667 -667
  491. package/docs/04-REFERENCE/testing/testing-readme.md +286 -286
  492. package/docs/04-REFERENCE/troubleshooting/readme.md +144 -144
  493. package/docs/04-REFERENCE/troubleshooting/troubleshooting-deno-sftp-compatibility.md +392 -392
  494. package/docs/template-loading-matrix.md +242 -242
  495. package/package.json +5 -3
@@ -1,1390 +1,1390 @@
1
- # Standalone: S3 CSV → Fluent Batch API
2
-
3
- **FC Connect SDK Use Case Guide**
4
-
5
- > **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
6
- > **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
7
-
8
- **Context**: Node.js script that reads CSV inventory files from S3 and updates Fluent Commerce inventory via Batch API
9
-
10
- **Complexity**: Medium
11
-
12
- **Runtime**: Node.js ≥18 / Deno
13
-
14
- **Estimated Lines**: ~500 lines
15
-
16
- ## What You'll Build
17
-
18
- - Standalone Node.js/Deno script (no Versori)
19
- - OAuth2 authentication with Fluent Commerce
20
- - S3 file listing and download with AWS SDK
21
- - CSV parsing with validation
22
- - UniversalMapper for field transformations
23
- - Batch API processing with job management
24
- - Error handling and logging
25
- - Optional: Schedule with cron or AWS EventBridge
26
-
27
- ## SDK Methods Used
28
-
29
- - `createClient({ config: { baseUrl, clientId, clientSecret, retailerId } })` - OAuth2 client
30
- - `S3DataSource(config, logger)` - S3 operations
31
- - `CSVParserService` - CSV parsing
32
- - `UniversalMapper(mappingConfig)` - Field mapping
33
- - `client.createJob({ name, retailerId })` - Create Batch job
34
- - `client.sendBatch(jobId, { entities })` - Send inventory batch
35
- - `client.getJobStatus(jobId)` - Check status
36
-
37
- ---
38
-
39
- ## Complete Working Implementation
40
-
41
- ### 1. Project Setup
42
-
43
- ```bash
44
- # Create project directory
45
- mkdir inventory-sync-standalone
46
- cd inventory-sync-standalone
47
-
48
- # Initialize Node.js project
49
- npm init -y
50
-
51
- # Install dependencies
52
- npm install @fluentcommerce/fc-connect-sdk dotenv
53
- ```
54
-
55
- ### Method selection (SDK)
56
-
57
- - Queries → `client.graphql`
58
- - Mutations → `client.graphql`
59
- - Advanced needs (pagination/telemetry) → `client.graphql` with pagination options
60
-
61
- ```typescript
62
- // Query (use graphql for both queries and mutations)
63
- const products = await client.graphql({ query, variables });
64
-
65
- // Mutation (use graphql for both queries and mutations)
66
- const result = await client.graphql({ query: mutation, variables });
67
-
68
- // Advanced pagination
69
- const res = await client.graphql({ query, variables, pagination: { maxPages: 50 } });
70
- ```
71
-
72
- ### 2. Environment Configuration
73
-
74
- Create `.env` file:
75
-
76
- ```env
77
- # Fluent Commerce Configuration
78
- FLUENT_BASE_URL=https://api.fluentcommerce.com
79
- FLUENT_CLIENT_ID=your-oauth2-client-id
80
- FLUENT_CLIENT_SECRET=your-oauth2-client-secret
81
- FLUENT_RETAILER_ID=your-retailer-id
82
-
83
- # AWS S3 Configuration
84
- AWS_ACCESS_KEY_ID=your-aws-access-key
85
- AWS_SECRET_ACCESS_KEY=your-aws-secret-key
86
- AWS_REGION=us-east-1
87
- AWS_BUCKET_NAME=your-inventory-bucket
88
- AWS_FILE_PREFIX=inventory/updates/
89
-
90
- # Processing Configuration
91
- BATCH_SIZE=100
92
- JOB_NAME_PREFIX=S3-CSV-Inventory-Update
93
- ARCHIVE_PREFIX=inventory/archive/
94
- ERROR_PREFIX=inventory/errors/
95
-
96
- # Optional: Scheduling
97
- ENABLE_SCHEDULING=false
98
- CRON_SCHEDULE=0 */4 * * *
99
- ```
100
-
101
- ### 3. Main Script Implementation
102
-
103
- Create `src/inventory-sync.ts`:
104
-
105
- ```typescript
106
- import 'dotenv/config';
107
- // FC Connect SDK+
108
- // Install: npm install @fluentcommerce/fc-connect-sdk@latest
109
- // Docs: https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk
110
- // GitHub: https://github.com/fluentcommerce/fc-connect-sdk
111
- import { createClient } from '@fluentcommerce/fc-connect-sdk';
112
- import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
113
- import { CSVParserService } from '@fluentcommerce/fc-connect-sdk';
114
- import { UniversalMapper } from '@fluentcommerce/fc-connect-sdk';
115
-
116
- /**
117
- * ARCHITECTURE:
118
- *
119
- * This standalone script implements a complete inventory sync workflow:
120
- * 1. List CSV files from S3 bucket
121
- * 2. Download and parse CSV files
122
- * 3. Map CSV fields to Fluent inventory format
123
- * 4. Create Batch API job
124
- * 5. Send inventory updates in batches
125
- * 6. Poll job status until complete
126
- * 7. Archive successful files, move errors
127
- *
128
- * ERROR HANDLING STRATEGY:
129
- * - File-level errors: Move to error folder, continue with other files
130
- * - Batch errors: Log and continue (Fluent API provides detailed error reports)
131
- * - Fatal errors: Exit with error code for monitoring/alerting
132
- *
133
- * STATE MANAGEMENT:
134
- * - No state tracking in this basic version (processes all files)
135
- * - Advanced: Use S3 metadata or external DB to track processed files
136
- */
137
-
138
- // ============================================================================
139
- // CONFIGURATION & SETUP
140
- // ============================================================================
141
-
142
- interface SyncConfig {
143
- fluent: {
144
- baseUrl: string;
145
- clientId: string;
146
- clientSecret: string;
147
- retailerId: string;
148
- };
149
- s3: {
150
- accessKeyId: string;
151
- secretAccessKey: string;
152
- region: string;
153
- bucket: string;
154
- prefix: string;
155
- };
156
- processing: {
157
- batchSize: number;
158
- jobNamePrefix: string;
159
- archivePrefix: string;
160
- errorPrefix: string;
161
- };
162
- }
163
-
164
- /**
165
- * Load configuration from environment variables
166
- * Validates all required config before proceeding
167
- */
168
- function loadConfig(): SyncConfig {
169
- const requiredEnvVars = [
170
- 'FLUENT_BASE_URL',
171
- 'FLUENT_CLIENT_ID',
172
- 'FLUENT_CLIENT_SECRET',
173
- 'FLUENT_RETAILER_ID',
174
- 'AWS_ACCESS_KEY_ID',
175
- 'AWS_SECRET_ACCESS_KEY',
176
- 'AWS_REGION',
177
- 'AWS_BUCKET_NAME',
178
- ];
179
-
180
- // Validate all required environment variables are present
181
- const missing = requiredEnvVars.filter(v => !process.env[v]);
182
- if (missing.length > 0) {
183
- throw new Error(`Missing required environment variables: ${missing.join(', ')}`);
184
- }
185
-
186
- return {
187
- fluent: {
188
- baseUrl: process.env.FLUENT_BASE_URL!,
189
- clientId: process.env.FLUENT_CLIENT_ID!,
190
- clientSecret: process.env.FLUENT_CLIENT_SECRET!,
191
- retailerId: process.env.FLUENT_RETAILER_ID!,
192
- },
193
- s3: {
194
- accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
195
- secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
196
- region: process.env.AWS_REGION!,
197
- bucket: process.env.AWS_BUCKET_NAME!,
198
- prefix: process.env.AWS_FILE_PREFIX || 'inventory/updates/',
199
- },
200
- processing: {
201
- batchSize: parseInt(process.env.BATCH_SIZE || '100'),
202
- jobNamePrefix: process.env.JOB_NAME_PREFIX || 'S3-CSV-Inventory-Update',
203
- archivePrefix: process.env.ARCHIVE_PREFIX || 'inventory/archive/',
204
- errorPrefix: process.env.ERROR_PREFIX || 'inventory/errors/',
205
- },
206
- };
207
- }
208
-
209
- // ============================================================================
210
- // FIELD MAPPING CONFIGURATION
211
- // ============================================================================
212
-
213
- /**
214
- * UniversalMapper configuration for CSV → Fluent Batch API transformation
215
- *
216
- * FIELD MAPPING RULES:
217
- * - ref: Product SKU (required) - uppercase transformation
218
- * - locationRef: Location code (required) - prefix with "LOC:"
219
- * - qty: Quantity (required) - parse as integer, default to 0
220
- * - type: Inventory type (required) - defaults to "ADJUSTMENT"
221
- * - status: Inventory status (optional) - defaults to "ACTIVE"
222
- *
223
- * SDK RESOLVERS USED:
224
- * - sdk.uppercase: Convert SKU to uppercase for consistency
225
- * - sdk.parseInt: Parse quantity string to integer
226
- * - sdk.trim: Remove whitespace from location codes
227
- * - sdk.defaultTo: Provide fallback values
228
- */
229
- const inventoryMappingConfig = {
230
- version: '1.0.0',
231
- description: 'S3 CSV to Fluent Commerce Inventory Mapping',
232
- fields: {
233
- // Product SKU (required field)
234
- ref: {
235
- source: 'sku',
236
- resolver: 'sdk.uppercase', // Normalize to uppercase
237
- required: true,
238
- },
239
- // Location reference (required field)
240
- locationRef: {
241
- source: 'location',
242
- resolver: 'sdk.trim', // Remove whitespace
243
- required: true,
244
- },
245
- // Quantity (required field)
246
- qty: {
247
- source: 'quantity',
248
- resolver: 'sdk.parseInt', // Parse as integer
249
- required: true,
250
- },
251
- // Inventory type (defaults to ADJUSTMENT)
252
- type: {
253
- value: 'ADJUSTMENT', // Static value
254
- },
255
- // Inventory status (optional, with default)
256
- status: {
257
- source: 'status',
258
- resolver: 'sdk.uppercase',
259
- defaultValue: 'ACTIVE', // Fallback if not provided
260
- },
261
- // Expected date (optional)
262
- expectedOn: {
263
- source: 'expected_date',
264
- resolver: 'sdk.formatDate', // Convert to ISO date format
265
- },
266
- },
267
- };
268
-
269
- // ============================================================================
270
- // LOGGER IMPLEMENTATION
271
- // ============================================================================
272
-
273
- /**
274
- * Simple console logger with timestamp and log levels
275
- * Production: Replace with Winston, Bunyan, or Pino
276
- */
277
- const logger = {
278
- debug: (message: string, meta?: any) => {
279
- if (process.env.LOG_LEVEL === 'debug') {
280
- console.log(`[${new Date().toISOString()}] DEBUG: ${message}`, meta || '');
281
- }
282
- },
283
- info: (message: string, meta?: any) => {
284
- console.log(`[${new Date().toISOString()}] INFO: ${message}`, meta || '');
285
- },
286
- warn: (message: string, meta?: any) => {
287
- console.warn(`[${new Date().toISOString()}] WARN: ${message}`, meta || '');
288
- },
289
- error: (message: string, error?: Error, meta?: any) => {
290
- console.error(
291
- `[${new Date().toISOString()}] ERROR: ${message}`,
292
- error?.message || '',
293
- meta || ''
294
- );
295
- },
296
- };
297
-
298
- // ============================================================================
299
- // MAIN SYNC ORCHESTRATION
300
- // ============================================================================
301
-
302
- /**
303
- * Main inventory sync function
304
- *
305
- * ALGORITHM:
306
- * 1. Initialize SDK clients (Fluent, S3, CSV parser, mapper)
307
- * 2. List CSV files from S3 prefix
308
- * 3. For each file:
309
- * a. Download file content
310
- * b. Parse CSV to records
311
- * c. Validate records (skip invalid)
312
- * d. Map records using UniversalMapper
313
- * e. Create Batch API job
314
- * f. Send batches (chunked by batch size)
315
- * g. Poll job status until complete
316
- * h. Archive successful files OR move to error folder
317
- * 4. Return summary statistics
318
- */
319
- async function syncInventory(): Promise<{
320
- filesProcessed: number;
321
- filesSucceeded: number;
322
- filesFailed: number;
323
- totalRecordsProcessed: number;
324
- totalRecordsFailed: number;
325
- }> {
326
- const startTime = Date.now();
327
- logger.info('='.repeat(80));
328
- logger.info('Starting S3 CSV → Fluent Batch API inventory sync');
329
- logger.info('='.repeat(80));
330
-
331
- // Load configuration
332
- const config = loadConfig();
333
- logger.info('Configuration loaded', {
334
- s3Bucket: config.s3.bucket,
335
- s3Prefix: config.s3.prefix,
336
- batchSize: config.processing.batchSize,
337
- });
338
-
339
- // ───────────────────────────────────────────────────────────────────────────
340
- // STEP 1: Initialize SDK Clients
341
- // ───────────────────────────────────────────────────────────────────────────
342
-
343
- // Initialize Fluent Client with OAuth2
344
- logger.info('Initializing Fluent Commerce client...');
345
- const fluentClient = await createClient({
346
- config: {
347
- baseUrl: config.fluent.baseUrl,
348
- clientId: config.fluent.clientId,
349
- clientSecret: config.fluent.clientSecret,
350
- retailerId: config.fluent.retailerId,
351
- },
352
- });
353
-
354
- // Initialize S3 Data Source
355
- logger.info('Initializing S3 data source...');
356
- const s3Source = new S3DataSource(
357
- {
358
- type: 'S3_CSV',
359
- connectionId: 'inventory-s3',
360
- name: 'Inventory S3 Source',
361
- s3Config: {
362
- bucket: config.s3.bucket,
363
- region: config.s3.region,
364
- accessKeyId: config.s3.accessKeyId,
365
- secretAccessKey: config.s3.secretAccessKey,
366
- },
367
- },
368
- logger
369
- );
370
-
371
- // Initialize CSV Parser
372
- const csvParser = new CSVParserService();
373
-
374
- // Initialize Universal Mapper
375
- const mapper = new UniversalMapper(inventoryMappingConfig, {
376
- logger,
377
- fluentClient,
378
- });
379
-
380
- // Validate S3 connection
381
- logger.info('Validating S3 connection...');
382
- const s3Connected = await s3Source.validateConnection();
383
- if (!s3Connected) {
384
- throw new Error('Failed to connect to S3 bucket');
385
- }
386
- logger.info('S3 connection validated successfully');
387
-
388
- // ───────────────────────────────────────────────────────────────────────────
389
- // STEP 2: List CSV Files from S3
390
- // ───────────────────────────────────────────────────────────────────────────
391
-
392
- logger.info(`Listing CSV files from S3 prefix: ${config.s3.prefix}`);
393
- const files = await s3Source.listFiles({
394
- prefix: config.s3.prefix,
395
- maxKeys: 1000,
396
- });
397
-
398
- // Filter for CSV files only
399
- // NOTE: S3 doesn't support glob patterns, so manual filtering is required
400
- // IMPORTANT: Use .endsWith('.csv') NOT .endsWith('*.csv')
401
- // ✓ Correct: f.name.endsWith('.csv')
402
- // ✗ Wrong: f.name.endsWith('*.csv') // Would never match!
403
- const csvFiles = files.filter(f => f.name.toLowerCase().endsWith('.csv'));
404
-
405
- logger.info(`Found ${csvFiles.length} CSV files to process`, {
406
- totalFiles: files.length,
407
- csvFiles: csvFiles.length,
408
- });
409
-
410
- if (csvFiles.length === 0) {
411
- logger.info('No CSV files found. Exiting.');
412
- return {
413
- filesProcessed: 0,
414
- filesSucceeded: 0,
415
- filesFailed: 0,
416
- totalRecordsProcessed: 0,
417
- totalRecordsFailed: 0,
418
- };
419
- }
420
-
421
- // ───────────────────────────────────────────────────────────────────────────
422
- // STEP 3: Process Each CSV File
423
- // ───────────────────────────────────────────────────────────────────────────
424
-
425
- let filesSucceeded = 0;
426
- let filesFailed = 0;
427
- let totalRecordsProcessed = 0;
428
- let totalRecordsFailed = 0;
429
-
430
- for (const file of csvFiles) {
431
- logger.info('-'.repeat(80));
432
- logger.info(`Processing file: ${file.name}`, {
433
- size: file.size,
434
- lastModified: file.lastModified,
435
- });
436
-
437
- try {
438
- // ─────────────────────────────────────────────────────────────────────
439
- // STEP 3a: Download CSV File
440
- // ─────────────────────────────────────────────────────────────────────
441
- logger.info(`Downloading file: ${file.path}`);
442
- const csvContent = await s3Source.downloadFile(file.path);
443
- logger.info(
444
- `Downloaded ${typeof csvContent === 'string' ? csvContent.length : csvContent.length} bytes`
445
- );
446
-
447
- // ─────────────────────────────────────────────────────────────────────
448
- // STEP 3b: Parse CSV to Records
449
- // ─────────────────────────────────────────────────────────────────────
450
- logger.info('Parsing CSV content...');
451
- const records = await csvParser.parse(csvContent as string);
452
- logger.info(`Parsed ${records.length} records from CSV`);
453
-
454
- if (records.length === 0) {
455
- logger.warn('CSV file is empty, skipping');
456
- continue;
457
- }
458
-
459
- // ─────────────────────────────────────────────────────────────────────
460
- // STEP 3c: Validate Records (Basic Validation)
461
- // ─────────────────────────────────────────────────────────────────────
462
- logger.info('Validating records...');
463
- const validRecords = records.filter((record: any, index: number) => {
464
- // Check for required fields
465
- if (!record.sku || !record.location || record.quantity === undefined) {
466
- logger.warn(`Record ${index + 1} missing required fields, skipping`, {
467
- record,
468
- });
469
- totalRecordsFailed++;
470
- return false;
471
- }
472
-
473
- // Validate quantity is numeric
474
- if (isNaN(Number(record.quantity))) {
475
- logger.warn(`Record ${index + 1} has invalid quantity, skipping`, {
476
- record,
477
- });
478
- totalRecordsFailed++;
479
- return false;
480
- }
481
-
482
- return true;
483
- });
484
-
485
- logger.info(
486
- `Validation complete: ${validRecords.length} valid, ${records.length - validRecords.length} invalid`
487
- );
488
-
489
- if (validRecords.length === 0) {
490
- logger.warn('No valid records found in file, skipping');
491
- continue;
492
- }
493
-
494
- // ─────────────────────────────────────────────────────────────────────
495
- // STEP 3d: Map Records Using UniversalMapper
496
- // ─────────────────────────────────────────────────────────────────────
497
- logger.info('Mapping records using UniversalMapper...');
498
- const mappedRecords: any[] = [];
499
- const mappingErrors: string[] = [];
500
-
501
- for (let i = 0; i < validRecords.length; i++) {
502
- const record = validRecords[i];
503
- const result = await mapper.map(record);
504
-
505
- if (result.success) {
506
- mappedRecords.push(result.data);
507
- } else {
508
- logger.warn(`Record ${i + 1} mapping failed`, {
509
- errors: result.errors,
510
- record,
511
- });
512
- mappingErrors.push(`Record ${i + 1}: ${result.errors?.join(', ')}`);
513
- totalRecordsFailed++;
514
- }
515
- }
516
-
517
- logger.info(
518
- `Mapping complete: ${mappedRecords.length} mapped, ${mappingErrors.length} failed`
519
- );
520
-
521
- if (mappedRecords.length === 0) {
522
- logger.warn('No records successfully mapped, skipping file');
523
- continue;
524
- }
525
-
526
- // ─────────────────────────────────────────────────────────────────────
527
- // STEP 3e: Create Batch API Job
528
- // ─────────────────────────────────────────────────────────────────────
529
- const jobName = `${config.processing.jobNamePrefix} - ${file.name} - ${new Date().toISOString()}`;
530
- logger.info(`Creating Batch API job: ${jobName}`);
531
-
532
- const job = await fluentClient.createJob({
533
- name: jobName,
534
- retailerId: config.fluent.retailerId,
535
- });
536
-
537
- logger.info(`Job created successfully`, {
538
- jobId: job.id,
539
- status: job.status,
540
- });
541
-
542
- // ─────────────────────────────────────────────────────────────────────
543
- // STEP 3f: Send Batches (Chunked by Batch Size)
544
- // ─────────────────────────────────────────────────────────────────────
545
- logger.info(
546
- `Sending ${mappedRecords.length} records in batches of ${config.processing.batchSize}`
547
- );
548
- const batches = chunk(mappedRecords, config.processing.batchSize);
549
-
550
- for (let i = 0; i < batches.length; i++) {
551
- const batch = batches[i];
552
- logger.info(`Sending batch ${i + 1}/${batches.length} (${batch.length} records)`);
553
-
554
- const batchResponse = await fluentClient.sendBatch(job.id, {
555
- entities: batch,
556
- });
557
-
558
- logger.info(`Batch ${i + 1} sent successfully`, {
559
- batchId: batchResponse.id,
560
- status: batchResponse.status,
561
- });
562
- }
563
-
564
- // ─────────────────────────────────────────────────────────────────────
565
- // STEP 3g: Poll Job Status Until Complete
566
- // ─────────────────────────────────────────────────────────────────────
567
- logger.info('Polling job status...');
568
- const finalStatus = await pollJobStatus(fluentClient, job.id, logger);
569
-
570
- logger.info(`Job completed with status: ${finalStatus.status}`, {
571
- jobId: job.id,
572
- status: finalStatus.status,
573
- });
574
-
575
- // Check if job succeeded
576
- if (finalStatus.status === 'COMPLETE') {
577
- totalRecordsProcessed += mappedRecords.length;
578
- filesSucceeded++;
579
-
580
- // ───────────────────────────────────────────────────────────────────
581
- // STEP 3h: Archive Successful File
582
- // ───────────────────────────────────────────────────────────────────
583
- const archivePath = `${config.processing.archivePrefix}${new Date().toISOString().split('T')[0]}/${file.name}`;
584
- logger.info(`Archiving file to: ${archivePath}`);
585
- await s3Source.moveFile(file.path, archivePath);
586
- logger.info('File archived successfully');
587
- } else {
588
- // Job failed or has errors
589
- logger.error('Job failed or has errors', undefined, {
590
- jobId: job.id,
591
- status: finalStatus.status,
592
- });
593
-
594
- // Move to error folder
595
- const errorPath = `${config.processing.errorPrefix}${new Date().toISOString().split('T')[0]}/${file.name}`;
596
- logger.info(`Moving file to error folder: ${errorPath}`);
597
- await s3Source.moveFile(file.path, errorPath);
598
- logger.info('File moved to error folder');
599
-
600
- filesFailed++;
601
- }
602
- } catch (error) {
603
- // File processing error
604
- logger.error(`Failed to process file: ${file.name}`, error as Error);
605
- filesFailed++;
606
-
607
- try {
608
- // Move to error folder
609
- const errorPath = `${config.processing.errorPrefix}${new Date().toISOString().split('T')[0]}/${file.name}`;
610
- await s3Source.moveFile(file.path, errorPath);
611
- logger.info('File moved to error folder after processing error');
612
- } catch (moveError) {
613
- logger.error('Failed to move file to error folder', moveError as Error);
614
- }
615
- }
616
- }
617
-
618
- // ───────────────────────────────────────────────────────────────────────────
619
- // STEP 4: Summary Statistics
620
- // ───────────────────────────────────────────────────────────────────────────
621
-
622
- const duration = Date.now() - startTime;
623
- logger.info('='.repeat(80));
624
- logger.info('Inventory sync completed', {
625
- duration: `${(duration / 1000).toFixed(2)}s`,
626
- filesProcessed: csvFiles.length,
627
- filesSucceeded,
628
- filesFailed,
629
- totalRecordsProcessed,
630
- totalRecordsFailed,
631
- });
632
- logger.info('='.repeat(80));
633
-
634
- return {
635
- filesProcessed: csvFiles.length,
636
- filesSucceeded,
637
- filesFailed,
638
- totalRecordsProcessed,
639
- totalRecordsFailed,
640
- };
641
- }
642
-
643
- // ============================================================================
644
- // HELPER FUNCTIONS
645
- // ============================================================================
646
-
647
- /**
648
- * Poll Batch API job status until completion
649
- *
650
- * POLLING STRATEGY:
651
- * - Initial delay: 2 seconds
652
- * - Max delay: 30 seconds
653
- * - Exponential backoff with 1.5x multiplier
654
- * - Timeout: 30 minutes
655
- *
656
- * TERMINAL STATES:
657
- * - COMPLETE: Job succeeded
658
- * - FAILED: Job failed (API error, validation errors, etc.)
659
- * - Other statuses: Continue polling
660
- */
661
- async function pollJobStatus(client: any, jobId: string, logger: any): Promise<any> {
662
- let delay = 2000; // Start with 2 second delay
663
- const maxDelay = 30000; // Cap at 30 seconds
664
- const timeout = 30 * 60 * 1000; // 30 minutes
665
- const startTime = Date.now();
666
-
667
- while (true) {
668
- // Check timeout
669
- if (Date.now() - startTime > timeout) {
670
- throw new Error('Job status polling timeout (30 minutes)');
671
- }
672
-
673
- // Get job status
674
- const status = await client.getJobStatus(jobId);
675
- logger.debug(`Job status: ${status.status}`, {
676
- jobId,
677
- status: status.status,
678
- });
679
-
680
- // Check terminal states
681
- if (status.status === 'COMPLETE' || status.status === 'FAILED') {
682
- return status;
683
- }
684
-
685
- // Wait before next poll
686
- await sleep(delay);
687
-
688
- // Increase delay with exponential backoff (1.5x multiplier)
689
- delay = Math.min(delay * 1.5, maxDelay);
690
- }
691
- }
692
-
693
- /**
694
- * Sleep helper function
695
- */
696
- function sleep(ms: number): Promise<void> {
697
- return new Promise(resolve => setTimeout(resolve, ms));
698
- }
699
-
700
- /**
701
- * Chunk array into smaller arrays
702
- */
703
- function chunk<T>(array: T[], size: number): T[][] {
704
- const chunks: T[][] = [];
705
- for (let i = 0; i < array.length; i += size) {
706
- chunks.push(array.slice(i, i + size));
707
- }
708
- return chunks;
709
- }
710
-
711
- // ============================================================================
712
- // ENTRY POINT
713
- // ============================================================================
714
-
715
- /**
716
- * Main entry point with error handling
717
- */
718
- async function main() {
719
- try {
720
- const result = await syncInventory();
721
-
722
- // Exit with success code if no files failed
723
- if (result.filesFailed === 0) {
724
- logger.info('All files processed successfully');
725
- process.exit(0);
726
- } else {
727
- logger.warn('Some files failed to process');
728
- process.exit(1);
729
- }
730
- } catch (error) {
731
- logger.error('Fatal error during inventory sync', error as Error);
732
- process.exit(1);
733
- }
734
- }
735
-
736
- // Run the sync
737
- main();
738
- ```
739
-
740
- ### 4. Package.json Configuration
741
-
742
- ```json
743
- {
744
- "name": "inventory-sync-standalone",
745
- "version": "1.0.0",
746
- "description": "Standalone S3 CSV to Fluent Commerce inventory sync",
747
- "main": "dist/inventory-sync.js",
748
- "type": "module",
749
- "scripts": {
750
- "build": "tsc",
751
- "start": "node dist/inventory-sync.js",
752
- "dev": "tsx src/inventory-sync.ts",
753
- "sync": "npm run dev",
754
- "lint": "eslint src --ext .ts",
755
- "type-check": "tsc --noEmit"
756
- },
757
- "keywords": ["fluent-commerce", "inventory", "s3", "batch-api"],
758
- "author": "",
759
- "license": "MIT",
760
- "dependencies": {
761
- "@fluentcommerce/fc-connect-sdk": "^0.1.39",
762
- "dotenv": "^16.0.0"
763
- },
764
- "devDependencies": {
765
- "@types/node": "^18.0.0",
766
- "tsx": "^4.0.0",
767
- "typescript": "^5.0.0"
768
- }
769
- }
770
- ```
771
-
772
- ### 5. TypeScript Configuration
773
-
774
- Create `tsconfig.json`:
775
-
776
- ```json
777
- {
778
- "compilerOptions": {
779
- "module": "ES2022",
780
- "target": "ES2024",
781
- "moduleResolution": "node"
782
- }
783
- }
784
- ```
785
-
786
- ---
787
-
788
- ## Key Patterns Explained
789
-
790
- ### Pattern 1: OAuth2 Authentication
791
-
792
- **Purpose**: Authenticate with Fluent Commerce without Versori connection
793
-
794
- **Implementation**:
795
-
796
- ```typescript
797
- const fluentClient = await createClient({
798
- config: {
799
- baseUrl: 'https://api.fluentcommerce.com',
800
- clientId: process.env.FLUENT_CLIENT_ID,
801
- clientSecret: process.env.FLUENT_CLIENT_SECRET,
802
- retailerId: process.env.FLUENT_RETAILER_ID,
803
- },
804
- });
805
- ```
806
-
807
- **Key Points**:
808
-
809
- - Uses client credentials OAuth2 grant type
810
- - Token automatically cached and refreshed by SDK
811
- - No manual token management needed
812
- - Token expires after ~1 hour, SDK handles refresh transparently
813
-
814
- **When to Use**:
815
-
816
- - Standalone Node.js/Deno scripts
817
- - Background jobs/cron tasks
818
- - CI/CD pipelines
819
- - Local development
820
-
821
- ### Pattern 2: S3 File Operations
822
-
823
- **Purpose**: List, download, and move files in S3 bucket
824
-
825
- **Implementation**:
826
-
827
- ```typescript
828
- // Initialize S3 Data Source
829
- const s3Source = new S3DataSource(
830
- {
831
- type: 'S3_CSV',
832
- connectionId: 'inventory-s3',
833
- name: 'Inventory S3 Source',
834
- s3Config: {
835
- bucket: 'my-bucket',
836
- region: 'us-east-1',
837
- accessKeyId: process.env.AWS_ACCESS_KEY_ID,
838
- secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
839
- },
840
- },
841
- logger
842
- );
843
-
844
- // List files with prefix filter
845
- const files = await s3Source.listFiles({
846
- prefix: 'inventory/updates/',
847
- maxKeys: 1000,
848
- });
849
-
850
- // Download file content
851
- const content = await s3Source.downloadFile(file.path);
852
-
853
- // Move file (copy + delete)
854
- await s3Source.moveFile(sourcePath, destPath);
855
- ```
856
-
857
- **Key Points**:
858
-
859
- - Uses presigned URLs (no AWS SDK required)
860
- - Automatic retry logic for transient failures
861
- - Streaming support for large files
862
- - Cross-bucket move support
863
-
864
- **Best Practices**:
865
-
866
- - Always use prefix filtering to limit results
867
- - Move files to archive/error folders (don't delete)
868
- - Use date-based folder structure for archiving
869
- - Validate S3 connection before processing
870
-
871
- ### Pattern 3: CSV Parsing & Validation
872
-
873
- **Purpose**: Parse CSV files and validate data quality
874
-
875
- **Implementation**:
876
-
877
- ```typescript
878
- // Initialize CSV Parser
879
- const csvParser = new CSVParserService();
880
-
881
- // Parse CSV content (returns array of objects)
882
- const records = await csvParser.parse(csvContent);
883
-
884
- // Validate records
885
- const validRecords = records.filter(record => {
886
- // Required field validation
887
- if (!record.sku || !record.location || record.quantity === undefined) {
888
- logger.warn('Missing required fields', { record });
889
- return false;
890
- }
891
-
892
- // Type validation
893
- if (isNaN(Number(record.quantity))) {
894
- logger.warn('Invalid quantity', { record });
895
- return false;
896
- }
897
-
898
- return true;
899
- });
900
- ```
901
-
902
- **Key Points**:
903
-
904
- - CSV parser auto-detects headers by default
905
- - Trims whitespace and skips empty lines
906
- - Returns array of objects (header → value mapping)
907
- - Validation happens before mapping (fail fast)
908
-
909
- **Validation Strategies**:
910
-
911
- 1. **Required Fields**: Check for null/undefined/empty
912
- 2. **Type Validation**: Ensure numeric fields are numbers
913
- 3. **Format Validation**: Regex for SKU, email, phone, etc.
914
- 4. **Business Rules**: Range checks, valid enum values
915
- 5. **Cross-field Validation**: Relationships between fields
916
-
917
- ### Pattern 4: Batch API Workflow
918
-
919
- **Purpose**: Send large datasets to Fluent Commerce efficiently
920
-
921
- **Implementation**:
922
-
923
- ```typescript
924
- // STEP 1: Create Job
925
- const job = await fluentClient.createJob({
926
- name: 'Inventory Update - file.csv',
927
- retailerId: 'my-retailer',
928
- });
929
-
930
- // STEP 2: Send Batches (chunked)
931
- const batchSize = 100;
932
- for (let i = 0; i < records.length; i += batchSize) {
933
- const batch = records.slice(i, i + batchSize);
934
- await fluentClient.sendBatch(job.id, {
935
- entities: batch,
936
- });
937
- }
938
-
939
- // STEP 3: Poll Job Status
940
- let status = await fluentClient.getJobStatus(job.id);
941
- while (status.status !== 'COMPLETE' && status.status !== 'FAILED') {
942
- await sleep(5000); // Wait 5 seconds
943
- status = await fluentClient.getJobStatus(job.id);
944
- }
945
-
946
- // STEP 4: Check Results
947
- if (status.status === 'COMPLETE') {
948
- console.log('Job succeeded!');
949
- } else {
950
- console.error('Job failed:', status);
951
- }
952
- ```
953
-
954
- **Key Points**:
955
-
956
- - One job per file (easier tracking)
957
- - Batch size 50-500 records (100 recommended)
958
- - Poll with exponential backoff
959
- - Handle both COMPLETE and FAILED states
960
-
961
- **Batch Size Guidelines**:
962
-
963
- - Small records (<10 fields): 200-500 per batch
964
- - Medium records (10-30 fields): 100-200 per batch
965
- - Large records (>30 fields): 50-100 per batch
966
- - API limit: Usually 1000 records per batch
967
-
968
- ### Pattern 5: Error Handling & Retry
969
-
970
- **Purpose**: Graceful degradation and recovery from failures
971
-
972
- **Implementation**:
973
-
974
- ```typescript
975
- // File-level try-catch
976
- for (const file of files) {
977
- try {
978
- await processFile(file);
979
- filesSucceeded++;
980
- } catch (error) {
981
- logger.error('File processing failed', error);
982
- filesFailed++;
983
-
984
- // Move to error folder
985
- try {
986
- await s3Source.moveFile(file.path, errorPath);
987
- } catch (moveError) {
988
- logger.error('Failed to move error file', moveError);
989
- }
990
- }
991
- }
992
-
993
- // Automatic retry with exponential backoff
994
- async function retryWithBackoff<T>(fn: () => Promise<T>, maxRetries: number = 3): Promise<T> {
995
- let lastError: Error;
996
- for (let attempt = 0; attempt < maxRetries; attempt++) {
997
- try {
998
- return await fn();
999
- } catch (error) {
1000
- lastError = error as Error;
1001
- if (attempt < maxRetries - 1) {
1002
- const delay = Math.pow(2, attempt) * 1000;
1003
- await sleep(delay);
1004
- }
1005
- }
1006
- }
1007
- throw lastError!;
1008
- }
1009
- ```
1010
-
1011
- **Error Categories**:
1012
-
1013
- 1. **Network Errors**: Retry with exponential backoff
1014
- 2. **4xx Client Errors**: Don't retry (bad request)
1015
- 3. **5xx Server Errors**: Retry (transient failure)
1016
- 4. **Validation Errors**: Skip record, continue processing
1017
- 5. **Fatal Errors**: Exit with error code
1018
-
1019
- **Retry Best Practices**:
1020
-
1021
- - Max 3-5 retry attempts
1022
- - Exponential backoff (1s, 2s, 4s, 8s, ...)
1023
- - Jitter to prevent thundering herd
1024
- - Log each retry attempt
1025
- - Give up after max retries
1026
-
1027
- ---
1028
-
1029
- ## Deployment Options
1030
-
1031
- ### Option 1: Local Execution
1032
-
1033
- ```bash
1034
- # One-time sync
1035
- npm run sync
1036
-
1037
- # With debug logging
1038
- LOG_LEVEL=debug npm run sync
1039
- ```
1040
-
1041
- ### Option 2: Docker Container
1042
-
1043
- Create `Dockerfile`:
1044
-
1045
- ```dockerfile
1046
- FROM node:18-alpine
1047
- WORKDIR /app
1048
-
1049
- # Copy package files
1050
- COPY package*.json ./
1051
-
1052
- # Install dependencies
1053
- RUN npm ci --only=production
1054
-
1055
- # Copy application code
1056
- COPY dist ./dist
1057
-
1058
- # Run the sync
1059
- CMD ["node", "dist/inventory-sync.js"]
1060
- ```
1061
-
1062
- Build and run:
1063
-
1064
- ```bash
1065
- # Build image
1066
- docker build -t inventory-sync .
1067
-
1068
- # Run container
1069
- docker run --env-file .env inventory-sync
1070
- ```
1071
-
1072
- ### Option 3: AWS Lambda
1073
-
1074
- Create `lambda-handler.ts`:
1075
-
1076
- ```typescript
1077
- import { syncInventory } from './inventory-sync';
1078
-
1079
- export const handler = async (event: any, context: any) => {
1080
- try {
1081
- const result = await syncInventory();
1082
- return {
1083
- statusCode: 200,
1084
- body: JSON.stringify(result),
1085
- };
1086
- } catch (error) {
1087
- console.error('Lambda error:', error);
1088
- return {
1089
- statusCode: 500,
1090
- body: JSON.stringify({ error: (error as Error).message }),
1091
- };
1092
- }
1093
- };
1094
- ```
1095
-
1096
- Deploy:
1097
-
1098
- ```bash
1099
- # Package for Lambda
1100
- npm run build
1101
- zip -r function.zip dist node_modules
1102
-
1103
- # Deploy with AWS CLI
1104
- aws lambda create-function \
1105
- --function-name inventory-sync \
1106
- --runtime nodejs18.x \
1107
- --handler dist/lambda-handler.handler \
1108
- --zip-file fileb://function.zip \
1109
- --role arn:aws:iam::123456789:role/lambda-execution \
1110
- --timeout 900 \
1111
- --memory-size 512
1112
- ```
1113
-
1114
- ### Option 4: Cron Scheduling
1115
-
1116
- **Linux/Mac cron**:
1117
-
1118
- ```bash
1119
- # Edit crontab
1120
- crontab -e
1121
-
1122
- # Run every 4 hours
1123
- 0 */4 * * * cd /path/to/inventory-sync && npm run sync >> /var/log/inventory-sync.log 2>&1
1124
- ```
1125
-
1126
- **Node.js cron**:
1127
-
1128
- Install: `npm install node-cron`
1129
-
1130
- ```typescript
1131
- import cron from 'node-cron';
1132
-
1133
- // Run every 4 hours
1134
- cron.schedule('0 */4 * * *', async () => {
1135
- logger.info('Starting scheduled inventory sync');
1136
- await syncInventory();
1137
- });
1138
-
1139
- logger.info('Cron scheduler started (every 4 hours)');
1140
- ```
1141
-
1142
- **AWS EventBridge**:
1143
-
1144
- ```bash
1145
- # Create rule
1146
- aws events put-rule \
1147
- --name inventory-sync-schedule \
1148
- --schedule-expression "rate(4 hours)"
1149
-
1150
- # Add Lambda target
1151
- aws events put-targets \
1152
- --rule inventory-sync-schedule \
1153
- --targets "Id"="1","Arn"="arn:aws:lambda:us-east-1:123:function:inventory-sync"
1154
- ```
1155
-
1156
- ---
1157
-
1158
- ## Testing
1159
-
1160
- ### Local Testing
1161
-
1162
- ```bash
1163
- # 1. Create test CSV file
1164
- cat > test-inventory.csv << EOF
1165
- sku,location,quantity,status,expected_date
1166
- TEST-SKU-001,WAREHOUSE-A,100,ACTIVE,2024-01-15
1167
- TEST-SKU-002,WAREHOUSE-B,50,ACTIVE,2024-01-15
1168
- TEST-SKU-003,STORE-001,25,ACTIVE,2024-01-16
1169
- EOF
1170
-
1171
- # 2. Upload to S3
1172
- aws s3 cp test-inventory.csv s3://your-bucket/inventory/updates/
1173
-
1174
- # 3. Run sync
1175
- npm run sync
1176
-
1177
- # 4. Check logs
1178
- cat logs/inventory-sync.log
1179
-
1180
- # 5. Verify in Fluent Commerce
1181
- # - Check job status in admin portal
1182
- # - Query inventory positions via GraphQL
1183
- ```
1184
-
1185
- ### Integration Testing
1186
-
1187
- Create `tests/integration.test.ts`:
1188
-
1189
- ```typescript
1190
- import { describe, it, expect, beforeAll } from '@jest/globals';
1191
- import { syncInventory } from '../src/inventory-sync';
1192
-
1193
- describe('Inventory Sync Integration', () => {
1194
- beforeAll(async () => {
1195
- // Setup test environment
1196
- });
1197
-
1198
- it('should process valid CSV file', async () => {
1199
- // Upload test file to S3
1200
- // Run sync
1201
- const result = await syncInventory();
1202
-
1203
- // Assertions
1204
- expect(result.filesSucceeded).toBe(1);
1205
- expect(result.filesFailed).toBe(0);
1206
- expect(result.totalRecordsProcessed).toBeGreaterThan(0);
1207
- });
1208
-
1209
- it('should handle invalid records gracefully', async () => {
1210
- // Upload CSV with invalid records
1211
- // Run sync
1212
- const result = await syncInventory();
1213
-
1214
- // Should still succeed but skip invalid records
1215
- expect(result.filesSucceeded).toBe(1);
1216
- expect(result.totalRecordsFailed).toBeGreaterThan(0);
1217
- });
1218
-
1219
- it('should move failed files to error folder', async () => {
1220
- // Upload malformed CSV
1221
- // Run sync
1222
- const result = await syncInventory();
1223
-
1224
- // File should be in error folder
1225
- expect(result.filesFailed).toBe(1);
1226
- // Check S3 error folder
1227
- });
1228
- });
1229
- ```
1230
-
1231
- Run tests:
1232
-
1233
- ```bash
1234
- npm install --save-dev jest @jest/globals @types/jest ts-jest
1235
- npm test
1236
- ```
1237
-
1238
- ---
1239
-
1240
- ## Common Issues
1241
-
1242
- ### Issue 1: OAuth2 Token Expired
1243
-
1244
- **Symptom**: `401 Unauthorized` errors
1245
-
1246
- **Cause**: Token expired and SDK failed to refresh
1247
-
1248
- **Solution**:
1249
-
1250
- ```typescript
1251
- // SDK handles refresh automatically, but you can force refresh
1252
- const client = await createClient({
1253
- config: {
1254
- baseUrl: 'https://api.fluentcommerce.com',
1255
- clientId: process.env.FLUENT_CLIENT_ID,
1256
- clientSecret: process.env.FLUENT_CLIENT_SECRET,
1257
- retailerId: process.env.FLUENT_RETAILER_ID,
1258
- retryAttempts: 3,
1259
- },
1260
- });
1261
- ```
1262
-
1263
- ### Issue 2: S3 Permission Denied
1264
-
1265
- **Symptom**: `AccessDenied` errors when listing/downloading files
1266
-
1267
- **Cause**: IAM policy missing required permissions
1268
-
1269
- **Solution**:
1270
-
1271
- ```json
1272
- {
1273
- "Version": "2012-10-17",
1274
- "Statement": [
1275
- {
1276
- "Effect": "Allow",
1277
- "Action": ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
1278
- "Resource": ["arn:aws:s3:::your-bucket", "arn:aws:s3:::your-bucket/*"]
1279
- }
1280
- ]
1281
- }
1282
- ```
1283
-
1284
- ### Issue 3: CSV Parsing Errors
1285
-
1286
- **Symptom**: `CSV parse error: Invalid record`
1287
-
1288
- **Cause**: Malformed CSV (unescaped quotes, inconsistent columns)
1289
-
1290
- **Solution**:
1291
-
1292
- ```typescript
1293
- // Use more lenient CSV parser options
1294
- const csvParser = CSVParserService.create({
1295
- columns: true,
1296
- skip_empty_lines: true,
1297
- trim: true,
1298
- relax_column_count: true, // Allow inconsistent column counts
1299
- quote: '"',
1300
- escape: '"',
1301
- delimiter: ',',
1302
- });
1303
- ```
1304
-
1305
- ### Issue 4: Batch API Timeout
1306
-
1307
- **Symptom**: Job stays in `PENDING` state indefinitely
1308
-
1309
- **Cause**: Fluent API processing delay or large batch size
1310
-
1311
- **Solution**:
1312
-
1313
- ```typescript
1314
- // Reduce batch size
1315
- const batchSize = 50; // Instead of 100
1316
-
1317
- // Increase polling timeout
1318
- const timeout = 60 * 60 * 1000; // 1 hour instead of 30 minutes
1319
-
1320
- // Add more aggressive polling
1321
- let delay = 1000; // Start with 1 second
1322
- const maxDelay = 15000; // Cap at 15 seconds
1323
- ```
1324
-
1325
- ### Issue 5: Memory Exhaustion
1326
-
1327
- **Symptom**: `JavaScript heap out of memory` error
1328
-
1329
- **Cause**: Loading entire large CSV file into memory
1330
-
1331
- **Solution**:
1332
-
1333
- ```typescript
1334
- // Use streaming CSV parsing instead
1335
- for await (const record of csvParser.parseStreaming(csvContent, {}, 100)) {
1336
- // Process records in batches of 100
1337
- const mappedBatch = await Promise.all(record.map(r => mapper.map(r)));
1338
-
1339
- // Send batch immediately (don't accumulate)
1340
- await fluentClient.sendBatch(jobId, {
1341
- entities: mappedBatch.filter(r => r.success).map(r => r.data),
1342
- });
1343
- }
1344
- ```
1345
-
1346
- ---
1347
-
1348
- ## Related Guides
1349
-
1350
- - **SDK Reference**: `../../readme.md` - Core SDK documentation
1351
- - **Universal Mapping Guide**: `../../02-CORE-GUIDES/mapping/readme.md` - Field mapping patterns
1352
- - **Batch API Guide**: `../../02-CORE-GUIDES/api-reference/modules/03-batch-processing.md` - Batch API details
1353
- - **S3 Data Source**: `../../02-CORE-GUIDES/data-sources/readme.md` - S3 operations
1354
- - **CSV Parser**: `../../02-CORE-GUIDES/parsers/readme.md` - CSV parsing options
1355
- - **Error Handling**: `../../03-PATTERN-GUIDES/error-handling/readme.md` - Error patterns
1356
-
1357
- ---
1358
-
1359
- ## Production Checklist
1360
-
1361
- Before deploying to production:
1362
-
1363
- - [ ] Environment variables secured (AWS Secrets Manager, vault, etc.)
1364
- - [ ] Logging configured (Winston, CloudWatch, Datadog, etc.)
1365
- - [ ] Error alerting setup (PagerDuty, Slack, email)
1366
- - [ ] Monitoring dashboard (metrics, job success rate, duration)
1367
- - [ ] Retry logic tested (network failures, API errors)
1368
- - [ ] Batch size optimized (tested with production file sizes)
1369
- - [ ] Memory profiling done (no leaks)
1370
- - [ ] S3 lifecycle policies configured (archive old files)
1371
- - [ ] IAM permissions minimized (least privilege)
1372
- - [ ] Health check endpoint added (for orchestration tools)
1373
- - [ ] Documentation updated (runbook, troubleshooting)
1374
- - [ ] Disaster recovery tested (restore from archive)
1375
-
1376
- ---
1377
-
1378
- **Next Steps**:
1379
-
1380
- 1. Customize field mappings for your inventory data structure
1381
- 2. Add custom validation rules for your business logic
1382
- 3. Implement state tracking to prevent duplicate processing
1383
- 4. Set up monitoring and alerting
1384
- 5. Deploy to your preferred environment (Lambda, ECS, K8s, etc.)
1385
-
1386
- **Support**:
1387
-
1388
- - SDK Issues: https://github.com/fluentcommerce/fc-connect-sdk/issues
1389
- - Fluent Commerce API: https://docs.fluentcommerce.com
1390
- - Community: https://community.fluentcommerce.com
1
+ # Standalone: S3 CSV → Fluent Batch API
2
+
3
+ **FC Connect SDK Use Case Guide**
4
+
5
+ > **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
6
+ > **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
7
+
8
+ **Context**: Node.js script that reads CSV inventory files from S3 and updates Fluent Commerce inventory via Batch API
9
+
10
+ **Complexity**: Medium
11
+
12
+ **Runtime**: Node.js ≥18 / Deno
13
+
14
+ **Estimated Lines**: ~500 lines
15
+
16
+ ## What You'll Build
17
+
18
+ - Standalone Node.js/Deno script (no Versori)
19
+ - OAuth2 authentication with Fluent Commerce
20
+ - S3 file listing and download with AWS SDK
21
+ - CSV parsing with validation
22
+ - UniversalMapper for field transformations
23
+ - Batch API processing with job management
24
+ - Error handling and logging
25
+ - Optional: Schedule with cron or AWS EventBridge
26
+
27
+ ## SDK Methods Used
28
+
29
+ - `createClient({ config: { baseUrl, clientId, clientSecret, retailerId } })` - OAuth2 client
30
+ - `S3DataSource(config, logger)` - S3 operations
31
+ - `CSVParserService` - CSV parsing
32
+ - `UniversalMapper(mappingConfig)` - Field mapping
33
+ - `client.createJob({ name, retailerId })` - Create Batch job
34
+ - `client.sendBatch(jobId, { entities })` - Send inventory batch
35
+ - `client.getJobStatus(jobId)` - Check status
36
+
37
+ ---
38
+
39
+ ## Complete Working Implementation
40
+
41
+ ### 1. Project Setup
42
+
43
+ ```bash
44
+ # Create project directory
45
+ mkdir inventory-sync-standalone
46
+ cd inventory-sync-standalone
47
+
48
+ # Initialize Node.js project
49
+ npm init -y
50
+
51
+ # Install dependencies
52
+ npm install @fluentcommerce/fc-connect-sdk dotenv
53
+ ```
54
+
55
+ ### Method selection (SDK)
56
+
57
+ - Queries → `client.graphql`
58
+ - Mutations → `client.graphql`
59
+ - Advanced needs (pagination/telemetry) → `client.graphql` with pagination options
60
+
61
+ ```typescript
62
+ // Query (use graphql for both queries and mutations)
63
+ const products = await client.graphql({ query, variables });
64
+
65
+ // Mutation (use graphql for both queries and mutations)
66
+ const result = await client.graphql({ query: mutation, variables });
67
+
68
+ // Advanced pagination
69
+ const res = await client.graphql({ query, variables, pagination: { maxPages: 50 } });
70
+ ```
71
+
72
+ ### 2. Environment Configuration
73
+
74
+ Create `.env` file:
75
+
76
+ ```env
77
+ # Fluent Commerce Configuration
78
+ FLUENT_BASE_URL=https://api.fluentcommerce.com
79
+ FLUENT_CLIENT_ID=your-oauth2-client-id
80
+ FLUENT_CLIENT_SECRET=your-oauth2-client-secret
81
+ FLUENT_RETAILER_ID=your-retailer-id
82
+
83
+ # AWS S3 Configuration
84
+ AWS_ACCESS_KEY_ID=your-aws-access-key
85
+ AWS_SECRET_ACCESS_KEY=your-aws-secret-key
86
+ AWS_REGION=us-east-1
87
+ AWS_BUCKET_NAME=your-inventory-bucket
88
+ AWS_FILE_PREFIX=inventory/updates/
89
+
90
+ # Processing Configuration
91
+ BATCH_SIZE=100
92
+ JOB_NAME_PREFIX=S3-CSV-Inventory-Update
93
+ ARCHIVE_PREFIX=inventory/archive/
94
+ ERROR_PREFIX=inventory/errors/
95
+
96
+ # Optional: Scheduling
97
+ ENABLE_SCHEDULING=false
98
+ CRON_SCHEDULE=0 */4 * * *
99
+ ```
100
+
101
+ ### 3. Main Script Implementation
102
+
103
+ Create `src/inventory-sync.ts`:
104
+
105
+ ```typescript
106
+ import 'dotenv/config';
107
+ // FC Connect SDK+
108
+ // Install: npm install @fluentcommerce/fc-connect-sdk@latest
109
+ // Docs: https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk
110
+ // GitHub: https://github.com/fluentcommerce/fc-connect-sdk
111
+ import { createClient } from '@fluentcommerce/fc-connect-sdk';
112
+ import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
113
+ import { CSVParserService } from '@fluentcommerce/fc-connect-sdk';
114
+ import { UniversalMapper } from '@fluentcommerce/fc-connect-sdk';
115
+
116
+ /**
117
+ * ARCHITECTURE:
118
+ *
119
+ * This standalone script implements a complete inventory sync workflow:
120
+ * 1. List CSV files from S3 bucket
121
+ * 2. Download and parse CSV files
122
+ * 3. Map CSV fields to Fluent inventory format
123
+ * 4. Create Batch API job
124
+ * 5. Send inventory updates in batches
125
+ * 6. Poll job status until complete
126
+ * 7. Archive successful files, move errors
127
+ *
128
+ * ERROR HANDLING STRATEGY:
129
+ * - File-level errors: Move to error folder, continue with other files
130
+ * - Batch errors: Log and continue (Fluent API provides detailed error reports)
131
+ * - Fatal errors: Exit with error code for monitoring/alerting
132
+ *
133
+ * STATE MANAGEMENT:
134
+ * - No state tracking in this basic version (processes all files)
135
+ * - Advanced: Use S3 metadata or external DB to track processed files
136
+ */
137
+
138
+ // ============================================================================
139
+ // CONFIGURATION & SETUP
140
+ // ============================================================================
141
+
142
+ interface SyncConfig {
143
+ fluent: {
144
+ baseUrl: string;
145
+ clientId: string;
146
+ clientSecret: string;
147
+ retailerId: string;
148
+ };
149
+ s3: {
150
+ accessKeyId: string;
151
+ secretAccessKey: string;
152
+ region: string;
153
+ bucket: string;
154
+ prefix: string;
155
+ };
156
+ processing: {
157
+ batchSize: number;
158
+ jobNamePrefix: string;
159
+ archivePrefix: string;
160
+ errorPrefix: string;
161
+ };
162
+ }
163
+
164
+ /**
165
+ * Load configuration from environment variables
166
+ * Validates all required config before proceeding
167
+ */
168
+ function loadConfig(): SyncConfig {
169
+ const requiredEnvVars = [
170
+ 'FLUENT_BASE_URL',
171
+ 'FLUENT_CLIENT_ID',
172
+ 'FLUENT_CLIENT_SECRET',
173
+ 'FLUENT_RETAILER_ID',
174
+ 'AWS_ACCESS_KEY_ID',
175
+ 'AWS_SECRET_ACCESS_KEY',
176
+ 'AWS_REGION',
177
+ 'AWS_BUCKET_NAME',
178
+ ];
179
+
180
+ // Validate all required environment variables are present
181
+ const missing = requiredEnvVars.filter(v => !process.env[v]);
182
+ if (missing.length > 0) {
183
+ throw new Error(`Missing required environment variables: ${missing.join(', ')}`);
184
+ }
185
+
186
+ return {
187
+ fluent: {
188
+ baseUrl: process.env.FLUENT_BASE_URL!,
189
+ clientId: process.env.FLUENT_CLIENT_ID!,
190
+ clientSecret: process.env.FLUENT_CLIENT_SECRET!,
191
+ retailerId: process.env.FLUENT_RETAILER_ID!,
192
+ },
193
+ s3: {
194
+ accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
195
+ secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
196
+ region: process.env.AWS_REGION!,
197
+ bucket: process.env.AWS_BUCKET_NAME!,
198
+ prefix: process.env.AWS_FILE_PREFIX || 'inventory/updates/',
199
+ },
200
+ processing: {
201
+ batchSize: parseInt(process.env.BATCH_SIZE || '100'),
202
+ jobNamePrefix: process.env.JOB_NAME_PREFIX || 'S3-CSV-Inventory-Update',
203
+ archivePrefix: process.env.ARCHIVE_PREFIX || 'inventory/archive/',
204
+ errorPrefix: process.env.ERROR_PREFIX || 'inventory/errors/',
205
+ },
206
+ };
207
+ }
208
+
209
+ // ============================================================================
210
+ // FIELD MAPPING CONFIGURATION
211
+ // ============================================================================
212
+
213
+ /**
214
+ * UniversalMapper configuration for CSV → Fluent Batch API transformation
215
+ *
216
+ * FIELD MAPPING RULES:
217
+ * - ref: Product SKU (required) - uppercase transformation
218
+ * - locationRef: Location code (required) - prefix with "LOC:"
219
+ * - qty: Quantity (required) - parse as integer, default to 0
220
+ * - type: Inventory type (required) - defaults to "ADJUSTMENT"
221
+ * - status: Inventory status (optional) - defaults to "ACTIVE"
222
+ *
223
+ * SDK RESOLVERS USED:
224
+ * - sdk.uppercase: Convert SKU to uppercase for consistency
225
+ * - sdk.parseInt: Parse quantity string to integer
226
+ * - sdk.trim: Remove whitespace from location codes
227
+ * - sdk.defaultTo: Provide fallback values
228
+ */
229
+ const inventoryMappingConfig = {
230
+ version: '1.0.0',
231
+ description: 'S3 CSV to Fluent Commerce Inventory Mapping',
232
+ fields: {
233
+ // Product SKU (required field)
234
+ ref: {
235
+ source: 'sku',
236
+ resolver: 'sdk.uppercase', // Normalize to uppercase
237
+ required: true,
238
+ },
239
+ // Location reference (required field)
240
+ locationRef: {
241
+ source: 'location',
242
+ resolver: 'sdk.trim', // Remove whitespace
243
+ required: true,
244
+ },
245
+ // Quantity (required field)
246
+ qty: {
247
+ source: 'quantity',
248
+ resolver: 'sdk.parseInt', // Parse as integer
249
+ required: true,
250
+ },
251
+ // Inventory type (defaults to ADJUSTMENT)
252
+ type: {
253
+ value: 'ADJUSTMENT', // Static value
254
+ },
255
+ // Inventory status (optional, with default)
256
+ status: {
257
+ source: 'status',
258
+ resolver: 'sdk.uppercase',
259
+ defaultValue: 'ACTIVE', // Fallback if not provided
260
+ },
261
+ // Expected date (optional)
262
+ expectedOn: {
263
+ source: 'expected_date',
264
+ resolver: 'sdk.formatDate', // Convert to ISO date format
265
+ },
266
+ },
267
+ };
268
+
269
+ // ============================================================================
270
+ // LOGGER IMPLEMENTATION
271
+ // ============================================================================
272
+
273
+ /**
274
+ * Simple console logger with timestamp and log levels
275
+ * Production: Replace with Winston, Bunyan, or Pino
276
+ */
277
+ const logger = {
278
+ debug: (message: string, meta?: any) => {
279
+ if (process.env.LOG_LEVEL === 'debug') {
280
+ console.log(`[${new Date().toISOString()}] DEBUG: ${message}`, meta || '');
281
+ }
282
+ },
283
+ info: (message: string, meta?: any) => {
284
+ console.log(`[${new Date().toISOString()}] INFO: ${message}`, meta || '');
285
+ },
286
+ warn: (message: string, meta?: any) => {
287
+ console.warn(`[${new Date().toISOString()}] WARN: ${message}`, meta || '');
288
+ },
289
+ error: (message: string, error?: Error, meta?: any) => {
290
+ console.error(
291
+ `[${new Date().toISOString()}] ERROR: ${message}`,
292
+ error?.message || '',
293
+ meta || ''
294
+ );
295
+ },
296
+ };
297
+
298
+ // ============================================================================
299
+ // MAIN SYNC ORCHESTRATION
300
+ // ============================================================================
301
+
302
+ /**
303
+ * Main inventory sync function
304
+ *
305
+ * ALGORITHM:
306
+ * 1. Initialize SDK clients (Fluent, S3, CSV parser, mapper)
307
+ * 2. List CSV files from S3 prefix
308
+ * 3. For each file:
309
+ * a. Download file content
310
+ * b. Parse CSV to records
311
+ * c. Validate records (skip invalid)
312
+ * d. Map records using UniversalMapper
313
+ * e. Create Batch API job
314
+ * f. Send batches (chunked by batch size)
315
+ * g. Poll job status until complete
316
+ * h. Archive successful files OR move to error folder
317
+ * 4. Return summary statistics
318
+ */
319
+ async function syncInventory(): Promise<{
320
+ filesProcessed: number;
321
+ filesSucceeded: number;
322
+ filesFailed: number;
323
+ totalRecordsProcessed: number;
324
+ totalRecordsFailed: number;
325
+ }> {
326
+ const startTime = Date.now();
327
+ logger.info('='.repeat(80));
328
+ logger.info('Starting S3 CSV → Fluent Batch API inventory sync');
329
+ logger.info('='.repeat(80));
330
+
331
+ // Load configuration
332
+ const config = loadConfig();
333
+ logger.info('Configuration loaded', {
334
+ s3Bucket: config.s3.bucket,
335
+ s3Prefix: config.s3.prefix,
336
+ batchSize: config.processing.batchSize,
337
+ });
338
+
339
+ // ───────────────────────────────────────────────────────────────────────────
340
+ // STEP 1: Initialize SDK Clients
341
+ // ───────────────────────────────────────────────────────────────────────────
342
+
343
+ // Initialize Fluent Client with OAuth2
344
+ logger.info('Initializing Fluent Commerce client...');
345
+ const fluentClient = await createClient({
346
+ config: {
347
+ baseUrl: config.fluent.baseUrl,
348
+ clientId: config.fluent.clientId,
349
+ clientSecret: config.fluent.clientSecret,
350
+ retailerId: config.fluent.retailerId,
351
+ },
352
+ });
353
+
354
+ // Initialize S3 Data Source
355
+ logger.info('Initializing S3 data source...');
356
+ const s3Source = new S3DataSource(
357
+ {
358
+ type: 'S3_CSV',
359
+ connectionId: 'inventory-s3',
360
+ name: 'Inventory S3 Source',
361
+ s3Config: {
362
+ bucket: config.s3.bucket,
363
+ region: config.s3.region,
364
+ accessKeyId: config.s3.accessKeyId,
365
+ secretAccessKey: config.s3.secretAccessKey,
366
+ },
367
+ },
368
+ logger
369
+ );
370
+
371
+ // Initialize CSV Parser
372
+ const csvParser = new CSVParserService();
373
+
374
+ // Initialize Universal Mapper
375
+ const mapper = new UniversalMapper(inventoryMappingConfig, {
376
+ logger,
377
+ fluentClient,
378
+ });
379
+
380
+ // Validate S3 connection
381
+ logger.info('Validating S3 connection...');
382
+ const s3Connected = await s3Source.validateConnection();
383
+ if (!s3Connected) {
384
+ throw new Error('Failed to connect to S3 bucket');
385
+ }
386
+ logger.info('S3 connection validated successfully');
387
+
388
+ // ───────────────────────────────────────────────────────────────────────────
389
+ // STEP 2: List CSV Files from S3
390
+ // ───────────────────────────────────────────────────────────────────────────
391
+
392
+ logger.info(`Listing CSV files from S3 prefix: ${config.s3.prefix}`);
393
+ const files = await s3Source.listFiles({
394
+ prefix: config.s3.prefix,
395
+ maxKeys: 1000,
396
+ });
397
+
398
+ // Filter for CSV files only
399
+ // NOTE: S3 doesn't support glob patterns, so manual filtering is required
400
+ // IMPORTANT: Use .endsWith('.csv') NOT .endsWith('*.csv')
401
+ // ✓ Correct: f.name.endsWith('.csv')
402
+ // ✗ Wrong: f.name.endsWith('*.csv') // Would never match!
403
+ const csvFiles = files.filter(f => f.name.toLowerCase().endsWith('.csv'));
404
+
405
+ logger.info(`Found ${csvFiles.length} CSV files to process`, {
406
+ totalFiles: files.length,
407
+ csvFiles: csvFiles.length,
408
+ });
409
+
410
+ if (csvFiles.length === 0) {
411
+ logger.info('No CSV files found. Exiting.');
412
+ return {
413
+ filesProcessed: 0,
414
+ filesSucceeded: 0,
415
+ filesFailed: 0,
416
+ totalRecordsProcessed: 0,
417
+ totalRecordsFailed: 0,
418
+ };
419
+ }
420
+
421
+ // ───────────────────────────────────────────────────────────────────────────
422
+ // STEP 3: Process Each CSV File
423
+ // ───────────────────────────────────────────────────────────────────────────
424
+
425
+ let filesSucceeded = 0;
426
+ let filesFailed = 0;
427
+ let totalRecordsProcessed = 0;
428
+ let totalRecordsFailed = 0;
429
+
430
+ for (const file of csvFiles) {
431
+ logger.info('-'.repeat(80));
432
+ logger.info(`Processing file: ${file.name}`, {
433
+ size: file.size,
434
+ lastModified: file.lastModified,
435
+ });
436
+
437
+ try {
438
+ // ─────────────────────────────────────────────────────────────────────
439
+ // STEP 3a: Download CSV File
440
+ // ─────────────────────────────────────────────────────────────────────
441
+ logger.info(`Downloading file: ${file.path}`);
442
+ const csvContent = await s3Source.downloadFile(file.path);
443
+ logger.info(
444
+ `Downloaded ${typeof csvContent === 'string' ? csvContent.length : csvContent.length} bytes`
445
+ );
446
+
447
+ // ─────────────────────────────────────────────────────────────────────
448
+ // STEP 3b: Parse CSV to Records
449
+ // ─────────────────────────────────────────────────────────────────────
450
+ logger.info('Parsing CSV content...');
451
+ const records = await csvParser.parse(csvContent as string);
452
+ logger.info(`Parsed ${records.length} records from CSV`);
453
+
454
+ if (records.length === 0) {
455
+ logger.warn('CSV file is empty, skipping');
456
+ continue;
457
+ }
458
+
459
+ // ─────────────────────────────────────────────────────────────────────
460
+ // STEP 3c: Validate Records (Basic Validation)
461
+ // ─────────────────────────────────────────────────────────────────────
462
+ logger.info('Validating records...');
463
+ const validRecords = records.filter((record: any, index: number) => {
464
+ // Check for required fields
465
+ if (!record.sku || !record.location || record.quantity === undefined) {
466
+ logger.warn(`Record ${index + 1} missing required fields, skipping`, {
467
+ record,
468
+ });
469
+ totalRecordsFailed++;
470
+ return false;
471
+ }
472
+
473
+ // Validate quantity is numeric
474
+ if (isNaN(Number(record.quantity))) {
475
+ logger.warn(`Record ${index + 1} has invalid quantity, skipping`, {
476
+ record,
477
+ });
478
+ totalRecordsFailed++;
479
+ return false;
480
+ }
481
+
482
+ return true;
483
+ });
484
+
485
+ logger.info(
486
+ `Validation complete: ${validRecords.length} valid, ${records.length - validRecords.length} invalid`
487
+ );
488
+
489
+ if (validRecords.length === 0) {
490
+ logger.warn('No valid records found in file, skipping');
491
+ continue;
492
+ }
493
+
494
+ // ─────────────────────────────────────────────────────────────────────
495
+ // STEP 3d: Map Records Using UniversalMapper
496
+ // ─────────────────────────────────────────────────────────────────────
497
+ logger.info('Mapping records using UniversalMapper...');
498
+ const mappedRecords: any[] = [];
499
+ const mappingErrors: string[] = [];
500
+
501
+ for (let i = 0; i < validRecords.length; i++) {
502
+ const record = validRecords[i];
503
+ const result = await mapper.map(record);
504
+
505
+ if (result.success) {
506
+ mappedRecords.push(result.data);
507
+ } else {
508
+ logger.warn(`Record ${i + 1} mapping failed`, {
509
+ errors: result.errors,
510
+ record,
511
+ });
512
+ mappingErrors.push(`Record ${i + 1}: ${result.errors?.join(', ')}`);
513
+ totalRecordsFailed++;
514
+ }
515
+ }
516
+
517
+ logger.info(
518
+ `Mapping complete: ${mappedRecords.length} mapped, ${mappingErrors.length} failed`
519
+ );
520
+
521
+ if (mappedRecords.length === 0) {
522
+ logger.warn('No records successfully mapped, skipping file');
523
+ continue;
524
+ }
525
+
526
+ // ─────────────────────────────────────────────────────────────────────
527
+ // STEP 3e: Create Batch API Job
528
+ // ─────────────────────────────────────────────────────────────────────
529
+ const jobName = `${config.processing.jobNamePrefix} - ${file.name} - ${new Date().toISOString()}`;
530
+ logger.info(`Creating Batch API job: ${jobName}`);
531
+
532
+ const job = await fluentClient.createJob({
533
+ name: jobName,
534
+ retailerId: config.fluent.retailerId,
535
+ });
536
+
537
+ logger.info(`Job created successfully`, {
538
+ jobId: job.id,
539
+ status: job.status,
540
+ });
541
+
542
+ // ─────────────────────────────────────────────────────────────────────
543
+ // STEP 3f: Send Batches (Chunked by Batch Size)
544
+ // ─────────────────────────────────────────────────────────────────────
545
+ logger.info(
546
+ `Sending ${mappedRecords.length} records in batches of ${config.processing.batchSize}`
547
+ );
548
+ const batches = chunk(mappedRecords, config.processing.batchSize);
549
+
550
+ for (let i = 0; i < batches.length; i++) {
551
+ const batch = batches[i];
552
+ logger.info(`Sending batch ${i + 1}/${batches.length} (${batch.length} records)`);
553
+
554
+ const batchResponse = await fluentClient.sendBatch(job.id, {
555
+ entities: batch,
556
+ });
557
+
558
+ logger.info(`Batch ${i + 1} sent successfully`, {
559
+ batchId: batchResponse.id,
560
+ status: batchResponse.status,
561
+ });
562
+ }
563
+
564
+ // ─────────────────────────────────────────────────────────────────────
565
+ // STEP 3g: Poll Job Status Until Complete
566
+ // ─────────────────────────────────────────────────────────────────────
567
+ logger.info('Polling job status...');
568
+ const finalStatus = await pollJobStatus(fluentClient, job.id, logger);
569
+
570
+ logger.info(`Job completed with status: ${finalStatus.status}`, {
571
+ jobId: job.id,
572
+ status: finalStatus.status,
573
+ });
574
+
575
+ // Check if job succeeded
576
+ if (finalStatus.status === 'COMPLETE') {
577
+ totalRecordsProcessed += mappedRecords.length;
578
+ filesSucceeded++;
579
+
580
+ // ───────────────────────────────────────────────────────────────────
581
+ // STEP 3h: Archive Successful File
582
+ // ───────────────────────────────────────────────────────────────────
583
+ const archivePath = `${config.processing.archivePrefix}${new Date().toISOString().split('T')[0]}/${file.name}`;
584
+ logger.info(`Archiving file to: ${archivePath}`);
585
+ await s3Source.moveFile(file.path, archivePath);
586
+ logger.info('File archived successfully');
587
+ } else {
588
+ // Job failed or has errors
589
+ logger.error('Job failed or has errors', undefined, {
590
+ jobId: job.id,
591
+ status: finalStatus.status,
592
+ });
593
+
594
+ // Move to error folder
595
+ const errorPath = `${config.processing.errorPrefix}${new Date().toISOString().split('T')[0]}/${file.name}`;
596
+ logger.info(`Moving file to error folder: ${errorPath}`);
597
+ await s3Source.moveFile(file.path, errorPath);
598
+ logger.info('File moved to error folder');
599
+
600
+ filesFailed++;
601
+ }
602
+ } catch (error) {
603
+ // File processing error
604
+ logger.error(`Failed to process file: ${file.name}`, error as Error);
605
+ filesFailed++;
606
+
607
+ try {
608
+ // Move to error folder
609
+ const errorPath = `${config.processing.errorPrefix}${new Date().toISOString().split('T')[0]}/${file.name}`;
610
+ await s3Source.moveFile(file.path, errorPath);
611
+ logger.info('File moved to error folder after processing error');
612
+ } catch (moveError) {
613
+ logger.error('Failed to move file to error folder', moveError as Error);
614
+ }
615
+ }
616
+ }
617
+
618
+ // ───────────────────────────────────────────────────────────────────────────
619
+ // STEP 4: Summary Statistics
620
+ // ───────────────────────────────────────────────────────────────────────────
621
+
622
+ const duration = Date.now() - startTime;
623
+ logger.info('='.repeat(80));
624
+ logger.info('Inventory sync completed', {
625
+ duration: `${(duration / 1000).toFixed(2)}s`,
626
+ filesProcessed: csvFiles.length,
627
+ filesSucceeded,
628
+ filesFailed,
629
+ totalRecordsProcessed,
630
+ totalRecordsFailed,
631
+ });
632
+ logger.info('='.repeat(80));
633
+
634
+ return {
635
+ filesProcessed: csvFiles.length,
636
+ filesSucceeded,
637
+ filesFailed,
638
+ totalRecordsProcessed,
639
+ totalRecordsFailed,
640
+ };
641
+ }
642
+
643
+ // ============================================================================
644
+ // HELPER FUNCTIONS
645
+ // ============================================================================
646
+
647
+ /**
648
+ * Poll Batch API job status until completion
649
+ *
650
+ * POLLING STRATEGY:
651
+ * - Initial delay: 2 seconds
652
+ * - Max delay: 30 seconds
653
+ * - Exponential backoff with 1.5x multiplier
654
+ * - Timeout: 30 minutes
655
+ *
656
+ * TERMINAL STATES:
657
+ * - COMPLETE: Job succeeded
658
+ * - FAILED: Job failed (API error, validation errors, etc.)
659
+ * - Other statuses: Continue polling
660
+ */
661
+ async function pollJobStatus(client: any, jobId: string, logger: any): Promise<any> {
662
+ let delay = 2000; // Start with 2 second delay
663
+ const maxDelay = 30000; // Cap at 30 seconds
664
+ const timeout = 30 * 60 * 1000; // 30 minutes
665
+ const startTime = Date.now();
666
+
667
+ while (true) {
668
+ // Check timeout
669
+ if (Date.now() - startTime > timeout) {
670
+ throw new Error('Job status polling timeout (30 minutes)');
671
+ }
672
+
673
+ // Get job status
674
+ const status = await client.getJobStatus(jobId);
675
+ logger.debug(`Job status: ${status.status}`, {
676
+ jobId,
677
+ status: status.status,
678
+ });
679
+
680
+ // Check terminal states
681
+ if (status.status === 'COMPLETE' || status.status === 'FAILED') {
682
+ return status;
683
+ }
684
+
685
+ // Wait before next poll
686
+ await sleep(delay);
687
+
688
+ // Increase delay with exponential backoff (1.5x multiplier)
689
+ delay = Math.min(delay * 1.5, maxDelay);
690
+ }
691
+ }
692
+
693
+ /**
694
+ * Sleep helper function
695
+ */
696
+ function sleep(ms: number): Promise<void> {
697
+ return new Promise(resolve => setTimeout(resolve, ms));
698
+ }
699
+
700
+ /**
701
+ * Chunk array into smaller arrays
702
+ */
703
+ function chunk<T>(array: T[], size: number): T[][] {
704
+ const chunks: T[][] = [];
705
+ for (let i = 0; i < array.length; i += size) {
706
+ chunks.push(array.slice(i, i + size));
707
+ }
708
+ return chunks;
709
+ }
710
+
711
+ // ============================================================================
712
+ // ENTRY POINT
713
+ // ============================================================================
714
+
715
+ /**
716
+ * Main entry point with error handling
717
+ */
718
+ async function main() {
719
+ try {
720
+ const result = await syncInventory();
721
+
722
+ // Exit with success code if no files failed
723
+ if (result.filesFailed === 0) {
724
+ logger.info('All files processed successfully');
725
+ process.exit(0);
726
+ } else {
727
+ logger.warn('Some files failed to process');
728
+ process.exit(1);
729
+ }
730
+ } catch (error) {
731
+ logger.error('Fatal error during inventory sync', error as Error);
732
+ process.exit(1);
733
+ }
734
+ }
735
+
736
+ // Run the sync
737
+ main();
738
+ ```
739
+
740
+ ### 4. Package.json Configuration
741
+
742
+ ```json
743
+ {
744
+ "name": "inventory-sync-standalone",
745
+ "version": "1.0.0",
746
+ "description": "Standalone S3 CSV to Fluent Commerce inventory sync",
747
+ "main": "dist/inventory-sync.js",
748
+ "type": "module",
749
+ "scripts": {
750
+ "build": "tsc",
751
+ "start": "node dist/inventory-sync.js",
752
+ "dev": "tsx src/inventory-sync.ts",
753
+ "sync": "npm run dev",
754
+ "lint": "eslint src --ext .ts",
755
+ "type-check": "tsc --noEmit"
756
+ },
757
+ "keywords": ["fluent-commerce", "inventory", "s3", "batch-api"],
758
+ "author": "",
759
+ "license": "MIT",
760
+ "dependencies": {
761
+ "@fluentcommerce/fc-connect-sdk": "^0.1.39",
762
+ "dotenv": "^16.0.0"
763
+ },
764
+ "devDependencies": {
765
+ "@types/node": "^18.0.0",
766
+ "tsx": "^4.0.0",
767
+ "typescript": "^5.0.0"
768
+ }
769
+ }
770
+ ```
771
+
772
+ ### 5. TypeScript Configuration
773
+
774
+ Create `tsconfig.json`:
775
+
776
+ ```json
777
+ {
778
+ "compilerOptions": {
779
+ "module": "ES2022",
780
+ "target": "ES2024",
781
+ "moduleResolution": "node"
782
+ }
783
+ }
784
+ ```
785
+
786
+ ---
787
+
788
+ ## Key Patterns Explained
789
+
790
+ ### Pattern 1: OAuth2 Authentication
791
+
792
+ **Purpose**: Authenticate with Fluent Commerce without Versori connection
793
+
794
+ **Implementation**:
795
+
796
+ ```typescript
797
+ const fluentClient = await createClient({
798
+ config: {
799
+ baseUrl: 'https://api.fluentcommerce.com',
800
+ clientId: process.env.FLUENT_CLIENT_ID,
801
+ clientSecret: process.env.FLUENT_CLIENT_SECRET,
802
+ retailerId: process.env.FLUENT_RETAILER_ID,
803
+ },
804
+ });
805
+ ```
806
+
807
+ **Key Points**:
808
+
809
+ - Uses client credentials OAuth2 grant type
810
+ - Token automatically cached and refreshed by SDK
811
+ - No manual token management needed
812
+ - Token expires after ~1 hour, SDK handles refresh transparently
813
+
814
+ **When to Use**:
815
+
816
+ - Standalone Node.js/Deno scripts
817
+ - Background jobs/cron tasks
818
+ - CI/CD pipelines
819
+ - Local development
820
+
821
+ ### Pattern 2: S3 File Operations
822
+
823
+ **Purpose**: List, download, and move files in S3 bucket
824
+
825
+ **Implementation**:
826
+
827
+ ```typescript
828
+ // Initialize S3 Data Source
829
+ const s3Source = new S3DataSource(
830
+ {
831
+ type: 'S3_CSV',
832
+ connectionId: 'inventory-s3',
833
+ name: 'Inventory S3 Source',
834
+ s3Config: {
835
+ bucket: 'my-bucket',
836
+ region: 'us-east-1',
837
+ accessKeyId: process.env.AWS_ACCESS_KEY_ID,
838
+ secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
839
+ },
840
+ },
841
+ logger
842
+ );
843
+
844
+ // List files with prefix filter
845
+ const files = await s3Source.listFiles({
846
+ prefix: 'inventory/updates/',
847
+ maxKeys: 1000,
848
+ });
849
+
850
+ // Download file content
851
+ const content = await s3Source.downloadFile(file.path);
852
+
853
+ // Move file (copy + delete)
854
+ await s3Source.moveFile(sourcePath, destPath);
855
+ ```
856
+
857
+ **Key Points**:
858
+
859
+ - Uses presigned URLs (no AWS SDK required)
860
+ - Automatic retry logic for transient failures
861
+ - Streaming support for large files
862
+ - Cross-bucket move support
863
+
864
+ **Best Practices**:
865
+
866
+ - Always use prefix filtering to limit results
867
+ - Move files to archive/error folders (don't delete)
868
+ - Use date-based folder structure for archiving
869
+ - Validate S3 connection before processing
870
+
871
+ ### Pattern 3: CSV Parsing & Validation
872
+
873
+ **Purpose**: Parse CSV files and validate data quality
874
+
875
+ **Implementation**:
876
+
877
+ ```typescript
878
+ // Initialize CSV Parser
879
+ const csvParser = new CSVParserService();
880
+
881
+ // Parse CSV content (returns array of objects)
882
+ const records = await csvParser.parse(csvContent);
883
+
884
+ // Validate records
885
+ const validRecords = records.filter(record => {
886
+ // Required field validation
887
+ if (!record.sku || !record.location || record.quantity === undefined) {
888
+ logger.warn('Missing required fields', { record });
889
+ return false;
890
+ }
891
+
892
+ // Type validation
893
+ if (isNaN(Number(record.quantity))) {
894
+ logger.warn('Invalid quantity', { record });
895
+ return false;
896
+ }
897
+
898
+ return true;
899
+ });
900
+ ```
901
+
902
+ **Key Points**:
903
+
904
+ - CSV parser auto-detects headers by default
905
+ - Trims whitespace and skips empty lines
906
+ - Returns array of objects (header → value mapping)
907
+ - Validation happens before mapping (fail fast)
908
+
909
+ **Validation Strategies**:
910
+
911
+ 1. **Required Fields**: Check for null/undefined/empty
912
+ 2. **Type Validation**: Ensure numeric fields are numbers
913
+ 3. **Format Validation**: Regex for SKU, email, phone, etc.
914
+ 4. **Business Rules**: Range checks, valid enum values
915
+ 5. **Cross-field Validation**: Relationships between fields
916
+
917
+ ### Pattern 4: Batch API Workflow
918
+
919
+ **Purpose**: Send large datasets to Fluent Commerce efficiently
920
+
921
+ **Implementation**:
922
+
923
+ ```typescript
924
+ // STEP 1: Create Job
925
+ const job = await fluentClient.createJob({
926
+ name: 'Inventory Update - file.csv',
927
+ retailerId: 'my-retailer',
928
+ });
929
+
930
+ // STEP 2: Send Batches (chunked)
931
+ const batchSize = 100;
932
+ for (let i = 0; i < records.length; i += batchSize) {
933
+ const batch = records.slice(i, i + batchSize);
934
+ await fluentClient.sendBatch(job.id, {
935
+ entities: batch,
936
+ });
937
+ }
938
+
939
+ // STEP 3: Poll Job Status
940
+ let status = await fluentClient.getJobStatus(job.id);
941
+ while (status.status !== 'COMPLETE' && status.status !== 'FAILED') {
942
+ await sleep(5000); // Wait 5 seconds
943
+ status = await fluentClient.getJobStatus(job.id);
944
+ }
945
+
946
+ // STEP 4: Check Results
947
+ if (status.status === 'COMPLETE') {
948
+ console.log('Job succeeded!');
949
+ } else {
950
+ console.error('Job failed:', status);
951
+ }
952
+ ```
953
+
954
+ **Key Points**:
955
+
956
+ - One job per file (easier tracking)
957
+ - Batch size 50-500 records (100 recommended)
958
+ - Poll with exponential backoff
959
+ - Handle both COMPLETE and FAILED states
960
+
961
+ **Batch Size Guidelines**:
962
+
963
+ - Small records (<10 fields): 200-500 per batch
964
+ - Medium records (10-30 fields): 100-200 per batch
965
+ - Large records (>30 fields): 50-100 per batch
966
+ - API limit: Usually 1000 records per batch
967
+
968
+ ### Pattern 5: Error Handling & Retry
969
+
970
+ **Purpose**: Graceful degradation and recovery from failures
971
+
972
+ **Implementation**:
973
+
974
+ ```typescript
975
+ // File-level try-catch
976
+ for (const file of files) {
977
+ try {
978
+ await processFile(file);
979
+ filesSucceeded++;
980
+ } catch (error) {
981
+ logger.error('File processing failed', error);
982
+ filesFailed++;
983
+
984
+ // Move to error folder
985
+ try {
986
+ await s3Source.moveFile(file.path, errorPath);
987
+ } catch (moveError) {
988
+ logger.error('Failed to move error file', moveError);
989
+ }
990
+ }
991
+ }
992
+
993
+ // Automatic retry with exponential backoff
994
+ async function retryWithBackoff<T>(fn: () => Promise<T>, maxRetries: number = 3): Promise<T> {
995
+ let lastError: Error;
996
+ for (let attempt = 0; attempt < maxRetries; attempt++) {
997
+ try {
998
+ return await fn();
999
+ } catch (error) {
1000
+ lastError = error as Error;
1001
+ if (attempt < maxRetries - 1) {
1002
+ const delay = Math.pow(2, attempt) * 1000;
1003
+ await sleep(delay);
1004
+ }
1005
+ }
1006
+ }
1007
+ throw lastError!;
1008
+ }
1009
+ ```
1010
+
1011
+ **Error Categories**:
1012
+
1013
+ 1. **Network Errors**: Retry with exponential backoff
1014
+ 2. **4xx Client Errors**: Don't retry (bad request)
1015
+ 3. **5xx Server Errors**: Retry (transient failure)
1016
+ 4. **Validation Errors**: Skip record, continue processing
1017
+ 5. **Fatal Errors**: Exit with error code
1018
+
1019
+ **Retry Best Practices**:
1020
+
1021
+ - Max 3-5 retry attempts
1022
+ - Exponential backoff (1s, 2s, 4s, 8s, ...)
1023
+ - Jitter to prevent thundering herd
1024
+ - Log each retry attempt
1025
+ - Give up after max retries
1026
+
1027
+ ---
1028
+
1029
+ ## Deployment Options
1030
+
1031
+ ### Option 1: Local Execution
1032
+
1033
+ ```bash
1034
+ # One-time sync
1035
+ npm run sync
1036
+
1037
+ # With debug logging
1038
+ LOG_LEVEL=debug npm run sync
1039
+ ```
1040
+
1041
+ ### Option 2: Docker Container
1042
+
1043
+ Create `Dockerfile`:
1044
+
1045
+ ```dockerfile
1046
+ FROM node:18-alpine
1047
+ WORKDIR /app
1048
+
1049
+ # Copy package files
1050
+ COPY package*.json ./
1051
+
1052
+ # Install dependencies
1053
+ RUN npm ci --only=production
1054
+
1055
+ # Copy application code
1056
+ COPY dist ./dist
1057
+
1058
+ # Run the sync
1059
+ CMD ["node", "dist/inventory-sync.js"]
1060
+ ```
1061
+
1062
+ Build and run:
1063
+
1064
+ ```bash
1065
+ # Build image
1066
+ docker build -t inventory-sync .
1067
+
1068
+ # Run container
1069
+ docker run --env-file .env inventory-sync
1070
+ ```
1071
+
1072
+ ### Option 3: AWS Lambda
1073
+
1074
+ Create `lambda-handler.ts`:
1075
+
1076
+ ```typescript
1077
+ import { syncInventory } from './inventory-sync';
1078
+
1079
+ export const handler = async (event: any, context: any) => {
1080
+ try {
1081
+ const result = await syncInventory();
1082
+ return {
1083
+ statusCode: 200,
1084
+ body: JSON.stringify(result),
1085
+ };
1086
+ } catch (error) {
1087
+ console.error('Lambda error:', error);
1088
+ return {
1089
+ statusCode: 500,
1090
+ body: JSON.stringify({ error: (error as Error).message }),
1091
+ };
1092
+ }
1093
+ };
1094
+ ```
1095
+
1096
+ Deploy:
1097
+
1098
+ ```bash
1099
+ # Package for Lambda
1100
+ npm run build
1101
+ zip -r function.zip dist node_modules
1102
+
1103
+ # Deploy with AWS CLI
1104
+ aws lambda create-function \
1105
+ --function-name inventory-sync \
1106
+ --runtime nodejs18.x \
1107
+ --handler dist/lambda-handler.handler \
1108
+ --zip-file fileb://function.zip \
1109
+ --role arn:aws:iam::123456789:role/lambda-execution \
1110
+ --timeout 900 \
1111
+ --memory-size 512
1112
+ ```
1113
+
1114
+ ### Option 4: Cron Scheduling
1115
+
1116
+ **Linux/Mac cron**:
1117
+
1118
+ ```bash
1119
+ # Edit crontab
1120
+ crontab -e
1121
+
1122
+ # Run every 4 hours
1123
+ 0 */4 * * * cd /path/to/inventory-sync && npm run sync >> /var/log/inventory-sync.log 2>&1
1124
+ ```
1125
+
1126
+ **Node.js cron**:
1127
+
1128
+ Install: `npm install node-cron`
1129
+
1130
+ ```typescript
1131
+ import cron from 'node-cron';
1132
+
1133
+ // Run every 4 hours
1134
+ cron.schedule('0 */4 * * *', async () => {
1135
+ logger.info('Starting scheduled inventory sync');
1136
+ await syncInventory();
1137
+ });
1138
+
1139
+ logger.info('Cron scheduler started (every 4 hours)');
1140
+ ```
1141
+
1142
+ **AWS EventBridge**:
1143
+
1144
+ ```bash
1145
+ # Create rule
1146
+ aws events put-rule \
1147
+ --name inventory-sync-schedule \
1148
+ --schedule-expression "rate(4 hours)"
1149
+
1150
+ # Add Lambda target
1151
+ aws events put-targets \
1152
+ --rule inventory-sync-schedule \
1153
+ --targets "Id"="1","Arn"="arn:aws:lambda:us-east-1:123:function:inventory-sync"
1154
+ ```
1155
+
1156
+ ---
1157
+
1158
+ ## Testing
1159
+
1160
+ ### Local Testing
1161
+
1162
+ ```bash
1163
+ # 1. Create test CSV file
1164
+ cat > test-inventory.csv << EOF
1165
+ sku,location,quantity,status,expected_date
1166
+ TEST-SKU-001,WAREHOUSE-A,100,ACTIVE,2024-01-15
1167
+ TEST-SKU-002,WAREHOUSE-B,50,ACTIVE,2024-01-15
1168
+ TEST-SKU-003,STORE-001,25,ACTIVE,2024-01-16
1169
+ EOF
1170
+
1171
+ # 2. Upload to S3
1172
+ aws s3 cp test-inventory.csv s3://your-bucket/inventory/updates/
1173
+
1174
+ # 3. Run sync
1175
+ npm run sync
1176
+
1177
+ # 4. Check logs
1178
+ cat logs/inventory-sync.log
1179
+
1180
+ # 5. Verify in Fluent Commerce
1181
+ # - Check job status in admin portal
1182
+ # - Query inventory positions via GraphQL
1183
+ ```
1184
+
1185
+ ### Integration Testing
1186
+
1187
+ Create `tests/integration.test.ts`:
1188
+
1189
+ ```typescript
1190
+ import { describe, it, expect, beforeAll } from '@jest/globals';
1191
+ import { syncInventory } from '../src/inventory-sync';
1192
+
1193
+ describe('Inventory Sync Integration', () => {
1194
+ beforeAll(async () => {
1195
+ // Setup test environment
1196
+ });
1197
+
1198
+ it('should process valid CSV file', async () => {
1199
+ // Upload test file to S3
1200
+ // Run sync
1201
+ const result = await syncInventory();
1202
+
1203
+ // Assertions
1204
+ expect(result.filesSucceeded).toBe(1);
1205
+ expect(result.filesFailed).toBe(0);
1206
+ expect(result.totalRecordsProcessed).toBeGreaterThan(0);
1207
+ });
1208
+
1209
+ it('should handle invalid records gracefully', async () => {
1210
+ // Upload CSV with invalid records
1211
+ // Run sync
1212
+ const result = await syncInventory();
1213
+
1214
+ // Should still succeed but skip invalid records
1215
+ expect(result.filesSucceeded).toBe(1);
1216
+ expect(result.totalRecordsFailed).toBeGreaterThan(0);
1217
+ });
1218
+
1219
+ it('should move failed files to error folder', async () => {
1220
+ // Upload malformed CSV
1221
+ // Run sync
1222
+ const result = await syncInventory();
1223
+
1224
+ // File should be in error folder
1225
+ expect(result.filesFailed).toBe(1);
1226
+ // Check S3 error folder
1227
+ });
1228
+ });
1229
+ ```
1230
+
1231
+ Run tests:
1232
+
1233
+ ```bash
1234
+ npm install --save-dev jest @jest/globals @types/jest ts-jest
1235
+ npm test
1236
+ ```
1237
+
1238
+ ---
1239
+
1240
+ ## Common Issues
1241
+
1242
+ ### Issue 1: OAuth2 Token Expired
1243
+
1244
+ **Symptom**: `401 Unauthorized` errors
1245
+
1246
+ **Cause**: Token expired and SDK failed to refresh
1247
+
1248
+ **Solution**:
1249
+
1250
+ ```typescript
1251
+ // SDK handles refresh automatically, but you can force refresh
1252
+ const client = await createClient({
1253
+ config: {
1254
+ baseUrl: 'https://api.fluentcommerce.com',
1255
+ clientId: process.env.FLUENT_CLIENT_ID,
1256
+ clientSecret: process.env.FLUENT_CLIENT_SECRET,
1257
+ retailerId: process.env.FLUENT_RETAILER_ID,
1258
+ retryAttempts: 3,
1259
+ },
1260
+ });
1261
+ ```
1262
+
1263
+ ### Issue 2: S3 Permission Denied
1264
+
1265
+ **Symptom**: `AccessDenied` errors when listing/downloading files
1266
+
1267
+ **Cause**: IAM policy missing required permissions
1268
+
1269
+ **Solution**:
1270
+
1271
+ ```json
1272
+ {
1273
+ "Version": "2012-10-17",
1274
+ "Statement": [
1275
+ {
1276
+ "Effect": "Allow",
1277
+ "Action": ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
1278
+ "Resource": ["arn:aws:s3:::your-bucket", "arn:aws:s3:::your-bucket/*"]
1279
+ }
1280
+ ]
1281
+ }
1282
+ ```
1283
+
1284
+ ### Issue 3: CSV Parsing Errors
1285
+
1286
+ **Symptom**: `CSV parse error: Invalid record`
1287
+
1288
+ **Cause**: Malformed CSV (unescaped quotes, inconsistent columns)
1289
+
1290
+ **Solution**:
1291
+
1292
+ ```typescript
1293
+ // Use more lenient CSV parser options
1294
+ const csvParser = CSVParserService.create({
1295
+ columns: true,
1296
+ skip_empty_lines: true,
1297
+ trim: true,
1298
+ relax_column_count: true, // Allow inconsistent column counts
1299
+ quote: '"',
1300
+ escape: '"',
1301
+ delimiter: ',',
1302
+ });
1303
+ ```
1304
+
1305
+ ### Issue 4: Batch API Timeout
1306
+
1307
+ **Symptom**: Job stays in `PENDING` state indefinitely
1308
+
1309
+ **Cause**: Fluent API processing delay or large batch size
1310
+
1311
+ **Solution**:
1312
+
1313
+ ```typescript
1314
+ // Reduce batch size
1315
+ const batchSize = 50; // Instead of 100
1316
+
1317
+ // Increase polling timeout
1318
+ const timeout = 60 * 60 * 1000; // 1 hour instead of 30 minutes
1319
+
1320
+ // Add more aggressive polling
1321
+ let delay = 1000; // Start with 1 second
1322
+ const maxDelay = 15000; // Cap at 15 seconds
1323
+ ```
1324
+
1325
+ ### Issue 5: Memory Exhaustion
1326
+
1327
+ **Symptom**: `JavaScript heap out of memory` error
1328
+
1329
+ **Cause**: Loading entire large CSV file into memory
1330
+
1331
+ **Solution**:
1332
+
1333
+ ```typescript
1334
+ // Use streaming CSV parsing instead
1335
+ for await (const record of csvParser.parseStreaming(csvContent, {}, 100)) {
1336
+ // Process records in batches of 100
1337
+ const mappedBatch = await Promise.all(record.map(r => mapper.map(r)));
1338
+
1339
+ // Send batch immediately (don't accumulate)
1340
+ await fluentClient.sendBatch(jobId, {
1341
+ entities: mappedBatch.filter(r => r.success).map(r => r.data),
1342
+ });
1343
+ }
1344
+ ```
1345
+
1346
+ ---
1347
+
1348
+ ## Related Guides
1349
+
1350
+ - **SDK Reference**: `../../readme.md` - Core SDK documentation
1351
+ - **Universal Mapping Guide**: `../../02-CORE-GUIDES/mapping/readme.md` - Field mapping patterns
1352
+ - **Batch API Guide**: `../../02-CORE-GUIDES/api-reference/modules/03-batch-processing.md` - Batch API details
1353
+ - **S3 Data Source**: `../../02-CORE-GUIDES/data-sources/readme.md` - S3 operations
1354
+ - **CSV Parser**: `../../02-CORE-GUIDES/parsers/readme.md` - CSV parsing options
1355
+ - **Error Handling**: `../../03-PATTERN-GUIDES/error-handling/readme.md` - Error patterns
1356
+
1357
+ ---
1358
+
1359
+ ## Production Checklist
1360
+
1361
+ Before deploying to production:
1362
+
1363
+ - [ ] Environment variables secured (AWS Secrets Manager, vault, etc.)
1364
+ - [ ] Logging configured (Winston, CloudWatch, Datadog, etc.)
1365
+ - [ ] Error alerting setup (PagerDuty, Slack, email)
1366
+ - [ ] Monitoring dashboard (metrics, job success rate, duration)
1367
+ - [ ] Retry logic tested (network failures, API errors)
1368
+ - [ ] Batch size optimized (tested with production file sizes)
1369
+ - [ ] Memory profiling done (no leaks)
1370
+ - [ ] S3 lifecycle policies configured (archive old files)
1371
+ - [ ] IAM permissions minimized (least privilege)
1372
+ - [ ] Health check endpoint added (for orchestration tools)
1373
+ - [ ] Documentation updated (runbook, troubleshooting)
1374
+ - [ ] Disaster recovery tested (restore from archive)
1375
+
1376
+ ---
1377
+
1378
+ **Next Steps**:
1379
+
1380
+ 1. Customize field mappings for your inventory data structure
1381
+ 2. Add custom validation rules for your business logic
1382
+ 3. Implement state tracking to prevent duplicate processing
1383
+ 4. Set up monitoring and alerting
1384
+ 5. Deploy to your preferred environment (Lambda, ECS, K8s, etc.)
1385
+
1386
+ **Support**:
1387
+
1388
+ - SDK Issues: https://github.com/fluentcommerce/fc-connect-sdk/issues
1389
+ - Fluent Commerce API: https://docs.fluentcommerce.com
1390
+ - Community: https://community.fluentcommerce.com