@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.56

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (476) hide show
  1. package/CHANGELOG.md +12 -0
  2. package/README.md +11 -0
  3. package/dist/cjs/clients/fluent-client.js +13 -6
  4. package/dist/cjs/utils/pagination-helpers.js +38 -2
  5. package/dist/cjs/versori/fluent-versori-client.js +11 -5
  6. package/dist/esm/clients/fluent-client.js +13 -6
  7. package/dist/esm/utils/pagination-helpers.js +38 -2
  8. package/dist/esm/versori/fluent-versori-client.js +11 -5
  9. package/dist/tsconfig.esm.tsbuildinfo +1 -1
  10. package/dist/tsconfig.tsbuildinfo +1 -1
  11. package/dist/tsconfig.types.tsbuildinfo +1 -1
  12. package/docs/00-START-HERE/EXPORT-VALIDATION.md +158 -158
  13. package/docs/00-START-HERE/cli-analyze-source-structure-guide.md +655 -655
  14. package/docs/00-START-HERE/cli-documentation-index.md +202 -202
  15. package/docs/00-START-HERE/cli-quick-reference.md +252 -252
  16. package/docs/00-START-HERE/decision-tree.md +552 -552
  17. package/docs/00-START-HERE/getting-started.md +1070 -1070
  18. package/docs/00-START-HERE/mapper-quick-decision-guide.md +235 -235
  19. package/docs/00-START-HERE/readme.md +237 -237
  20. package/docs/00-START-HERE/retailerid-configuration.md +404 -404
  21. package/docs/00-START-HERE/sdk-philosophy.md +794 -794
  22. package/docs/00-START-HERE/troubleshooting-quick-reference.md +1086 -1086
  23. package/docs/01-TEMPLATES/faq.md +686 -686
  24. package/docs/01-TEMPLATES/patterns/pattern-templates-guide.md +68 -68
  25. package/docs/01-TEMPLATES/patterns/patterns-csv-schema-validation-and-rejection-report.md +233 -233
  26. package/docs/01-TEMPLATES/patterns/patterns-custom-resolvers.md +407 -407
  27. package/docs/01-TEMPLATES/patterns/patterns-error-handling-retry.md +511 -511
  28. package/docs/01-TEMPLATES/patterns/patterns-field-mapping-universal.md +701 -701
  29. package/docs/01-TEMPLATES/patterns/patterns-large-file-splitting.md +1430 -1430
  30. package/docs/01-TEMPLATES/patterns/patterns-master-data-etl.md +2399 -2399
  31. package/docs/01-TEMPLATES/patterns/patterns-pagination-streaming.md +447 -447
  32. package/docs/01-TEMPLATES/patterns/patterns-state-duplicate-prevention.md +385 -385
  33. package/docs/01-TEMPLATES/readme.md +957 -957
  34. package/docs/01-TEMPLATES/standalone/standalone-asn-inbound-processing.md +1209 -1209
  35. package/docs/01-TEMPLATES/standalone/standalone-graphql-query-export.md +1140 -1140
  36. package/docs/01-TEMPLATES/standalone/standalone-graphql-to-parquet-partitioned-s3.md +432 -432
  37. package/docs/01-TEMPLATES/standalone/standalone-multi-channel-inventory-sync.md +1185 -1185
  38. package/docs/01-TEMPLATES/standalone/standalone-multi-source-aggregation.md +1462 -1462
  39. package/docs/01-TEMPLATES/standalone/standalone-s3-csv-batch-api.md +1390 -1390
  40. package/docs/01-TEMPLATES/standalone/standalone-s3-csv-inventory-to-batch.md +330 -330
  41. package/docs/01-TEMPLATES/standalone/standalone-scripts-guide.md +87 -87
  42. package/docs/01-TEMPLATES/standalone/standalone-sftp-xml-graphql.md +1444 -1444
  43. package/docs/01-TEMPLATES/standalone/standalone-webhook-payload-processing.md +688 -688
  44. package/docs/01-TEMPLATES/versori/business-examples/business-examples-dropship-order-routing.md +193 -193
  45. package/docs/01-TEMPLATES/versori/business-examples/business-examples-graphql-parquet-extraction.md +518 -518
  46. package/docs/01-TEMPLATES/versori/business-examples/business-examples-inter-location-transfers.md +2162 -2162
  47. package/docs/01-TEMPLATES/versori/business-examples/business-examples-pre-order-allocation.md +2226 -2226
  48. package/docs/01-TEMPLATES/versori/business-examples/business-scenarios-guide.md +87 -87
  49. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-connection-validation-pattern.md +656 -656
  50. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-dual-workflow-connector.md +835 -835
  51. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-guide.md +108 -108
  52. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-kv-state-management.md +1533 -1533
  53. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-xml-response-patterns.md +1160 -1160
  54. package/docs/01-TEMPLATES/versori/versori-platform-guide.md +201 -201
  55. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-asn-purchase-order.md +1906 -1906
  56. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-dropship-routing.md +1074 -1074
  57. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-flash-sale-reserve.md +1395 -1395
  58. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-generic-xml-order.md +888 -888
  59. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-payment-gateway-integration.md +2478 -2478
  60. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-rma-returns-comprehensive.md +2240 -2240
  61. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-xml-order-ingestion.md +2029 -2029
  62. package/docs/01-TEMPLATES/versori/webhooks/webhook-templates-guide.md +140 -140
  63. package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/inventory-mapping.json +20 -20
  64. package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/products_2025-01-22.csv +11 -11
  65. package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/sample-data-guide.md +34 -34
  66. package/docs/01-TEMPLATES/versori/workflows/_examples/workflow-examples-guide.md +36 -36
  67. package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-modes-guide.md +1038 -1038
  68. package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-workflows-guide.md +138 -138
  69. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/graphql-extraction-guide.md +63 -63
  70. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-csv.md +2062 -2062
  71. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-xml.md +2294 -2294
  72. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-s3-csv.md +2461 -2461
  73. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-sftp-xml.md +2529 -2529
  74. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-csv.md +2464 -2464
  75. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-json.md +1959 -1959
  76. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-s3-csv.md +1953 -1953
  77. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-sftp-xml.md +2541 -2541
  78. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-s3-json.md +2384 -2384
  79. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-sftp-xml.md +2445 -2445
  80. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-csv.md +2355 -2355
  81. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-json.md +2042 -2042
  82. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-sftp-xml.md +2726 -2726
  83. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/batch-api-guide.md +206 -206
  84. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-cycle-count-reconciliation.md +2030 -2030
  85. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-multi-channel-inventory-sync.md +1882 -1882
  86. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-csv-inventory-batch.md +2827 -2827
  87. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-json-inventory-batch.md +1952 -1952
  88. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-xml-inventory-batch.md +3289 -3289
  89. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-csv-inventory-batch.md +3064 -3064
  90. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-json-inventory-batch.md +3238 -3238
  91. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-xml-inventory-batch.md +2977 -2977
  92. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/event-api-guide.md +321 -321
  93. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-json-order-cancel-event.md +959 -959
  94. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-xml-order-cancel-event.md +1170 -1170
  95. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-csv-product-event.md +2312 -2312
  96. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-json-product-event.md +2999 -2999
  97. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-parquet-product-event.md +2836 -2836
  98. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-xml-product-event.md +2395 -2395
  99. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-csv-product-event.md +2295 -2295
  100. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-json-product-event.md +2602 -2602
  101. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-parquet-product-event.md +2589 -2589
  102. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-xml-product-event.md +3578 -3578
  103. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/graphql-mutations-guide.md +93 -93
  104. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-json-order-update-graphql.md +1260 -1260
  105. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-xml-order-update-graphql.md +1472 -1472
  106. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-control-graphql.md +2417 -2417
  107. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-location-graphql.md +2811 -2811
  108. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-price-graphql.md +2619 -2619
  109. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-json-location-graphql.md +2807 -2807
  110. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-xml-location-graphql.md +2373 -2373
  111. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-control-graphql.md +2740 -2740
  112. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-location-graphql.md +2760 -2760
  113. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-json-location-graphql.md +1710 -1710
  114. package/docs/01-TEMPLATES/versori/workflows/ingestion/ingestion-workflows-guide.md +136 -136
  115. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/rubix-webhooks-guide.md +520 -520
  116. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-inline.md +1418 -1418
  117. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-universal-mapper.md +1785 -1785
  118. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-order-attribute-update.md +824 -824
  119. package/docs/01-TEMPLATES/versori/workflows/workflows-overview-guide.md +646 -646
  120. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-batch-archival.md +724 -724
  121. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-job-tracker.md +627 -627
  122. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-partial-batch-recovery.md +561 -561
  123. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-quick-reference.md +367 -367
  124. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-readme.md +407 -407
  125. package/docs/02-CORE-GUIDES/advanced-services/readme.md +49 -49
  126. package/docs/02-CORE-GUIDES/api-reference/api-reference-quick-reference.md +548 -548
  127. package/docs/02-CORE-GUIDES/api-reference/event-api-input-output-reference.md +702 -1171
  128. package/docs/02-CORE-GUIDES/api-reference/examples/client-initialization.ts +286 -286
  129. package/docs/02-CORE-GUIDES/api-reference/graphql-error-classification.md +337 -337
  130. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-01-client-api.md +399 -520
  131. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-03-authentication.md +199 -199
  132. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-04-graphql-mapping.md +925 -925
  133. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-05-services.md +1198 -1198
  134. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-06-data-sources.md +1083 -1083
  135. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-07-parsers.md +1097 -1097
  136. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-pagination.md +513 -513
  137. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-types.md +545 -597
  138. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-error-handling.md +527 -527
  139. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-webhook-validation.md +514 -514
  140. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-extraction.md +557 -557
  141. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-utilities.md +412 -412
  142. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-cli-tools.md +423 -423
  143. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-error-handling.md +716 -716
  144. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-analyze-source-structure.md +518 -518
  145. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-partial-responses.md +212 -212
  146. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-testing.md +300 -300
  147. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-13-resolver-builder.md +322 -322
  148. package/docs/02-CORE-GUIDES/api-reference/readme.md +279 -279
  149. package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-quick-reference.md +351 -351
  150. package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-readme.md +277 -277
  151. package/docs/02-CORE-GUIDES/auto-pagination/examples/auto-pagination-readme.md +178 -178
  152. package/docs/02-CORE-GUIDES/auto-pagination/examples/common-patterns.ts +351 -351
  153. package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-products.ts +384 -384
  154. package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-virtual-positions.ts +308 -308
  155. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-01-foundations.md +470 -470
  156. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-02-quick-start.md +713 -713
  157. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-03-configuration.md +754 -754
  158. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-04-advanced-patterns.md +732 -732
  159. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-05-sdk-integration.md +847 -847
  160. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-06-troubleshooting.md +359 -359
  161. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-07-api-reference.md +462 -462
  162. package/docs/02-CORE-GUIDES/auto-pagination/readme.md +54 -54
  163. package/docs/02-CORE-GUIDES/data-sources/data-sources-file-operations-error-handling.md +1487 -1487
  164. package/docs/02-CORE-GUIDES/data-sources/data-sources-quick-reference.md +836 -836
  165. package/docs/02-CORE-GUIDES/data-sources/data-sources-readme.md +276 -276
  166. package/docs/02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md +553 -553
  167. package/docs/02-CORE-GUIDES/data-sources/examples/common-patterns.ts +409 -409
  168. package/docs/02-CORE-GUIDES/data-sources/examples/data-sources-readme.md +178 -178
  169. package/docs/02-CORE-GUIDES/data-sources/examples/s3-operations.ts +308 -308
  170. package/docs/02-CORE-GUIDES/data-sources/examples/sftp-operations.ts +371 -371
  171. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-01-foundations.md +735 -735
  172. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-02-s3-operations.md +1302 -1302
  173. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-03-sftp-operations.md +1379 -1379
  174. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-04-file-patterns.md +941 -941
  175. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-05-advanced-topics.md +813 -813
  176. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-06-integration-patterns.md +486 -486
  177. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-07-troubleshooting.md +387 -387
  178. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-08-api-reference.md +417 -417
  179. package/docs/02-CORE-GUIDES/data-sources/readme.md +77 -77
  180. package/docs/02-CORE-GUIDES/error-handling-guide.md +936 -936
  181. package/docs/02-CORE-GUIDES/extraction/examples/02-core-guides-extraction-readme.md +116 -116
  182. package/docs/02-CORE-GUIDES/extraction/examples/common-patterns.ts +428 -428
  183. package/docs/02-CORE-GUIDES/extraction/examples/extract-inventory-basic.ts +187 -187
  184. package/docs/02-CORE-GUIDES/extraction/extraction-quick-reference.md +596 -596
  185. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-01-foundations.md +514 -514
  186. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-02-basic-extraction.md +823 -823
  187. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-03-parquet-processing.md +507 -507
  188. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-04-data-enrichment.md +546 -546
  189. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-05-transformation.md +494 -494
  190. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-export-formats.md +458 -458
  191. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-performance.md +138 -138
  192. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-api-reference.md +148 -148
  193. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-optimization.md +692 -692
  194. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-08-extraction-orchestrator.md +1008 -1008
  195. package/docs/02-CORE-GUIDES/extraction/readme.md +151 -151
  196. package/docs/02-CORE-GUIDES/ingestion/examples/_simple-kv-store.ts +40 -40
  197. package/docs/02-CORE-GUIDES/ingestion/examples/error-recovery.ts +728 -728
  198. package/docs/02-CORE-GUIDES/ingestion/examples/event-driven.ts +501 -501
  199. package/docs/02-CORE-GUIDES/ingestion/examples/local-file-ingestion.ts +88 -88
  200. package/docs/02-CORE-GUIDES/ingestion/examples/parquet-ingestion.ts +117 -117
  201. package/docs/02-CORE-GUIDES/ingestion/examples/performance-optimized.ts +647 -647
  202. package/docs/02-CORE-GUIDES/ingestion/examples/s3-csv-ingestion.ts +169 -169
  203. package/docs/02-CORE-GUIDES/ingestion/examples/sftp-csv-ingestion.ts +134 -134
  204. package/docs/02-CORE-GUIDES/ingestion/ingestion-quick-reference.md +546 -546
  205. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-01-introduction.md +626 -626
  206. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-02-quick-start.md +658 -658
  207. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-03-data-sources.md +1052 -1052
  208. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-04-field-mapping.md +763 -763
  209. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-05-advanced-parsers.md +676 -676
  210. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-06-batch-api.md +1295 -1295
  211. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-api-reference.md +138 -138
  212. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-state-management.md +1037 -1037
  213. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-08-performance-optimization.md +1349 -1349
  214. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-09-best-practices.md +1893 -1893
  215. package/docs/02-CORE-GUIDES/ingestion/readme.md +160 -160
  216. package/docs/02-CORE-GUIDES/logging-guide.md +585 -585
  217. package/docs/02-CORE-GUIDES/mapping/error-handling-patterns.md +401 -401
  218. package/docs/02-CORE-GUIDES/mapping/examples/02-core-guides-mapping-readme.md +128 -128
  219. package/docs/02-CORE-GUIDES/mapping/examples/common-patterns.ts +273 -273
  220. package/docs/02-CORE-GUIDES/mapping/examples/csv-location-ingestion.json +36 -36
  221. package/docs/02-CORE-GUIDES/mapping/examples/csv-mapping.ts +242 -242
  222. package/docs/02-CORE-GUIDES/mapping/examples/graphql-to-parquet-extraction.json +36 -36
  223. package/docs/02-CORE-GUIDES/mapping/examples/json-mapping.ts +213 -213
  224. package/docs/02-CORE-GUIDES/mapping/examples/json-product-to-mutation.json +48 -48
  225. package/docs/02-CORE-GUIDES/mapping/examples/xml-mapping.ts +291 -291
  226. package/docs/02-CORE-GUIDES/mapping/examples/xml-order-to-mutation.json +45 -45
  227. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-quick-reference.md +463 -463
  228. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-readme.md +227 -227
  229. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-01-introduction.md +222 -222
  230. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-02-quick-start.md +351 -351
  231. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-03-schema-validation.md +569 -569
  232. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-04-mapping-patterns.md +471 -471
  233. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-05-configuration-reference.md +611 -611
  234. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-advanced-xpath.md +148 -148
  235. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-path-syntax.md +464 -464
  236. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-api-reference.md +94 -94
  237. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-array-handling.md +307 -307
  238. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-08-custom-resolvers.md +544 -544
  239. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-09-advanced-patterns.md +427 -427
  240. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-10-hooks-and-variables.md +336 -336
  241. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-11-error-handling.md +488 -488
  242. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-12-arguments-vs-nodes.md +383 -383
  243. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-13-best-practices.md +477 -477
  244. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/readme.md +62 -62
  245. package/docs/02-CORE-GUIDES/mapping/mapping-format-decision-tree.md +480 -480
  246. package/docs/02-CORE-GUIDES/mapping/mapping-graphql-alias-batching-guide.md +820 -820
  247. package/docs/02-CORE-GUIDES/mapping/mapping-javascript-objects.md +2369 -2369
  248. package/docs/02-CORE-GUIDES/mapping/mapping-mapper-comparison-guide.md +682 -682
  249. package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-07-api-reference.md +1327 -1327
  250. package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-08-error-handling.md +1142 -1142
  251. package/docs/02-CORE-GUIDES/mapping/modules/mapping-04-use-cases.md +891 -891
  252. package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-helpers-resolvers.md +1126 -1126
  253. package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-sdk-resolvers.md +199 -199
  254. package/docs/02-CORE-GUIDES/mapping/modules/mapping-07-api-reference.md +1319 -1319
  255. package/docs/02-CORE-GUIDES/mapping/readme.md +178 -178
  256. package/docs/02-CORE-GUIDES/mapping/resolver-registration.md +410 -410
  257. package/docs/02-CORE-GUIDES/mapping/resolvers/examples/common-patterns.ts +226 -226
  258. package/docs/02-CORE-GUIDES/mapping/resolvers/examples/custom-resolvers.ts +227 -227
  259. package/docs/02-CORE-GUIDES/mapping/resolvers/examples/sdk-resolvers-usage.ts +203 -203
  260. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-readme.md +274 -274
  261. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-api-reference.md +679 -679
  262. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-cookbook.md +826 -826
  263. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-guide.md +1330 -1330
  264. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-helpers-reference.md +1437 -1437
  265. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-parameters-reference.md +553 -553
  266. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-troubleshooting.md +854 -854
  267. package/docs/02-CORE-GUIDES/mapping/resolvers/readme.md +75 -75
  268. package/docs/02-CORE-GUIDES/parsers/examples/02-core-guides-parsers-readme.md +161 -161
  269. package/docs/02-CORE-GUIDES/parsers/examples/csv-parser-examples.ts +110 -110
  270. package/docs/02-CORE-GUIDES/parsers/examples/json-parser-examples.ts +33 -33
  271. package/docs/02-CORE-GUIDES/parsers/examples/parquet-parser-examples.ts +47 -47
  272. package/docs/02-CORE-GUIDES/parsers/examples/xml-parser-examples.ts +38 -38
  273. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-01-foundations.md +355 -355
  274. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-02-csv-parser.md +772 -772
  275. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-03-json-parser.md +789 -789
  276. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-04-xml-parser.md +857 -857
  277. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-05-parquet-parser.md +603 -603
  278. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-integration-patterns.md +702 -702
  279. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-streaming.md +121 -121
  280. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-api-reference.md +89 -89
  281. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-troubleshooting.md +727 -727
  282. package/docs/02-CORE-GUIDES/parsers/parsers-quick-reference.md +482 -482
  283. package/docs/02-CORE-GUIDES/parsers/parsers-readme.md +258 -258
  284. package/docs/02-CORE-GUIDES/parsers/readme.md +65 -65
  285. package/docs/02-CORE-GUIDES/readme.md +194 -194
  286. package/docs/02-CORE-GUIDES/webhook-validation/examples/basic-validation.ts +108 -108
  287. package/docs/02-CORE-GUIDES/webhook-validation/examples/common-patterns.ts +316 -316
  288. package/docs/02-CORE-GUIDES/webhook-validation/examples/webhook-validation-readme.md +61 -61
  289. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-01-foundations.md +440 -440
  290. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-02-quick-start.md +525 -525
  291. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-03-versori-integration.md +741 -741
  292. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-04-platform-integration.md +629 -629
  293. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-05-configuration.md +535 -535
  294. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-error-handling.md +611 -611
  295. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-troubleshooting.md +124 -124
  296. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-07-api-reference.md +511 -511
  297. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-08-rubix-webhooks.md +590 -590
  298. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-09-rubix-event-vs-http-call.md +432 -432
  299. package/docs/02-CORE-GUIDES/webhook-validation/readme.md +239 -239
  300. package/docs/02-CORE-GUIDES/webhook-validation/webhook-validation-quick-reference.md +392 -392
  301. package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-quick-reference.md +498 -498
  302. package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-readme.md +313 -313
  303. package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/common-patterns.ts +612 -612
  304. package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/connector-scenarios-readme.md +253 -253
  305. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-01-foundations.md +452 -452
  306. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-02-simple-scenarios.md +681 -681
  307. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-03-intermediate-scenarios.md +637 -637
  308. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-04-advanced-scenarios.md +650 -650
  309. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-05-bidirectional-sync.md +233 -233
  310. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-06-production-patterns.md +442 -442
  311. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-07-reference.md +445 -445
  312. package/docs/03-PATTERN-GUIDES/connector-scenarios/readme.md +31 -31
  313. package/docs/03-PATTERN-GUIDES/enterprise-integration-patterns.md +1528 -1528
  314. package/docs/03-PATTERN-GUIDES/error-handling/comprehensive-error-handling-guide.md +1437 -1437
  315. package/docs/03-PATTERN-GUIDES/error-handling/error-handling-quick-reference.md +390 -390
  316. package/docs/03-PATTERN-GUIDES/error-handling/examples/common-patterns.ts +438 -438
  317. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-01-foundations.md +362 -362
  318. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-02-error-types.md +850 -850
  319. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-03-utf8-handling.md +456 -456
  320. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-04-error-scenarios.md +658 -658
  321. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-05-calling-patterns.md +671 -671
  322. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-06-retry-strategies.md +1034 -1034
  323. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-07-monitoring.md +653 -653
  324. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-08-api-reference.md +847 -847
  325. package/docs/03-PATTERN-GUIDES/error-handling/readme.md +36 -36
  326. package/docs/03-PATTERN-GUIDES/examples/__tests__/readme.md +40 -40
  327. package/docs/03-PATTERN-GUIDES/examples/__tests__/resolver-examples.test.js +282 -282
  328. package/docs/03-PATTERN-GUIDES/examples/test-data/03-pattern-guides-readme.md +110 -110
  329. package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-inventory.json +123 -123
  330. package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-order.json +171 -171
  331. package/docs/03-PATTERN-GUIDES/examples/test-data/readme.md +28 -28
  332. package/docs/03-PATTERN-GUIDES/extraction/extraction-readme.md +15 -15
  333. package/docs/03-PATTERN-GUIDES/extraction/readme.md +25 -25
  334. package/docs/03-PATTERN-GUIDES/file-operations/examples/common-patterns.ts +407 -407
  335. package/docs/03-PATTERN-GUIDES/file-operations/examples/file-operations-readme.md +142 -142
  336. package/docs/03-PATTERN-GUIDES/file-operations/file-operations-quick-reference.md +462 -462
  337. package/docs/03-PATTERN-GUIDES/file-operations/file-operations-readme.md +379 -379
  338. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-01-foundations.md +430 -430
  339. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-02-quick-start.md +484 -484
  340. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-03-s3-operations.md +507 -507
  341. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-04-sftp-operations.md +963 -963
  342. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-05-streaming-performance.md +503 -503
  343. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-archive-patterns.md +386 -386
  344. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-error-handling.md +117 -117
  345. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-api-reference.md +78 -78
  346. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-testing-troubleshooting.md +567 -567
  347. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-08-api-reference.md +1055 -1055
  348. package/docs/03-PATTERN-GUIDES/file-operations/readme.md +32 -32
  349. package/docs/03-PATTERN-GUIDES/ingestion/ingestion-readme.md +15 -15
  350. package/docs/03-PATTERN-GUIDES/ingestion/readme.md +25 -25
  351. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/batch-processing.ts +130 -130
  352. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/common-patterns.ts +360 -360
  353. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/delta-sync.ts +130 -130
  354. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/integration-patterns-readme.md +100 -100
  355. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/real-time-webhook.ts +398 -398
  356. package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-quick-reference.md +962 -962
  357. package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-readme.md +134 -134
  358. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-01-real-time-processing.md +991 -991
  359. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-02-batch-processing.md +1547 -1547
  360. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-03-delta-sync.md +1108 -1108
  361. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-04-webhook-patterns.md +1181 -1181
  362. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-05-error-handling.md +1061 -1061
  363. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-advanced-integration-services.md +1547 -1547
  364. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-performance.md +109 -109
  365. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-07-api-reference.md +34 -34
  366. package/docs/03-PATTERN-GUIDES/integration-patterns/readme.md +30 -30
  367. package/docs/03-PATTERN-GUIDES/logging-minimal-mode.md +128 -128
  368. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/common-patterns.ts +380 -380
  369. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/multiple-connections-readme.md +139 -139
  370. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/parallel-root-connections.ts +149 -149
  371. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/real-world-scenarios.ts +405 -405
  372. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-01-foundations.md +378 -378
  373. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-02-quick-start.md +566 -566
  374. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-03-targeting-connections.md +659 -659
  375. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-04-parallel-queries.md +656 -656
  376. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-05-best-practices.md +624 -624
  377. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-api-reference.md +824 -824
  378. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-versori.md +119 -119
  379. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-07-api-reference.md +87 -87
  380. package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-quick-reference.md +353 -353
  381. package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-readme.md +270 -270
  382. package/docs/03-PATTERN-GUIDES/multiple-connections/readme.md +30 -30
  383. package/docs/03-PATTERN-GUIDES/pagination/pagination-readme.md +14 -14
  384. package/docs/03-PATTERN-GUIDES/pagination/readme.md +24 -24
  385. package/docs/03-PATTERN-GUIDES/parquet/examples/common-patterns.ts +180 -180
  386. package/docs/03-PATTERN-GUIDES/parquet/examples/read-parquet.ts +48 -48
  387. package/docs/03-PATTERN-GUIDES/parquet/examples/write-parquet.ts +65 -65
  388. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-01-introduction.md +393 -393
  389. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-02-quick-start.md +572 -572
  390. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-03-reading-parquet.md +525 -525
  391. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-04-writing-parquet.md +554 -554
  392. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-05-graphql-extraction.md +405 -405
  393. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-performance.md +104 -104
  394. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-s3-integration.md +511 -511
  395. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-api-reference.md +90 -90
  396. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-performance-optimization.md +525 -525
  397. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-08-best-practices.md +712 -712
  398. package/docs/03-PATTERN-GUIDES/parquet/parquet-quick-reference.md +683 -683
  399. package/docs/03-PATTERN-GUIDES/parquet/parquet-readme.md +248 -248
  400. package/docs/03-PATTERN-GUIDES/parquet/readme.md +32 -32
  401. package/docs/03-PATTERN-GUIDES/parsers/parsers-readme.md +12 -12
  402. package/docs/03-PATTERN-GUIDES/parsers/readme.md +24 -24
  403. package/docs/03-PATTERN-GUIDES/readme.md +159 -159
  404. package/docs/03-PATTERN-GUIDES/webhooks/readme.md +24 -24
  405. package/docs/03-PATTERN-GUIDES/webhooks/webhooks-readme.md +8 -8
  406. package/docs/04-REFERENCE/architecture/architecture-01-overview.md +427 -427
  407. package/docs/04-REFERENCE/architecture/architecture-02-client-architecture.md +424 -424
  408. package/docs/04-REFERENCE/architecture/architecture-03-data-flow.md +690 -690
  409. package/docs/04-REFERENCE/architecture/architecture-04-service-layer.md +834 -834
  410. package/docs/04-REFERENCE/architecture/architecture-05-integration-architecture.md +655 -655
  411. package/docs/04-REFERENCE/architecture/architecture-06-state-management.md +653 -653
  412. package/docs/04-REFERENCE/architecture/architecture-adding-new-data-sources.md +686 -686
  413. package/docs/04-REFERENCE/architecture/readme.md +279 -279
  414. package/docs/04-REFERENCE/platforms/deno/readme.md +117 -117
  415. package/docs/04-REFERENCE/platforms/nodejs/readme.md +146 -146
  416. package/docs/04-REFERENCE/platforms/readme.md +135 -135
  417. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-01-introduction.md +398 -398
  418. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-02-quick-start.md +560 -560
  419. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-03-authentication.md +757 -757
  420. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-04-workflows.md +2476 -2476
  421. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-05-connections.md +1167 -1167
  422. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-kv-storage.md +990 -990
  423. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-state-management.md +121 -121
  424. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-api-reference.md +68 -68
  425. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-deployment.md +731 -731
  426. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-08-best-practices.md +1111 -1111
  427. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-09-signature-reference.md +766 -766
  428. package/docs/04-REFERENCE/platforms/versori/platforms-versori-readme.md +299 -299
  429. package/docs/04-REFERENCE/platforms/versori/platforms-versori-s3-sftp-configuration-guide.md +1425 -1425
  430. package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-api-key-security.md +816 -816
  431. package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-connection-security.md +681 -681
  432. package/docs/04-REFERENCE/platforms/versori/platforms-versori-workflow-task-types.md +708 -708
  433. package/docs/04-REFERENCE/platforms/versori/readme.md +108 -108
  434. package/docs/04-REFERENCE/readme.md +148 -148
  435. package/docs/04-REFERENCE/resolver-signature/examples/advanced-resolvers.ts +482 -482
  436. package/docs/04-REFERENCE/resolver-signature/examples/async-resolvers.ts +496 -496
  437. package/docs/04-REFERENCE/resolver-signature/examples/basic-resolvers.ts +343 -343
  438. package/docs/04-REFERENCE/resolver-signature/examples/resolver-signature-readme.md +188 -188
  439. package/docs/04-REFERENCE/resolver-signature/examples/testing-resolvers.ts +463 -463
  440. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-01-foundations.md +286 -286
  441. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-02-parameter-reference.md +643 -643
  442. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-03-basic-examples.md +521 -521
  443. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-04-advanced-patterns.md +739 -739
  444. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-05-sdk-resolvers.md +531 -531
  445. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-migration-guide.md +650 -650
  446. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-testing.md +125 -125
  447. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-07-api-reference.md +794 -794
  448. package/docs/04-REFERENCE/resolver-signature/readme.md +64 -64
  449. package/docs/04-REFERENCE/resolver-signature/resolver-signature-quick-reference.md +270 -270
  450. package/docs/04-REFERENCE/resolver-signature/resolver-signature-readme.md +351 -351
  451. package/docs/04-REFERENCE/schema/fluent-commerce-schema.json +764 -764
  452. package/docs/04-REFERENCE/schema/readme.md +141 -141
  453. package/docs/04-REFERENCE/testing/examples/04-reference-testing-readme.md +158 -158
  454. package/docs/04-REFERENCE/testing/examples/fluent-testing.ts +62 -62
  455. package/docs/04-REFERENCE/testing/examples/health-check.ts +155 -155
  456. package/docs/04-REFERENCE/testing/examples/integration-test.ts +119 -119
  457. package/docs/04-REFERENCE/testing/examples/performance-test.ts +183 -183
  458. package/docs/04-REFERENCE/testing/examples/s3-testing.ts +127 -127
  459. package/docs/04-REFERENCE/testing/modules/04-reference-testing-01-foundations.md +267 -267
  460. package/docs/04-REFERENCE/testing/modules/04-reference-testing-02-s3-testing.md +599 -599
  461. package/docs/04-REFERENCE/testing/modules/04-reference-testing-03-fluent-testing.md +589 -589
  462. package/docs/04-REFERENCE/testing/modules/04-reference-testing-04-integration-testing.md +699 -699
  463. package/docs/04-REFERENCE/testing/modules/04-reference-testing-05-debugging.md +478 -478
  464. package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-cicd-integration.md +463 -463
  465. package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-preflight-validation.md +131 -131
  466. package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-best-practices.md +499 -499
  467. package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-coverage-ci.md +165 -165
  468. package/docs/04-REFERENCE/testing/modules/04-reference-testing-08-api-reference.md +634 -634
  469. package/docs/04-REFERENCE/testing/readme.md +86 -86
  470. package/docs/04-REFERENCE/testing/testing-quick-reference.md +667 -667
  471. package/docs/04-REFERENCE/testing/testing-readme.md +286 -286
  472. package/docs/04-REFERENCE/troubleshooting/readme.md +144 -144
  473. package/docs/04-REFERENCE/troubleshooting/troubleshooting-deno-sftp-compatibility.md +392 -392
  474. package/docs/template-loading-matrix.md +242 -242
  475. package/package.json +5 -3
  476. package/docs/02-CORE-GUIDES/api-reference/cli-profile-integration.md +0 -377
@@ -1,2464 +1,2464 @@
1
- ---
2
- template_id: tpl-extract-inventory-quantities-to-s3-csv
3
- canonical_filename: template-extraction-inventory-quantities-to-s3-csv.md
4
- version: 2.0.0
5
- sdk_version: ^0.1.39
6
- runtime: versori
7
- direction: extraction
8
- source: fluent-graphql
9
- destination: s3-csv
10
- entity: inventoryQuantities
11
- format: csv
12
- logging: versori
13
- status: stable
14
- features:
15
- - memory-management
16
- - enhanced-logging
17
- - pagination-progress
18
- ---
19
-
20
- # Template: Extraction - Inventory Quantities to S3 CSV
21
-
22
- **Template Version:** 2.0.0
23
- **SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
24
- **Last Updated:** 2025-01-24
25
- **Deployment Target:** Versori Platform
26
-
27
- **🆕 Version 2.0.0 Enhancements:**
28
- - ✅ **Memory Management** - Clear large result sets after processing batches
29
- - ✅ **Enhanced Logging** - Pagination progress tracking with emoji indicators (📊, 📥, ✅)
30
- - ✅ **Pagination Progress** - Real-time page-by-page progress logging with metrics
31
-
32
- ---
33
-
34
- ## 📚 STEP 1: Load These Docs (Human Checklist)
35
-
36
- 1. REQUIRED (load all)
37
- - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
38
- - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
39
- - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
40
- - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
41
- - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
42
- - [ ] fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
43
-
44
- Copy-paste list (open these):
45
- fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
46
- fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
47
- fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
48
- fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
49
- fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
50
- fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
51
-
52
- ---
53
-
54
- ## 📋 STEP 2: Tell Your AI (Prompt)
55
-
56
- Copy/paste this prompt into your AI tool after loading the documentation above:
57
-
58
- ```
59
- I need a Versori scheduled extractor that:
60
-
61
- 1) Queries Fluent Commerce GraphQL for inventoryQuantities with auto-pagination
62
- 2) Uses incremental mode with a 60-second overlap buffer stored in Versori KV
63
- 3) Transforms results using UniversalMapper per mapping JSON
64
- 4) Generates CSV with CSVParserService and uploads to S3
65
- 5) Uses native Versori log (LoggingService removed - use native log)
66
-
67
- Use the loaded docs for SDK specifics and best practices. Keep structure identical to the template.
68
- ```
69
-
70
- ---
71
-
72
- ## 📦 SDK Imports (Verified - Versori Optimized)
73
-
74
- ```typescript
75
- import { Buffer } from 'node:buffer';
76
- import {
77
- createClient,
78
- UniversalMapper,
79
- S3DataSource,
80
- CSVParserService,
81
- } from '@fluentcommerce/fc-connect-sdk';
82
-
83
- import { schedule, http } from '@versori/run';
84
- ```
85
-
86
- ---
87
-
88
- # Versori Scheduled: Inventory Quantities Extraction to S3 CSV (Configurable)
89
-
90
- **FC Connect SDK Use Case Guide**
91
-
92
- > SDK: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
93
- > Version: `npm install @fluentcommerce/fc-connect-sdk@^0.1.39`
94
-
95
- Context: Scheduled Versori workflow that extracts inventory quantities (detailed quantity records) from Fluent Commerce via GraphQL query with **configurable extraction modes**, transforms with `UniversalMapper`, and writes CSV files to S3 for analytics and reporting.
96
-
97
- **Pattern**: EXTRACTION (Fluent → S3 CSV)
98
- **Entity**: inventoryQuantities
99
- **Complexity**: High | Runtime: Versori Platform (Scheduled)
100
-
101
- ---
102
-
103
- ## ⚠️ IMPORTANT: Sample Code for SDK Demonstration Only
104
-
105
- > **🔴 PRODUCTION WARNING**
106
- >
107
- > This guide demonstrates FC Connect SDK capabilities for **extraction and mapping workflows**. The multiple extraction modes (incremental, dateRange, historical) are included to show SDK flexibility and serve as **reference examples**.
108
- >
109
- > **✅ PRODUCTION RECOMMENDATION:**
110
- >
111
- > - **ONLY use INCREMENTAL mode with scheduled runs** (e.g., daily/hourly)
112
- > - Incremental mode is safe, efficient, and production-ready
113
- > - Uses overlap buffer to prevent missed records
114
- > - Natural rate limiting via timestamps
115
- >
116
- > **🚫 DO NOT USE IN PRODUCTION:**
117
- >
118
- > - **dateRange mode** - High risk of platform overload with large date windows
119
- > - **historical mode** - Extremely dangerous, can fetch millions of records
120
- > - These modes are **demonstration only** to show SDK query patterns
121
- > - Using these modes on large inventory datasets can crash your runtime and impact platform stability
122
- >
123
- > **📝 If you need historical data:**
124
- >
125
- > - Run multiple small incremental extractions (e.g., daily for past 30 days)
126
- > - Use one-time migration scripts with proper monitoring (not scheduled workflows)
127
- > - Always validate date ranges and implement file splitting
128
- > - Get explicit approval before running large extractions
129
- >
130
- > **This sample code shows HOW to use the SDK - not WHAT to use in production.**
131
-
132
- ---
133
-
134
- ## What You'll Build
135
-
136
- - **Three extraction modes**: Incremental, Date Range, or Historical
137
- - **State management** with VersoriKVAdapter to track last successful run
138
- - GraphQL query with auto-pagination
139
- - UniversalMapper transformation for reporting schema
140
- - CSV file generation with CSVParserService
141
- - S3 upload to analytics system
142
- - **Failure recovery** with timestamp tracking
143
-
144
- ## Business Use Cases
145
-
146
- **1. Incremental Daily Sync (Analytics)**
147
-
148
- - Extract only changed inventory quantities since last run
149
- - Run daily at 2 AM
150
- - Minimize data transfer
151
- - Track changes over time
152
-
153
- **2. Date Range Extract (Audit)**
154
-
155
- - Extract quantity changes within specific date window
156
- - For audits, reconciliation, historical analysis
157
- - Example: "Show all quantity changes between Jan 1-15"
158
-
159
- **3. Historical Backfill**
160
-
161
- - Extract all quantities created within date range
162
- - For initial data warehouse load
163
- - One-time backfill operation
164
-
165
- ## Inventory Quantities vs Positions
166
-
167
- **InventoryQuantity** = Specific quantity record (retailer-defined types)
168
-
169
- - Individual records: e.g., LAST_ON_HAND, RESERVED, DELTA, SALE, CORRECTION (plus any custom IQ types)
170
- - Multiple quantities per product/location
171
- - Fields: locationRef, skuRef, qty, type, status, expectedOn (if applicable)
172
- - Used for: Detailed tracking, audit trails
173
-
174
- **InventoryPosition** = Aggregated on-hand calculation
175
-
176
- - One position per product/location
177
- - Calculated `onHand` from all associated quantities
178
- - Used for: Stock availability, reporting
179
-
180
- ## SDK Methods Used
181
-
182
- ```typescript
183
- import { Buffer } from 'node:buffer';
184
- import {
185
- createClient,
186
- UniversalMapper,
187
- S3DataSource,
188
- VersoriKVAdapter,
189
- CSVParserService,
190
- } from '@fluentcommerce/fc-connect-sdk';
191
-
192
- await createClient(ctx);
193
- await client.graphql({ query, variables, pagination });
194
- new VersoriKVAdapter(ctx.openKv(':project:'));
195
- new UniversalMapper(exportMapping);
196
- const csvParser = new CSVParserService({ includeHeaders: true });
197
- const csvContent = await csvParser.stringify(rows);
198
- await s3.uploadFile(key, Buffer.from(csvContent, 'utf8'), options);
199
- ```
200
-
201
- ## Activation Variables
202
-
203
- ```json
204
- {
205
- "retailerId": "your-retailer-id",
206
- "s3BucketName": "inventory-audit-exports",
207
- "awsAccessKeyId": "AKIAXXXXXXXXXXXX",
208
- "awsSecretAccessKey": "********",
209
- "awsRegion": "us-east-1",
210
- "s3Prefix": "inventory-quantities/daily/",
211
- "fileNamePrefix": "inventoryquantities",
212
- "catalogueRef": "DEFAULT_CATALOGUE",
213
- "pageSize": 200,
214
- "maxRecords": 100000,
215
- "extractionMode": "incremental",
216
- "fallbackStartDate": "2024-01-01T00:00:00Z",
217
- "overlapBufferSeconds": "60",
218
- "startDate": "",
219
- "endDate": ""
220
- }
221
- ```
222
-
223
- ### Variable Reference
224
-
225
- | Variable | Type | Required | Default | Description |
226
- |----------|------|----------|---------|-------------|
227
- | `retailerId` | string | Yes | - | Fluent Commerce retailer ID |
228
- | `s3BucketName` | string | Yes | - | S3 bucket for CSV export |
229
- | `awsAccessKeyId` | string | Yes | - | AWS access key with S3 write permissions |
230
- | `awsSecretAccessKey` | string | Yes | - | AWS secret access key |
231
- | `awsRegion` | string | Yes | - | AWS region (e.g., `us-east-1`) |
232
- | `s3Prefix` | string | No | `""` | S3 key prefix (e.g., `inventory-quantities/daily/`) |
233
- | `fileNamePrefix` | string | No | `"inventoryquantities"` | CSV filename prefix |
234
- | `catalogueRef` | string | No | - | Filter by catalogue reference (optional) |
235
- | `pageSize` | number | No | `200` | GraphQL page size (max 500) |
236
- | `maxRecords` | number | No | `100000` | Maximum records per extraction |
237
- | `extractionMode` | string | No | `"incremental"` | Extraction mode: `incremental`, `dateRange`, or `historical` |
238
- | `fallbackStartDate` | string | No | `"2024-01-01T00:00:00Z"` | Fallback date if no state exists |
239
- | `overlapBufferSeconds` | number | No | `60` | Overlap buffer to prevent missed records (seconds) |
240
- | `startDate` | string | No | - | Manual start date (for `dateRange`/`historical` modes) |
241
- | `endDate` | string | No | - | Manual end date (for `dateRange`/`historical` modes) |
242
-
243
- ### Extraction Mode Configuration
244
-
245
- **Mode 1: Incremental (default)**
246
-
247
- ```json
248
- {
249
- "extractionMode": "incremental",
250
- "fallbackStartDate": "2024-01-01T00:00:00Z"
251
- }
252
- ```
253
-
254
- Extracts quantities with `updatedOn > lastRunTime`. Ideal for daily syncs.
255
-
256
- **Mode 2: Date Range**
257
-
258
- ```json
259
- {
260
- "extractionMode": "dateRange",
261
- "startDate": "2025-01-01T00:00:00Z",
262
- "endDate": "2025-01-15T23:59:59Z"
263
- }
264
- ```
265
-
266
- Extracts quantities updated between `startDate` and `endDate`. Ideal for audits.
267
-
268
- **Mode 3: Historical**
269
-
270
- ```json
271
- {
272
- "extractionMode": "historical",
273
- "startDate": "2024-01-01T00:00:00Z",
274
- "endDate": "2024-12-31T23:59:59Z"
275
- }
276
- ```
277
-
278
- Extracts quantities created between `startDate` and `endDate` using `createdOn` filter.
279
-
280
- ## ⚠️ Production Safety & Guardrails
281
-
282
- ### Critical: Extraction Mode Selection
283
-
284
- **🟢 RECOMMENDED: Incremental Mode (Production)**
285
-
286
- - Safe for automated schedules
287
- - Natural rate limiting via timestamps
288
- - Predictable resource usage
289
- - **Use this for all production workflows**
290
-
291
- **🟡 CAUTION: Date Range Mode (Audit/Backfill)**
292
-
293
- - **Maximum 30-day window enforced**
294
- - Use for specific audit requests only
295
- - Run during off-peak hours
296
- - Monitor resource usage
297
-
298
- **🔴 DANGER: Historical Mode (One-Time Only)**
299
-
300
- - **Maximum 90-day window enforced**
301
- - **Requires explicit approval**
302
- - **Risk of platform overload**
303
- - Can fetch millions of records
304
- - Use multiple small incremental runs instead
305
- - Only for initial data migration
306
-
307
- ### Date Range Validation (Required)
308
-
309
- ```typescript
310
- // Validate date range limits to prevent platform overload
311
- function validateDateRange(mode, startDate, endDate) {
312
- if (mode === 'incremental') return { valid: true };
313
-
314
- if (!startDate || !endDate) {
315
- return {
316
- valid: false,
317
- error: `${mode} mode requires both startDate and endDate`,
318
- };
319
- }
320
-
321
- const start = new Date(startDate);
322
- const end = new Date(endDate);
323
- const daysDiff = (end - start) / (1000 * 60 * 60 * 24);
324
-
325
- // Guardrail: Maximum date ranges
326
- const maxDays = mode === 'dateRange' ? 30 : 90;
327
-
328
- if (daysDiff > maxDays) {
329
- return {
330
- valid: false,
331
- error: `${mode} mode: Maximum ${maxDays}-day window. Requested: ${Math.round(daysDiff)} days. Use multiple smaller extractions or incremental mode.`,
332
- recommendation: `Split into ${Math.ceil(daysDiff / maxDays)} separate extractions of ${maxDays} days each.`,
333
- };
334
- }
335
-
336
- if (daysDiff < 0) {
337
- return { valid: false, error: 'endDate must be after startDate' };
338
- }
339
-
340
- return { valid: true };
341
- }
342
- ```
343
-
344
- ### File Splitting Configuration
345
-
346
- Large extractions must split into multiple files to prevent memory issues and upload failures.
347
-
348
- ```json
349
- {
350
- "maxRecordsPerFile": 50000,
351
- "maxFileSizeMB": 100,
352
- "enableFileSplitting": true
353
- }
354
- ```
355
-
356
- **File Naming Pattern:**
357
-
358
- ```
359
- {prefix}inventory-quantities-{timestamp}-part-{sequence}.csv
360
-
361
- Examples:
362
- inventory-quantities/daily/inventory-quantities-2025-01-22T14-30-00Z-part-001.csv
363
- inventory-quantities/daily/inventory-quantities-2025-01-22T14-30-00Z-part-002.csv
364
- inventory-quantities/daily/inventory-quantities-2025-01-22T14-30-00Z-manifest.json
365
- ```
366
-
367
- **Manifest File (auto-generated):**
368
-
369
- ```json
370
- {
371
- "extractionId": "inventory-quantities-2025-01-22T14-30-00Z",
372
- "totalRecords": 127543,
373
- "totalFiles": 3,
374
- "files": [
375
- {
376
- "filename": "inventory-quantities-2025-01-22T14-30-00Z-part-001.csv",
377
- "recordCount": 50000,
378
- "s3Key": "inventory-quantities/daily/inventory-quantities-2025-01-22T14-30-00Z-part-001.csv"
379
- },
380
- {
381
- "filename": "inventory-quantities-2025-01-22T14-30-00Z-part-002.csv",
382
- "recordCount": 50000,
383
- "s3Key": "inventory-quantities/daily/inventory-quantities-2025-01-22T14-30-00Z-part-002.csv"
384
- },
385
- {
386
- "filename": "inventory-quantities-2025-01-22T14-30-00Z-part-003.csv",
387
- "recordCount": 27543,
388
- "s3Key": "inventory-quantities/daily/inventory-quantities-2025-01-22T14-30-00Z-part-003.csv"
389
- }
390
- ],
391
- "extractionMode": "dateRange",
392
- "dateRange": {
393
- "from": "2025-01-01T00:00:00Z",
394
- "to": "2025-01-31T23:59:59Z"
395
- },
396
- "completedAt": "2025-01-22T14:35:27Z"
397
- }
398
- ```
399
-
400
- ### Hard Limits (Enforced)
401
-
402
- ```typescript
403
- const SAFETY_LIMITS = {
404
- // Maximum records per single extraction
405
- MAX_RECORDS_TOTAL: 500000, // 500k hard limit
406
-
407
- // Maximum records per file before splitting
408
- MAX_RECORDS_PER_FILE: 50000, // 50k per file
409
-
410
- // Maximum file size before splitting
411
- MAX_FILE_SIZE_MB: 100, // 100MB per file
412
-
413
- // Date range limits
414
- MAX_DATE_RANGE_DAYS: 30, // dateRange mode
415
- MAX_HISTORICAL_DAYS: 90, // historical mode
416
-
417
- // Pagination limits
418
- MAX_PAGE_SIZE: 500, // Fluent API limit
419
- RECOMMENDED_PAGE_SIZE: 200, // Balance throughput/memory
420
-
421
- // Memory management
422
- CHUNK_SIZE: 10000, // Process in chunks
423
- };
424
- ```
425
-
426
- ### Memory-Safe Implementation Pattern
427
-
428
- ```typescript
429
- // Process large extractions in chunks to prevent OOM
430
- async function processLargeExtraction(edges, mapper, csvParser, s3, options) {
431
- const CHUNK_SIZE = 10000;
432
- const MAX_RECORDS_PER_FILE = options.maxRecordsPerFile || 50000;
433
-
434
- let fileSequence = 1;
435
- let currentFileRecords = [];
436
- const manifestFiles = [];
437
-
438
- for (let i = 0; i < edges.length; i += CHUNK_SIZE) {
439
- const chunk = edges.slice(i, i + CHUNK_SIZE);
440
-
441
- // Bulk mapping for chunk
442
- const chunkNodes = chunk.map(edge => edge.node);
443
- const mappingResult = await mapper.map(chunkNodes);
444
-
445
- if (!mappingResult.success) {
446
- const mappingErrors = mappingResult.errors || ['Unknown mapping failure'];
447
- log.error('Chunk mapping failed', {
448
- chunkIndex: i / CHUNK_SIZE,
449
- errorCount: mappingErrors.length,
450
- sampleErrors: mappingErrors.slice(0, 3),
451
- });
452
- throw new Error(`Mapping failed: ${mappingErrors[0] || 'Unknown error'}`);
453
- }
454
-
455
- const transformedChunk = mappingResult.data || [];
456
- const mappingErrors = mappingResult.errors || [];
457
-
458
- if (mappingErrors.length > 0) {
459
- log.warn('Some records in chunk failed transformation', {
460
- chunkIndex: i / CHUNK_SIZE,
461
- errorCount: mappingErrors.length,
462
- });
463
- }
464
-
465
- if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
466
- log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
467
- chunkIndex: i / CHUNK_SIZE,
468
- skippedFields: mappingResult.skippedFields,
469
- note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
470
- });
471
- }
472
-
473
- // Add to current file, handling splits
474
- for (const record of transformedChunk) {
475
- currentFileRecords.push(record);
476
-
477
- // Split file when limit reached
478
- if (currentFileRecords.length >= MAX_RECORDS_PER_FILE) {
479
- const fileInfo = await writeFileToS3(
480
- currentFileRecords,
481
- fileSequence++,
482
- csvParser,
483
- s3,
484
- options
485
- );
486
- manifestFiles.push(fileInfo);
487
- currentFileRecords = []; // Reset for next file
488
- }
489
- }
490
- }
491
-
492
- // Write remaining records
493
- if (currentFileRecords.length > 0) {
494
- const fileInfo = await writeFileToS3(
495
- currentFileRecords,
496
- fileSequence++,
497
- csvParser,
498
- s3,
499
- options
500
- );
501
- manifestFiles.push(fileInfo);
502
- }
503
-
504
- // Write manifest
505
- await writeManifest(manifestFiles, s3, options);
506
-
507
- return manifestFiles;
508
- }
509
- ```
510
-
511
- ### Enterprise Time Buffer Configuration
512
-
513
- ```json
514
- {
515
- "overlapBufferSeconds": "60"
516
- }
517
- ```
518
-
519
- **Default: 60 seconds (recommended for most deployments)**
520
-
521
- **Purpose**: Prevents missed records due to:
522
-
523
- - **Clock skew** between Fluent API servers (typically 1-5 seconds)
524
- - **Transaction timing** - records updated during query execution
525
- - **Race conditions** - records updated between extraction runs
526
-
527
- **How It Works**:
528
-
529
- - **Query**: Uses `updatedOn >= (lastRunTime - 60 seconds)`
530
- - **Save**: Stores `MAX(updatedOn)` WITHOUT buffer
531
- - **Result**: Records from the last minute of previous extraction are included again
532
-
533
- **Buffer Sizes by Deployment**:
534
-
535
- - `30` - Low-latency single-region (minimal clock skew expected)
536
- - `60` - **Standard production** (recommended default)
537
- - `300` - Cross-region deployments or high-latency networks
538
-
539
- **Duplicate Handling**: Downstream systems should upsert by `quantity_id` (idempotent). Duplicates are safe and expected.
540
-
541
- ### Timezone Handling
542
-
543
- **All timestamps are in ISO 8601 format (UTC)**:
544
-
545
- ```typescript
546
- // Input: ISO 8601 UTC timestamp
547
- const timestamp = '2025-01-22T14:30:00.000Z';
548
-
549
- // JavaScript Date operations preserve UTC
550
- new Date(timestamp).toISOString();
551
- // Returns: "2025-01-22T14:30:00.000Z" (same format)
552
-
553
- new Date(timestamp).getTime();
554
- // Returns: 1737558600000 (UTC epoch milliseconds)
555
-
556
- // Subtract 60 seconds for buffer
557
- const buffered = new Date(new Date(timestamp).getTime() - 60000).toISOString();
558
- // Returns: "2025-01-22T14:29:00.000Z"
559
- ```
560
-
561
- **Key Points**:
562
-
563
- - Fluent API returns all timestamps in UTC
564
- - `.getTime()` returns UTC epoch milliseconds
565
- - Buffer arithmetic is done in milliseconds
566
- - `.toISOString()` converts back to ISO 8601 UTC
567
- - No timezone conversion needed
568
-
569
- ## Export Mapping Configuration
570
-
571
- Create file: `./config/inventory-quantities.export.json`
572
-
573
- ```json
574
- {
575
- "name": "inventory-quantities.export",
576
- "version": "1.0.0",
577
- "description": "Fluent Inventory Quantities → CSV Export Mapping",
578
- "fields": {
579
- "quantity_id": { "source": "id", "required": true, "resolver": "sdk.trim" },
580
- "quantity_ref": { "source": "ref", "required": true, "resolver": "sdk.trim" },
581
- "catalogue_ref": { "source": "catalogue.ref", "required": false, "resolver": "sdk.trim" },
582
- "catalogue_name": { "source": "catalogue.name", "required": false, "resolver": "sdk.trim" },
583
- "location": { "source": "locationRef", "required": true, "resolver": "sdk.trim" },
584
- "sku": { "source": "skuRef", "required": true, "resolver": "sdk.trim" },
585
- "quantity": { "source": "qty", "required": true, "resolver": "sdk.parseInt" },
586
- "type": { "source": "type", "required": true, "resolver": "sdk.uppercase" },
587
- "status": { "source": "status", "required": true, "resolver": "sdk.uppercase" },
588
- "expected_on": { "source": "expectedOn", "resolver": "sdk.formatDate" },
589
- "created_on": { "source": "createdOn", "resolver": "sdk.formatDate" },
590
- "updated_on": { "source": "updatedOn", "required": true, "resolver": "sdk.formatDate" }
591
- }
592
- }
593
- ```
594
-
595
- ## Mapping & Resolvers Explained
596
-
597
- This section explains how the SDK transforms raw GraphQL data into your CSV export format using **UniversalMapper** and **SDK resolvers**.
598
-
599
- ### SDK Resolvers Used
600
-
601
- | Field | Resolver | Why? | Example Transformation |
602
- | ---------------- | ---------------- | ------------------------------------------ | ----------------------------------------------- |
603
- | `quantity_id` | `sdk.trim` | Clean quantity IDs from whitespace | `" Q001 "` → `"Q001"` |
604
- | `quantity_ref` | `sdk.trim` | Clean quantity references | `" QTY-REF-001 "` → `"QTY-REF-001"` |
605
- | `catalogue_ref` | `sdk.trim` | Clean catalogue references | `" DEFAULT_CATALOGUE "` → `"DEFAULT_CATALOGUE"` |
606
- | `catalogue_name` | `sdk.trim` | Clean catalogue names | `" Default Catalogue "` → `"Default Catalogue"` |
607
- | `location` | `sdk.trim` | Clean location references | `" DC01 "` → `"DC01"` |
608
- | `sku` | `sdk.trim` | Clean SKU references | `" SKU-001 "` → `"SKU-001"` |
609
- | `quantity` | `sdk.parseInt` | Parse quantity as integer for calculations | `"100"` → `100` |
610
- | `type` | `sdk.uppercase` | Normalize type codes | `"available"` → `"AVAILABLE"` |
611
- | `status` | `sdk.uppercase` | Normalize status codes | `"active"` → `"ACTIVE"` |
612
- | `expected_on` | `sdk.formatDate` | Format dates for CSV export | `"2025-01-30T00:00:00.000Z"` → `"2025-01-30"` |
613
- | `created_on` | `sdk.formatDate` | Format created timestamps | `"2025-01-15T10:00:00.000Z"` → `"2025-01-15"` |
614
- | `updated_on` | `sdk.formatDate` | Format updated timestamps for tracking | `"2025-01-22T08:30:00.000Z"` → `"2025-01-22"` |
615
-
616
- ### Transformation Flow
617
-
618
- ```typescript
619
- // 1. GraphQL Response (raw data from Fluent Commerce)
620
- const rawQuantity = {
621
- id: ' Q001 ',
622
- ref: ' QTY-REF-001 ',
623
- locationRef: ' DC01 ',
624
- skuRef: ' SKU-001 ',
625
- qty: '100',
626
- type: 'available',
627
- status: 'active',
628
- expectedOn: null,
629
- createdOn: '2025-01-15T10:00:00.000Z',
630
- updatedOn: '2025-01-22T08:30:00.000Z',
631
- catalogue: {
632
- ref: ' DEFAULT_CATALOGUE ',
633
- name: ' Default Catalogue ',
634
- },
635
- };
636
-
637
- // 2. UniversalMapper applies SDK resolvers
638
- const mapper = new UniversalMapper(inventoryQuantitiesExportMapping);
639
- const result = await mapper.map(rawQuantity);
640
-
641
- // 3. Transformed Output (clean, normalized for CSV)
642
- const transformedQuantity = {
643
- quantity_id: 'Q001',
644
- quantity_ref: 'QTY-REF-001',
645
- catalogue_ref: 'DEFAULT_CATALOGUE',
646
- catalogue_name: 'Default Catalogue',
647
- location: 'DC01',
648
- sku: 'SKU-001',
649
- quantity: 100,
650
- type: 'AVAILABLE',
651
- status: 'ACTIVE',
652
- expected_on: '', // null → empty string
653
- created_on: '2025-01-15',
654
- updated_on: '2025-01-22',
655
- };
656
- ```
657
-
658
- ### Custom Resolvers for Inventory Quantity-Specific Logic
659
-
660
- While the mapping above uses built-in SDK resolvers, you can extend with custom business logic:
661
-
662
- ```typescript
663
- const customResolvers = {
664
- /**
665
- * Validate that quantity values are positive
666
- */
667
- 'custom.validateQuantity': (qty: any) => {
668
- const parsed = parseInt(qty) || 0;
669
- return parsed >= 0 ? parsed : 0; // Ensure non-negative
670
- },
671
-
672
- /**
673
- * Add human-readable type descriptions for reporting
674
- */
675
- 'custom.enrichQuantityType': (type: string) => {
676
- const typeDescriptions: Record<string, string> = {
677
- LAST_ON_HAND: 'Last recorded on-hand quantity',
678
- RESERVED: 'Reserved against orders',
679
- DELTA: 'Incremental change (adjustment delta)',
680
- SALE: 'Quantity decreased due to sale',
681
- CORRECTION: 'Manual correction entry',
682
- };
683
- return typeDescriptions[(type || '').toUpperCase()] || type;
684
- },
685
-
686
- /**
687
- * Check if expected date is in the future
688
- */
689
- 'custom.isExpectedInFuture': (expectedOn: string) => {
690
- if (!expectedOn) return false;
691
- return new Date(expectedOn) > new Date();
692
- },
693
-
694
- /**
695
- * Calculate days until expected arrival
696
- */
697
- 'custom.calculateDaysUntilExpected': (expectedOn: string) => {
698
- if (!expectedOn) return null;
699
- const expected = new Date(expectedOn);
700
- const today = new Date();
701
- const diffMs = expected.getTime() - today.getTime();
702
- return Math.ceil(diffMs / (1000 * 60 * 60 * 24));
703
- },
704
-
705
- /**
706
- * Validate status-type combinations
707
- */
708
- 'custom.validateQuantityStatus': (_quantity: any) => {
709
- // Example placeholder – adapt rules to your retailer-defined IQ types
710
- return 'VALID';
711
- },
712
- };
713
-
714
- // Use custom resolvers with UniversalMapper
715
- const mapper = new UniversalMapper(inventoryQuantitiesExportMapping, {
716
- customResolvers,
717
- });
718
- ```
719
-
720
- ### Available SDK Resolvers
721
-
722
- The SDK provides these built-in resolvers (no custom code needed):
723
-
724
- **String Transformations:**
725
-
726
- - `sdk.trim` - Remove leading/trailing whitespace
727
- - `sdk.uppercase` - Convert to uppercase
728
- - `sdk.lowercase` - Convert to lowercase
729
- - `sdk.toString` - Convert to string
730
-
731
- **Number Parsing:**
732
-
733
- - `sdk.parseInt` - Parse as integer
734
- - `sdk.parseFloat` - Parse as decimal
735
- - `sdk.number` - Parse as number (auto-detect int/float)
736
-
737
- **Date Formatting:**
738
-
739
- - `sdk.formatDate` - ISO 8601 date only (YYYY-MM-DD)
740
- - `sdk.formatDateShort` - Short date format
741
- - `sdk.parseDate` - Parse various date formats
742
-
743
- **Type Conversions:**
744
-
745
- - `sdk.boolean` - Convert to boolean
746
- - `sdk.parseJson` - Parse JSON strings
747
- - `sdk.toJson` - Convert to JSON string
748
-
749
- **Utilities:**
750
-
751
- - `sdk.identity` - Return value unchanged
752
- - `sdk.coalesce` - Return first non-null value
753
-
754
- See [Universal Mapping Guide](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) for complete resolver documentation.
755
-
756
- ## GraphQL Query
757
-
758
- ```graphql
759
- query GetInventoryQuantities(
760
- $retailerId: ID!
761
- $updatedAfter: DateTime
762
- $createdAfter: DateTime
763
- $first: Int!
764
- $after: String
765
- ) {
766
- inventoryQuantities(
767
- retailerId: $retailerId
768
- updatedOn: { after: $updatedAfter }
769
- createdOn: { after: $createdAfter }
770
- first: $first
771
- after: $after
772
- ) {
773
- edges {
774
- node {
775
- id
776
- ref
777
- locationRef
778
- skuRef
779
- qty
780
- type
781
- status
782
- expectedOn
783
- createdOn
784
- updatedOn
785
- catalogue {
786
- ref
787
- name
788
- }
789
- }
790
- cursor
791
- }
792
- pageInfo {
793
- hasNextPage
794
- # Note: Fluent doesn't return endCursor/startCursor - cursors are in edges[].cursor
795
- }
796
- }
797
- }
798
- ```
799
-
800
- ## Guardrails Implementation (Required)
801
-
802
- ```typescript
803
- // Overlap buffer (safety window)
804
- const overlapBufferSeconds = parseInt(
805
- ctx.activation?.getVariable('overlapBufferSeconds') || '60',
806
- 10
807
- );
808
- const OVERLAP_BUFFER_MS = overlapBufferSeconds * 1000;
809
-
810
- // Read last successful run and apply buffer
811
- const kv = new VersoriKVAdapter(ctx.openKv(':project:'));
812
- const stateKey = ['extraction', 'inventory-quantities-csv', 'lastRunTime'];
813
- const lastRunState = await kv.get(stateKey);
814
- const rawLastRunTime = lastRunState?.value?.timestamp || fallbackStartDate;
815
- const bufferedLastRunTime = new Date(
816
- new Date(rawLastRunTime).getTime() - OVERLAP_BUFFER_MS
817
- ).toISOString();
818
-
819
- // Query WITH buffer
820
- const result = await client.graphql({
821
- query: INVENTORY_QUANTITIES_QUERY,
822
- variables: {
823
- retailerId,
824
- updatedAfter: bufferedLastRunTime,
825
- first: pageSize,
826
- },
827
- pagination: { maxRecords },
828
- });
829
-
830
- const edges = result.data?.inventoryQuantities?.edges || [];
831
-
832
- // 🛡️ GUARDRAIL: Validate extraction size limits
833
- const MAX_RECORDS_PER_RUN = 500000;
834
- const ESTIMATED_BYTES_PER_RECORD = 300; // Smaller than positions
835
- const estimatedSizeMB = (edges.length * ESTIMATED_BYTES_PER_RECORD) / (1024 * 1024);
836
- const MAX_CSV_SIZE_MB = 100;
837
-
838
- if (edges.length > MAX_RECORDS_PER_RUN) {
839
- log.error('Extraction limit exceeded', {
840
- recordCount: edges.length,
841
- maxAllowed: MAX_RECORDS_PER_RUN,
842
- });
843
- return {
844
- success: false,
845
- error: `Extraction limit exceeded: ${edges.length} records (max: ${MAX_RECORDS_PER_RUN})`,
846
- recommendation: `Split into smaller extractions or increase extraction frequency`,
847
- recordCount: edges.length,
848
- maxAllowed: MAX_RECORDS_PER_RUN,
849
- };
850
- }
851
-
852
- if (estimatedSizeMB > MAX_CSV_SIZE_MB) {
853
- log.warn('CSV size approaching limit', {
854
- estimatedSizeMB: estimatedSizeMB.toFixed(2),
855
- maxAllowed: MAX_CSV_SIZE_MB,
856
- });
857
- }
858
-
859
- log.info('Extraction limits validated', {
860
- recordCount: edges.length,
861
- estimatedSizeMB: estimatedSizeMB.toFixed(2),
862
- withinLimits: true,
863
- });
864
-
865
- // Transform with UniversalMapper
866
- const mapper = new UniversalMapper(inventoryQuantitiesExportMapping);
867
- const transformedRecords: any[] = [];
868
- for (const edge of edges) {
869
- const mapped = await mapper.map(edge.node);
870
- if (mapped.success) {
871
- transformedRecords.push(mapped.data);
872
- }
873
- }
874
-
875
- // Save state WITHOUT buffer (use MAX(updatedOn))
876
- const maxUpdatedOn = transformedRecords.reduce((max, r) => {
877
- const t = new Date(r.updated_on).getTime();
878
- return t > max ? t : max;
879
- }, new Date(rawLastRunTime).getTime());
880
-
881
- await kv.set(stateKey, {
882
- timestamp: new Date(maxUpdatedOn).toISOString(),
883
- recordCount: transformedRecords.length,
884
- extractedAt: new Date().toISOString(),
885
- overlapBufferSeconds,
886
- });
887
-
888
- // Date range guardrails (if you add dateRange/historical modes)
889
- function validateDateRange(mode: 'dateRange' | 'historical', from: string, to: string) {
890
- const start = new Date(from);
891
- const end = new Date(to);
892
- const daysDiff = (end.getTime() - start.getTime()) / (1000 * 60 * 60 * 24);
893
- const maxDays = mode === 'dateRange' ? 30 : 90;
894
- if (daysDiff > maxDays) {
895
- return {
896
- valid: false,
897
- error: `${mode} mode: Maximum ${maxDays}-day window. Requested: ${Math.round(daysDiff)} days.`,
898
- };
899
- }
900
- if (daysDiff < 0) return { valid: false, error: 'endDate must be after startDate' };
901
- return { valid: true };
902
- }
903
- ```
904
-
905
- ---
906
-
907
- ## Versori Workflows Structure
908
-
909
- **Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
910
-
911
- **Trigger Types:**
912
- - **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
913
- - **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
914
- - **`workflow()`** → Durable workflows (advanced, rarely used)
915
-
916
- **Execution Steps (chained to triggers):**
917
- - **`http()`** → External API calls (chained from schedule/webhook)
918
- - **`fn()`** → Internal processing (chained from schedule/webhook)
919
-
920
- ### Recommended Project Structure
921
-
922
- ```
923
- inventory-quantities-extraction/
924
- ├── index.ts # Entry point - exports all workflows
925
- └── src/
926
- ├── workflows/
927
- │ ├── scheduled/
928
- │ │ └── daily-inventory-quantities-extraction.ts # Scheduled: Daily extraction
929
- │ │
930
- │ └── webhook/
931
- │ ├── adhoc-inventory-quantities-extraction.ts # Webhook: Manual trigger
932
- │ └── job-status-check.ts # Webhook: Status query
933
-
934
- ├── services/
935
- │ └── inventory-quantities-extraction.service.ts # Shared orchestration logic (reusable)
936
-
937
- └── config/
938
- └── inventory-quantities.export.csv.json # Mapping configuration
939
- ```
940
-
941
- ---
942
-
943
- ````csv
944
- quantity_id,quantity_ref,catalogue_ref,catalogue_name,location,sku,quantity,type,status,expected_on,created_on,updated_on
945
- Q001,QTY-REF-001,DEFAULT_CATALOGUE,Default Catalogue,DC01,SKU-001,100,AVAILABLE,ACTIVE,,2025-01-15T10:00:00Z,2025-01-22T08:30:00Z
946
- Q002,QTY-REF-002,DEFAULT_CATALOGUE,Default Catalogue,DC01,SKU-001,50,RESERVED,ACTIVE,,2025-01-16T11:00:00Z,2025-01-22T09:15:00Z
947
- Q003,QTY-REF-003,DEFAULT_CATALOGUE,Default Catalogue,DC02,SKU-002,200,EXPECTED,CREATED,2025-01-30T00:00:00Z,2025-01-17T12:00:00Z,2025-01-22T10:00:00Z
948
- Q004,QTY-REF-004,DEFAULT_CATALOGUE,Default Catalogue,STORE-NYC,SKU-003,25,AVAILABLE,ACTIVE,,2025-01-18T13:00:00Z,2025-01-22T11:00:00Z
949
-
950
- ## Advanced Mapping Patterns
951
-
952
- ### Array Mapping (Preserving Nested Structure)
953
-
954
- For nested data structures, use `isArray: true` pattern:
955
-
956
- ```json
957
- {
958
- "fields": {
959
- "ref": { "source": "ref", "required": true },
960
- "relatedItems": {
961
- "source": "items",
962
- "isArray": true,
963
- "fields": {
964
- "itemRef": { "source": "ref", "required": true },
965
- "value": { "source": "value", "resolver": "sdk.parseFloat" }
966
- }
967
- }
968
- }
969
- }
970
- ````
971
-
972
- **When to use**:
973
-
974
- - **Flattened structure**: Simpler, easier for downstream systems
975
- - **Nested with arrays**: Complex data, preserves relationships
976
-
977
- ### Nested Object Mapping
978
-
979
- **Option 1: Flattened paths** (recommended):
980
-
981
- ```json
982
- {
983
- "fields": {
984
- "location_ref": { "source": "location.ref" },
985
- "location_name": { "source": "location.name" }
986
- }
987
- }
988
- ```
989
-
990
- **Option 2: Nested object definition**:
991
-
992
- ```json
993
- {
994
- "fields": {
995
- "location": {
996
- "fields": {
997
- "ref": { "source": "location.ref" },
998
- "name": { "source": "location.name" }
999
- }
1000
- }
1001
- }
1002
- }
1003
- ```
1004
-
1005
- ## Error Handling Strategies
1006
-
1007
- ### Handling Mapping Failures
1008
-
1009
- **Strategy 1: Fail-fast (strict)**:
1010
-
1011
- ```typescript
1012
- if (errors.length > 0) {
1013
- throw new Error(`${errors.length} records failed mapping validation`);
1014
- }
1015
- ```
1016
-
1017
- **Strategy 2: Threshold-based (recommended)**:
1018
-
1019
- ```typescript
1020
- const errorRate = errors.length / transformed.length;
1021
- if (errorRate > 0.05) {
1022
- // 5% threshold
1023
- throw new Error(`Error rate too high: ${(errorRate * 100).toFixed(1)}%`);
1024
- }
1025
- ```
1026
-
1027
- **Strategy 3: Upload error manifest**:
1028
-
1029
- ```typescript
1030
- if (errors.length > 0) {
1031
- const errorManifest = {
1032
- extractionTimestamp: new Date().toISOString(),
1033
- totalErrors: errors.length,
1034
- errors: errors.map(e => ({ record: e.record, errors: e.errors })),
1035
- };
1036
- // Upload to storage for review
1037
- }
1038
- ```
1039
-
1040
- ### State Management with Partial Failures
1041
-
1042
- **Recommended**: Only update state if extraction succeeded:
1043
-
1044
- ```typescript
1045
- if (errors.length === 0) {
1046
- await kv.set(stateKey, { timestamp: newTimestamp });
1047
- log.info('State updated - all records successful');
1048
- } else {
1049
- log.warn('State NOT updated - will retry next run', {
1050
- failedRecords: errors.length,
1051
- willRetryNextRun: true,
1052
- });
1053
- }
1054
- ```
1055
-
1056
- ## GraphQL Query Validation & Testing
1057
-
1058
- ### Schema Validation Workflow
1059
-
1060
- **Step 1: Introspect schema**
1061
-
1062
- ```bash
1063
- npx fc-connect introspect-schema \
1064
- --url https://your-instance.api.fluentcommerce.com/graphql \
1065
- --output fluent-schema.json
1066
- ```
1067
-
1068
- **Step 2: Validate mapping**
1069
-
1070
- ```bash
1071
- npx fc-connect validate-schema \
1072
- --mapping ./config/mapping.json \
1073
- --schema ./fluent-schema.json
1074
- ```
1075
-
1076
- **Step 3: Analyze coverage**
1077
-
1078
- ```bash
1079
- npx fc-connect analyze-coverage \
1080
- --mapping ./config/mapping.json \
1081
- --schema ./fluent-schema.json
1082
- ```
1083
-
1084
- ### GraphQL Pagination Explained
1085
-
1086
- The SDK handles pagination automatically:
1087
-
1088
- ```typescript
1089
- await client.graphql({
1090
- query: QUERY,
1091
- variables: { first: pageSize },
1092
- pagination: { maxRecords }, // SDK handles cursors automatically
1093
- });
1094
- ```
1095
-
1096
- ## Date Format Handling
1097
-
1098
- | Format | Resolver | Output | Use Case |
1099
- | -------- | --------------------- | -------------------------- | --------- |
1100
- | CSV/JSON | `sdk.formatDate` | `2025-01-22T14:30:00.000Z` | ISO 8601 |
1101
- | CSV/JSON | `sdk.formatDateShort` | `2025-01-22` | Date only |
1102
- | CSV/JSON | `sdk.toString` | Pass through | As-is |
1103
-
1104
- ## Monitoring & Alerting
1105
-
1106
- ### Key Metrics to Track
1107
-
1108
- ```typescript
1109
- const metrics = {
1110
- extractionDurationMs: Date.now() - startTime,
1111
- recordCount: edges.length,
1112
- transformedCount: transformed.length,
1113
- failedCount: errors.length,
1114
- errorRate: ((errors.length / edges.length) * 100).toFixed(2) + '%',
1115
- fileSizeMB: (buffer.length / (1024 * 1024)).toFixed(2),
1116
- lastRunTime: rawLastRunTime,
1117
- newTimestamp: newTimestamp,
1118
- };
1119
- log.info('Extraction complete', metrics);
1120
- ```
1121
-
1122
- ### Alert Thresholds
1123
-
1124
- ```typescript
1125
- const ALERTS = {
1126
- EXTRACTION_DURATION_MS: 5 * 60 * 1000, // 5 minutes
1127
- MAX_ERROR_RATE: 0.05, // 5%
1128
- MAX_FILE_SIZE_MB: 150, // 150MB
1129
- MAX_RECORDS_PER_RUN: 100000, // Adjust per entity
1130
- };
1131
- ```
1132
-
1133
- ## Testing Checklist
1134
-
1135
- **Before production deployment:**
1136
-
1137
- ### 1. Schema Validation
1138
-
1139
- - [ ] Run `npx fc-connect introspect-schema`
1140
- - [ ] Run `npx fc-connect validate-schema`
1141
- - [ ] Run `npx fc-connect analyze-coverage`
1142
- - [ ] Verify all `source` paths exist
1143
-
1144
- ### 2. Mapping Testing
1145
-
1146
- - [ ] Test with sample data (maxRecords=10)
1147
- - [ ] Verify required fields populated
1148
- - [ ] Verify SDK resolvers work correctly
1149
- - [ ] Test custom resolvers with edge cases
1150
-
1151
- ### 3. Error Handling
1152
-
1153
- - [ ] Test with invalid data
1154
- - [ ] Verify error collection
1155
- - [ ] Test error threshold logic
1156
-
1157
- ### 4. State Management
1158
-
1159
- - [ ] Verify overlap buffer prevents misses
1160
- - [ ] Test state recovery after failure
1161
- - [ ] Verify timestamp saved WITHOUT buffer
1162
-
1163
- ### 5. File Operations
1164
-
1165
- - [ ] Test connection and upload
1166
- - [ ] Verify file format validity
1167
- - [ ] Test with large files (>50MB)
1168
-
1169
- ### 6. Staging Environment
1170
-
1171
- - [ ] Run full extraction in staging
1172
- - [ ] Verify file format with downstream system
1173
- - [ ] Monitor duration and resource usage
1174
-
1175
- ## Troubleshooting Guide
1176
-
1177
- **Issue**: "Extraction timeout after 10 minutes"
1178
-
1179
- - **Cause**: Too many records
1180
- - **Fix**: Reduce maxRecords, increase frequency
1181
-
1182
- **Issue**: "Mapping errors for 50% of records"
1183
-
1184
- - **Cause**: Schema mismatch
1185
- - **Fix**: Run schema validation, check field names
1186
-
1187
- **Issue**: "State not updating"
1188
-
1189
- - **Cause**: KV write failure or intentional retry
1190
- - **Fix**: Check KV logs, verify state update code
1191
-
1192
- **Issue**: "First run exceeds limits"
1193
-
1194
- - **Cause**: No previous timestamp, fetches all
1195
- - **Fix**: Set fallbackStartDate close to current, apply filters
1196
-
1197
- **Issue**: "Excessive duplicates"
1198
-
1199
- - **Cause**: Overlap buffer (expected) or timestamp not saved
1200
- - **Fix**: Verify newTimestamp saved WITHOUT buffer
1201
-
1202
- ## Security Best Practices
1203
-
1204
- ### Credential Management
1205
-
1206
- **✅ DO**:
1207
-
1208
- - Store credentials in Versori activation variables
1209
- - Rotate credentials quarterly
1210
- - Use least-privilege accounts
1211
-
1212
- **❌ DON'T**:
1213
-
1214
- - Never log credentials
1215
- - Never commit to git
1216
- - Never share across environments
1217
-
1218
- ### Data Security
1219
-
1220
- - Enable encryption in transit and at rest
1221
- - Apply data retention policies
1222
- - Monitor access logs
1223
- - Use VPC/private networks for sensitive data
1224
-
1225
- ---
1226
-
1227
- ```
1228
-
1229
- ---
1230
-
1231
- **Pattern**: Enterprise incremental extraction with overlap buffer for inventory quantities
1232
- **⚠️ Sample Code**: For SDK demonstration only - **ONLY use incremental mode in production**
1233
- **Key Learning**: Use VersoriKVAdapter for state management with 60-second overlap buffer
1234
- **Critical**: Apply overlap buffer to prevent missed records due to clock skew (default: 60 seconds)
1235
- **Buffer Pattern**: Query WITH buffer (`updatedOn >= lastRunTime - 60s`), save WITHOUT buffer (`MAX(updatedOn)`)
1236
- **Timezone**: All timestamps are ISO 8601 UTC format - no conversion needed
1237
- ```
1238
-
1239
- ---
1240
-
1241
- ## 🔧 Complete Production Code
1242
-
1243
- ### 1. Entry Point (src/index.ts)
1244
-
1245
- ```typescript
1246
- /**
1247
- * Entry Point - Registers all workflows with Versori platform
1248
- *
1249
- * This file is the entry point for the Versori deployment.
1250
- * It imports and re-exports workflows from their respective files:
1251
- * 1. Scheduled extraction (runs automatically on cron schedule)
1252
- * 2. Ad hoc webhook (manual trigger with optional date override)
1253
- * 3. Job status webhook (query job progress)
1254
- *
1255
- * AI CUSTOMIZATION:
1256
- * - Add new workflows by importing from their respective files
1257
- * - Remove workflows by commenting out imports/exports
1258
- * - Organize workflows by type (scheduled vs webhook) for clarity
1259
- */
1260
-
1261
- import { scheduledInventoryQuantitiesExtraction } from './workflows/scheduled/daily-inventory-quantities-extraction';
1262
- import { adhocInventoryQuantitiesExtraction } from './workflows/webhook/adhoc-inventory-quantities-extraction';
1263
- import { inventoryQuantitiesJobStatus } from './workflows/webhook/job-status-check';
1264
-
1265
- // Register workflows with Versori platform
1266
- // The platform will expose webhooks as HTTP endpoints and run scheduled workflows on cron schedule
1267
-
1268
- export {
1269
- scheduledInventoryQuantitiesExtraction, // Cron-based auto-run (NOT exposed as HTTP endpoint)
1270
- adhocInventoryQuantitiesExtraction, // Manual webhook trigger (HTTP endpoint)
1271
- inventoryQuantitiesJobStatus, // Job status query (HTTP endpoint)
1272
- };
1273
- ```
1274
-
1275
- ---
1276
-
1277
- ### 2. Workflows
1278
-
1279
- #### src/workflows/scheduled/daily-inventory-quantities-extraction.ts
1280
-
1281
- ```typescript
1282
- /**
1283
- * WORKFLOW 1: Scheduled Extraction
1284
- *
1285
- * Purpose: Automated hourly extraction for incremental sync
1286
- * Trigger: Cron schedule (every hour at minute 0)
1287
- * State Update: Always updates lastSync timestamp
1288
- *
1289
- * AI CUSTOMIZATION:
1290
- * - Change schedule: Replace '0 * * * *' with your cron expression
1291
- * Examples:
1292
- * - Every 30 min: '*/30 * * * *'
1293
- * - Daily at 2 AM: '0 2 * * *'
1294
- * - Every 15 min: '*/15 * * * *'
1295
- */
1296
-
1297
- import { schedule, fn } from '@versori/run';
1298
- import { executeInventoryQuantityExtraction } from '../../services/extraction-orchestration';
1299
- import { generateJobId } from '../../utils/job-id-generator';
1300
-
1301
- /**
1302
- * WORKFLOW 1: Scheduled Extraction
1303
- *
1304
- * Purpose: Automated hourly extraction for incremental sync
1305
- * Trigger: Cron schedule (every hour at minute 0)
1306
- * State Update: Always updates lastSync timestamp
1307
- *
1308
- * AI CUSTOMIZATION:
1309
- * - Change schedule: Replace '0 * * * *' with your cron expression
1310
- * Examples:
1311
- * - Every 30 min: '*/30 * * * *'
1312
- * - Daily at 2 AM: '0 2 * * *'
1313
- * - Every 15 min: '*/15 * * * *'
1314
- */
1315
- export const scheduledInventoryQuantitiesExtraction = schedule(
1316
- 'inventory-quantities-scheduled',
1317
- '0 * * * *', // ← CUSTOMIZE: Cron expression
1318
- fn('execute-scheduled-extraction', async (ctx) => {
1319
- const { log, activation } = ctx;
1320
- const startTime = Date.now();
1321
-
1322
- // Generate unique job ID for tracking
1323
- // Format: SCHEDULED_IQ_YYYYMMDD_HHMMSS_random
1324
- const jobId = generateJobId('SCHEDULED', 'INVENTORY_QUANTITIES');
1325
-
1326
- log.info('🚀 [START] Scheduled extraction triggered', { jobId });
1327
-
1328
- try {
1329
- // Execute main workflow (extraction → transform → upload)
1330
- const result = await executeInventoryQuantityExtraction(ctx, {
1331
- jobId,
1332
- triggeredBy: 'schedule',
1333
- updateState: true, // Always update state for scheduled runs
1334
- });
1335
-
1336
- const durationMs = Date.now() - startTime;
1337
-
1338
- log.info('✅ [END] Scheduled extraction completed', {
1339
- jobId,
1340
- recordCount: result.recordsExtracted,
1341
- fileName: result.fileName,
1342
- durationMs,
1343
- durationSec: (durationMs / 1000).toFixed(2)
1344
- });
1345
-
1346
- return result;
1347
-
1348
- } catch (error: any) {
1349
- const durationMs = Date.now() - startTime;
1350
-
1351
- log.error('❌ [ERROR] Scheduled extraction failed', {
1352
- jobId,
1353
- message: error instanceof Error ? error.message : String(error),
1354
- stack: error instanceof Error ? error.stack : undefined,
1355
- errorType: error instanceof Error ? error.constructor.name : 'Error',
1356
- durationMs,
1357
- recommendation: 'Check Fluent API connectivity, S3 credentials, and date range configuration'
1358
- });
1359
- throw error;
1360
- }
1361
- }));
1362
- ```
1363
-
1364
- ---
1365
-
1366
- #### src/workflows/webhook/adhoc-inventory-quantities-extraction.ts
1367
-
1368
- ```typescript
1369
- /**
1370
- * WORKFLOW 2: Ad hoc Extraction (Manual Trigger)
1371
- *
1372
- * Purpose: Manual extraction with optional date range override
1373
- * Trigger: Webhook POST to /webhooks/inventory-quantities-adhoc
1374
- * State Update: Optional (controlled by request payload)
1375
- *
1376
- * WEBHOOK PAYLOAD EXAMPLES:
1377
- *
1378
- * 1. Incremental (use last sync timestamp):
1379
- * {}
1380
- *
1381
- * 2. Date range (manual override):
1382
- * {
1383
- * "fromDate": "2025-01-01T00:00:00Z",
1384
- * "toDate": "2025-01-31T23:59:59Z",
1385
- * "updateState": false
1386
- * }
1387
- *
1388
- * AI CUSTOMIZATION:
1389
- * - Add request validation
1390
- * - Add authentication check
1391
- * - Add custom filters from payload
1392
- */
1393
-
1394
- import { webhook, fn } from '@versori/run';
1395
- import { executeInventoryQuantityExtraction } from '../../services/extraction-orchestration';
1396
- import { generateJobId } from '../../utils/job-id-generator';
1397
-
1398
- export const adhocInventoryQuantitiesExtraction = webhook(
1399
- 'inventory-quantities-adhoc',
1400
- { connection: 'inventory-quantities-adhoc', response: { mode: 'sync' } },
1401
- fn('execute-adhoc-extraction', async (ctx) => {
1402
- const { data, log, connections, activation } = ctx;
1403
- const startTime = Date.now();
1404
-
1405
- // Generate unique job ID
1406
- const jobId = generateJobId('ADHOC', 'INVENTORY_QUANTITIES');
1407
-
1408
- // SECURITY: Authentication is enforced by Versori connection configuration
1409
- // Configure auth on the connection and reference it in webhook({ connection: '...' })
1410
-
1411
- // Extract optional date override from webhook payload
1412
- const fromDate = data.fromDate as string | undefined;
1413
- const toDate = data.toDate as string | undefined;
1414
- const updateState = data.updateState === true; // Default false; advance state only if explicitly true
1415
-
1416
- log.info('🌐 [START] Ad hoc extraction triggered via webhook', {
1417
- jobId,
1418
- hasDateOverride: !!fromDate,
1419
- fromDate: fromDate || 'not specified',
1420
- toDate: toDate || 'not specified',
1421
- updateState
1422
- });
1423
-
1424
- try {
1425
- // Execute main workflow with optional overrides
1426
- const result = await executeInventoryQuantityExtraction(ctx, {
1427
- jobId,
1428
- triggeredBy: 'webhook',
1429
- fromDate, // Optional: override start date
1430
- toDate, // Optional: override end date
1431
- updateState, // Optional: skip state update for historical queries
1432
- });
1433
-
1434
- const durationMs = Date.now() - startTime;
1435
-
1436
- log.info('✅ [END] Ad hoc extraction completed', {
1437
- jobId,
1438
- recordCount: result.recordsExtracted,
1439
- fileName: result.fileName,
1440
- isManualOverride: !!fromDate,
1441
- stateUpdated: result.stateUpdated,
1442
- durationMs,
1443
- durationSec: (durationMs / 1000).toFixed(2)
1444
- });
1445
-
1446
- // Return success with job details
1447
- return {
1448
- success: true,
1449
- jobId,
1450
- recordsExtracted: result.recordsExtracted,
1451
- fileName: result.fileName,
1452
- s3Path: result.s3Path,
1453
- statusUrl: `/webhooks/inventory-quantities-job-status?jobId=${jobId}`,
1454
- durationMs,
1455
- dateRange: fromDate ? {
1456
- from: fromDate,
1457
- to: toDate || 'not specified',
1458
- updateState
1459
- } : undefined
1460
- };
1461
-
1462
- } catch (error: any) {
1463
- const durationMs = Date.now() - startTime;
1464
-
1465
- log.error('❌ [ERROR] Ad hoc extraction failed', {
1466
- jobId,
1467
- message: error instanceof Error ? error.message : String(error),
1468
- stack: error instanceof Error ? error.stack : undefined,
1469
- errorType: error instanceof Error ? error.constructor.name : 'Error',
1470
- durationMs,
1471
- recommendation: 'Verify date range format (ISO 8601), check S3 credentials, and ensure Fluent API access'
1472
- });
1473
-
1474
- return {
1475
- success: false,
1476
- jobId,
1477
- message: error instanceof Error ? error.message : String(error),
1478
- stack: error instanceof Error ? error.stack : undefined,
1479
- errorType: error instanceof Error ? error.constructor.name : 'Error',
1480
- durationMs,
1481
- recommendation: 'Verify date range format (ISO 8601), check S3 credentials, and ensure Fluent API access'
1482
- };
1483
- }
1484
- }));
1485
- ```
1486
-
1487
- ---
1488
-
1489
- #### src/workflows/webhook/job-status-check.ts
1490
-
1491
- ```typescript
1492
- /**
1493
- * WORKFLOW 3: Job Status Query
1494
- *
1495
- * Purpose: Check job progress and status
1496
- * Trigger: Webhook GET/POST to /webhooks/inventory-quantities-job-status?jobId=xxx
1497
- * Returns: Current job status, stage, progress
1498
- *
1499
- * QUERY EXAMPLES:
1500
- *
1501
- * 1. HTTP GET:
1502
- * GET /webhooks/inventory-quantities-job-status?jobId=ADHOC_IQ_20251027_183045_abc123
1503
- *
1504
- * 2. HTTP POST:
1505
- * POST /webhooks/inventory-quantities-job-status
1506
- * { "jobId": "ADHOC_IQ_20251027_183045_abc123" }
1507
- */
1508
-
1509
- import { webhook, fn } from '@versori/run';
1510
- import { getJobStatus } from '../../services/extraction-orchestration';
1511
-
1512
- export const inventoryQuantitiesJobStatus = webhook(
1513
- 'inventory-quantities-job-status',
1514
- { connection: 'inventory-quantities-job-status', response: { mode: 'sync' } },
1515
- fn('query-job-status', async (ctx) => {
1516
- const { data, log, openKv, activation } = ctx;
1517
- const startTime = Date.now();
1518
-
1519
- // SECURITY: Authentication is enforced by Versori connection configuration
1520
- // Configure auth on the connection and reference it in webhook({ connection: '...' })
1521
-
1522
- // Get jobId from query param or POST body
1523
- const jobId = data.jobId as string;
1524
-
1525
- if (!jobId) {
1526
- log.error('❌ Job ID not provided in request');
1527
- return {
1528
- success: false,
1529
- error: 'Job ID is required. Provide jobId in query param or request body.'
1530
- };
1531
- }
1532
-
1533
- log.info('🔍 [START] Querying job status', { jobId });
1534
-
1535
- try {
1536
- // Query job status from KV store
1537
- const status = await getJobStatus(openKv(':project:'), jobId, log);
1538
-
1539
- const durationMs = Date.now() - startTime;
1540
-
1541
- if (!status) {
1542
- log.info('⚠️ Job not found', { jobId, durationMs });
1543
- return {
1544
- success: false,
1545
- error: 'Job not found',
1546
- jobId,
1547
- durationMs
1548
- };
1549
- }
1550
-
1551
- log.info('✅ [END] Job status retrieved', {
1552
- jobId,
1553
- status: status.status,
1554
- durationMs
1555
- });
1556
-
1557
- return {
1558
- success: true,
1559
- jobId,
1560
- ...status,
1561
- queryDurationMs: durationMs
1562
- };
1563
-
1564
- } catch (error: any) {
1565
- const durationMs = Date.now() - startTime;
1566
-
1567
- log.error('❌ [ERROR] Failed to query job status', {
1568
- jobId,
1569
- message: error instanceof Error ? error.message : String(error),
1570
- stack: error instanceof Error ? error.stack : undefined,
1571
- errorType: error instanceof Error ? error.constructor.name : 'Error',
1572
- durationMs,
1573
- recommendation: 'Verify KV store access and job ID format'
1574
- });
1575
-
1576
- return {
1577
- success: false,
1578
- jobId,
1579
- message: error instanceof Error ? error.message : String(error),
1580
- stack: error instanceof Error ? error.stack : undefined,
1581
- errorType: error instanceof Error ? error.constructor.name : 'Error',
1582
- durationMs,
1583
- recommendation: 'Verify KV store access and job ID format'
1584
- };
1585
- }
1586
- }));
1587
- ```
1588
-
1589
- ---
1590
-
1591
- ### 3. Main Orchestration Service (src/services/extraction-orchestration.ts)
1592
-
1593
- ```typescript
1594
- /**
1595
- * MAIN ORCHESTRATION SERVICE
1596
- *
1597
- * This is the heart of the extraction workflow. It coordinates all steps:
1598
- * 1. Initialize clients and services
1599
- * 2. Determine date range (incremental vs manual)
1600
- * 3. Extract data using ExtractionOrchestrator
1601
- * 4. Transform using UniversalMapper
1602
- * 5. Generate CSV using CSVParserService
1603
- * 6. Upload to S3
1604
- * 7. Track job progress with JobTracker
1605
- * 8. Update state for next run
1606
- *
1607
- * NAMING PATTERN (consistent across all use cases):
1608
- * - Interface: {Entity}ExtractionParams (e.g., InventoryQuantityExtractionParams)
1609
- * - Result: {Entity}ExtractionResult (e.g., InventoryQuantityExtractionResult)
1610
- * - Main function: execute{Entity}Extraction (e.g., executeInventoryQuantityExtraction)
1611
- *
1612
- * AI CUSTOMIZATION HINTS:
1613
- * - Change entity: Replace "InventoryQuantity" with "Order", "Product", etc.
1614
- * - Change output: Replace CSVParserService with XMLBuilder
1615
- * - Change destination: Replace S3DataSource with SftpDataSource
1616
- * - Add steps: Insert new service calls between existing steps
1617
- */
1618
-
1619
- import { Buffer } from 'node:buffer';
1620
- import {
1621
- createClient,
1622
- ExtractionOrchestrator,
1623
- JobTracker,
1624
- UniversalMapper,
1625
- CSVParserService,
1626
- S3DataSource,
1627
- } from '@fluentcommerce/fc-connect-sdk';
1628
-
1629
- import mappingConfig from '../../config/inventory-quantities.export.csv.json' with { type: 'json' };
1630
-
1631
- // ✅ VERSORI PLATFORM: Use native log from context, not LoggingService
1632
-
1633
- /**
1634
- * Parameters for extraction workflow
1635
- *
1636
- * NAMING: {Entity}ExtractionParams
1637
- */
1638
- export interface InventoryQuantityExtractionParams {
1639
- jobId: string;
1640
- triggeredBy: 'schedule' | 'webhook';
1641
- fromDate?: string; // Optional: manual date override
1642
- toDate?: string; // Optional: manual date override
1643
- updateState: boolean; // Whether to update lastSync timestamp
1644
-
1645
- // AI CUSTOMIZATION: Add filters specific to entity
1646
- quantityTypes?: string[]; // e.g., ['LAST_ON_HAND', 'RESERVED']
1647
- catalogueRef?: string; // e.g., 'DEFAULT_CATALOGUE'
1648
- }
1649
-
1650
- /**
1651
- * Result from extraction workflow
1652
- *
1653
- * NAMING: {Entity}ExtractionResult
1654
- */
1655
- export interface InventoryQuantityExtractionResult {
1656
- success: boolean;
1657
- jobId: string;
1658
- recordsExtracted: number;
1659
- fileName?: string;
1660
- s3Path?: string;
1661
- error?: string;
1662
- errors?: any[];
1663
- isManualOverride?: boolean;
1664
- stateUpdated?: boolean;
1665
- newTimestamp?: string;
1666
- }
1667
-
1668
- /**
1669
- * GraphQL Query for Inventory Quantities
1670
- *
1671
- * NAMING: {ENTITY}_EXTRACTION_QUERY (uppercase constant)
1672
- */
1673
- const INVENTORY_QUANTITIES_EXTRACTION_QUERY = `
1674
- query GetInventoryQuantities(
1675
- $catalogues: [InventoryCatalogueKey]
1676
- $dateRangeFilter: DateRange
1677
- $productRefs: [String!]
1678
- $types: [String!]
1679
- $first: Int!
1680
- $after: String
1681
- ) {
1682
- inventoryQuantities(
1683
- catalogues: $catalogues
1684
- updatedOn: $dateRangeFilter
1685
- productRef: $productRefs
1686
- type: $types
1687
- first: $first
1688
- after: $after
1689
- ) {
1690
- edges {
1691
- node {
1692
- id
1693
- ref
1694
- locationRef
1695
- productRef
1696
- qty
1697
- type
1698
- status
1699
- expectedOn
1700
- createdOn
1701
- updatedOn
1702
- catalogue {
1703
- ref
1704
- name
1705
- }
1706
- }
1707
- cursor
1708
- }
1709
- pageInfo {
1710
- hasNextPage
1711
- }
1712
- }
1713
- }
1714
- `;
1715
-
1716
- /**
1717
- * Query job status from KV store
1718
- *
1719
- * ✅ VERSORI PLATFORM: Uses native log from context (passed as parameter)
1720
- */
1721
- export async function getJobStatus(
1722
- kv: any, // ✅ Versori KV (compatible with JobTracker's KVAdapter interface)
1723
- jobId: string,
1724
- log: any // ✅ Native Versori log from context
1725
- ): Promise<any | undefined> {
1726
- try {
1727
- const tracker = new JobTracker(kv, log);
1728
- return await tracker.getJob(jobId);
1729
- } catch (error: any) {
1730
- log.error('Failed to get job status', { jobId, message: error instanceof Error ? error.message : String(error),
1731
- stack: error instanceof Error ? error.stack : undefined,
1732
- errorType: error instanceof Error ? error.constructor.name : 'Error', });
1733
- return undefined;
1734
- }
1735
- }
1736
-
1737
- /**
1738
- * MAIN ORCHESTRATION FUNCTION
1739
- *
1740
- * NAMING: execute{Entity}Extraction (e.g., executeInventoryQuantityExtraction)
1741
- *
1742
- * This function implements the complete workflow in steps.
1743
- * Each step is clearly commented for AI understanding.
1744
- */
1745
- export async function executeInventoryQuantityExtraction(
1746
- ctx: any,
1747
- params: InventoryQuantityExtractionParams
1748
- ): Promise<InventoryQuantityExtractionResult> {
1749
- // ✅ VERSORI PLATFORM: Extract native log from context
1750
- const { log, openKv, activation } = ctx;
1751
- const { jobId, triggeredBy, fromDate, toDate, updateState } = params;
1752
-
1753
- // Open KV store for state management and job tracking
1754
- // ✅ Pass raw Versori KV directly - it matches KVAdapter interface
1755
- // ✅ Pass native log to JobTracker
1756
- const kv = openKv(':project:');
1757
- const tracker = new JobTracker(kv, log);
1758
-
1759
- try {
1760
- // ═══════════════════════════════════════════════════════════
1761
- // STEP 1/8: Initialize Job Tracking
1762
- // ═══════════════════════════════════════════════════════════
1763
- log.info('📝 [STEP 1/8] Initializing job tracking', { jobId });
1764
-
1765
- await tracker.createJob(jobId, {
1766
- triggeredBy,
1767
- hasDateOverride: !!fromDate,
1768
- fromDate,
1769
- toDate,
1770
- updateStateAfterRun: updateState,
1771
- });
1772
-
1773
- // ═══════════════════════════════════════════════════════════
1774
- // STEP 2/8: Initialize Fluent Client
1775
- // ═══════════════════════════════════════════════════════════
1776
- log.info('📡 [STEP 2/8] Initializing Fluent Commerce client', { jobId });
1777
-
1778
- const client = await createClient(ctx, { validateConnection: true });
1779
-
1780
- if (!client) {
1781
- throw new Error('Failed to create Fluent Commerce client');
1782
- }
1783
-
1784
- log.info('✅ Fluent client initialized and connection validated', { jobId });
1785
-
1786
- // ═══════════════════════════════════════════════════════════
1787
- // STEP 3/8: Determine Date Range
1788
- // ═══════════════════════════════════════════════════════════
1789
- log.info('📅 [STEP 3/8] Determining date range for extraction', { jobId });
1790
-
1791
- // State key for incremental sync tracking
1792
- // NAMING: last{Entity}Sync (e.g., lastInventoryQuantitySync)
1793
- const STATE_KEY = 'lastInventoryQuantitySync';
1794
- const DEFAULT_FALLBACK = '2024-01-01T00:00:00Z';
1795
- const OVERLAP_BUFFER_SECONDS = parseInt(
1796
- activation.getVariable('overlapBufferSeconds') || '60',
1797
- 10
1798
- );
1799
-
1800
- let dateRangeFilter: { from?: string; to?: string } | null = null;
1801
- const isManualOverride = !!fromDate;
1802
-
1803
- if (isManualOverride) {
1804
- // Manual date override from webhook
1805
- dateRangeFilter = { from: fromDate, to: toDate };
1806
- log.info('Using manual date override', { fromDate, toDate });
1807
- } else {
1808
- // Incremental sync - get last sync timestamp
1809
- const rawLastRunTime = (await kv.get<string>(STATE_KEY)) || DEFAULT_FALLBACK;
1810
-
1811
- // Apply overlap buffer (prevents missed records)
1812
- const bufferedLastRunTime = new Date(
1813
- new Date(rawLastRunTime).getTime() - OVERLAP_BUFFER_SECONDS * 1000
1814
- ).toISOString();
1815
-
1816
- const effectiveEndTime = toDate || new Date().toISOString();
1817
-
1818
- dateRangeFilter = {
1819
- from: bufferedLastRunTime,
1820
- to: effectiveEndTime, // End of extraction window
1821
- };
1822
-
1823
- log.info('Using incremental sync with overlap buffer', {
1824
- rawLastRunTime,
1825
- bufferedLastRunTime,
1826
- effectiveEndTime,
1827
- overlapBufferSeconds: OVERLAP_BUFFER_SECONDS,
1828
- });
1829
- }
1830
-
1831
- // ═══════════════════════════════════════════════════════════
1832
- // STEP 4/8: Extract Data (ExtractionOrchestrator)
1833
- // ═══════════════════════════════════════════════════════════
1834
- log.info('🔄 [STEP 4/8] Extracting data from Fluent Commerce', { jobId });
1835
-
1836
- await tracker.updateJob(jobId, {
1837
- status: 'processing',
1838
- stage: 'extraction',
1839
- message: 'Extracting data with auto-pagination',
1840
- });
1841
-
1842
- // Build catalogues array from config
1843
- const catalogueRef = params.catalogueRef || activation.getVariable('catalogueRef');
1844
- const catalogues = catalogueRef ? [{ ref: catalogueRef }] : [];
1845
-
1846
- // Configure extraction
1847
- const pageSize = parseInt(activation.getVariable('pageSize') || '200', 10);
1848
- const maxRecords = parseInt(activation.getVariable('maxRecords') || '100000', 10);
1849
-
1850
- // Initialize ExtractionOrchestrator
1851
- const orchestrator = new ExtractionOrchestrator(client, log);
1852
-
1853
- // ? Enhanced: Extract context for progress logging
1854
- const dateRangeInfo = {
1855
- start: dateRangeFilter?.from || 'N/A',
1856
- end: dateRangeFilter?.to || 'N/A',
1857
- catalogues: catalogues.map((c: any) => c.ref).join(', ') || 'all',
1858
- types: params.quantityTypes?.join(', ') || 'all'
1859
- };
1860
-
1861
- // ? Enhanced: Start logging with context
1862
- log.info(`🔍 [ExtractionOrchestrator] Starting extraction`, {
1863
- query: 'inventoryQuantities',
1864
- pageSize,
1865
- maxRecords,
1866
- dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
1867
- catalogues: dateRangeInfo.catalogues,
1868
- quantityTypes: dateRangeInfo.types,
1869
- jobId
1870
- });
1871
-
1872
- // Execute extraction with auto-pagination
1873
- const extractionResult = await orchestrator.extract({
1874
- query: INVENTORY_QUANTITIES_EXTRACTION_QUERY,
1875
- resultPath: 'inventoryQuantities.edges.node',
1876
- variables: {
1877
- catalogues,
1878
- dateRangeFilter,
1879
- types: params.quantityTypes,
1880
- // Note: Don't include 'first' or 'after' here; orchestrator injects them
1881
- },
1882
- pageSize,
1883
- maxRecords,
1884
- // Optional: validate each record
1885
- validateItem: (item: any) => {
1886
- return !!(item.ref && item.productRef);
1887
- },
1888
- });
1889
-
1890
- const records = extractionResult.data || [];
1891
-
1892
- log.info('Extraction complete', {
1893
- totalRecords: extractionResult.stats.totalRecords,
1894
- totalPages: extractionResult.stats.totalPages,
1895
- validRecords: extractionResult.stats.validRecords ?? records.length,
1896
- failedValidations: extractionResult.stats.failedValidations,
1897
- errors: extractionResult.errors ? extractionResult.errors.length : 0,
1898
- });
1899
-
1900
- // ? Enhanced: Completion logging with summary
1901
- log.info(`✅ [ExtractionOrchestrator] Extraction completed`, {
1902
- totalRecords: extractionResult.stats.totalRecords,
1903
- totalPages: extractionResult.stats.totalPages,
1904
- validRecords: extractionResult.stats.validRecords ?? records.length,
1905
- failedValidations: extractionResult.stats.failedValidations,
1906
- truncated: extractionResult.stats.truncated,
1907
- truncationReason: extractionResult.stats.truncationReason,
1908
- dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
1909
- jobId
1910
- });
1911
-
1912
- if (extractionResult.errors && extractionResult.errors.length > 0) {
1913
- log.warn('Non-fatal extraction errors encountered', {
1914
- errorCount: extractionResult.errors.length,
1915
- sampleErrors: extractionResult.errors.slice(0, 3),
1916
- });
1917
- }
1918
-
1919
- // Handle empty result
1920
- if (records.length === 0) {
1921
- log.info('No records to process');
1922
-
1923
- // Update state even with no records (prevents re-querying empty window)
1924
- if (updateState && !isManualOverride) {
1925
- await kv.set(STATE_KEY, new Date().toISOString());
1926
- }
1927
-
1928
- await tracker.markCompleted(jobId, {
1929
- recordCount: 0,
1930
- message: 'No records to extract',
1931
- });
1932
-
1933
- return {
1934
- success: true,
1935
- jobId,
1936
- recordsExtracted: 0,
1937
- };
1938
- }
1939
-
1940
- // ═══════════════════════════════════════════════════════════
1941
- // STEP 5/8: Transform Data (UniversalMapper)
1942
- // ═══════════════════════════════════════════════════════════
1943
- log.info('🔧 [STEP 5/8] Transforming data with UniversalMapper', {
1944
- jobId,
1945
- recordCount: records.length,
1946
- });
1947
-
1948
- await tracker.updateJob(jobId, {
1949
- status: 'processing',
1950
- stage: 'transformation',
1951
- message: `Transforming ${records.length} records`,
1952
- });
1953
-
1954
- const mapper = new UniversalMapper(mappingConfig);
1955
- const mappingResult = await mapper.map(records);
1956
-
1957
- if (!mappingResult.success) {
1958
- const mappingErrors = mappingResult.errors || ['Unknown mapping failure'];
1959
- await tracker.markFailed(jobId, {
1960
- error: mappingErrors[0] || 'UniversalMapper returned unsuccessful result',
1961
- failedCount: mappingErrors.length,
1962
- });
1963
- return {
1964
- success: false,
1965
- error: `Transformation failed: ${mappingErrors[0] || 'Unknown error'}`,
1966
- jobId,
1967
- errors: mappingErrors,
1968
- };
1969
- }
1970
-
1971
- const transformedRecords = Array.isArray(mappingResult.data) ? mappingResult.data : [];
1972
- const mappingErrors = mappingResult.errors || [];
1973
-
1974
- if (mappingErrors.length > 0) {
1975
- log.warn('Some records failed transformation', {
1976
- jobId,
1977
- errorCount: mappingErrors.length,
1978
- sampleErrors: mappingErrors.slice(0, 3),
1979
- });
1980
- }
1981
-
1982
- if (transformedRecords.length === 0) {
1983
- await tracker.markFailed(jobId, {
1984
- error: 'All records failed mapping',
1985
- failedCount: mappingErrors.length,
1986
- errors: mappingErrors,
1987
- });
1988
- return {
1989
- success: false,
1990
- error: 'All records failed mapping',
1991
- jobId,
1992
- errors: mappingErrors,
1993
- };
1994
- }
1995
-
1996
- log.info('Transformation complete', {
1997
- successful: transformedRecords.length,
1998
- failed: mappingErrors.length,
1999
- skippedRecords: records.length - transformedRecords.length,
2000
- });
2001
-
2002
- // ═══════════════════════════════════════════════════════════
2003
- // STEP 6/8: Generate CSV (CSVParserService)
2004
- // ═══════════════════════════════════════════════════════════
2005
- log.info('📄 [STEP 6/8] Generating CSV file', { jobId });
2006
-
2007
- await tracker.updateJob(jobId, {
2008
- status: 'processing',
2009
- stage: 'csv_generation',
2010
- message: `Generating CSV for ${transformedRecords.length} records`,
2011
- });
2012
-
2013
- // Initialize CSVParserService
2014
- const csvParser = new CSVParserService({ includeHeaders: true });
2015
-
2016
- // Generate CSV content
2017
- const csvContent = await csvParser.stringify(transformedRecords);
2018
-
2019
- // Generate filename
2020
- const fileNamePrefix = activation.getVariable('fileNamePrefix') || 'inventoryquantities';
2021
- const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
2022
- const fileName = `${fileNamePrefix}-${timestamp}.csv`;
2023
-
2024
- log.info('CSV file generated', {
2025
- fileName,
2026
- sizeBytes: csvContent.length,
2027
- recordCount: transformedRecords.length,
2028
- });
2029
-
2030
- // ═══════════════════════════════════════════════════════════
2031
- // STEP 7/8: Upload to S3 (S3DataSource)
2032
- // ═══════════════════════════════════════════════════════════
2033
- log.info('☁️ [STEP 7/8] Uploading to S3', { jobId, fileName });
2034
-
2035
- await tracker.updateJob(jobId, {
2036
- status: 'processing',
2037
- stage: 's3_upload',
2038
- message: `Uploading ${fileName} to S3`,
2039
- });
2040
-
2041
- // Get S3 configuration from activation variables
2042
- const s3Config = {
2043
- bucket: activation.getVariable('s3BucketName'),
2044
- region: activation.getVariable('awsRegion') || 'us-east-1',
2045
- accessKeyId: activation.getVariable('awsAccessKeyId'),
2046
- secretAccessKey: activation.getVariable('awsSecretAccessKey'),
2047
- };
2048
- const s3Prefix = activation.getVariable('s3Prefix') || 'inventory-quantities/daily/';
2049
-
2050
- // Validate S3 config
2051
- if (!s3Config.bucket || !s3Config.accessKeyId || !s3Config.secretAccessKey) {
2052
- throw new Error(
2053
- 'S3 configuration incomplete: missing bucket, accessKeyId, or secretAccessKey'
2054
- );
2055
- }
2056
-
2057
- // Initialize S3 data source
2058
- // ✅ VERSORI PLATFORM: Pass native log from context
2059
- const s3 = new S3DataSource(
2060
- {
2061
- type: 'S3_CSV',
2062
- connectionId: 'inventory-quantities-s3',
2063
- name: 'Inventory Quantities S3 Upload',
2064
- s3Config,
2065
- },
2066
- log
2067
- );
2068
-
2069
- // Construct S3 key
2070
- const s3Key = `${s3Prefix}${fileName}`;
2071
-
2072
- // Upload with retry logic (built into S3DataSource)
2073
- await s3.uploadFile(s3Key, Buffer.from(csvContent, 'utf-8'), {
2074
- contentType: 'text/csv',
2075
- metadata: {
2076
- recordCount: String(transformedRecords.length),
2077
- extractedAt: new Date().toISOString(),
2078
- jobId,
2079
- mappingErrors: mappingErrors.length > 0 ? String(mappingErrors.length) : undefined,
2080
- },
2081
- });
2082
-
2083
- log.info('S3 upload successful', { fileName, s3Key });
2084
-
2085
- // ═══════════════════════════════════════════════════════════
2086
- // STEP 8/8: Update State & Complete Job
2087
- // ═══════════════════════════════════════════════════════════
2088
- log.info('💾 [STEP 8/8] Updating state and completing job', { jobId });
2089
-
2090
- // Calculate new timestamp for next incremental run
2091
- let newTimestamp: string | undefined;
2092
-
2093
- if (updateState && !isManualOverride) {
2094
- // Find max updatedOn from extracted records
2095
- const maxUpdatedOn = records.reduce(
2096
- (max, record) => {
2097
- const recordTime = new Date(record.updatedOn).getTime();
2098
- return recordTime > max ? recordTime : max;
2099
- },
2100
- new Date(dateRangeFilter?.from || DEFAULT_FALLBACK).getTime()
2101
- );
2102
-
2103
- newTimestamp = new Date(maxUpdatedOn).toISOString();
2104
-
2105
- // Store new timestamp (WITHOUT buffer - buffer only applied on read)
2106
- await kv.set(STATE_KEY, newTimestamp);
2107
-
2108
- log.info('State updated', {
2109
- oldTimestamp: dateRangeFilter?.from,
2110
- newTimestamp,
2111
- });
2112
- }
2113
-
2114
- // Mark job as completed
2115
- await tracker.markCompleted(jobId, {
2116
- recordCount: transformedRecords.length,
2117
- fileName,
2118
- s3Key,
2119
- errorCount: mappingErrors.length,
2120
- errors: mappingErrors,
2121
- isManualOverride,
2122
- stateUpdated: updateState,
2123
- newTimestamp,
2124
- });
2125
-
2126
- return {
2127
- success: true,
2128
- jobId,
2129
- recordsExtracted: transformedRecords.length,
2130
- fileName,
2131
- s3Path: s3Key,
2132
- isManualOverride,
2133
- stateUpdated: updateState,
2134
- newTimestamp,
2135
- errors: mappingErrors.length > 0 ? mappingErrors : undefined,
2136
- };
2137
- } catch (error: any) {
2138
- log.error('Extraction workflow failed', {
2139
- jobId,
2140
- message: error instanceof Error ? error.message : String(error),
2141
-
2142
- stack: error instanceof Error ? error.stack : undefined,
2143
-
2144
- errorType: error instanceof Error ? error.constructor.name : 'Error',
2145
- });
2146
-
2147
- // Mark job as failed
2148
- await tracker.markFailed(jobId, error);
2149
-
2150
- return {
2151
- success: false,
2152
- jobId,
2153
- recordsExtracted: 0,
2154
- message: error instanceof Error ? error.message : String(error),
2155
-
2156
- stack: error instanceof Error ? error.stack : undefined,
2157
-
2158
- errorType: error instanceof Error ? error.constructor.name : 'Error',
2159
- };
2160
- }
2161
- }
2162
- ```
2163
-
2164
- ---
2165
-
2166
- ### 4. Utility Functions (src/utils/job-id-generator.ts)
2167
-
2168
- ```typescript
2169
- /**
2170
- * Job ID Generator
2171
- *
2172
- * Generates unique job IDs for tracking extraction workflows
2173
- *
2174
- * FORMAT: {TYPE}_{ENTITY}_{YYYYMMDD}_{HHMMSS}_{RANDOM}
2175
- * Example: SCHEDULED_IQ_20251027_183045_a1b2c3
2176
- */
2177
-
2178
- /**
2179
- * Generate unique job ID
2180
- *
2181
- * @param type - Job trigger type (SCHEDULED, ADHOC, MANUAL)
2182
- * @param entity - Entity abbreviation (IQ=Inventory Quantities, IP, VP, ORD, PRD)
2183
- * @returns Unique job ID string
2184
- */
2185
- export function generateJobId(type: string, entity: string): string {
2186
- const now = new Date();
2187
-
2188
- // Format: YYYYMMDD
2189
- const dateStr = now.toISOString().slice(0, 10).replace(/-/g, '');
2190
-
2191
- // Format: HHMMSS
2192
- const timeStr = now.toISOString().slice(11, 19).replace(/:/g, '');
2193
-
2194
- // Random suffix (6 chars)
2195
- const randomStr = Math.random().toString(36).substring(2, 8);
2196
-
2197
- return `${type}_${entity}_${dateStr}_${timeStr}_${randomStr}`;
2198
- }
2199
-
2200
- /**
2201
- * Parse job ID components
2202
- */
2203
- export function parseJobId(jobId: string): {
2204
- type: string;
2205
- entity: string;
2206
- date: string;
2207
- time: string;
2208
- random: string;
2209
- } | null {
2210
- const parts = jobId.split('_');
2211
-
2212
- if (parts.length !== 5) {
2213
- return null;
2214
- }
2215
-
2216
- return {
2217
- type: parts[0],
2218
- entity: parts[1],
2219
- date: parts[2],
2220
- time: parts[3],
2221
- random: parts[4],
2222
- };
2223
- }
2224
- ```
2225
-
2226
- ---
2227
-
2228
- ### 5. Package Configuration
2229
-
2230
- #### package.json
2231
-
2232
- ```json
2233
- {
2234
- "name": "inventory-quantities-to-s3-csv",
2235
- "version": "1.0.0",
2236
- "description": "Extract inventory quantities from Fluent Commerce and export to S3 as CSV",
2237
- "type": "module",
2238
- "main": "src/index.ts",
2239
- "scripts": {
2240
- "dev": "versori dev",
2241
- "build": "versori build",
2242
- "deploy": "versori deploy"
2243
- },
2244
- "dependencies": {
2245
- "@fluentcommerce/fc-connect-sdk": "^0.1.39",
2246
- "@versori/run": "latest"
2247
- },
2248
- "devDependencies": {
2249
- "@types/node": "^20.0.0",
2250
- "typescript": "^5.0.0"
2251
- }
2252
- }
2253
- ```
2254
-
2255
- #### tsconfig.json
2256
-
2257
- ```json
2258
- {
2259
- "compilerOptions": {
2260
- "module": "ES2022",
2261
- "target": "ES2024",
2262
- "moduleResolution": "node"
2263
- }
2264
- }
2265
- ```
2266
-
2267
- ---
2268
-
2269
- ## 6. Deployment Instructions
2270
-
2271
- ### Deploy to Versori
2272
-
2273
- ```bash
2274
- # 1. Install dependencies
2275
- npm install
2276
-
2277
- # 2. Test locally (if using Versori CLI)
2278
- npm run dev
2279
-
2280
- # 3. Deploy to Versori platform
2281
- npm run deploy
2282
- ```
2283
-
2284
- ### Configure Activation Variables
2285
-
2286
- In Versori platform settings, configure:
2287
-
2288
- ```json
2289
- {
2290
- "catalogueRef": "DEFAULT_CATALOGUE",
2291
- "s3BucketName": "inventory-audit-exports",
2292
- "awsAccessKeyId": "AKIAXXXXXXXXXXXX",
2293
- "awsSecretAccessKey": "********",
2294
- "awsRegion": "us-east-1",
2295
- "s3Prefix": "inventory-quantities/daily/",
2296
- "fileNamePrefix": "inventoryquantities",
2297
- "pageSize": 200,
2298
- "maxRecords": 100000,
2299
- "overlapBufferSeconds": 60,
2300
- "webhookApiKey": "your-secure-api-key-here"
2301
- }
2302
- ```
2303
-
2304
- ---
2305
-
2306
- ## 7. Testing
2307
-
2308
- ### Test Scheduled Extraction
2309
-
2310
- The scheduled workflow runs automatically based on cron schedule.
2311
-
2312
- **Check logs:**
2313
-
2314
- ```
2315
- [STEP 1/8] Initializing job tracking
2316
- [STEP 2/8] Initializing Fluent Commerce client
2317
- [STEP 3/8] Determining date range for extraction
2318
- [STEP 4/8] Extracting data from Fluent Commerce
2319
- [STEP 5/8] Transforming data with UniversalMapper
2320
- [STEP 6/8] Generating CSV file
2321
- [STEP 7/8] Uploading to S3
2322
- [STEP 8/8] Updating state and completing job
2323
- ```
2324
-
2325
- ### Test Ad hoc Extraction
2326
-
2327
- ```bash
2328
- # Incremental (uses last sync timestamp)
2329
- curl -X POST https://api.versori.com/webhooks/inventory-quantities-adhoc \
2330
- -H "X-API-Key: your-api-key" \
2331
- -H "Content-Type: application/json" \
2332
- -d '{}'
2333
-
2334
- # Date range override
2335
- curl -X POST https://api.versori.com/webhooks/inventory-quantities-adhoc \
2336
- -H "X-API-Key: your-api-key" \
2337
- -H "Content-Type: application/json" \
2338
- -d '{
2339
- "fromDate": "2025-01-01T00:00:00Z",
2340
- "toDate": "2025-01-31T23:59:59Z",
2341
- "updateState": false
2342
- }'
2343
- ```
2344
-
2345
- ### Test Job Status Query
2346
-
2347
- ```bash
2348
- curl -X POST https://api.versori.com/webhooks/inventory-quantities-job-status \
2349
- -H "X-API-Key: your-api-key" \
2350
- -H "Content-Type: application/json" \
2351
- -d '{
2352
- "jobId": "ADHOC_IQ_20251027_183045_abc123"
2353
- }'
2354
- ```
2355
-
2356
- **Response:**
2357
-
2358
- ```json
2359
- {
2360
- "success": true,
2361
- "jobId": "ADHOC_IQ_20251027_183045_abc123",
2362
- "status": "processing",
2363
- "stage": "transformation",
2364
- "message": "Transforming 15000 records",
2365
- "createdAt": "2025-10-27T18:30:45.000Z",
2366
- "startedAt": "2025-10-27T18:30:46.000Z"
2367
- }
2368
- ```
2369
-
2370
- ---
2371
-
2372
- ## 8. Troubleshooting
2373
-
2374
- **Issue**: "No records extracted"
2375
-
2376
- - Check dateRange (manual override vs incremental)
2377
- - Check catalogueRef filter
2378
- - Verify quantity types filter
2379
-
2380
- **Issue**: "S3 upload failed"
2381
-
2382
- - Job fails; state not advanced
2383
- - Next run retries same window
2384
- - Check S3 credentials and bucket permissions
2385
-
2386
- **Issue**: "GraphQL pagination error"
2387
-
2388
- - Ensure edges.cursor and pageInfo.hasNextPage are in query
2389
-
2390
- **Issue**: "Memory pressure"
2391
-
2392
- - Lower pageSize or maxRecords
2393
- - Consider file splitting for large extractions
2394
-
2395
- **Issue**: "Transformation errors"
2396
-
2397
- - Check mapping config field paths
2398
- - Verify required fields are present in GraphQL response
2399
- - Review transformation error details in logs
2400
-
2401
- ---
2402
-
2403
- ## 9. Replication Checklist
2404
-
2405
- **To replicate this template for other entities/formats:**
2406
-
2407
- 1. **File Naming:** Replace `inventory-quantities`, `IQ`, `InventoryQuantity` with your entity name
2408
- 2. **GraphQL Query:** Update query constant and field selection to match your entity schema
2409
- 3. **Mapping Config:** Create new mapping file in `config/` with correct field paths
2410
- 4. **Workflows:** Rename workflow exports to match entity (e.g., `scheduledOrdersExtraction`)
2411
- 5. **Service Function:** Rename main function (e.g., `executeOrderExtraction`)
2412
- 6. **State Key:** Update KV key (e.g., `lastOrderSync`)
2413
- 7. **Output Format:** For XML use `XMLBuilder`, for JSON use `JSON.stringify()`, for CSV use `CSVParserService`
2414
- 8. **Upload Destination:** For SFTP replace `S3DataSource` with `SftpDataSource` (and add `dispose()` in finally block)
2415
- 9. **Job ID Entity Code:** Update entity abbreviation in generateJobId() (e.g., 'ORD' for orders)
2416
- 10. **Result Path:** Update `resultPath` in ExtractionOrchestrator (e.g., `'orders.edges.node'`)
2417
-
2418
- ---
2419
-
2420
- **Pattern**: Enterprise incremental extraction with overlap buffer for inventory quantities
2421
- **Key Learning**: Use ExtractionOrchestrator for auto-pagination, JobTracker for job status, CSVParserService for CSV generation
2422
- **Critical**: Apply 60-second overlap buffer to prevent missed records due to clock skew
2423
- **Buffer Pattern**: Query WITH buffer (`updatedOn >= lastRunTime - 60s`), save WITHOUT buffer (`MAX(updatedOn)`)
2424
- **SDK Services**: ExtractionOrchestrator, UniversalMapper, CSVParserService, S3DataSource, JobTracker
2425
- **Entity-Specific**: Query uses `inventoryQuantities`, resultPath is `'inventoryQuantities.edges.node'`, state key is `lastInventoryQuantitySync`
2426
-
2427
- ---
2428
-
2429
- ### Optional: Backward Pagination (Advanced)
2430
-
2431
- - Default: forward ($first/$after) + pageInfo.hasNextPage.
2432
- - Reverse: define $last/$before and include pageInfo.hasPreviousPage; set direction='backward'.
2433
-
2434
- GraphQL:
2435
-
2436
- ```graphql
2437
- query GetInventoryQuantitiesBackward($retailerId: ID!, $last: Int!, $before: String) {
2438
- inventoryQuantities(retailerId: $retailerId, last: $last, before: $before) {
2439
- edges {
2440
- cursor
2441
- node {
2442
- id
2443
- ref
2444
- updatedOn
2445
- }
2446
- }
2447
- pageInfo {
2448
- hasPreviousPage
2449
- }
2450
- }
2451
- }
2452
- ```
2453
-
2454
- SDK:
2455
-
2456
- ```typescript
2457
- await orchestrator.extract({
2458
- query: INVENTORY_QUANTITIES_BACKWARD_QUERY,
2459
- resultPath: 'inventoryQuantities.edges.node',
2460
- variables: { retailerId },
2461
- pageSize,
2462
- direction: 'backward',
2463
- });
2464
- ```
1
+ ---
2
+ template_id: tpl-extract-inventory-quantities-to-s3-csv
3
+ canonical_filename: template-extraction-inventory-quantities-to-s3-csv.md
4
+ version: 2.0.0
5
+ sdk_version: ^0.1.39
6
+ runtime: versori
7
+ direction: extraction
8
+ source: fluent-graphql
9
+ destination: s3-csv
10
+ entity: inventoryQuantities
11
+ format: csv
12
+ logging: versori
13
+ status: stable
14
+ features:
15
+ - memory-management
16
+ - enhanced-logging
17
+ - pagination-progress
18
+ ---
19
+
20
+ # Template: Extraction - Inventory Quantities to S3 CSV
21
+
22
+ **Template Version:** 2.0.0
23
+ **SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
24
+ **Last Updated:** 2025-01-24
25
+ **Deployment Target:** Versori Platform
26
+
27
+ **🆕 Version 2.0.0 Enhancements:**
28
+ - ✅ **Memory Management** - Clear large result sets after processing batches
29
+ - ✅ **Enhanced Logging** - Pagination progress tracking with emoji indicators (📊, 📥, ✅)
30
+ - ✅ **Pagination Progress** - Real-time page-by-page progress logging with metrics
31
+
32
+ ---
33
+
34
+ ## 📚 STEP 1: Load These Docs (Human Checklist)
35
+
36
+ 1. REQUIRED (load all)
37
+ - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
38
+ - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
39
+ - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
40
+ - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
41
+ - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
42
+ - [ ] fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
43
+
44
+ Copy-paste list (open these):
45
+ fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
46
+ fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
47
+ fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
48
+ fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
49
+ fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
50
+ fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
51
+
52
+ ---
53
+
54
+ ## 📋 STEP 2: Tell Your AI (Prompt)
55
+
56
+ Copy/paste this prompt into your AI tool after loading the documentation above:
57
+
58
+ ```
59
+ I need a Versori scheduled extractor that:
60
+
61
+ 1) Queries Fluent Commerce GraphQL for inventoryQuantities with auto-pagination
62
+ 2) Uses incremental mode with a 60-second overlap buffer stored in Versori KV
63
+ 3) Transforms results using UniversalMapper per mapping JSON
64
+ 4) Generates CSV with CSVParserService and uploads to S3
65
+ 5) Uses native Versori log (LoggingService removed - use native log)
66
+
67
+ Use the loaded docs for SDK specifics and best practices. Keep structure identical to the template.
68
+ ```
69
+
70
+ ---
71
+
72
+ ## 📦 SDK Imports (Verified - Versori Optimized)
73
+
74
+ ```typescript
75
+ import { Buffer } from 'node:buffer';
76
+ import {
77
+ createClient,
78
+ UniversalMapper,
79
+ S3DataSource,
80
+ CSVParserService,
81
+ } from '@fluentcommerce/fc-connect-sdk';
82
+
83
+ import { schedule, http } from '@versori/run';
84
+ ```
85
+
86
+ ---
87
+
88
+ # Versori Scheduled: Inventory Quantities Extraction to S3 CSV (Configurable)
89
+
90
+ **FC Connect SDK Use Case Guide**
91
+
92
+ > SDK: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
93
+ > Version: `npm install @fluentcommerce/fc-connect-sdk@^0.1.39`
94
+
95
+ Context: Scheduled Versori workflow that extracts inventory quantities (detailed quantity records) from Fluent Commerce via GraphQL query with **configurable extraction modes**, transforms with `UniversalMapper`, and writes CSV files to S3 for analytics and reporting.
96
+
97
+ **Pattern**: EXTRACTION (Fluent → S3 CSV)
98
+ **Entity**: inventoryQuantities
99
+ **Complexity**: High | Runtime: Versori Platform (Scheduled)
100
+
101
+ ---
102
+
103
+ ## ⚠️ IMPORTANT: Sample Code for SDK Demonstration Only
104
+
105
+ > **🔴 PRODUCTION WARNING**
106
+ >
107
+ > This guide demonstrates FC Connect SDK capabilities for **extraction and mapping workflows**. The multiple extraction modes (incremental, dateRange, historical) are included to show SDK flexibility and serve as **reference examples**.
108
+ >
109
+ > **✅ PRODUCTION RECOMMENDATION:**
110
+ >
111
+ > - **ONLY use INCREMENTAL mode with scheduled runs** (e.g., daily/hourly)
112
+ > - Incremental mode is safe, efficient, and production-ready
113
+ > - Uses overlap buffer to prevent missed records
114
+ > - Natural rate limiting via timestamps
115
+ >
116
+ > **🚫 DO NOT USE IN PRODUCTION:**
117
+ >
118
+ > - **dateRange mode** - High risk of platform overload with large date windows
119
+ > - **historical mode** - Extremely dangerous, can fetch millions of records
120
+ > - These modes are **demonstration only** to show SDK query patterns
121
+ > - Using these modes on large inventory datasets can crash your runtime and impact platform stability
122
+ >
123
+ > **📝 If you need historical data:**
124
+ >
125
+ > - Run multiple small incremental extractions (e.g., daily for past 30 days)
126
+ > - Use one-time migration scripts with proper monitoring (not scheduled workflows)
127
+ > - Always validate date ranges and implement file splitting
128
+ > - Get explicit approval before running large extractions
129
+ >
130
+ > **This sample code shows HOW to use the SDK - not WHAT to use in production.**
131
+
132
+ ---
133
+
134
+ ## What You'll Build
135
+
136
+ - **Three extraction modes**: Incremental, Date Range, or Historical
137
+ - **State management** with VersoriKVAdapter to track last successful run
138
+ - GraphQL query with auto-pagination
139
+ - UniversalMapper transformation for reporting schema
140
+ - CSV file generation with CSVParserService
141
+ - S3 upload to analytics system
142
+ - **Failure recovery** with timestamp tracking
143
+
144
+ ## Business Use Cases
145
+
146
+ **1. Incremental Daily Sync (Analytics)**
147
+
148
+ - Extract only changed inventory quantities since last run
149
+ - Run daily at 2 AM
150
+ - Minimize data transfer
151
+ - Track changes over time
152
+
153
+ **2. Date Range Extract (Audit)**
154
+
155
+ - Extract quantity changes within specific date window
156
+ - For audits, reconciliation, historical analysis
157
+ - Example: "Show all quantity changes between Jan 1-15"
158
+
159
+ **3. Historical Backfill**
160
+
161
+ - Extract all quantities created within date range
162
+ - For initial data warehouse load
163
+ - One-time backfill operation
164
+
165
+ ## Inventory Quantities vs Positions
166
+
167
+ **InventoryQuantity** = Specific quantity record (retailer-defined types)
168
+
169
+ - Individual records: e.g., LAST_ON_HAND, RESERVED, DELTA, SALE, CORRECTION (plus any custom IQ types)
170
+ - Multiple quantities per product/location
171
+ - Fields: locationRef, skuRef, qty, type, status, expectedOn (if applicable)
172
+ - Used for: Detailed tracking, audit trails
173
+
174
+ **InventoryPosition** = Aggregated on-hand calculation
175
+
176
+ - One position per product/location
177
+ - Calculated `onHand` from all associated quantities
178
+ - Used for: Stock availability, reporting
179
+
180
+ ## SDK Methods Used
181
+
182
+ ```typescript
183
+ import { Buffer } from 'node:buffer';
184
+ import {
185
+ createClient,
186
+ UniversalMapper,
187
+ S3DataSource,
188
+ VersoriKVAdapter,
189
+ CSVParserService,
190
+ } from '@fluentcommerce/fc-connect-sdk';
191
+
192
+ await createClient(ctx);
193
+ await client.graphql({ query, variables, pagination });
194
+ new VersoriKVAdapter(ctx.openKv(':project:'));
195
+ new UniversalMapper(exportMapping);
196
+ const csvParser = new CSVParserService({ includeHeaders: true });
197
+ const csvContent = await csvParser.stringify(rows);
198
+ await s3.uploadFile(key, Buffer.from(csvContent, 'utf8'), options);
199
+ ```
200
+
201
+ ## Activation Variables
202
+
203
+ ```json
204
+ {
205
+ "retailerId": "your-retailer-id",
206
+ "s3BucketName": "inventory-audit-exports",
207
+ "awsAccessKeyId": "AKIAXXXXXXXXXXXX",
208
+ "awsSecretAccessKey": "********",
209
+ "awsRegion": "us-east-1",
210
+ "s3Prefix": "inventory-quantities/daily/",
211
+ "fileNamePrefix": "inventoryquantities",
212
+ "catalogueRef": "DEFAULT_CATALOGUE",
213
+ "pageSize": 200,
214
+ "maxRecords": 100000,
215
+ "extractionMode": "incremental",
216
+ "fallbackStartDate": "2024-01-01T00:00:00Z",
217
+ "overlapBufferSeconds": "60",
218
+ "startDate": "",
219
+ "endDate": ""
220
+ }
221
+ ```
222
+
223
+ ### Variable Reference
224
+
225
+ | Variable | Type | Required | Default | Description |
226
+ |----------|------|----------|---------|-------------|
227
+ | `retailerId` | string | Yes | - | Fluent Commerce retailer ID |
228
+ | `s3BucketName` | string | Yes | - | S3 bucket for CSV export |
229
+ | `awsAccessKeyId` | string | Yes | - | AWS access key with S3 write permissions |
230
+ | `awsSecretAccessKey` | string | Yes | - | AWS secret access key |
231
+ | `awsRegion` | string | Yes | - | AWS region (e.g., `us-east-1`) |
232
+ | `s3Prefix` | string | No | `""` | S3 key prefix (e.g., `inventory-quantities/daily/`) |
233
+ | `fileNamePrefix` | string | No | `"inventoryquantities"` | CSV filename prefix |
234
+ | `catalogueRef` | string | No | - | Filter by catalogue reference (optional) |
235
+ | `pageSize` | number | No | `200` | GraphQL page size (max 500) |
236
+ | `maxRecords` | number | No | `100000` | Maximum records per extraction |
237
+ | `extractionMode` | string | No | `"incremental"` | Extraction mode: `incremental`, `dateRange`, or `historical` |
238
+ | `fallbackStartDate` | string | No | `"2024-01-01T00:00:00Z"` | Fallback date if no state exists |
239
+ | `overlapBufferSeconds` | number | No | `60` | Overlap buffer to prevent missed records (seconds) |
240
+ | `startDate` | string | No | - | Manual start date (for `dateRange`/`historical` modes) |
241
+ | `endDate` | string | No | - | Manual end date (for `dateRange`/`historical` modes) |
242
+
243
+ ### Extraction Mode Configuration
244
+
245
+ **Mode 1: Incremental (default)**
246
+
247
+ ```json
248
+ {
249
+ "extractionMode": "incremental",
250
+ "fallbackStartDate": "2024-01-01T00:00:00Z"
251
+ }
252
+ ```
253
+
254
+ Extracts quantities with `updatedOn > lastRunTime`. Ideal for daily syncs.
255
+
256
+ **Mode 2: Date Range**
257
+
258
+ ```json
259
+ {
260
+ "extractionMode": "dateRange",
261
+ "startDate": "2025-01-01T00:00:00Z",
262
+ "endDate": "2025-01-15T23:59:59Z"
263
+ }
264
+ ```
265
+
266
+ Extracts quantities updated between `startDate` and `endDate`. Ideal for audits.
267
+
268
+ **Mode 3: Historical**
269
+
270
+ ```json
271
+ {
272
+ "extractionMode": "historical",
273
+ "startDate": "2024-01-01T00:00:00Z",
274
+ "endDate": "2024-12-31T23:59:59Z"
275
+ }
276
+ ```
277
+
278
+ Extracts quantities created between `startDate` and `endDate` using `createdOn` filter.
279
+
280
+ ## ⚠️ Production Safety & Guardrails
281
+
282
+ ### Critical: Extraction Mode Selection
283
+
284
+ **🟢 RECOMMENDED: Incremental Mode (Production)**
285
+
286
+ - Safe for automated schedules
287
+ - Natural rate limiting via timestamps
288
+ - Predictable resource usage
289
+ - **Use this for all production workflows**
290
+
291
+ **🟡 CAUTION: Date Range Mode (Audit/Backfill)**
292
+
293
+ - **Maximum 30-day window enforced**
294
+ - Use for specific audit requests only
295
+ - Run during off-peak hours
296
+ - Monitor resource usage
297
+
298
+ **🔴 DANGER: Historical Mode (One-Time Only)**
299
+
300
+ - **Maximum 90-day window enforced**
301
+ - **Requires explicit approval**
302
+ - **Risk of platform overload**
303
+ - Can fetch millions of records
304
+ - Use multiple small incremental runs instead
305
+ - Only for initial data migration
306
+
307
+ ### Date Range Validation (Required)
308
+
309
+ ```typescript
310
+ // Validate date range limits to prevent platform overload
311
+ function validateDateRange(mode, startDate, endDate) {
312
+ if (mode === 'incremental') return { valid: true };
313
+
314
+ if (!startDate || !endDate) {
315
+ return {
316
+ valid: false,
317
+ error: `${mode} mode requires both startDate and endDate`,
318
+ };
319
+ }
320
+
321
+ const start = new Date(startDate);
322
+ const end = new Date(endDate);
323
+ const daysDiff = (end - start) / (1000 * 60 * 60 * 24);
324
+
325
+ // Guardrail: Maximum date ranges
326
+ const maxDays = mode === 'dateRange' ? 30 : 90;
327
+
328
+ if (daysDiff > maxDays) {
329
+ return {
330
+ valid: false,
331
+ error: `${mode} mode: Maximum ${maxDays}-day window. Requested: ${Math.round(daysDiff)} days. Use multiple smaller extractions or incremental mode.`,
332
+ recommendation: `Split into ${Math.ceil(daysDiff / maxDays)} separate extractions of ${maxDays} days each.`,
333
+ };
334
+ }
335
+
336
+ if (daysDiff < 0) {
337
+ return { valid: false, error: 'endDate must be after startDate' };
338
+ }
339
+
340
+ return { valid: true };
341
+ }
342
+ ```
343
+
344
+ ### File Splitting Configuration
345
+
346
+ Large extractions must split into multiple files to prevent memory issues and upload failures.
347
+
348
+ ```json
349
+ {
350
+ "maxRecordsPerFile": 50000,
351
+ "maxFileSizeMB": 100,
352
+ "enableFileSplitting": true
353
+ }
354
+ ```
355
+
356
+ **File Naming Pattern:**
357
+
358
+ ```
359
+ {prefix}inventory-quantities-{timestamp}-part-{sequence}.csv
360
+
361
+ Examples:
362
+ inventory-quantities/daily/inventory-quantities-2025-01-22T14-30-00Z-part-001.csv
363
+ inventory-quantities/daily/inventory-quantities-2025-01-22T14-30-00Z-part-002.csv
364
+ inventory-quantities/daily/inventory-quantities-2025-01-22T14-30-00Z-manifest.json
365
+ ```
366
+
367
+ **Manifest File (auto-generated):**
368
+
369
+ ```json
370
+ {
371
+ "extractionId": "inventory-quantities-2025-01-22T14-30-00Z",
372
+ "totalRecords": 127543,
373
+ "totalFiles": 3,
374
+ "files": [
375
+ {
376
+ "filename": "inventory-quantities-2025-01-22T14-30-00Z-part-001.csv",
377
+ "recordCount": 50000,
378
+ "s3Key": "inventory-quantities/daily/inventory-quantities-2025-01-22T14-30-00Z-part-001.csv"
379
+ },
380
+ {
381
+ "filename": "inventory-quantities-2025-01-22T14-30-00Z-part-002.csv",
382
+ "recordCount": 50000,
383
+ "s3Key": "inventory-quantities/daily/inventory-quantities-2025-01-22T14-30-00Z-part-002.csv"
384
+ },
385
+ {
386
+ "filename": "inventory-quantities-2025-01-22T14-30-00Z-part-003.csv",
387
+ "recordCount": 27543,
388
+ "s3Key": "inventory-quantities/daily/inventory-quantities-2025-01-22T14-30-00Z-part-003.csv"
389
+ }
390
+ ],
391
+ "extractionMode": "dateRange",
392
+ "dateRange": {
393
+ "from": "2025-01-01T00:00:00Z",
394
+ "to": "2025-01-31T23:59:59Z"
395
+ },
396
+ "completedAt": "2025-01-22T14:35:27Z"
397
+ }
398
+ ```
399
+
400
+ ### Hard Limits (Enforced)
401
+
402
+ ```typescript
403
+ const SAFETY_LIMITS = {
404
+ // Maximum records per single extraction
405
+ MAX_RECORDS_TOTAL: 500000, // 500k hard limit
406
+
407
+ // Maximum records per file before splitting
408
+ MAX_RECORDS_PER_FILE: 50000, // 50k per file
409
+
410
+ // Maximum file size before splitting
411
+ MAX_FILE_SIZE_MB: 100, // 100MB per file
412
+
413
+ // Date range limits
414
+ MAX_DATE_RANGE_DAYS: 30, // dateRange mode
415
+ MAX_HISTORICAL_DAYS: 90, // historical mode
416
+
417
+ // Pagination limits
418
+ MAX_PAGE_SIZE: 500, // Fluent API limit
419
+ RECOMMENDED_PAGE_SIZE: 200, // Balance throughput/memory
420
+
421
+ // Memory management
422
+ CHUNK_SIZE: 10000, // Process in chunks
423
+ };
424
+ ```
425
+
426
+ ### Memory-Safe Implementation Pattern
427
+
428
+ ```typescript
429
+ // Process large extractions in chunks to prevent OOM
430
+ async function processLargeExtraction(edges, mapper, csvParser, s3, options) {
431
+ const CHUNK_SIZE = 10000;
432
+ const MAX_RECORDS_PER_FILE = options.maxRecordsPerFile || 50000;
433
+
434
+ let fileSequence = 1;
435
+ let currentFileRecords = [];
436
+ const manifestFiles = [];
437
+
438
+ for (let i = 0; i < edges.length; i += CHUNK_SIZE) {
439
+ const chunk = edges.slice(i, i + CHUNK_SIZE);
440
+
441
+ // Bulk mapping for chunk
442
+ const chunkNodes = chunk.map(edge => edge.node);
443
+ const mappingResult = await mapper.map(chunkNodes);
444
+
445
+ if (!mappingResult.success) {
446
+ const mappingErrors = mappingResult.errors || ['Unknown mapping failure'];
447
+ log.error('Chunk mapping failed', {
448
+ chunkIndex: i / CHUNK_SIZE,
449
+ errorCount: mappingErrors.length,
450
+ sampleErrors: mappingErrors.slice(0, 3),
451
+ });
452
+ throw new Error(`Mapping failed: ${mappingErrors[0] || 'Unknown error'}`);
453
+ }
454
+
455
+ const transformedChunk = mappingResult.data || [];
456
+ const mappingErrors = mappingResult.errors || [];
457
+
458
+ if (mappingErrors.length > 0) {
459
+ log.warn('Some records in chunk failed transformation', {
460
+ chunkIndex: i / CHUNK_SIZE,
461
+ errorCount: mappingErrors.length,
462
+ });
463
+ }
464
+
465
+ if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
466
+ log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
467
+ chunkIndex: i / CHUNK_SIZE,
468
+ skippedFields: mappingResult.skippedFields,
469
+ note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
470
+ });
471
+ }
472
+
473
+ // Add to current file, handling splits
474
+ for (const record of transformedChunk) {
475
+ currentFileRecords.push(record);
476
+
477
+ // Split file when limit reached
478
+ if (currentFileRecords.length >= MAX_RECORDS_PER_FILE) {
479
+ const fileInfo = await writeFileToS3(
480
+ currentFileRecords,
481
+ fileSequence++,
482
+ csvParser,
483
+ s3,
484
+ options
485
+ );
486
+ manifestFiles.push(fileInfo);
487
+ currentFileRecords = []; // Reset for next file
488
+ }
489
+ }
490
+ }
491
+
492
+ // Write remaining records
493
+ if (currentFileRecords.length > 0) {
494
+ const fileInfo = await writeFileToS3(
495
+ currentFileRecords,
496
+ fileSequence++,
497
+ csvParser,
498
+ s3,
499
+ options
500
+ );
501
+ manifestFiles.push(fileInfo);
502
+ }
503
+
504
+ // Write manifest
505
+ await writeManifest(manifestFiles, s3, options);
506
+
507
+ return manifestFiles;
508
+ }
509
+ ```
510
+
511
+ ### Enterprise Time Buffer Configuration
512
+
513
+ ```json
514
+ {
515
+ "overlapBufferSeconds": "60"
516
+ }
517
+ ```
518
+
519
+ **Default: 60 seconds (recommended for most deployments)**
520
+
521
+ **Purpose**: Prevents missed records due to:
522
+
523
+ - **Clock skew** between Fluent API servers (typically 1-5 seconds)
524
+ - **Transaction timing** - records updated during query execution
525
+ - **Race conditions** - records updated between extraction runs
526
+
527
+ **How It Works**:
528
+
529
+ - **Query**: Uses `updatedOn >= (lastRunTime - 60 seconds)`
530
+ - **Save**: Stores `MAX(updatedOn)` WITHOUT buffer
531
+ - **Result**: Records from the last minute of previous extraction are included again
532
+
533
+ **Buffer Sizes by Deployment**:
534
+
535
+ - `30` - Low-latency single-region (minimal clock skew expected)
536
+ - `60` - **Standard production** (recommended default)
537
+ - `300` - Cross-region deployments or high-latency networks
538
+
539
+ **Duplicate Handling**: Downstream systems should upsert by `quantity_id` (idempotent). Duplicates are safe and expected.
540
+
541
+ ### Timezone Handling
542
+
543
+ **All timestamps are in ISO 8601 format (UTC)**:
544
+
545
+ ```typescript
546
+ // Input: ISO 8601 UTC timestamp
547
+ const timestamp = '2025-01-22T14:30:00.000Z';
548
+
549
+ // JavaScript Date operations preserve UTC
550
+ new Date(timestamp).toISOString();
551
+ // Returns: "2025-01-22T14:30:00.000Z" (same format)
552
+
553
+ new Date(timestamp).getTime();
554
+ // Returns: 1737558600000 (UTC epoch milliseconds)
555
+
556
+ // Subtract 60 seconds for buffer
557
+ const buffered = new Date(new Date(timestamp).getTime() - 60000).toISOString();
558
+ // Returns: "2025-01-22T14:29:00.000Z"
559
+ ```
560
+
561
+ **Key Points**:
562
+
563
+ - Fluent API returns all timestamps in UTC
564
+ - `.getTime()` returns UTC epoch milliseconds
565
+ - Buffer arithmetic is done in milliseconds
566
+ - `.toISOString()` converts back to ISO 8601 UTC
567
+ - No timezone conversion needed
568
+
569
+ ## Export Mapping Configuration
570
+
571
+ Create file: `./config/inventory-quantities.export.json`
572
+
573
+ ```json
574
+ {
575
+ "name": "inventory-quantities.export",
576
+ "version": "1.0.0",
577
+ "description": "Fluent Inventory Quantities → CSV Export Mapping",
578
+ "fields": {
579
+ "quantity_id": { "source": "id", "required": true, "resolver": "sdk.trim" },
580
+ "quantity_ref": { "source": "ref", "required": true, "resolver": "sdk.trim" },
581
+ "catalogue_ref": { "source": "catalogue.ref", "required": false, "resolver": "sdk.trim" },
582
+ "catalogue_name": { "source": "catalogue.name", "required": false, "resolver": "sdk.trim" },
583
+ "location": { "source": "locationRef", "required": true, "resolver": "sdk.trim" },
584
+ "sku": { "source": "skuRef", "required": true, "resolver": "sdk.trim" },
585
+ "quantity": { "source": "qty", "required": true, "resolver": "sdk.parseInt" },
586
+ "type": { "source": "type", "required": true, "resolver": "sdk.uppercase" },
587
+ "status": { "source": "status", "required": true, "resolver": "sdk.uppercase" },
588
+ "expected_on": { "source": "expectedOn", "resolver": "sdk.formatDate" },
589
+ "created_on": { "source": "createdOn", "resolver": "sdk.formatDate" },
590
+ "updated_on": { "source": "updatedOn", "required": true, "resolver": "sdk.formatDate" }
591
+ }
592
+ }
593
+ ```
594
+
595
+ ## Mapping & Resolvers Explained
596
+
597
+ This section explains how the SDK transforms raw GraphQL data into your CSV export format using **UniversalMapper** and **SDK resolvers**.
598
+
599
+ ### SDK Resolvers Used
600
+
601
+ | Field | Resolver | Why? | Example Transformation |
602
+ | ---------------- | ---------------- | ------------------------------------------ | ----------------------------------------------- |
603
+ | `quantity_id` | `sdk.trim` | Clean quantity IDs from whitespace | `" Q001 "` → `"Q001"` |
604
+ | `quantity_ref` | `sdk.trim` | Clean quantity references | `" QTY-REF-001 "` → `"QTY-REF-001"` |
605
+ | `catalogue_ref` | `sdk.trim` | Clean catalogue references | `" DEFAULT_CATALOGUE "` → `"DEFAULT_CATALOGUE"` |
606
+ | `catalogue_name` | `sdk.trim` | Clean catalogue names | `" Default Catalogue "` → `"Default Catalogue"` |
607
+ | `location` | `sdk.trim` | Clean location references | `" DC01 "` → `"DC01"` |
608
+ | `sku` | `sdk.trim` | Clean SKU references | `" SKU-001 "` → `"SKU-001"` |
609
+ | `quantity` | `sdk.parseInt` | Parse quantity as integer for calculations | `"100"` → `100` |
610
+ | `type` | `sdk.uppercase` | Normalize type codes | `"available"` → `"AVAILABLE"` |
611
+ | `status` | `sdk.uppercase` | Normalize status codes | `"active"` → `"ACTIVE"` |
612
+ | `expected_on` | `sdk.formatDate` | Format dates for CSV export | `"2025-01-30T00:00:00.000Z"` → `"2025-01-30"` |
613
+ | `created_on` | `sdk.formatDate` | Format created timestamps | `"2025-01-15T10:00:00.000Z"` → `"2025-01-15"` |
614
+ | `updated_on` | `sdk.formatDate` | Format updated timestamps for tracking | `"2025-01-22T08:30:00.000Z"` → `"2025-01-22"` |
615
+
616
+ ### Transformation Flow
617
+
618
+ ```typescript
619
+ // 1. GraphQL Response (raw data from Fluent Commerce)
620
+ const rawQuantity = {
621
+ id: ' Q001 ',
622
+ ref: ' QTY-REF-001 ',
623
+ locationRef: ' DC01 ',
624
+ skuRef: ' SKU-001 ',
625
+ qty: '100',
626
+ type: 'available',
627
+ status: 'active',
628
+ expectedOn: null,
629
+ createdOn: '2025-01-15T10:00:00.000Z',
630
+ updatedOn: '2025-01-22T08:30:00.000Z',
631
+ catalogue: {
632
+ ref: ' DEFAULT_CATALOGUE ',
633
+ name: ' Default Catalogue ',
634
+ },
635
+ };
636
+
637
+ // 2. UniversalMapper applies SDK resolvers
638
+ const mapper = new UniversalMapper(inventoryQuantitiesExportMapping);
639
+ const result = await mapper.map(rawQuantity);
640
+
641
+ // 3. Transformed Output (clean, normalized for CSV)
642
+ const transformedQuantity = {
643
+ quantity_id: 'Q001',
644
+ quantity_ref: 'QTY-REF-001',
645
+ catalogue_ref: 'DEFAULT_CATALOGUE',
646
+ catalogue_name: 'Default Catalogue',
647
+ location: 'DC01',
648
+ sku: 'SKU-001',
649
+ quantity: 100,
650
+ type: 'AVAILABLE',
651
+ status: 'ACTIVE',
652
+ expected_on: '', // null → empty string
653
+ created_on: '2025-01-15',
654
+ updated_on: '2025-01-22',
655
+ };
656
+ ```
657
+
658
+ ### Custom Resolvers for Inventory Quantity-Specific Logic
659
+
660
+ While the mapping above uses built-in SDK resolvers, you can extend with custom business logic:
661
+
662
+ ```typescript
663
+ const customResolvers = {
664
+ /**
665
+ * Validate that quantity values are positive
666
+ */
667
+ 'custom.validateQuantity': (qty: any) => {
668
+ const parsed = parseInt(qty) || 0;
669
+ return parsed >= 0 ? parsed : 0; // Ensure non-negative
670
+ },
671
+
672
+ /**
673
+ * Add human-readable type descriptions for reporting
674
+ */
675
+ 'custom.enrichQuantityType': (type: string) => {
676
+ const typeDescriptions: Record<string, string> = {
677
+ LAST_ON_HAND: 'Last recorded on-hand quantity',
678
+ RESERVED: 'Reserved against orders',
679
+ DELTA: 'Incremental change (adjustment delta)',
680
+ SALE: 'Quantity decreased due to sale',
681
+ CORRECTION: 'Manual correction entry',
682
+ };
683
+ return typeDescriptions[(type || '').toUpperCase()] || type;
684
+ },
685
+
686
+ /**
687
+ * Check if expected date is in the future
688
+ */
689
+ 'custom.isExpectedInFuture': (expectedOn: string) => {
690
+ if (!expectedOn) return false;
691
+ return new Date(expectedOn) > new Date();
692
+ },
693
+
694
+ /**
695
+ * Calculate days until expected arrival
696
+ */
697
+ 'custom.calculateDaysUntilExpected': (expectedOn: string) => {
698
+ if (!expectedOn) return null;
699
+ const expected = new Date(expectedOn);
700
+ const today = new Date();
701
+ const diffMs = expected.getTime() - today.getTime();
702
+ return Math.ceil(diffMs / (1000 * 60 * 60 * 24));
703
+ },
704
+
705
+ /**
706
+ * Validate status-type combinations
707
+ */
708
+ 'custom.validateQuantityStatus': (_quantity: any) => {
709
+ // Example placeholder – adapt rules to your retailer-defined IQ types
710
+ return 'VALID';
711
+ },
712
+ };
713
+
714
+ // Use custom resolvers with UniversalMapper
715
+ const mapper = new UniversalMapper(inventoryQuantitiesExportMapping, {
716
+ customResolvers,
717
+ });
718
+ ```
719
+
720
+ ### Available SDK Resolvers
721
+
722
+ The SDK provides these built-in resolvers (no custom code needed):
723
+
724
+ **String Transformations:**
725
+
726
+ - `sdk.trim` - Remove leading/trailing whitespace
727
+ - `sdk.uppercase` - Convert to uppercase
728
+ - `sdk.lowercase` - Convert to lowercase
729
+ - `sdk.toString` - Convert to string
730
+
731
+ **Number Parsing:**
732
+
733
+ - `sdk.parseInt` - Parse as integer
734
+ - `sdk.parseFloat` - Parse as decimal
735
+ - `sdk.number` - Parse as number (auto-detect int/float)
736
+
737
+ **Date Formatting:**
738
+
739
+ - `sdk.formatDate` - ISO 8601 date only (YYYY-MM-DD)
740
+ - `sdk.formatDateShort` - Short date format
741
+ - `sdk.parseDate` - Parse various date formats
742
+
743
+ **Type Conversions:**
744
+
745
+ - `sdk.boolean` - Convert to boolean
746
+ - `sdk.parseJson` - Parse JSON strings
747
+ - `sdk.toJson` - Convert to JSON string
748
+
749
+ **Utilities:**
750
+
751
+ - `sdk.identity` - Return value unchanged
752
+ - `sdk.coalesce` - Return first non-null value
753
+
754
+ See [Universal Mapping Guide](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) for complete resolver documentation.
755
+
756
+ ## GraphQL Query
757
+
758
+ ```graphql
759
+ query GetInventoryQuantities(
760
+ $retailerId: ID!
761
+ $updatedAfter: DateTime
762
+ $createdAfter: DateTime
763
+ $first: Int!
764
+ $after: String
765
+ ) {
766
+ inventoryQuantities(
767
+ retailerId: $retailerId
768
+ updatedOn: { after: $updatedAfter }
769
+ createdOn: { after: $createdAfter }
770
+ first: $first
771
+ after: $after
772
+ ) {
773
+ edges {
774
+ node {
775
+ id
776
+ ref
777
+ locationRef
778
+ skuRef
779
+ qty
780
+ type
781
+ status
782
+ expectedOn
783
+ createdOn
784
+ updatedOn
785
+ catalogue {
786
+ ref
787
+ name
788
+ }
789
+ }
790
+ cursor
791
+ }
792
+ pageInfo {
793
+ hasNextPage
794
+ # Note: Fluent doesn't return endCursor/startCursor - cursors are in edges[].cursor
795
+ }
796
+ }
797
+ }
798
+ ```
799
+
800
+ ## Guardrails Implementation (Required)
801
+
802
+ ```typescript
803
+ // Overlap buffer (safety window)
804
+ const overlapBufferSeconds = parseInt(
805
+ ctx.activation?.getVariable('overlapBufferSeconds') || '60',
806
+ 10
807
+ );
808
+ const OVERLAP_BUFFER_MS = overlapBufferSeconds * 1000;
809
+
810
+ // Read last successful run and apply buffer
811
+ const kv = new VersoriKVAdapter(ctx.openKv(':project:'));
812
+ const stateKey = ['extraction', 'inventory-quantities-csv', 'lastRunTime'];
813
+ const lastRunState = await kv.get(stateKey);
814
+ const rawLastRunTime = lastRunState?.value?.timestamp || fallbackStartDate;
815
+ const bufferedLastRunTime = new Date(
816
+ new Date(rawLastRunTime).getTime() - OVERLAP_BUFFER_MS
817
+ ).toISOString();
818
+
819
+ // Query WITH buffer
820
+ const result = await client.graphql({
821
+ query: INVENTORY_QUANTITIES_QUERY,
822
+ variables: {
823
+ retailerId,
824
+ updatedAfter: bufferedLastRunTime,
825
+ first: pageSize,
826
+ },
827
+ pagination: { maxRecords },
828
+ });
829
+
830
+ const edges = result.data?.inventoryQuantities?.edges || [];
831
+
832
+ // 🛡️ GUARDRAIL: Validate extraction size limits
833
+ const MAX_RECORDS_PER_RUN = 500000;
834
+ const ESTIMATED_BYTES_PER_RECORD = 300; // Smaller than positions
835
+ const estimatedSizeMB = (edges.length * ESTIMATED_BYTES_PER_RECORD) / (1024 * 1024);
836
+ const MAX_CSV_SIZE_MB = 100;
837
+
838
+ if (edges.length > MAX_RECORDS_PER_RUN) {
839
+ log.error('Extraction limit exceeded', {
840
+ recordCount: edges.length,
841
+ maxAllowed: MAX_RECORDS_PER_RUN,
842
+ });
843
+ return {
844
+ success: false,
845
+ error: `Extraction limit exceeded: ${edges.length} records (max: ${MAX_RECORDS_PER_RUN})`,
846
+ recommendation: `Split into smaller extractions or increase extraction frequency`,
847
+ recordCount: edges.length,
848
+ maxAllowed: MAX_RECORDS_PER_RUN,
849
+ };
850
+ }
851
+
852
+ if (estimatedSizeMB > MAX_CSV_SIZE_MB) {
853
+ log.warn('CSV size approaching limit', {
854
+ estimatedSizeMB: estimatedSizeMB.toFixed(2),
855
+ maxAllowed: MAX_CSV_SIZE_MB,
856
+ });
857
+ }
858
+
859
+ log.info('Extraction limits validated', {
860
+ recordCount: edges.length,
861
+ estimatedSizeMB: estimatedSizeMB.toFixed(2),
862
+ withinLimits: true,
863
+ });
864
+
865
+ // Transform with UniversalMapper
866
+ const mapper = new UniversalMapper(inventoryQuantitiesExportMapping);
867
+ const transformedRecords: any[] = [];
868
+ for (const edge of edges) {
869
+ const mapped = await mapper.map(edge.node);
870
+ if (mapped.success) {
871
+ transformedRecords.push(mapped.data);
872
+ }
873
+ }
874
+
875
+ // Save state WITHOUT buffer (use MAX(updatedOn))
876
+ const maxUpdatedOn = transformedRecords.reduce((max, r) => {
877
+ const t = new Date(r.updated_on).getTime();
878
+ return t > max ? t : max;
879
+ }, new Date(rawLastRunTime).getTime());
880
+
881
+ await kv.set(stateKey, {
882
+ timestamp: new Date(maxUpdatedOn).toISOString(),
883
+ recordCount: transformedRecords.length,
884
+ extractedAt: new Date().toISOString(),
885
+ overlapBufferSeconds,
886
+ });
887
+
888
+ // Date range guardrails (if you add dateRange/historical modes)
889
+ function validateDateRange(mode: 'dateRange' | 'historical', from: string, to: string) {
890
+ const start = new Date(from);
891
+ const end = new Date(to);
892
+ const daysDiff = (end.getTime() - start.getTime()) / (1000 * 60 * 60 * 24);
893
+ const maxDays = mode === 'dateRange' ? 30 : 90;
894
+ if (daysDiff > maxDays) {
895
+ return {
896
+ valid: false,
897
+ error: `${mode} mode: Maximum ${maxDays}-day window. Requested: ${Math.round(daysDiff)} days.`,
898
+ };
899
+ }
900
+ if (daysDiff < 0) return { valid: false, error: 'endDate must be after startDate' };
901
+ return { valid: true };
902
+ }
903
+ ```
904
+
905
+ ---
906
+
907
+ ## Versori Workflows Structure
908
+
909
+ **Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
910
+
911
+ **Trigger Types:**
912
+ - **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
913
+ - **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
914
+ - **`workflow()`** → Durable workflows (advanced, rarely used)
915
+
916
+ **Execution Steps (chained to triggers):**
917
+ - **`http()`** → External API calls (chained from schedule/webhook)
918
+ - **`fn()`** → Internal processing (chained from schedule/webhook)
919
+
920
+ ### Recommended Project Structure
921
+
922
+ ```
923
+ inventory-quantities-extraction/
924
+ ├── index.ts # Entry point - exports all workflows
925
+ └── src/
926
+ ├── workflows/
927
+ │ ├── scheduled/
928
+ │ │ └── daily-inventory-quantities-extraction.ts # Scheduled: Daily extraction
929
+ │ │
930
+ │ └── webhook/
931
+ │ ├── adhoc-inventory-quantities-extraction.ts # Webhook: Manual trigger
932
+ │ └── job-status-check.ts # Webhook: Status query
933
+
934
+ ├── services/
935
+ │ └── inventory-quantities-extraction.service.ts # Shared orchestration logic (reusable)
936
+
937
+ └── config/
938
+ └── inventory-quantities.export.csv.json # Mapping configuration
939
+ ```
940
+
941
+ ---
942
+
943
+ ````csv
944
+ quantity_id,quantity_ref,catalogue_ref,catalogue_name,location,sku,quantity,type,status,expected_on,created_on,updated_on
945
+ Q001,QTY-REF-001,DEFAULT_CATALOGUE,Default Catalogue,DC01,SKU-001,100,AVAILABLE,ACTIVE,,2025-01-15T10:00:00Z,2025-01-22T08:30:00Z
946
+ Q002,QTY-REF-002,DEFAULT_CATALOGUE,Default Catalogue,DC01,SKU-001,50,RESERVED,ACTIVE,,2025-01-16T11:00:00Z,2025-01-22T09:15:00Z
947
+ Q003,QTY-REF-003,DEFAULT_CATALOGUE,Default Catalogue,DC02,SKU-002,200,EXPECTED,CREATED,2025-01-30T00:00:00Z,2025-01-17T12:00:00Z,2025-01-22T10:00:00Z
948
+ Q004,QTY-REF-004,DEFAULT_CATALOGUE,Default Catalogue,STORE-NYC,SKU-003,25,AVAILABLE,ACTIVE,,2025-01-18T13:00:00Z,2025-01-22T11:00:00Z
949
+
950
+ ## Advanced Mapping Patterns
951
+
952
+ ### Array Mapping (Preserving Nested Structure)
953
+
954
+ For nested data structures, use `isArray: true` pattern:
955
+
956
+ ```json
957
+ {
958
+ "fields": {
959
+ "ref": { "source": "ref", "required": true },
960
+ "relatedItems": {
961
+ "source": "items",
962
+ "isArray": true,
963
+ "fields": {
964
+ "itemRef": { "source": "ref", "required": true },
965
+ "value": { "source": "value", "resolver": "sdk.parseFloat" }
966
+ }
967
+ }
968
+ }
969
+ }
970
+ ````
971
+
972
+ **When to use**:
973
+
974
+ - **Flattened structure**: Simpler, easier for downstream systems
975
+ - **Nested with arrays**: Complex data, preserves relationships
976
+
977
+ ### Nested Object Mapping
978
+
979
+ **Option 1: Flattened paths** (recommended):
980
+
981
+ ```json
982
+ {
983
+ "fields": {
984
+ "location_ref": { "source": "location.ref" },
985
+ "location_name": { "source": "location.name" }
986
+ }
987
+ }
988
+ ```
989
+
990
+ **Option 2: Nested object definition**:
991
+
992
+ ```json
993
+ {
994
+ "fields": {
995
+ "location": {
996
+ "fields": {
997
+ "ref": { "source": "location.ref" },
998
+ "name": { "source": "location.name" }
999
+ }
1000
+ }
1001
+ }
1002
+ }
1003
+ ```
1004
+
1005
+ ## Error Handling Strategies
1006
+
1007
+ ### Handling Mapping Failures
1008
+
1009
+ **Strategy 1: Fail-fast (strict)**:
1010
+
1011
+ ```typescript
1012
+ if (errors.length > 0) {
1013
+ throw new Error(`${errors.length} records failed mapping validation`);
1014
+ }
1015
+ ```
1016
+
1017
+ **Strategy 2: Threshold-based (recommended)**:
1018
+
1019
+ ```typescript
1020
+ const errorRate = errors.length / transformed.length;
1021
+ if (errorRate > 0.05) {
1022
+ // 5% threshold
1023
+ throw new Error(`Error rate too high: ${(errorRate * 100).toFixed(1)}%`);
1024
+ }
1025
+ ```
1026
+
1027
+ **Strategy 3: Upload error manifest**:
1028
+
1029
+ ```typescript
1030
+ if (errors.length > 0) {
1031
+ const errorManifest = {
1032
+ extractionTimestamp: new Date().toISOString(),
1033
+ totalErrors: errors.length,
1034
+ errors: errors.map(e => ({ record: e.record, errors: e.errors })),
1035
+ };
1036
+ // Upload to storage for review
1037
+ }
1038
+ ```
1039
+
1040
+ ### State Management with Partial Failures
1041
+
1042
+ **Recommended**: Only update state if extraction succeeded:
1043
+
1044
+ ```typescript
1045
+ if (errors.length === 0) {
1046
+ await kv.set(stateKey, { timestamp: newTimestamp });
1047
+ log.info('State updated - all records successful');
1048
+ } else {
1049
+ log.warn('State NOT updated - will retry next run', {
1050
+ failedRecords: errors.length,
1051
+ willRetryNextRun: true,
1052
+ });
1053
+ }
1054
+ ```
1055
+
1056
+ ## GraphQL Query Validation & Testing
1057
+
1058
+ ### Schema Validation Workflow
1059
+
1060
+ **Step 1: Introspect schema**
1061
+
1062
+ ```bash
1063
+ npx fc-connect introspect-schema \
1064
+ --url https://your-instance.api.fluentcommerce.com/graphql \
1065
+ --output fluent-schema.json
1066
+ ```
1067
+
1068
+ **Step 2: Validate mapping**
1069
+
1070
+ ```bash
1071
+ npx fc-connect validate-schema \
1072
+ --mapping ./config/mapping.json \
1073
+ --schema ./fluent-schema.json
1074
+ ```
1075
+
1076
+ **Step 3: Analyze coverage**
1077
+
1078
+ ```bash
1079
+ npx fc-connect analyze-coverage \
1080
+ --mapping ./config/mapping.json \
1081
+ --schema ./fluent-schema.json
1082
+ ```
1083
+
1084
+ ### GraphQL Pagination Explained
1085
+
1086
+ The SDK handles pagination automatically:
1087
+
1088
+ ```typescript
1089
+ await client.graphql({
1090
+ query: QUERY,
1091
+ variables: { first: pageSize },
1092
+ pagination: { maxRecords }, // SDK handles cursors automatically
1093
+ });
1094
+ ```
1095
+
1096
+ ## Date Format Handling
1097
+
1098
+ | Format | Resolver | Output | Use Case |
1099
+ | -------- | --------------------- | -------------------------- | --------- |
1100
+ | CSV/JSON | `sdk.formatDate` | `2025-01-22T14:30:00.000Z` | ISO 8601 |
1101
+ | CSV/JSON | `sdk.formatDateShort` | `2025-01-22` | Date only |
1102
+ | CSV/JSON | `sdk.toString` | Pass through | As-is |
1103
+
1104
+ ## Monitoring & Alerting
1105
+
1106
+ ### Key Metrics to Track
1107
+
1108
+ ```typescript
1109
+ const metrics = {
1110
+ extractionDurationMs: Date.now() - startTime,
1111
+ recordCount: edges.length,
1112
+ transformedCount: transformed.length,
1113
+ failedCount: errors.length,
1114
+ errorRate: ((errors.length / edges.length) * 100).toFixed(2) + '%',
1115
+ fileSizeMB: (buffer.length / (1024 * 1024)).toFixed(2),
1116
+ lastRunTime: rawLastRunTime,
1117
+ newTimestamp: newTimestamp,
1118
+ };
1119
+ log.info('Extraction complete', metrics);
1120
+ ```
1121
+
1122
+ ### Alert Thresholds
1123
+
1124
+ ```typescript
1125
+ const ALERTS = {
1126
+ EXTRACTION_DURATION_MS: 5 * 60 * 1000, // 5 minutes
1127
+ MAX_ERROR_RATE: 0.05, // 5%
1128
+ MAX_FILE_SIZE_MB: 150, // 150MB
1129
+ MAX_RECORDS_PER_RUN: 100000, // Adjust per entity
1130
+ };
1131
+ ```
1132
+
1133
+ ## Testing Checklist
1134
+
1135
+ **Before production deployment:**
1136
+
1137
+ ### 1. Schema Validation
1138
+
1139
+ - [ ] Run `npx fc-connect introspect-schema`
1140
+ - [ ] Run `npx fc-connect validate-schema`
1141
+ - [ ] Run `npx fc-connect analyze-coverage`
1142
+ - [ ] Verify all `source` paths exist
1143
+
1144
+ ### 2. Mapping Testing
1145
+
1146
+ - [ ] Test with sample data (maxRecords=10)
1147
+ - [ ] Verify required fields populated
1148
+ - [ ] Verify SDK resolvers work correctly
1149
+ - [ ] Test custom resolvers with edge cases
1150
+
1151
+ ### 3. Error Handling
1152
+
1153
+ - [ ] Test with invalid data
1154
+ - [ ] Verify error collection
1155
+ - [ ] Test error threshold logic
1156
+
1157
+ ### 4. State Management
1158
+
1159
+ - [ ] Verify overlap buffer prevents misses
1160
+ - [ ] Test state recovery after failure
1161
+ - [ ] Verify timestamp saved WITHOUT buffer
1162
+
1163
+ ### 5. File Operations
1164
+
1165
+ - [ ] Test connection and upload
1166
+ - [ ] Verify file format validity
1167
+ - [ ] Test with large files (>50MB)
1168
+
1169
+ ### 6. Staging Environment
1170
+
1171
+ - [ ] Run full extraction in staging
1172
+ - [ ] Verify file format with downstream system
1173
+ - [ ] Monitor duration and resource usage
1174
+
1175
+ ## Troubleshooting Guide
1176
+
1177
+ **Issue**: "Extraction timeout after 10 minutes"
1178
+
1179
+ - **Cause**: Too many records
1180
+ - **Fix**: Reduce maxRecords, increase frequency
1181
+
1182
+ **Issue**: "Mapping errors for 50% of records"
1183
+
1184
+ - **Cause**: Schema mismatch
1185
+ - **Fix**: Run schema validation, check field names
1186
+
1187
+ **Issue**: "State not updating"
1188
+
1189
+ - **Cause**: KV write failure or intentional retry
1190
+ - **Fix**: Check KV logs, verify state update code
1191
+
1192
+ **Issue**: "First run exceeds limits"
1193
+
1194
+ - **Cause**: No previous timestamp, fetches all
1195
+ - **Fix**: Set fallbackStartDate close to current, apply filters
1196
+
1197
+ **Issue**: "Excessive duplicates"
1198
+
1199
+ - **Cause**: Overlap buffer (expected) or timestamp not saved
1200
+ - **Fix**: Verify newTimestamp saved WITHOUT buffer
1201
+
1202
+ ## Security Best Practices
1203
+
1204
+ ### Credential Management
1205
+
1206
+ **✅ DO**:
1207
+
1208
+ - Store credentials in Versori activation variables
1209
+ - Rotate credentials quarterly
1210
+ - Use least-privilege accounts
1211
+
1212
+ **❌ DON'T**:
1213
+
1214
+ - Never log credentials
1215
+ - Never commit to git
1216
+ - Never share across environments
1217
+
1218
+ ### Data Security
1219
+
1220
+ - Enable encryption in transit and at rest
1221
+ - Apply data retention policies
1222
+ - Monitor access logs
1223
+ - Use VPC/private networks for sensitive data
1224
+
1225
+ ---
1226
+
1227
+ ```
1228
+
1229
+ ---
1230
+
1231
+ **Pattern**: Enterprise incremental extraction with overlap buffer for inventory quantities
1232
+ **⚠️ Sample Code**: For SDK demonstration only - **ONLY use incremental mode in production**
1233
+ **Key Learning**: Use VersoriKVAdapter for state management with 60-second overlap buffer
1234
+ **Critical**: Apply overlap buffer to prevent missed records due to clock skew (default: 60 seconds)
1235
+ **Buffer Pattern**: Query WITH buffer (`updatedOn >= lastRunTime - 60s`), save WITHOUT buffer (`MAX(updatedOn)`)
1236
+ **Timezone**: All timestamps are ISO 8601 UTC format - no conversion needed
1237
+ ```
1238
+
1239
+ ---
1240
+
1241
+ ## 🔧 Complete Production Code
1242
+
1243
+ ### 1. Entry Point (src/index.ts)
1244
+
1245
+ ```typescript
1246
+ /**
1247
+ * Entry Point - Registers all workflows with Versori platform
1248
+ *
1249
+ * This file is the entry point for the Versori deployment.
1250
+ * It imports and re-exports workflows from their respective files:
1251
+ * 1. Scheduled extraction (runs automatically on cron schedule)
1252
+ * 2. Ad hoc webhook (manual trigger with optional date override)
1253
+ * 3. Job status webhook (query job progress)
1254
+ *
1255
+ * AI CUSTOMIZATION:
1256
+ * - Add new workflows by importing from their respective files
1257
+ * - Remove workflows by commenting out imports/exports
1258
+ * - Organize workflows by type (scheduled vs webhook) for clarity
1259
+ */
1260
+
1261
+ import { scheduledInventoryQuantitiesExtraction } from './workflows/scheduled/daily-inventory-quantities-extraction';
1262
+ import { adhocInventoryQuantitiesExtraction } from './workflows/webhook/adhoc-inventory-quantities-extraction';
1263
+ import { inventoryQuantitiesJobStatus } from './workflows/webhook/job-status-check';
1264
+
1265
+ // Register workflows with Versori platform
1266
+ // The platform will expose webhooks as HTTP endpoints and run scheduled workflows on cron schedule
1267
+
1268
+ export {
1269
+ scheduledInventoryQuantitiesExtraction, // Cron-based auto-run (NOT exposed as HTTP endpoint)
1270
+ adhocInventoryQuantitiesExtraction, // Manual webhook trigger (HTTP endpoint)
1271
+ inventoryQuantitiesJobStatus, // Job status query (HTTP endpoint)
1272
+ };
1273
+ ```
1274
+
1275
+ ---
1276
+
1277
+ ### 2. Workflows
1278
+
1279
+ #### src/workflows/scheduled/daily-inventory-quantities-extraction.ts
1280
+
1281
+ ```typescript
1282
+ /**
1283
+ * WORKFLOW 1: Scheduled Extraction
1284
+ *
1285
+ * Purpose: Automated hourly extraction for incremental sync
1286
+ * Trigger: Cron schedule (every hour at minute 0)
1287
+ * State Update: Always updates lastSync timestamp
1288
+ *
1289
+ * AI CUSTOMIZATION:
1290
+ * - Change schedule: Replace '0 * * * *' with your cron expression
1291
+ * Examples:
1292
+ * - Every 30 min: '*/30 * * * *'
1293
+ * - Daily at 2 AM: '0 2 * * *'
1294
+ * - Every 15 min: '*/15 * * * *'
1295
+ */
1296
+
1297
+ import { schedule, fn } from '@versori/run';
1298
+ import { executeInventoryQuantityExtraction } from '../../services/extraction-orchestration';
1299
+ import { generateJobId } from '../../utils/job-id-generator';
1300
+
1301
+ /**
1302
+ * WORKFLOW 1: Scheduled Extraction
1303
+ *
1304
+ * Purpose: Automated hourly extraction for incremental sync
1305
+ * Trigger: Cron schedule (every hour at minute 0)
1306
+ * State Update: Always updates lastSync timestamp
1307
+ *
1308
+ * AI CUSTOMIZATION:
1309
+ * - Change schedule: Replace '0 * * * *' with your cron expression
1310
+ * Examples:
1311
+ * - Every 30 min: '*/30 * * * *'
1312
+ * - Daily at 2 AM: '0 2 * * *'
1313
+ * - Every 15 min: '*/15 * * * *'
1314
+ */
1315
+ export const scheduledInventoryQuantitiesExtraction = schedule(
1316
+ 'inventory-quantities-scheduled',
1317
+ '0 * * * *', // ← CUSTOMIZE: Cron expression
1318
+ fn('execute-scheduled-extraction', async (ctx) => {
1319
+ const { log, activation } = ctx;
1320
+ const startTime = Date.now();
1321
+
1322
+ // Generate unique job ID for tracking
1323
+ // Format: SCHEDULED_IQ_YYYYMMDD_HHMMSS_random
1324
+ const jobId = generateJobId('SCHEDULED', 'INVENTORY_QUANTITIES');
1325
+
1326
+ log.info('🚀 [START] Scheduled extraction triggered', { jobId });
1327
+
1328
+ try {
1329
+ // Execute main workflow (extraction → transform → upload)
1330
+ const result = await executeInventoryQuantityExtraction(ctx, {
1331
+ jobId,
1332
+ triggeredBy: 'schedule',
1333
+ updateState: true, // Always update state for scheduled runs
1334
+ });
1335
+
1336
+ const durationMs = Date.now() - startTime;
1337
+
1338
+ log.info('✅ [END] Scheduled extraction completed', {
1339
+ jobId,
1340
+ recordCount: result.recordsExtracted,
1341
+ fileName: result.fileName,
1342
+ durationMs,
1343
+ durationSec: (durationMs / 1000).toFixed(2)
1344
+ });
1345
+
1346
+ return result;
1347
+
1348
+ } catch (error: any) {
1349
+ const durationMs = Date.now() - startTime;
1350
+
1351
+ log.error('❌ [ERROR] Scheduled extraction failed', {
1352
+ jobId,
1353
+ message: error instanceof Error ? error.message : String(error),
1354
+ stack: error instanceof Error ? error.stack : undefined,
1355
+ errorType: error instanceof Error ? error.constructor.name : 'Error',
1356
+ durationMs,
1357
+ recommendation: 'Check Fluent API connectivity, S3 credentials, and date range configuration'
1358
+ });
1359
+ throw error;
1360
+ }
1361
+ }));
1362
+ ```
1363
+
1364
+ ---
1365
+
1366
+ #### src/workflows/webhook/adhoc-inventory-quantities-extraction.ts
1367
+
1368
+ ```typescript
1369
+ /**
1370
+ * WORKFLOW 2: Ad hoc Extraction (Manual Trigger)
1371
+ *
1372
+ * Purpose: Manual extraction with optional date range override
1373
+ * Trigger: Webhook POST to /webhooks/inventory-quantities-adhoc
1374
+ * State Update: Optional (controlled by request payload)
1375
+ *
1376
+ * WEBHOOK PAYLOAD EXAMPLES:
1377
+ *
1378
+ * 1. Incremental (use last sync timestamp):
1379
+ * {}
1380
+ *
1381
+ * 2. Date range (manual override):
1382
+ * {
1383
+ * "fromDate": "2025-01-01T00:00:00Z",
1384
+ * "toDate": "2025-01-31T23:59:59Z",
1385
+ * "updateState": false
1386
+ * }
1387
+ *
1388
+ * AI CUSTOMIZATION:
1389
+ * - Add request validation
1390
+ * - Add authentication check
1391
+ * - Add custom filters from payload
1392
+ */
1393
+
1394
+ import { webhook, fn } from '@versori/run';
1395
+ import { executeInventoryQuantityExtraction } from '../../services/extraction-orchestration';
1396
+ import { generateJobId } from '../../utils/job-id-generator';
1397
+
1398
+ export const adhocInventoryQuantitiesExtraction = webhook(
1399
+ 'inventory-quantities-adhoc',
1400
+ { connection: 'inventory-quantities-adhoc', response: { mode: 'sync' } },
1401
+ fn('execute-adhoc-extraction', async (ctx) => {
1402
+ const { data, log, connections, activation } = ctx;
1403
+ const startTime = Date.now();
1404
+
1405
+ // Generate unique job ID
1406
+ const jobId = generateJobId('ADHOC', 'INVENTORY_QUANTITIES');
1407
+
1408
+ // SECURITY: Authentication is enforced by Versori connection configuration
1409
+ // Configure auth on the connection and reference it in webhook({ connection: '...' })
1410
+
1411
+ // Extract optional date override from webhook payload
1412
+ const fromDate = data.fromDate as string | undefined;
1413
+ const toDate = data.toDate as string | undefined;
1414
+ const updateState = data.updateState === true; // Default false; advance state only if explicitly true
1415
+
1416
+ log.info('🌐 [START] Ad hoc extraction triggered via webhook', {
1417
+ jobId,
1418
+ hasDateOverride: !!fromDate,
1419
+ fromDate: fromDate || 'not specified',
1420
+ toDate: toDate || 'not specified',
1421
+ updateState
1422
+ });
1423
+
1424
+ try {
1425
+ // Execute main workflow with optional overrides
1426
+ const result = await executeInventoryQuantityExtraction(ctx, {
1427
+ jobId,
1428
+ triggeredBy: 'webhook',
1429
+ fromDate, // Optional: override start date
1430
+ toDate, // Optional: override end date
1431
+ updateState, // Optional: skip state update for historical queries
1432
+ });
1433
+
1434
+ const durationMs = Date.now() - startTime;
1435
+
1436
+ log.info('✅ [END] Ad hoc extraction completed', {
1437
+ jobId,
1438
+ recordCount: result.recordsExtracted,
1439
+ fileName: result.fileName,
1440
+ isManualOverride: !!fromDate,
1441
+ stateUpdated: result.stateUpdated,
1442
+ durationMs,
1443
+ durationSec: (durationMs / 1000).toFixed(2)
1444
+ });
1445
+
1446
+ // Return success with job details
1447
+ return {
1448
+ success: true,
1449
+ jobId,
1450
+ recordsExtracted: result.recordsExtracted,
1451
+ fileName: result.fileName,
1452
+ s3Path: result.s3Path,
1453
+ statusUrl: `/webhooks/inventory-quantities-job-status?jobId=${jobId}`,
1454
+ durationMs,
1455
+ dateRange: fromDate ? {
1456
+ from: fromDate,
1457
+ to: toDate || 'not specified',
1458
+ updateState
1459
+ } : undefined
1460
+ };
1461
+
1462
+ } catch (error: any) {
1463
+ const durationMs = Date.now() - startTime;
1464
+
1465
+ log.error('❌ [ERROR] Ad hoc extraction failed', {
1466
+ jobId,
1467
+ message: error instanceof Error ? error.message : String(error),
1468
+ stack: error instanceof Error ? error.stack : undefined,
1469
+ errorType: error instanceof Error ? error.constructor.name : 'Error',
1470
+ durationMs,
1471
+ recommendation: 'Verify date range format (ISO 8601), check S3 credentials, and ensure Fluent API access'
1472
+ });
1473
+
1474
+ return {
1475
+ success: false,
1476
+ jobId,
1477
+ message: error instanceof Error ? error.message : String(error),
1478
+ stack: error instanceof Error ? error.stack : undefined,
1479
+ errorType: error instanceof Error ? error.constructor.name : 'Error',
1480
+ durationMs,
1481
+ recommendation: 'Verify date range format (ISO 8601), check S3 credentials, and ensure Fluent API access'
1482
+ };
1483
+ }
1484
+ }));
1485
+ ```
1486
+
1487
+ ---
1488
+
1489
+ #### src/workflows/webhook/job-status-check.ts
1490
+
1491
+ ```typescript
1492
+ /**
1493
+ * WORKFLOW 3: Job Status Query
1494
+ *
1495
+ * Purpose: Check job progress and status
1496
+ * Trigger: Webhook GET/POST to /webhooks/inventory-quantities-job-status?jobId=xxx
1497
+ * Returns: Current job status, stage, progress
1498
+ *
1499
+ * QUERY EXAMPLES:
1500
+ *
1501
+ * 1. HTTP GET:
1502
+ * GET /webhooks/inventory-quantities-job-status?jobId=ADHOC_IQ_20251027_183045_abc123
1503
+ *
1504
+ * 2. HTTP POST:
1505
+ * POST /webhooks/inventory-quantities-job-status
1506
+ * { "jobId": "ADHOC_IQ_20251027_183045_abc123" }
1507
+ */
1508
+
1509
+ import { webhook, fn } from '@versori/run';
1510
+ import { getJobStatus } from '../../services/extraction-orchestration';
1511
+
1512
+ export const inventoryQuantitiesJobStatus = webhook(
1513
+ 'inventory-quantities-job-status',
1514
+ { connection: 'inventory-quantities-job-status', response: { mode: 'sync' } },
1515
+ fn('query-job-status', async (ctx) => {
1516
+ const { data, log, openKv, activation } = ctx;
1517
+ const startTime = Date.now();
1518
+
1519
+ // SECURITY: Authentication is enforced by Versori connection configuration
1520
+ // Configure auth on the connection and reference it in webhook({ connection: '...' })
1521
+
1522
+ // Get jobId from query param or POST body
1523
+ const jobId = data.jobId as string;
1524
+
1525
+ if (!jobId) {
1526
+ log.error('❌ Job ID not provided in request');
1527
+ return {
1528
+ success: false,
1529
+ error: 'Job ID is required. Provide jobId in query param or request body.'
1530
+ };
1531
+ }
1532
+
1533
+ log.info('🔍 [START] Querying job status', { jobId });
1534
+
1535
+ try {
1536
+ // Query job status from KV store
1537
+ const status = await getJobStatus(openKv(':project:'), jobId, log);
1538
+
1539
+ const durationMs = Date.now() - startTime;
1540
+
1541
+ if (!status) {
1542
+ log.info('⚠️ Job not found', { jobId, durationMs });
1543
+ return {
1544
+ success: false,
1545
+ error: 'Job not found',
1546
+ jobId,
1547
+ durationMs
1548
+ };
1549
+ }
1550
+
1551
+ log.info('✅ [END] Job status retrieved', {
1552
+ jobId,
1553
+ status: status.status,
1554
+ durationMs
1555
+ });
1556
+
1557
+ return {
1558
+ success: true,
1559
+ jobId,
1560
+ ...status,
1561
+ queryDurationMs: durationMs
1562
+ };
1563
+
1564
+ } catch (error: any) {
1565
+ const durationMs = Date.now() - startTime;
1566
+
1567
+ log.error('❌ [ERROR] Failed to query job status', {
1568
+ jobId,
1569
+ message: error instanceof Error ? error.message : String(error),
1570
+ stack: error instanceof Error ? error.stack : undefined,
1571
+ errorType: error instanceof Error ? error.constructor.name : 'Error',
1572
+ durationMs,
1573
+ recommendation: 'Verify KV store access and job ID format'
1574
+ });
1575
+
1576
+ return {
1577
+ success: false,
1578
+ jobId,
1579
+ message: error instanceof Error ? error.message : String(error),
1580
+ stack: error instanceof Error ? error.stack : undefined,
1581
+ errorType: error instanceof Error ? error.constructor.name : 'Error',
1582
+ durationMs,
1583
+ recommendation: 'Verify KV store access and job ID format'
1584
+ };
1585
+ }
1586
+ }));
1587
+ ```
1588
+
1589
+ ---
1590
+
1591
+ ### 3. Main Orchestration Service (src/services/extraction-orchestration.ts)
1592
+
1593
+ ```typescript
1594
+ /**
1595
+ * MAIN ORCHESTRATION SERVICE
1596
+ *
1597
+ * This is the heart of the extraction workflow. It coordinates all steps:
1598
+ * 1. Initialize clients and services
1599
+ * 2. Determine date range (incremental vs manual)
1600
+ * 3. Extract data using ExtractionOrchestrator
1601
+ * 4. Transform using UniversalMapper
1602
+ * 5. Generate CSV using CSVParserService
1603
+ * 6. Upload to S3
1604
+ * 7. Track job progress with JobTracker
1605
+ * 8. Update state for next run
1606
+ *
1607
+ * NAMING PATTERN (consistent across all use cases):
1608
+ * - Interface: {Entity}ExtractionParams (e.g., InventoryQuantityExtractionParams)
1609
+ * - Result: {Entity}ExtractionResult (e.g., InventoryQuantityExtractionResult)
1610
+ * - Main function: execute{Entity}Extraction (e.g., executeInventoryQuantityExtraction)
1611
+ *
1612
+ * AI CUSTOMIZATION HINTS:
1613
+ * - Change entity: Replace "InventoryQuantity" with "Order", "Product", etc.
1614
+ * - Change output: Replace CSVParserService with XMLBuilder
1615
+ * - Change destination: Replace S3DataSource with SftpDataSource
1616
+ * - Add steps: Insert new service calls between existing steps
1617
+ */
1618
+
1619
+ import { Buffer } from 'node:buffer';
1620
+ import {
1621
+ createClient,
1622
+ ExtractionOrchestrator,
1623
+ JobTracker,
1624
+ UniversalMapper,
1625
+ CSVParserService,
1626
+ S3DataSource,
1627
+ } from '@fluentcommerce/fc-connect-sdk';
1628
+
1629
+ import mappingConfig from '../../config/inventory-quantities.export.csv.json' with { type: 'json' };
1630
+
1631
+ // ✅ VERSORI PLATFORM: Use native log from context, not LoggingService
1632
+
1633
+ /**
1634
+ * Parameters for extraction workflow
1635
+ *
1636
+ * NAMING: {Entity}ExtractionParams
1637
+ */
1638
+ export interface InventoryQuantityExtractionParams {
1639
+ jobId: string;
1640
+ triggeredBy: 'schedule' | 'webhook';
1641
+ fromDate?: string; // Optional: manual date override
1642
+ toDate?: string; // Optional: manual date override
1643
+ updateState: boolean; // Whether to update lastSync timestamp
1644
+
1645
+ // AI CUSTOMIZATION: Add filters specific to entity
1646
+ quantityTypes?: string[]; // e.g., ['LAST_ON_HAND', 'RESERVED']
1647
+ catalogueRef?: string; // e.g., 'DEFAULT_CATALOGUE'
1648
+ }
1649
+
1650
+ /**
1651
+ * Result from extraction workflow
1652
+ *
1653
+ * NAMING: {Entity}ExtractionResult
1654
+ */
1655
+ export interface InventoryQuantityExtractionResult {
1656
+ success: boolean;
1657
+ jobId: string;
1658
+ recordsExtracted: number;
1659
+ fileName?: string;
1660
+ s3Path?: string;
1661
+ error?: string;
1662
+ errors?: any[];
1663
+ isManualOverride?: boolean;
1664
+ stateUpdated?: boolean;
1665
+ newTimestamp?: string;
1666
+ }
1667
+
1668
+ /**
1669
+ * GraphQL Query for Inventory Quantities
1670
+ *
1671
+ * NAMING: {ENTITY}_EXTRACTION_QUERY (uppercase constant)
1672
+ */
1673
+ const INVENTORY_QUANTITIES_EXTRACTION_QUERY = `
1674
+ query GetInventoryQuantities(
1675
+ $catalogues: [InventoryCatalogueKey]
1676
+ $dateRangeFilter: DateRange
1677
+ $productRefs: [String!]
1678
+ $types: [String!]
1679
+ $first: Int!
1680
+ $after: String
1681
+ ) {
1682
+ inventoryQuantities(
1683
+ catalogues: $catalogues
1684
+ updatedOn: $dateRangeFilter
1685
+ productRef: $productRefs
1686
+ type: $types
1687
+ first: $first
1688
+ after: $after
1689
+ ) {
1690
+ edges {
1691
+ node {
1692
+ id
1693
+ ref
1694
+ locationRef
1695
+ productRef
1696
+ qty
1697
+ type
1698
+ status
1699
+ expectedOn
1700
+ createdOn
1701
+ updatedOn
1702
+ catalogue {
1703
+ ref
1704
+ name
1705
+ }
1706
+ }
1707
+ cursor
1708
+ }
1709
+ pageInfo {
1710
+ hasNextPage
1711
+ }
1712
+ }
1713
+ }
1714
+ `;
1715
+
1716
+ /**
1717
+ * Query job status from KV store
1718
+ *
1719
+ * ✅ VERSORI PLATFORM: Uses native log from context (passed as parameter)
1720
+ */
1721
+ export async function getJobStatus(
1722
+ kv: any, // ✅ Versori KV (compatible with JobTracker's KVAdapter interface)
1723
+ jobId: string,
1724
+ log: any // ✅ Native Versori log from context
1725
+ ): Promise<any | undefined> {
1726
+ try {
1727
+ const tracker = new JobTracker(kv, log);
1728
+ return await tracker.getJob(jobId);
1729
+ } catch (error: any) {
1730
+ log.error('Failed to get job status', { jobId, message: error instanceof Error ? error.message : String(error),
1731
+ stack: error instanceof Error ? error.stack : undefined,
1732
+ errorType: error instanceof Error ? error.constructor.name : 'Error', });
1733
+ return undefined;
1734
+ }
1735
+ }
1736
+
1737
+ /**
1738
+ * MAIN ORCHESTRATION FUNCTION
1739
+ *
1740
+ * NAMING: execute{Entity}Extraction (e.g., executeInventoryQuantityExtraction)
1741
+ *
1742
+ * This function implements the complete workflow in steps.
1743
+ * Each step is clearly commented for AI understanding.
1744
+ */
1745
+ export async function executeInventoryQuantityExtraction(
1746
+ ctx: any,
1747
+ params: InventoryQuantityExtractionParams
1748
+ ): Promise<InventoryQuantityExtractionResult> {
1749
+ // ✅ VERSORI PLATFORM: Extract native log from context
1750
+ const { log, openKv, activation } = ctx;
1751
+ const { jobId, triggeredBy, fromDate, toDate, updateState } = params;
1752
+
1753
+ // Open KV store for state management and job tracking
1754
+ // ✅ Pass raw Versori KV directly - it matches KVAdapter interface
1755
+ // ✅ Pass native log to JobTracker
1756
+ const kv = openKv(':project:');
1757
+ const tracker = new JobTracker(kv, log);
1758
+
1759
+ try {
1760
+ // ═══════════════════════════════════════════════════════════
1761
+ // STEP 1/8: Initialize Job Tracking
1762
+ // ═══════════════════════════════════════════════════════════
1763
+ log.info('📝 [STEP 1/8] Initializing job tracking', { jobId });
1764
+
1765
+ await tracker.createJob(jobId, {
1766
+ triggeredBy,
1767
+ hasDateOverride: !!fromDate,
1768
+ fromDate,
1769
+ toDate,
1770
+ updateStateAfterRun: updateState,
1771
+ });
1772
+
1773
+ // ═══════════════════════════════════════════════════════════
1774
+ // STEP 2/8: Initialize Fluent Client
1775
+ // ═══════════════════════════════════════════════════════════
1776
+ log.info('📡 [STEP 2/8] Initializing Fluent Commerce client', { jobId });
1777
+
1778
+ const client = await createClient(ctx, { validateConnection: true });
1779
+
1780
+ if (!client) {
1781
+ throw new Error('Failed to create Fluent Commerce client');
1782
+ }
1783
+
1784
+ log.info('✅ Fluent client initialized and connection validated', { jobId });
1785
+
1786
+ // ═══════════════════════════════════════════════════════════
1787
+ // STEP 3/8: Determine Date Range
1788
+ // ═══════════════════════════════════════════════════════════
1789
+ log.info('📅 [STEP 3/8] Determining date range for extraction', { jobId });
1790
+
1791
+ // State key for incremental sync tracking
1792
+ // NAMING: last{Entity}Sync (e.g., lastInventoryQuantitySync)
1793
+ const STATE_KEY = 'lastInventoryQuantitySync';
1794
+ const DEFAULT_FALLBACK = '2024-01-01T00:00:00Z';
1795
+ const OVERLAP_BUFFER_SECONDS = parseInt(
1796
+ activation.getVariable('overlapBufferSeconds') || '60',
1797
+ 10
1798
+ );
1799
+
1800
+ let dateRangeFilter: { from?: string; to?: string } | null = null;
1801
+ const isManualOverride = !!fromDate;
1802
+
1803
+ if (isManualOverride) {
1804
+ // Manual date override from webhook
1805
+ dateRangeFilter = { from: fromDate, to: toDate };
1806
+ log.info('Using manual date override', { fromDate, toDate });
1807
+ } else {
1808
+ // Incremental sync - get last sync timestamp
1809
+ const rawLastRunTime = (await kv.get<string>(STATE_KEY)) || DEFAULT_FALLBACK;
1810
+
1811
+ // Apply overlap buffer (prevents missed records)
1812
+ const bufferedLastRunTime = new Date(
1813
+ new Date(rawLastRunTime).getTime() - OVERLAP_BUFFER_SECONDS * 1000
1814
+ ).toISOString();
1815
+
1816
+ const effectiveEndTime = toDate || new Date().toISOString();
1817
+
1818
+ dateRangeFilter = {
1819
+ from: bufferedLastRunTime,
1820
+ to: effectiveEndTime, // End of extraction window
1821
+ };
1822
+
1823
+ log.info('Using incremental sync with overlap buffer', {
1824
+ rawLastRunTime,
1825
+ bufferedLastRunTime,
1826
+ effectiveEndTime,
1827
+ overlapBufferSeconds: OVERLAP_BUFFER_SECONDS,
1828
+ });
1829
+ }
1830
+
1831
+ // ═══════════════════════════════════════════════════════════
1832
+ // STEP 4/8: Extract Data (ExtractionOrchestrator)
1833
+ // ═══════════════════════════════════════════════════════════
1834
+ log.info('🔄 [STEP 4/8] Extracting data from Fluent Commerce', { jobId });
1835
+
1836
+ await tracker.updateJob(jobId, {
1837
+ status: 'processing',
1838
+ stage: 'extraction',
1839
+ message: 'Extracting data with auto-pagination',
1840
+ });
1841
+
1842
+ // Build catalogues array from config
1843
+ const catalogueRef = params.catalogueRef || activation.getVariable('catalogueRef');
1844
+ const catalogues = catalogueRef ? [{ ref: catalogueRef }] : [];
1845
+
1846
+ // Configure extraction
1847
+ const pageSize = parseInt(activation.getVariable('pageSize') || '200', 10);
1848
+ const maxRecords = parseInt(activation.getVariable('maxRecords') || '100000', 10);
1849
+
1850
+ // Initialize ExtractionOrchestrator
1851
+ const orchestrator = new ExtractionOrchestrator(client, log);
1852
+
1853
+ // ? Enhanced: Extract context for progress logging
1854
+ const dateRangeInfo = {
1855
+ start: dateRangeFilter?.from || 'N/A',
1856
+ end: dateRangeFilter?.to || 'N/A',
1857
+ catalogues: catalogues.map((c: any) => c.ref).join(', ') || 'all',
1858
+ types: params.quantityTypes?.join(', ') || 'all'
1859
+ };
1860
+
1861
+ // ? Enhanced: Start logging with context
1862
+ log.info(`🔍 [ExtractionOrchestrator] Starting extraction`, {
1863
+ query: 'inventoryQuantities',
1864
+ pageSize,
1865
+ maxRecords,
1866
+ dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
1867
+ catalogues: dateRangeInfo.catalogues,
1868
+ quantityTypes: dateRangeInfo.types,
1869
+ jobId
1870
+ });
1871
+
1872
+ // Execute extraction with auto-pagination
1873
+ const extractionResult = await orchestrator.extract({
1874
+ query: INVENTORY_QUANTITIES_EXTRACTION_QUERY,
1875
+ resultPath: 'inventoryQuantities.edges.node',
1876
+ variables: {
1877
+ catalogues,
1878
+ dateRangeFilter,
1879
+ types: params.quantityTypes,
1880
+ // Note: Don't include 'first' or 'after' here; orchestrator injects them
1881
+ },
1882
+ pageSize,
1883
+ maxRecords,
1884
+ // Optional: validate each record
1885
+ validateItem: (item: any) => {
1886
+ return !!(item.ref && item.productRef);
1887
+ },
1888
+ });
1889
+
1890
+ const records = extractionResult.data || [];
1891
+
1892
+ log.info('Extraction complete', {
1893
+ totalRecords: extractionResult.stats.totalRecords,
1894
+ totalPages: extractionResult.stats.totalPages,
1895
+ validRecords: extractionResult.stats.validRecords ?? records.length,
1896
+ failedValidations: extractionResult.stats.failedValidations,
1897
+ errors: extractionResult.errors ? extractionResult.errors.length : 0,
1898
+ });
1899
+
1900
+ // ? Enhanced: Completion logging with summary
1901
+ log.info(`✅ [ExtractionOrchestrator] Extraction completed`, {
1902
+ totalRecords: extractionResult.stats.totalRecords,
1903
+ totalPages: extractionResult.stats.totalPages,
1904
+ validRecords: extractionResult.stats.validRecords ?? records.length,
1905
+ failedValidations: extractionResult.stats.failedValidations,
1906
+ truncated: extractionResult.stats.truncated,
1907
+ truncationReason: extractionResult.stats.truncationReason,
1908
+ dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
1909
+ jobId
1910
+ });
1911
+
1912
+ if (extractionResult.errors && extractionResult.errors.length > 0) {
1913
+ log.warn('Non-fatal extraction errors encountered', {
1914
+ errorCount: extractionResult.errors.length,
1915
+ sampleErrors: extractionResult.errors.slice(0, 3),
1916
+ });
1917
+ }
1918
+
1919
+ // Handle empty result
1920
+ if (records.length === 0) {
1921
+ log.info('No records to process');
1922
+
1923
+ // Update state even with no records (prevents re-querying empty window)
1924
+ if (updateState && !isManualOverride) {
1925
+ await kv.set(STATE_KEY, new Date().toISOString());
1926
+ }
1927
+
1928
+ await tracker.markCompleted(jobId, {
1929
+ recordCount: 0,
1930
+ message: 'No records to extract',
1931
+ });
1932
+
1933
+ return {
1934
+ success: true,
1935
+ jobId,
1936
+ recordsExtracted: 0,
1937
+ };
1938
+ }
1939
+
1940
+ // ═══════════════════════════════════════════════════════════
1941
+ // STEP 5/8: Transform Data (UniversalMapper)
1942
+ // ═══════════════════════════════════════════════════════════
1943
+ log.info('🔧 [STEP 5/8] Transforming data with UniversalMapper', {
1944
+ jobId,
1945
+ recordCount: records.length,
1946
+ });
1947
+
1948
+ await tracker.updateJob(jobId, {
1949
+ status: 'processing',
1950
+ stage: 'transformation',
1951
+ message: `Transforming ${records.length} records`,
1952
+ });
1953
+
1954
+ const mapper = new UniversalMapper(mappingConfig);
1955
+ const mappingResult = await mapper.map(records);
1956
+
1957
+ if (!mappingResult.success) {
1958
+ const mappingErrors = mappingResult.errors || ['Unknown mapping failure'];
1959
+ await tracker.markFailed(jobId, {
1960
+ error: mappingErrors[0] || 'UniversalMapper returned unsuccessful result',
1961
+ failedCount: mappingErrors.length,
1962
+ });
1963
+ return {
1964
+ success: false,
1965
+ error: `Transformation failed: ${mappingErrors[0] || 'Unknown error'}`,
1966
+ jobId,
1967
+ errors: mappingErrors,
1968
+ };
1969
+ }
1970
+
1971
+ const transformedRecords = Array.isArray(mappingResult.data) ? mappingResult.data : [];
1972
+ const mappingErrors = mappingResult.errors || [];
1973
+
1974
+ if (mappingErrors.length > 0) {
1975
+ log.warn('Some records failed transformation', {
1976
+ jobId,
1977
+ errorCount: mappingErrors.length,
1978
+ sampleErrors: mappingErrors.slice(0, 3),
1979
+ });
1980
+ }
1981
+
1982
+ if (transformedRecords.length === 0) {
1983
+ await tracker.markFailed(jobId, {
1984
+ error: 'All records failed mapping',
1985
+ failedCount: mappingErrors.length,
1986
+ errors: mappingErrors,
1987
+ });
1988
+ return {
1989
+ success: false,
1990
+ error: 'All records failed mapping',
1991
+ jobId,
1992
+ errors: mappingErrors,
1993
+ };
1994
+ }
1995
+
1996
+ log.info('Transformation complete', {
1997
+ successful: transformedRecords.length,
1998
+ failed: mappingErrors.length,
1999
+ skippedRecords: records.length - transformedRecords.length,
2000
+ });
2001
+
2002
+ // ═══════════════════════════════════════════════════════════
2003
+ // STEP 6/8: Generate CSV (CSVParserService)
2004
+ // ═══════════════════════════════════════════════════════════
2005
+ log.info('📄 [STEP 6/8] Generating CSV file', { jobId });
2006
+
2007
+ await tracker.updateJob(jobId, {
2008
+ status: 'processing',
2009
+ stage: 'csv_generation',
2010
+ message: `Generating CSV for ${transformedRecords.length} records`,
2011
+ });
2012
+
2013
+ // Initialize CSVParserService
2014
+ const csvParser = new CSVParserService({ includeHeaders: true });
2015
+
2016
+ // Generate CSV content
2017
+ const csvContent = await csvParser.stringify(transformedRecords);
2018
+
2019
+ // Generate filename
2020
+ const fileNamePrefix = activation.getVariable('fileNamePrefix') || 'inventoryquantities';
2021
+ const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
2022
+ const fileName = `${fileNamePrefix}-${timestamp}.csv`;
2023
+
2024
+ log.info('CSV file generated', {
2025
+ fileName,
2026
+ sizeBytes: csvContent.length,
2027
+ recordCount: transformedRecords.length,
2028
+ });
2029
+
2030
+ // ═══════════════════════════════════════════════════════════
2031
+ // STEP 7/8: Upload to S3 (S3DataSource)
2032
+ // ═══════════════════════════════════════════════════════════
2033
+ log.info('☁️ [STEP 7/8] Uploading to S3', { jobId, fileName });
2034
+
2035
+ await tracker.updateJob(jobId, {
2036
+ status: 'processing',
2037
+ stage: 's3_upload',
2038
+ message: `Uploading ${fileName} to S3`,
2039
+ });
2040
+
2041
+ // Get S3 configuration from activation variables
2042
+ const s3Config = {
2043
+ bucket: activation.getVariable('s3BucketName'),
2044
+ region: activation.getVariable('awsRegion') || 'us-east-1',
2045
+ accessKeyId: activation.getVariable('awsAccessKeyId'),
2046
+ secretAccessKey: activation.getVariable('awsSecretAccessKey'),
2047
+ };
2048
+ const s3Prefix = activation.getVariable('s3Prefix') || 'inventory-quantities/daily/';
2049
+
2050
+ // Validate S3 config
2051
+ if (!s3Config.bucket || !s3Config.accessKeyId || !s3Config.secretAccessKey) {
2052
+ throw new Error(
2053
+ 'S3 configuration incomplete: missing bucket, accessKeyId, or secretAccessKey'
2054
+ );
2055
+ }
2056
+
2057
+ // Initialize S3 data source
2058
+ // ✅ VERSORI PLATFORM: Pass native log from context
2059
+ const s3 = new S3DataSource(
2060
+ {
2061
+ type: 'S3_CSV',
2062
+ connectionId: 'inventory-quantities-s3',
2063
+ name: 'Inventory Quantities S3 Upload',
2064
+ s3Config,
2065
+ },
2066
+ log
2067
+ );
2068
+
2069
+ // Construct S3 key
2070
+ const s3Key = `${s3Prefix}${fileName}`;
2071
+
2072
+ // Upload with retry logic (built into S3DataSource)
2073
+ await s3.uploadFile(s3Key, Buffer.from(csvContent, 'utf-8'), {
2074
+ contentType: 'text/csv',
2075
+ metadata: {
2076
+ recordCount: String(transformedRecords.length),
2077
+ extractedAt: new Date().toISOString(),
2078
+ jobId,
2079
+ mappingErrors: mappingErrors.length > 0 ? String(mappingErrors.length) : undefined,
2080
+ },
2081
+ });
2082
+
2083
+ log.info('S3 upload successful', { fileName, s3Key });
2084
+
2085
+ // ═══════════════════════════════════════════════════════════
2086
+ // STEP 8/8: Update State & Complete Job
2087
+ // ═══════════════════════════════════════════════════════════
2088
+ log.info('💾 [STEP 8/8] Updating state and completing job', { jobId });
2089
+
2090
+ // Calculate new timestamp for next incremental run
2091
+ let newTimestamp: string | undefined;
2092
+
2093
+ if (updateState && !isManualOverride) {
2094
+ // Find max updatedOn from extracted records
2095
+ const maxUpdatedOn = records.reduce(
2096
+ (max, record) => {
2097
+ const recordTime = new Date(record.updatedOn).getTime();
2098
+ return recordTime > max ? recordTime : max;
2099
+ },
2100
+ new Date(dateRangeFilter?.from || DEFAULT_FALLBACK).getTime()
2101
+ );
2102
+
2103
+ newTimestamp = new Date(maxUpdatedOn).toISOString();
2104
+
2105
+ // Store new timestamp (WITHOUT buffer - buffer only applied on read)
2106
+ await kv.set(STATE_KEY, newTimestamp);
2107
+
2108
+ log.info('State updated', {
2109
+ oldTimestamp: dateRangeFilter?.from,
2110
+ newTimestamp,
2111
+ });
2112
+ }
2113
+
2114
+ // Mark job as completed
2115
+ await tracker.markCompleted(jobId, {
2116
+ recordCount: transformedRecords.length,
2117
+ fileName,
2118
+ s3Key,
2119
+ errorCount: mappingErrors.length,
2120
+ errors: mappingErrors,
2121
+ isManualOverride,
2122
+ stateUpdated: updateState,
2123
+ newTimestamp,
2124
+ });
2125
+
2126
+ return {
2127
+ success: true,
2128
+ jobId,
2129
+ recordsExtracted: transformedRecords.length,
2130
+ fileName,
2131
+ s3Path: s3Key,
2132
+ isManualOverride,
2133
+ stateUpdated: updateState,
2134
+ newTimestamp,
2135
+ errors: mappingErrors.length > 0 ? mappingErrors : undefined,
2136
+ };
2137
+ } catch (error: any) {
2138
+ log.error('Extraction workflow failed', {
2139
+ jobId,
2140
+ message: error instanceof Error ? error.message : String(error),
2141
+
2142
+ stack: error instanceof Error ? error.stack : undefined,
2143
+
2144
+ errorType: error instanceof Error ? error.constructor.name : 'Error',
2145
+ });
2146
+
2147
+ // Mark job as failed
2148
+ await tracker.markFailed(jobId, error);
2149
+
2150
+ return {
2151
+ success: false,
2152
+ jobId,
2153
+ recordsExtracted: 0,
2154
+ message: error instanceof Error ? error.message : String(error),
2155
+
2156
+ stack: error instanceof Error ? error.stack : undefined,
2157
+
2158
+ errorType: error instanceof Error ? error.constructor.name : 'Error',
2159
+ };
2160
+ }
2161
+ }
2162
+ ```
2163
+
2164
+ ---
2165
+
2166
+ ### 4. Utility Functions (src/utils/job-id-generator.ts)
2167
+
2168
+ ```typescript
2169
+ /**
2170
+ * Job ID Generator
2171
+ *
2172
+ * Generates unique job IDs for tracking extraction workflows
2173
+ *
2174
+ * FORMAT: {TYPE}_{ENTITY}_{YYYYMMDD}_{HHMMSS}_{RANDOM}
2175
+ * Example: SCHEDULED_IQ_20251027_183045_a1b2c3
2176
+ */
2177
+
2178
+ /**
2179
+ * Generate unique job ID
2180
+ *
2181
+ * @param type - Job trigger type (SCHEDULED, ADHOC, MANUAL)
2182
+ * @param entity - Entity abbreviation (IQ=Inventory Quantities, IP, VP, ORD, PRD)
2183
+ * @returns Unique job ID string
2184
+ */
2185
+ export function generateJobId(type: string, entity: string): string {
2186
+ const now = new Date();
2187
+
2188
+ // Format: YYYYMMDD
2189
+ const dateStr = now.toISOString().slice(0, 10).replace(/-/g, '');
2190
+
2191
+ // Format: HHMMSS
2192
+ const timeStr = now.toISOString().slice(11, 19).replace(/:/g, '');
2193
+
2194
+ // Random suffix (6 chars)
2195
+ const randomStr = Math.random().toString(36).substring(2, 8);
2196
+
2197
+ return `${type}_${entity}_${dateStr}_${timeStr}_${randomStr}`;
2198
+ }
2199
+
2200
+ /**
2201
+ * Parse job ID components
2202
+ */
2203
+ export function parseJobId(jobId: string): {
2204
+ type: string;
2205
+ entity: string;
2206
+ date: string;
2207
+ time: string;
2208
+ random: string;
2209
+ } | null {
2210
+ const parts = jobId.split('_');
2211
+
2212
+ if (parts.length !== 5) {
2213
+ return null;
2214
+ }
2215
+
2216
+ return {
2217
+ type: parts[0],
2218
+ entity: parts[1],
2219
+ date: parts[2],
2220
+ time: parts[3],
2221
+ random: parts[4],
2222
+ };
2223
+ }
2224
+ ```
2225
+
2226
+ ---
2227
+
2228
+ ### 5. Package Configuration
2229
+
2230
+ #### package.json
2231
+
2232
+ ```json
2233
+ {
2234
+ "name": "inventory-quantities-to-s3-csv",
2235
+ "version": "1.0.0",
2236
+ "description": "Extract inventory quantities from Fluent Commerce and export to S3 as CSV",
2237
+ "type": "module",
2238
+ "main": "src/index.ts",
2239
+ "scripts": {
2240
+ "dev": "versori dev",
2241
+ "build": "versori build",
2242
+ "deploy": "versori deploy"
2243
+ },
2244
+ "dependencies": {
2245
+ "@fluentcommerce/fc-connect-sdk": "^0.1.39",
2246
+ "@versori/run": "latest"
2247
+ },
2248
+ "devDependencies": {
2249
+ "@types/node": "^20.0.0",
2250
+ "typescript": "^5.0.0"
2251
+ }
2252
+ }
2253
+ ```
2254
+
2255
+ #### tsconfig.json
2256
+
2257
+ ```json
2258
+ {
2259
+ "compilerOptions": {
2260
+ "module": "ES2022",
2261
+ "target": "ES2024",
2262
+ "moduleResolution": "node"
2263
+ }
2264
+ }
2265
+ ```
2266
+
2267
+ ---
2268
+
2269
+ ## 6. Deployment Instructions
2270
+
2271
+ ### Deploy to Versori
2272
+
2273
+ ```bash
2274
+ # 1. Install dependencies
2275
+ npm install
2276
+
2277
+ # 2. Test locally (if using Versori CLI)
2278
+ npm run dev
2279
+
2280
+ # 3. Deploy to Versori platform
2281
+ npm run deploy
2282
+ ```
2283
+
2284
+ ### Configure Activation Variables
2285
+
2286
+ In Versori platform settings, configure:
2287
+
2288
+ ```json
2289
+ {
2290
+ "catalogueRef": "DEFAULT_CATALOGUE",
2291
+ "s3BucketName": "inventory-audit-exports",
2292
+ "awsAccessKeyId": "AKIAXXXXXXXXXXXX",
2293
+ "awsSecretAccessKey": "********",
2294
+ "awsRegion": "us-east-1",
2295
+ "s3Prefix": "inventory-quantities/daily/",
2296
+ "fileNamePrefix": "inventoryquantities",
2297
+ "pageSize": 200,
2298
+ "maxRecords": 100000,
2299
+ "overlapBufferSeconds": 60,
2300
+ "webhookApiKey": "your-secure-api-key-here"
2301
+ }
2302
+ ```
2303
+
2304
+ ---
2305
+
2306
+ ## 7. Testing
2307
+
2308
+ ### Test Scheduled Extraction
2309
+
2310
+ The scheduled workflow runs automatically based on cron schedule.
2311
+
2312
+ **Check logs:**
2313
+
2314
+ ```
2315
+ [STEP 1/8] Initializing job tracking
2316
+ [STEP 2/8] Initializing Fluent Commerce client
2317
+ [STEP 3/8] Determining date range for extraction
2318
+ [STEP 4/8] Extracting data from Fluent Commerce
2319
+ [STEP 5/8] Transforming data with UniversalMapper
2320
+ [STEP 6/8] Generating CSV file
2321
+ [STEP 7/8] Uploading to S3
2322
+ [STEP 8/8] Updating state and completing job
2323
+ ```
2324
+
2325
+ ### Test Ad hoc Extraction
2326
+
2327
+ ```bash
2328
+ # Incremental (uses last sync timestamp)
2329
+ curl -X POST https://api.versori.com/webhooks/inventory-quantities-adhoc \
2330
+ -H "X-API-Key: your-api-key" \
2331
+ -H "Content-Type: application/json" \
2332
+ -d '{}'
2333
+
2334
+ # Date range override
2335
+ curl -X POST https://api.versori.com/webhooks/inventory-quantities-adhoc \
2336
+ -H "X-API-Key: your-api-key" \
2337
+ -H "Content-Type: application/json" \
2338
+ -d '{
2339
+ "fromDate": "2025-01-01T00:00:00Z",
2340
+ "toDate": "2025-01-31T23:59:59Z",
2341
+ "updateState": false
2342
+ }'
2343
+ ```
2344
+
2345
+ ### Test Job Status Query
2346
+
2347
+ ```bash
2348
+ curl -X POST https://api.versori.com/webhooks/inventory-quantities-job-status \
2349
+ -H "X-API-Key: your-api-key" \
2350
+ -H "Content-Type: application/json" \
2351
+ -d '{
2352
+ "jobId": "ADHOC_IQ_20251027_183045_abc123"
2353
+ }'
2354
+ ```
2355
+
2356
+ **Response:**
2357
+
2358
+ ```json
2359
+ {
2360
+ "success": true,
2361
+ "jobId": "ADHOC_IQ_20251027_183045_abc123",
2362
+ "status": "processing",
2363
+ "stage": "transformation",
2364
+ "message": "Transforming 15000 records",
2365
+ "createdAt": "2025-10-27T18:30:45.000Z",
2366
+ "startedAt": "2025-10-27T18:30:46.000Z"
2367
+ }
2368
+ ```
2369
+
2370
+ ---
2371
+
2372
+ ## 8. Troubleshooting
2373
+
2374
+ **Issue**: "No records extracted"
2375
+
2376
+ - Check dateRange (manual override vs incremental)
2377
+ - Check catalogueRef filter
2378
+ - Verify quantity types filter
2379
+
2380
+ **Issue**: "S3 upload failed"
2381
+
2382
+ - Job fails; state not advanced
2383
+ - Next run retries same window
2384
+ - Check S3 credentials and bucket permissions
2385
+
2386
+ **Issue**: "GraphQL pagination error"
2387
+
2388
+ - Ensure edges.cursor and pageInfo.hasNextPage are in query
2389
+
2390
+ **Issue**: "Memory pressure"
2391
+
2392
+ - Lower pageSize or maxRecords
2393
+ - Consider file splitting for large extractions
2394
+
2395
+ **Issue**: "Transformation errors"
2396
+
2397
+ - Check mapping config field paths
2398
+ - Verify required fields are present in GraphQL response
2399
+ - Review transformation error details in logs
2400
+
2401
+ ---
2402
+
2403
+ ## 9. Replication Checklist
2404
+
2405
+ **To replicate this template for other entities/formats:**
2406
+
2407
+ 1. **File Naming:** Replace `inventory-quantities`, `IQ`, `InventoryQuantity` with your entity name
2408
+ 2. **GraphQL Query:** Update query constant and field selection to match your entity schema
2409
+ 3. **Mapping Config:** Create new mapping file in `config/` with correct field paths
2410
+ 4. **Workflows:** Rename workflow exports to match entity (e.g., `scheduledOrdersExtraction`)
2411
+ 5. **Service Function:** Rename main function (e.g., `executeOrderExtraction`)
2412
+ 6. **State Key:** Update KV key (e.g., `lastOrderSync`)
2413
+ 7. **Output Format:** For XML use `XMLBuilder`, for JSON use `JSON.stringify()`, for CSV use `CSVParserService`
2414
+ 8. **Upload Destination:** For SFTP replace `S3DataSource` with `SftpDataSource` (and add `dispose()` in finally block)
2415
+ 9. **Job ID Entity Code:** Update entity abbreviation in generateJobId() (e.g., 'ORD' for orders)
2416
+ 10. **Result Path:** Update `resultPath` in ExtractionOrchestrator (e.g., `'orders.edges.node'`)
2417
+
2418
+ ---
2419
+
2420
+ **Pattern**: Enterprise incremental extraction with overlap buffer for inventory quantities
2421
+ **Key Learning**: Use ExtractionOrchestrator for auto-pagination, JobTracker for job status, CSVParserService for CSV generation
2422
+ **Critical**: Apply 60-second overlap buffer to prevent missed records due to clock skew
2423
+ **Buffer Pattern**: Query WITH buffer (`updatedOn >= lastRunTime - 60s`), save WITHOUT buffer (`MAX(updatedOn)`)
2424
+ **SDK Services**: ExtractionOrchestrator, UniversalMapper, CSVParserService, S3DataSource, JobTracker
2425
+ **Entity-Specific**: Query uses `inventoryQuantities`, resultPath is `'inventoryQuantities.edges.node'`, state key is `lastInventoryQuantitySync`
2426
+
2427
+ ---
2428
+
2429
+ ### Optional: Backward Pagination (Advanced)
2430
+
2431
+ - Default: forward ($first/$after) + pageInfo.hasNextPage.
2432
+ - Reverse: define $last/$before and include pageInfo.hasPreviousPage; set direction='backward'.
2433
+
2434
+ GraphQL:
2435
+
2436
+ ```graphql
2437
+ query GetInventoryQuantitiesBackward($retailerId: ID!, $last: Int!, $before: String) {
2438
+ inventoryQuantities(retailerId: $retailerId, last: $last, before: $before) {
2439
+ edges {
2440
+ cursor
2441
+ node {
2442
+ id
2443
+ ref
2444
+ updatedOn
2445
+ }
2446
+ }
2447
+ pageInfo {
2448
+ hasPreviousPage
2449
+ }
2450
+ }
2451
+ }
2452
+ ```
2453
+
2454
+ SDK:
2455
+
2456
+ ```typescript
2457
+ await orchestrator.extract({
2458
+ query: INVENTORY_QUANTITIES_BACKWARD_QUERY,
2459
+ resultPath: 'inventoryQuantities.edges.node',
2460
+ variables: { retailerId },
2461
+ pageSize,
2462
+ direction: 'backward',
2463
+ });
2464
+ ```