@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.55

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (475) hide show
  1. package/CHANGELOG.md +12 -0
  2. package/dist/cjs/clients/fluent-client.js +13 -6
  3. package/dist/cjs/utils/pagination-helpers.js +38 -2
  4. package/dist/cjs/versori/fluent-versori-client.js +11 -5
  5. package/dist/esm/clients/fluent-client.js +13 -6
  6. package/dist/esm/utils/pagination-helpers.js +38 -2
  7. package/dist/esm/versori/fluent-versori-client.js +11 -5
  8. package/dist/tsconfig.esm.tsbuildinfo +1 -1
  9. package/dist/tsconfig.tsbuildinfo +1 -1
  10. package/dist/tsconfig.types.tsbuildinfo +1 -1
  11. package/docs/00-START-HERE/EXPORT-VALIDATION.md +158 -158
  12. package/docs/00-START-HERE/cli-analyze-source-structure-guide.md +655 -655
  13. package/docs/00-START-HERE/cli-documentation-index.md +202 -202
  14. package/docs/00-START-HERE/cli-quick-reference.md +252 -252
  15. package/docs/00-START-HERE/decision-tree.md +552 -552
  16. package/docs/00-START-HERE/getting-started.md +1070 -1070
  17. package/docs/00-START-HERE/mapper-quick-decision-guide.md +235 -235
  18. package/docs/00-START-HERE/readme.md +237 -237
  19. package/docs/00-START-HERE/retailerid-configuration.md +404 -404
  20. package/docs/00-START-HERE/sdk-philosophy.md +794 -794
  21. package/docs/00-START-HERE/troubleshooting-quick-reference.md +1086 -1086
  22. package/docs/01-TEMPLATES/faq.md +686 -686
  23. package/docs/01-TEMPLATES/patterns/pattern-templates-guide.md +68 -68
  24. package/docs/01-TEMPLATES/patterns/patterns-csv-schema-validation-and-rejection-report.md +233 -233
  25. package/docs/01-TEMPLATES/patterns/patterns-custom-resolvers.md +407 -407
  26. package/docs/01-TEMPLATES/patterns/patterns-error-handling-retry.md +511 -511
  27. package/docs/01-TEMPLATES/patterns/patterns-field-mapping-universal.md +701 -701
  28. package/docs/01-TEMPLATES/patterns/patterns-large-file-splitting.md +1430 -1430
  29. package/docs/01-TEMPLATES/patterns/patterns-master-data-etl.md +2399 -2399
  30. package/docs/01-TEMPLATES/patterns/patterns-pagination-streaming.md +447 -447
  31. package/docs/01-TEMPLATES/patterns/patterns-state-duplicate-prevention.md +385 -385
  32. package/docs/01-TEMPLATES/readme.md +957 -957
  33. package/docs/01-TEMPLATES/standalone/standalone-asn-inbound-processing.md +1209 -1209
  34. package/docs/01-TEMPLATES/standalone/standalone-graphql-query-export.md +1140 -1140
  35. package/docs/01-TEMPLATES/standalone/standalone-graphql-to-parquet-partitioned-s3.md +432 -432
  36. package/docs/01-TEMPLATES/standalone/standalone-multi-channel-inventory-sync.md +1185 -1185
  37. package/docs/01-TEMPLATES/standalone/standalone-multi-source-aggregation.md +1462 -1462
  38. package/docs/01-TEMPLATES/standalone/standalone-s3-csv-batch-api.md +1390 -1390
  39. package/docs/01-TEMPLATES/standalone/standalone-s3-csv-inventory-to-batch.md +330 -330
  40. package/docs/01-TEMPLATES/standalone/standalone-scripts-guide.md +87 -87
  41. package/docs/01-TEMPLATES/standalone/standalone-sftp-xml-graphql.md +1444 -1444
  42. package/docs/01-TEMPLATES/standalone/standalone-webhook-payload-processing.md +688 -688
  43. package/docs/01-TEMPLATES/versori/business-examples/business-examples-dropship-order-routing.md +193 -193
  44. package/docs/01-TEMPLATES/versori/business-examples/business-examples-graphql-parquet-extraction.md +518 -518
  45. package/docs/01-TEMPLATES/versori/business-examples/business-examples-inter-location-transfers.md +2162 -2162
  46. package/docs/01-TEMPLATES/versori/business-examples/business-examples-pre-order-allocation.md +2226 -2226
  47. package/docs/01-TEMPLATES/versori/business-examples/business-scenarios-guide.md +87 -87
  48. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-connection-validation-pattern.md +656 -656
  49. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-dual-workflow-connector.md +835 -835
  50. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-guide.md +108 -108
  51. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-kv-state-management.md +1533 -1533
  52. package/docs/01-TEMPLATES/versori/patterns/versori-patterns-xml-response-patterns.md +1160 -1160
  53. package/docs/01-TEMPLATES/versori/versori-platform-guide.md +201 -201
  54. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-asn-purchase-order.md +1906 -1906
  55. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-dropship-routing.md +1074 -1074
  56. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-flash-sale-reserve.md +1395 -1395
  57. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-generic-xml-order.md +888 -888
  58. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-payment-gateway-integration.md +2478 -2478
  59. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-rma-returns-comprehensive.md +2240 -2240
  60. package/docs/01-TEMPLATES/versori/webhooks/template-webhook-xml-order-ingestion.md +2029 -2029
  61. package/docs/01-TEMPLATES/versori/webhooks/webhook-templates-guide.md +140 -140
  62. package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/inventory-mapping.json +20 -20
  63. package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/products_2025-01-22.csv +11 -11
  64. package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/sample-data-guide.md +34 -34
  65. package/docs/01-TEMPLATES/versori/workflows/_examples/workflow-examples-guide.md +36 -36
  66. package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-modes-guide.md +1038 -1038
  67. package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-workflows-guide.md +138 -138
  68. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/graphql-extraction-guide.md +63 -63
  69. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-csv.md +2062 -2062
  70. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-xml.md +2294 -2294
  71. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-s3-csv.md +2461 -2461
  72. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-sftp-xml.md +2529 -2529
  73. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-csv.md +2464 -2464
  74. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-json.md +1959 -1959
  75. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-s3-csv.md +1953 -1953
  76. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-sftp-xml.md +2541 -2541
  77. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-s3-json.md +2384 -2384
  78. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-sftp-xml.md +2445 -2445
  79. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-csv.md +2355 -2355
  80. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-json.md +2042 -2042
  81. package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-sftp-xml.md +2726 -2726
  82. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/batch-api-guide.md +206 -206
  83. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-cycle-count-reconciliation.md +2030 -2030
  84. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-multi-channel-inventory-sync.md +1882 -1882
  85. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-csv-inventory-batch.md +2827 -2827
  86. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-json-inventory-batch.md +1952 -1952
  87. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-xml-inventory-batch.md +3289 -3289
  88. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-csv-inventory-batch.md +3064 -3064
  89. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-json-inventory-batch.md +3238 -3238
  90. package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-xml-inventory-batch.md +2977 -2977
  91. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/event-api-guide.md +321 -321
  92. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-json-order-cancel-event.md +959 -959
  93. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-xml-order-cancel-event.md +1170 -1170
  94. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-csv-product-event.md +2312 -2312
  95. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-json-product-event.md +2999 -2999
  96. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-parquet-product-event.md +2836 -2836
  97. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-xml-product-event.md +2395 -2395
  98. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-csv-product-event.md +2295 -2295
  99. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-json-product-event.md +2602 -2602
  100. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-parquet-product-event.md +2589 -2589
  101. package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-xml-product-event.md +3578 -3578
  102. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/graphql-mutations-guide.md +93 -93
  103. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-json-order-update-graphql.md +1260 -1260
  104. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-xml-order-update-graphql.md +1472 -1472
  105. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-control-graphql.md +2417 -2417
  106. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-location-graphql.md +2811 -2811
  107. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-price-graphql.md +2619 -2619
  108. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-json-location-graphql.md +2807 -2807
  109. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-xml-location-graphql.md +2373 -2373
  110. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-control-graphql.md +2740 -2740
  111. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-location-graphql.md +2760 -2760
  112. package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-json-location-graphql.md +1710 -1710
  113. package/docs/01-TEMPLATES/versori/workflows/ingestion/ingestion-workflows-guide.md +136 -136
  114. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/rubix-webhooks-guide.md +520 -520
  115. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-inline.md +1418 -1418
  116. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-universal-mapper.md +1785 -1785
  117. package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-order-attribute-update.md +824 -824
  118. package/docs/01-TEMPLATES/versori/workflows/workflows-overview-guide.md +646 -646
  119. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-batch-archival.md +724 -724
  120. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-job-tracker.md +627 -627
  121. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-partial-batch-recovery.md +561 -561
  122. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-quick-reference.md +367 -367
  123. package/docs/02-CORE-GUIDES/advanced-services/advanced-services-readme.md +407 -407
  124. package/docs/02-CORE-GUIDES/advanced-services/readme.md +49 -49
  125. package/docs/02-CORE-GUIDES/api-reference/api-reference-quick-reference.md +548 -548
  126. package/docs/02-CORE-GUIDES/api-reference/event-api-input-output-reference.md +702 -1171
  127. package/docs/02-CORE-GUIDES/api-reference/examples/client-initialization.ts +286 -286
  128. package/docs/02-CORE-GUIDES/api-reference/graphql-error-classification.md +337 -337
  129. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-01-client-api.md +399 -520
  130. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-03-authentication.md +199 -199
  131. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-04-graphql-mapping.md +925 -925
  132. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-05-services.md +1198 -1198
  133. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-06-data-sources.md +1083 -1083
  134. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-07-parsers.md +1097 -1097
  135. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-pagination.md +513 -513
  136. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-types.md +545 -597
  137. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-error-handling.md +527 -527
  138. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-webhook-validation.md +514 -514
  139. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-extraction.md +557 -557
  140. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-utilities.md +412 -412
  141. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-cli-tools.md +423 -423
  142. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-error-handling.md +716 -716
  143. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-analyze-source-structure.md +518 -518
  144. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-partial-responses.md +212 -212
  145. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-testing.md +300 -300
  146. package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-13-resolver-builder.md +322 -322
  147. package/docs/02-CORE-GUIDES/api-reference/readme.md +279 -279
  148. package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-quick-reference.md +351 -351
  149. package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-readme.md +277 -277
  150. package/docs/02-CORE-GUIDES/auto-pagination/examples/auto-pagination-readme.md +178 -178
  151. package/docs/02-CORE-GUIDES/auto-pagination/examples/common-patterns.ts +351 -351
  152. package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-products.ts +384 -384
  153. package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-virtual-positions.ts +308 -308
  154. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-01-foundations.md +470 -470
  155. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-02-quick-start.md +713 -713
  156. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-03-configuration.md +754 -754
  157. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-04-advanced-patterns.md +732 -732
  158. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-05-sdk-integration.md +847 -847
  159. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-06-troubleshooting.md +359 -359
  160. package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-07-api-reference.md +462 -462
  161. package/docs/02-CORE-GUIDES/auto-pagination/readme.md +54 -54
  162. package/docs/02-CORE-GUIDES/data-sources/data-sources-file-operations-error-handling.md +1487 -1487
  163. package/docs/02-CORE-GUIDES/data-sources/data-sources-quick-reference.md +836 -836
  164. package/docs/02-CORE-GUIDES/data-sources/data-sources-readme.md +276 -276
  165. package/docs/02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md +553 -553
  166. package/docs/02-CORE-GUIDES/data-sources/examples/common-patterns.ts +409 -409
  167. package/docs/02-CORE-GUIDES/data-sources/examples/data-sources-readme.md +178 -178
  168. package/docs/02-CORE-GUIDES/data-sources/examples/s3-operations.ts +308 -308
  169. package/docs/02-CORE-GUIDES/data-sources/examples/sftp-operations.ts +371 -371
  170. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-01-foundations.md +735 -735
  171. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-02-s3-operations.md +1302 -1302
  172. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-03-sftp-operations.md +1379 -1379
  173. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-04-file-patterns.md +941 -941
  174. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-05-advanced-topics.md +813 -813
  175. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-06-integration-patterns.md +486 -486
  176. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-07-troubleshooting.md +387 -387
  177. package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-08-api-reference.md +417 -417
  178. package/docs/02-CORE-GUIDES/data-sources/readme.md +77 -77
  179. package/docs/02-CORE-GUIDES/error-handling-guide.md +936 -936
  180. package/docs/02-CORE-GUIDES/extraction/examples/02-core-guides-extraction-readme.md +116 -116
  181. package/docs/02-CORE-GUIDES/extraction/examples/common-patterns.ts +428 -428
  182. package/docs/02-CORE-GUIDES/extraction/examples/extract-inventory-basic.ts +187 -187
  183. package/docs/02-CORE-GUIDES/extraction/extraction-quick-reference.md +596 -596
  184. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-01-foundations.md +514 -514
  185. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-02-basic-extraction.md +823 -823
  186. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-03-parquet-processing.md +507 -507
  187. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-04-data-enrichment.md +546 -546
  188. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-05-transformation.md +494 -494
  189. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-export-formats.md +458 -458
  190. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-performance.md +138 -138
  191. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-api-reference.md +148 -148
  192. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-optimization.md +692 -692
  193. package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-08-extraction-orchestrator.md +1008 -1008
  194. package/docs/02-CORE-GUIDES/extraction/readme.md +151 -151
  195. package/docs/02-CORE-GUIDES/ingestion/examples/_simple-kv-store.ts +40 -40
  196. package/docs/02-CORE-GUIDES/ingestion/examples/error-recovery.ts +728 -728
  197. package/docs/02-CORE-GUIDES/ingestion/examples/event-driven.ts +501 -501
  198. package/docs/02-CORE-GUIDES/ingestion/examples/local-file-ingestion.ts +88 -88
  199. package/docs/02-CORE-GUIDES/ingestion/examples/parquet-ingestion.ts +117 -117
  200. package/docs/02-CORE-GUIDES/ingestion/examples/performance-optimized.ts +647 -647
  201. package/docs/02-CORE-GUIDES/ingestion/examples/s3-csv-ingestion.ts +169 -169
  202. package/docs/02-CORE-GUIDES/ingestion/examples/sftp-csv-ingestion.ts +134 -134
  203. package/docs/02-CORE-GUIDES/ingestion/ingestion-quick-reference.md +546 -546
  204. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-01-introduction.md +626 -626
  205. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-02-quick-start.md +658 -658
  206. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-03-data-sources.md +1052 -1052
  207. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-04-field-mapping.md +763 -763
  208. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-05-advanced-parsers.md +676 -676
  209. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-06-batch-api.md +1295 -1295
  210. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-api-reference.md +138 -138
  211. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-state-management.md +1037 -1037
  212. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-08-performance-optimization.md +1349 -1349
  213. package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-09-best-practices.md +1893 -1893
  214. package/docs/02-CORE-GUIDES/ingestion/readme.md +160 -160
  215. package/docs/02-CORE-GUIDES/logging-guide.md +585 -585
  216. package/docs/02-CORE-GUIDES/mapping/error-handling-patterns.md +401 -401
  217. package/docs/02-CORE-GUIDES/mapping/examples/02-core-guides-mapping-readme.md +128 -128
  218. package/docs/02-CORE-GUIDES/mapping/examples/common-patterns.ts +273 -273
  219. package/docs/02-CORE-GUIDES/mapping/examples/csv-location-ingestion.json +36 -36
  220. package/docs/02-CORE-GUIDES/mapping/examples/csv-mapping.ts +242 -242
  221. package/docs/02-CORE-GUIDES/mapping/examples/graphql-to-parquet-extraction.json +36 -36
  222. package/docs/02-CORE-GUIDES/mapping/examples/json-mapping.ts +213 -213
  223. package/docs/02-CORE-GUIDES/mapping/examples/json-product-to-mutation.json +48 -48
  224. package/docs/02-CORE-GUIDES/mapping/examples/xml-mapping.ts +291 -291
  225. package/docs/02-CORE-GUIDES/mapping/examples/xml-order-to-mutation.json +45 -45
  226. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-quick-reference.md +463 -463
  227. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-readme.md +227 -227
  228. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-01-introduction.md +222 -222
  229. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-02-quick-start.md +351 -351
  230. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-03-schema-validation.md +569 -569
  231. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-04-mapping-patterns.md +471 -471
  232. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-05-configuration-reference.md +611 -611
  233. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-advanced-xpath.md +148 -148
  234. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-path-syntax.md +464 -464
  235. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-api-reference.md +94 -94
  236. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-array-handling.md +307 -307
  237. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-08-custom-resolvers.md +544 -544
  238. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-09-advanced-patterns.md +427 -427
  239. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-10-hooks-and-variables.md +336 -336
  240. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-11-error-handling.md +488 -488
  241. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-12-arguments-vs-nodes.md +383 -383
  242. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-13-best-practices.md +477 -477
  243. package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/readme.md +62 -62
  244. package/docs/02-CORE-GUIDES/mapping/mapping-format-decision-tree.md +480 -480
  245. package/docs/02-CORE-GUIDES/mapping/mapping-graphql-alias-batching-guide.md +820 -820
  246. package/docs/02-CORE-GUIDES/mapping/mapping-javascript-objects.md +2369 -2369
  247. package/docs/02-CORE-GUIDES/mapping/mapping-mapper-comparison-guide.md +682 -682
  248. package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-07-api-reference.md +1327 -1327
  249. package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-08-error-handling.md +1142 -1142
  250. package/docs/02-CORE-GUIDES/mapping/modules/mapping-04-use-cases.md +891 -891
  251. package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-helpers-resolvers.md +1126 -1126
  252. package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-sdk-resolvers.md +199 -199
  253. package/docs/02-CORE-GUIDES/mapping/modules/mapping-07-api-reference.md +1319 -1319
  254. package/docs/02-CORE-GUIDES/mapping/readme.md +178 -178
  255. package/docs/02-CORE-GUIDES/mapping/resolver-registration.md +410 -410
  256. package/docs/02-CORE-GUIDES/mapping/resolvers/examples/common-patterns.ts +226 -226
  257. package/docs/02-CORE-GUIDES/mapping/resolvers/examples/custom-resolvers.ts +227 -227
  258. package/docs/02-CORE-GUIDES/mapping/resolvers/examples/sdk-resolvers-usage.ts +203 -203
  259. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-readme.md +274 -274
  260. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-api-reference.md +679 -679
  261. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-cookbook.md +826 -826
  262. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-guide.md +1330 -1330
  263. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-helpers-reference.md +1437 -1437
  264. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-parameters-reference.md +553 -553
  265. package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-troubleshooting.md +854 -854
  266. package/docs/02-CORE-GUIDES/mapping/resolvers/readme.md +75 -75
  267. package/docs/02-CORE-GUIDES/parsers/examples/02-core-guides-parsers-readme.md +161 -161
  268. package/docs/02-CORE-GUIDES/parsers/examples/csv-parser-examples.ts +110 -110
  269. package/docs/02-CORE-GUIDES/parsers/examples/json-parser-examples.ts +33 -33
  270. package/docs/02-CORE-GUIDES/parsers/examples/parquet-parser-examples.ts +47 -47
  271. package/docs/02-CORE-GUIDES/parsers/examples/xml-parser-examples.ts +38 -38
  272. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-01-foundations.md +355 -355
  273. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-02-csv-parser.md +772 -772
  274. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-03-json-parser.md +789 -789
  275. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-04-xml-parser.md +857 -857
  276. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-05-parquet-parser.md +603 -603
  277. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-integration-patterns.md +702 -702
  278. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-streaming.md +121 -121
  279. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-api-reference.md +89 -89
  280. package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-troubleshooting.md +727 -727
  281. package/docs/02-CORE-GUIDES/parsers/parsers-quick-reference.md +482 -482
  282. package/docs/02-CORE-GUIDES/parsers/parsers-readme.md +258 -258
  283. package/docs/02-CORE-GUIDES/parsers/readme.md +65 -65
  284. package/docs/02-CORE-GUIDES/readme.md +194 -194
  285. package/docs/02-CORE-GUIDES/webhook-validation/examples/basic-validation.ts +108 -108
  286. package/docs/02-CORE-GUIDES/webhook-validation/examples/common-patterns.ts +316 -316
  287. package/docs/02-CORE-GUIDES/webhook-validation/examples/webhook-validation-readme.md +61 -61
  288. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-01-foundations.md +440 -440
  289. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-02-quick-start.md +525 -525
  290. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-03-versori-integration.md +741 -741
  291. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-04-platform-integration.md +629 -629
  292. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-05-configuration.md +535 -535
  293. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-error-handling.md +611 -611
  294. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-troubleshooting.md +124 -124
  295. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-07-api-reference.md +511 -511
  296. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-08-rubix-webhooks.md +590 -590
  297. package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-09-rubix-event-vs-http-call.md +432 -432
  298. package/docs/02-CORE-GUIDES/webhook-validation/readme.md +239 -239
  299. package/docs/02-CORE-GUIDES/webhook-validation/webhook-validation-quick-reference.md +392 -392
  300. package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-quick-reference.md +498 -498
  301. package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-readme.md +313 -313
  302. package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/common-patterns.ts +612 -612
  303. package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/connector-scenarios-readme.md +253 -253
  304. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-01-foundations.md +452 -452
  305. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-02-simple-scenarios.md +681 -681
  306. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-03-intermediate-scenarios.md +637 -637
  307. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-04-advanced-scenarios.md +650 -650
  308. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-05-bidirectional-sync.md +233 -233
  309. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-06-production-patterns.md +442 -442
  310. package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-07-reference.md +445 -445
  311. package/docs/03-PATTERN-GUIDES/connector-scenarios/readme.md +31 -31
  312. package/docs/03-PATTERN-GUIDES/enterprise-integration-patterns.md +1528 -1528
  313. package/docs/03-PATTERN-GUIDES/error-handling/comprehensive-error-handling-guide.md +1437 -1437
  314. package/docs/03-PATTERN-GUIDES/error-handling/error-handling-quick-reference.md +390 -390
  315. package/docs/03-PATTERN-GUIDES/error-handling/examples/common-patterns.ts +438 -438
  316. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-01-foundations.md +362 -362
  317. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-02-error-types.md +850 -850
  318. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-03-utf8-handling.md +456 -456
  319. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-04-error-scenarios.md +658 -658
  320. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-05-calling-patterns.md +671 -671
  321. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-06-retry-strategies.md +1034 -1034
  322. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-07-monitoring.md +653 -653
  323. package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-08-api-reference.md +847 -847
  324. package/docs/03-PATTERN-GUIDES/error-handling/readme.md +36 -36
  325. package/docs/03-PATTERN-GUIDES/examples/__tests__/readme.md +40 -40
  326. package/docs/03-PATTERN-GUIDES/examples/__tests__/resolver-examples.test.js +282 -282
  327. package/docs/03-PATTERN-GUIDES/examples/test-data/03-pattern-guides-readme.md +110 -110
  328. package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-inventory.json +123 -123
  329. package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-order.json +171 -171
  330. package/docs/03-PATTERN-GUIDES/examples/test-data/readme.md +28 -28
  331. package/docs/03-PATTERN-GUIDES/extraction/extraction-readme.md +15 -15
  332. package/docs/03-PATTERN-GUIDES/extraction/readme.md +25 -25
  333. package/docs/03-PATTERN-GUIDES/file-operations/examples/common-patterns.ts +407 -407
  334. package/docs/03-PATTERN-GUIDES/file-operations/examples/file-operations-readme.md +142 -142
  335. package/docs/03-PATTERN-GUIDES/file-operations/file-operations-quick-reference.md +462 -462
  336. package/docs/03-PATTERN-GUIDES/file-operations/file-operations-readme.md +379 -379
  337. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-01-foundations.md +430 -430
  338. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-02-quick-start.md +484 -484
  339. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-03-s3-operations.md +507 -507
  340. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-04-sftp-operations.md +963 -963
  341. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-05-streaming-performance.md +503 -503
  342. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-archive-patterns.md +386 -386
  343. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-error-handling.md +117 -117
  344. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-api-reference.md +78 -78
  345. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-testing-troubleshooting.md +567 -567
  346. package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-08-api-reference.md +1055 -1055
  347. package/docs/03-PATTERN-GUIDES/file-operations/readme.md +32 -32
  348. package/docs/03-PATTERN-GUIDES/ingestion/ingestion-readme.md +15 -15
  349. package/docs/03-PATTERN-GUIDES/ingestion/readme.md +25 -25
  350. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/batch-processing.ts +130 -130
  351. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/common-patterns.ts +360 -360
  352. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/delta-sync.ts +130 -130
  353. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/integration-patterns-readme.md +100 -100
  354. package/docs/03-PATTERN-GUIDES/integration-patterns/examples/real-time-webhook.ts +398 -398
  355. package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-quick-reference.md +962 -962
  356. package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-readme.md +134 -134
  357. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-01-real-time-processing.md +991 -991
  358. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-02-batch-processing.md +1547 -1547
  359. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-03-delta-sync.md +1108 -1108
  360. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-04-webhook-patterns.md +1181 -1181
  361. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-05-error-handling.md +1061 -1061
  362. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-advanced-integration-services.md +1547 -1547
  363. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-performance.md +109 -109
  364. package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-07-api-reference.md +34 -34
  365. package/docs/03-PATTERN-GUIDES/integration-patterns/readme.md +30 -30
  366. package/docs/03-PATTERN-GUIDES/logging-minimal-mode.md +128 -128
  367. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/common-patterns.ts +380 -380
  368. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/multiple-connections-readme.md +139 -139
  369. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/parallel-root-connections.ts +149 -149
  370. package/docs/03-PATTERN-GUIDES/multiple-connections/examples/real-world-scenarios.ts +405 -405
  371. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-01-foundations.md +378 -378
  372. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-02-quick-start.md +566 -566
  373. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-03-targeting-connections.md +659 -659
  374. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-04-parallel-queries.md +656 -656
  375. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-05-best-practices.md +624 -624
  376. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-api-reference.md +824 -824
  377. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-versori.md +119 -119
  378. package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-07-api-reference.md +87 -87
  379. package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-quick-reference.md +353 -353
  380. package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-readme.md +270 -270
  381. package/docs/03-PATTERN-GUIDES/multiple-connections/readme.md +30 -30
  382. package/docs/03-PATTERN-GUIDES/pagination/pagination-readme.md +14 -14
  383. package/docs/03-PATTERN-GUIDES/pagination/readme.md +24 -24
  384. package/docs/03-PATTERN-GUIDES/parquet/examples/common-patterns.ts +180 -180
  385. package/docs/03-PATTERN-GUIDES/parquet/examples/read-parquet.ts +48 -48
  386. package/docs/03-PATTERN-GUIDES/parquet/examples/write-parquet.ts +65 -65
  387. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-01-introduction.md +393 -393
  388. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-02-quick-start.md +572 -572
  389. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-03-reading-parquet.md +525 -525
  390. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-04-writing-parquet.md +554 -554
  391. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-05-graphql-extraction.md +405 -405
  392. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-performance.md +104 -104
  393. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-s3-integration.md +511 -511
  394. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-api-reference.md +90 -90
  395. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-performance-optimization.md +525 -525
  396. package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-08-best-practices.md +712 -712
  397. package/docs/03-PATTERN-GUIDES/parquet/parquet-quick-reference.md +683 -683
  398. package/docs/03-PATTERN-GUIDES/parquet/parquet-readme.md +248 -248
  399. package/docs/03-PATTERN-GUIDES/parquet/readme.md +32 -32
  400. package/docs/03-PATTERN-GUIDES/parsers/parsers-readme.md +12 -12
  401. package/docs/03-PATTERN-GUIDES/parsers/readme.md +24 -24
  402. package/docs/03-PATTERN-GUIDES/readme.md +159 -159
  403. package/docs/03-PATTERN-GUIDES/webhooks/readme.md +24 -24
  404. package/docs/03-PATTERN-GUIDES/webhooks/webhooks-readme.md +8 -8
  405. package/docs/04-REFERENCE/architecture/architecture-01-overview.md +427 -427
  406. package/docs/04-REFERENCE/architecture/architecture-02-client-architecture.md +424 -424
  407. package/docs/04-REFERENCE/architecture/architecture-03-data-flow.md +690 -690
  408. package/docs/04-REFERENCE/architecture/architecture-04-service-layer.md +834 -834
  409. package/docs/04-REFERENCE/architecture/architecture-05-integration-architecture.md +655 -655
  410. package/docs/04-REFERENCE/architecture/architecture-06-state-management.md +653 -653
  411. package/docs/04-REFERENCE/architecture/architecture-adding-new-data-sources.md +686 -686
  412. package/docs/04-REFERENCE/architecture/readme.md +279 -279
  413. package/docs/04-REFERENCE/platforms/deno/readme.md +117 -117
  414. package/docs/04-REFERENCE/platforms/nodejs/readme.md +146 -146
  415. package/docs/04-REFERENCE/platforms/readme.md +135 -135
  416. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-01-introduction.md +398 -398
  417. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-02-quick-start.md +560 -560
  418. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-03-authentication.md +757 -757
  419. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-04-workflows.md +2476 -2476
  420. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-05-connections.md +1167 -1167
  421. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-kv-storage.md +990 -990
  422. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-state-management.md +121 -121
  423. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-api-reference.md +68 -68
  424. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-deployment.md +731 -731
  425. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-08-best-practices.md +1111 -1111
  426. package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-09-signature-reference.md +766 -766
  427. package/docs/04-REFERENCE/platforms/versori/platforms-versori-readme.md +299 -299
  428. package/docs/04-REFERENCE/platforms/versori/platforms-versori-s3-sftp-configuration-guide.md +1425 -1425
  429. package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-api-key-security.md +816 -816
  430. package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-connection-security.md +681 -681
  431. package/docs/04-REFERENCE/platforms/versori/platforms-versori-workflow-task-types.md +708 -708
  432. package/docs/04-REFERENCE/platforms/versori/readme.md +108 -108
  433. package/docs/04-REFERENCE/readme.md +148 -148
  434. package/docs/04-REFERENCE/resolver-signature/examples/advanced-resolvers.ts +482 -482
  435. package/docs/04-REFERENCE/resolver-signature/examples/async-resolvers.ts +496 -496
  436. package/docs/04-REFERENCE/resolver-signature/examples/basic-resolvers.ts +343 -343
  437. package/docs/04-REFERENCE/resolver-signature/examples/resolver-signature-readme.md +188 -188
  438. package/docs/04-REFERENCE/resolver-signature/examples/testing-resolvers.ts +463 -463
  439. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-01-foundations.md +286 -286
  440. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-02-parameter-reference.md +643 -643
  441. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-03-basic-examples.md +521 -521
  442. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-04-advanced-patterns.md +739 -739
  443. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-05-sdk-resolvers.md +531 -531
  444. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-migration-guide.md +650 -650
  445. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-testing.md +125 -125
  446. package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-07-api-reference.md +794 -794
  447. package/docs/04-REFERENCE/resolver-signature/readme.md +64 -64
  448. package/docs/04-REFERENCE/resolver-signature/resolver-signature-quick-reference.md +270 -270
  449. package/docs/04-REFERENCE/resolver-signature/resolver-signature-readme.md +351 -351
  450. package/docs/04-REFERENCE/schema/fluent-commerce-schema.json +764 -764
  451. package/docs/04-REFERENCE/schema/readme.md +141 -141
  452. package/docs/04-REFERENCE/testing/examples/04-reference-testing-readme.md +158 -158
  453. package/docs/04-REFERENCE/testing/examples/fluent-testing.ts +62 -62
  454. package/docs/04-REFERENCE/testing/examples/health-check.ts +155 -155
  455. package/docs/04-REFERENCE/testing/examples/integration-test.ts +119 -119
  456. package/docs/04-REFERENCE/testing/examples/performance-test.ts +183 -183
  457. package/docs/04-REFERENCE/testing/examples/s3-testing.ts +127 -127
  458. package/docs/04-REFERENCE/testing/modules/04-reference-testing-01-foundations.md +267 -267
  459. package/docs/04-REFERENCE/testing/modules/04-reference-testing-02-s3-testing.md +599 -599
  460. package/docs/04-REFERENCE/testing/modules/04-reference-testing-03-fluent-testing.md +589 -589
  461. package/docs/04-REFERENCE/testing/modules/04-reference-testing-04-integration-testing.md +699 -699
  462. package/docs/04-REFERENCE/testing/modules/04-reference-testing-05-debugging.md +478 -478
  463. package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-cicd-integration.md +463 -463
  464. package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-preflight-validation.md +131 -131
  465. package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-best-practices.md +499 -499
  466. package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-coverage-ci.md +165 -165
  467. package/docs/04-REFERENCE/testing/modules/04-reference-testing-08-api-reference.md +634 -634
  468. package/docs/04-REFERENCE/testing/readme.md +86 -86
  469. package/docs/04-REFERENCE/testing/testing-quick-reference.md +667 -667
  470. package/docs/04-REFERENCE/testing/testing-readme.md +286 -286
  471. package/docs/04-REFERENCE/troubleshooting/readme.md +144 -144
  472. package/docs/04-REFERENCE/troubleshooting/troubleshooting-deno-sftp-compatibility.md +392 -392
  473. package/docs/template-loading-matrix.md +242 -242
  474. package/package.json +5 -3
  475. package/docs/02-CORE-GUIDES/api-reference/cli-profile-integration.md +0 -377
@@ -1,2464 +1,2464 @@
1
- ---
2
- template_id: tpl-extract-inventory-quantities-to-s3-csv
3
- canonical_filename: template-extraction-inventory-quantities-to-s3-csv.md
4
- version: 2.0.0
5
- sdk_version: ^0.1.39
6
- runtime: versori
7
- direction: extraction
8
- source: fluent-graphql
9
- destination: s3-csv
10
- entity: inventoryQuantities
11
- format: csv
12
- logging: versori
13
- status: stable
14
- features:
15
- - memory-management
16
- - enhanced-logging
17
- - pagination-progress
18
- ---
19
-
20
- # Template: Extraction - Inventory Quantities to S3 CSV
21
-
22
- **Template Version:** 2.0.0
23
- **SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
24
- **Last Updated:** 2025-01-24
25
- **Deployment Target:** Versori Platform
26
-
27
- **🆕 Version 2.0.0 Enhancements:**
28
- - ✅ **Memory Management** - Clear large result sets after processing batches
29
- - ✅ **Enhanced Logging** - Pagination progress tracking with emoji indicators (📊, 📥, ✅)
30
- - ✅ **Pagination Progress** - Real-time page-by-page progress logging with metrics
31
-
32
- ---
33
-
34
- ## 📚 STEP 1: Load These Docs (Human Checklist)
35
-
36
- 1. REQUIRED (load all)
37
- - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
38
- - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
39
- - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
40
- - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
41
- - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
42
- - [ ] fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
43
-
44
- Copy-paste list (open these):
45
- fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
46
- fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
47
- fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
48
- fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
49
- fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
50
- fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
51
-
52
- ---
53
-
54
- ## 📋 STEP 2: Tell Your AI (Prompt)
55
-
56
- Copy/paste this prompt into your AI tool after loading the documentation above:
57
-
58
- ```
59
- I need a Versori scheduled extractor that:
60
-
61
- 1) Queries Fluent Commerce GraphQL for inventoryQuantities with auto-pagination
62
- 2) Uses incremental mode with a 60-second overlap buffer stored in Versori KV
63
- 3) Transforms results using UniversalMapper per mapping JSON
64
- 4) Generates CSV with CSVParserService and uploads to S3
65
- 5) Uses native Versori log (LoggingService removed - use native log)
66
-
67
- Use the loaded docs for SDK specifics and best practices. Keep structure identical to the template.
68
- ```
69
-
70
- ---
71
-
72
- ## 📦 SDK Imports (Verified - Versori Optimized)
73
-
74
- ```typescript
75
- import { Buffer } from 'node:buffer';
76
- import {
77
- createClient,
78
- UniversalMapper,
79
- S3DataSource,
80
- CSVParserService,
81
- } from '@fluentcommerce/fc-connect-sdk';
82
-
83
- import { schedule, http } from '@versori/run';
84
- ```
85
-
86
- ---
87
-
88
- # Versori Scheduled: Inventory Quantities Extraction to S3 CSV (Configurable)
89
-
90
- **FC Connect SDK Use Case Guide**
91
-
92
- > SDK: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
93
- > Version: `npm install @fluentcommerce/fc-connect-sdk@^0.1.39`
94
-
95
- Context: Scheduled Versori workflow that extracts inventory quantities (detailed quantity records) from Fluent Commerce via GraphQL query with **configurable extraction modes**, transforms with `UniversalMapper`, and writes CSV files to S3 for analytics and reporting.
96
-
97
- **Pattern**: EXTRACTION (Fluent → S3 CSV)
98
- **Entity**: inventoryQuantities
99
- **Complexity**: High | Runtime: Versori Platform (Scheduled)
100
-
101
- ---
102
-
103
- ## ⚠️ IMPORTANT: Sample Code for SDK Demonstration Only
104
-
105
- > **🔴 PRODUCTION WARNING**
106
- >
107
- > This guide demonstrates FC Connect SDK capabilities for **extraction and mapping workflows**. The multiple extraction modes (incremental, dateRange, historical) are included to show SDK flexibility and serve as **reference examples**.
108
- >
109
- > **✅ PRODUCTION RECOMMENDATION:**
110
- >
111
- > - **ONLY use INCREMENTAL mode with scheduled runs** (e.g., daily/hourly)
112
- > - Incremental mode is safe, efficient, and production-ready
113
- > - Uses overlap buffer to prevent missed records
114
- > - Natural rate limiting via timestamps
115
- >
116
- > **🚫 DO NOT USE IN PRODUCTION:**
117
- >
118
- > - **dateRange mode** - High risk of platform overload with large date windows
119
- > - **historical mode** - Extremely dangerous, can fetch millions of records
120
- > - These modes are **demonstration only** to show SDK query patterns
121
- > - Using these modes on large inventory datasets can crash your runtime and impact platform stability
122
- >
123
- > **📝 If you need historical data:**
124
- >
125
- > - Run multiple small incremental extractions (e.g., daily for past 30 days)
126
- > - Use one-time migration scripts with proper monitoring (not scheduled workflows)
127
- > - Always validate date ranges and implement file splitting
128
- > - Get explicit approval before running large extractions
129
- >
130
- > **This sample code shows HOW to use the SDK - not WHAT to use in production.**
131
-
132
- ---
133
-
134
- ## What You'll Build
135
-
136
- - **Three extraction modes**: Incremental, Date Range, or Historical
137
- - **State management** with VersoriKVAdapter to track last successful run
138
- - GraphQL query with auto-pagination
139
- - UniversalMapper transformation for reporting schema
140
- - CSV file generation with CSVParserService
141
- - S3 upload to analytics system
142
- - **Failure recovery** with timestamp tracking
143
-
144
- ## Business Use Cases
145
-
146
- **1. Incremental Daily Sync (Analytics)**
147
-
148
- - Extract only changed inventory quantities since last run
149
- - Run daily at 2 AM
150
- - Minimize data transfer
151
- - Track changes over time
152
-
153
- **2. Date Range Extract (Audit)**
154
-
155
- - Extract quantity changes within specific date window
156
- - For audits, reconciliation, historical analysis
157
- - Example: "Show all quantity changes between Jan 1-15"
158
-
159
- **3. Historical Backfill**
160
-
161
- - Extract all quantities created within date range
162
- - For initial data warehouse load
163
- - One-time backfill operation
164
-
165
- ## Inventory Quantities vs Positions
166
-
167
- **InventoryQuantity** = Specific quantity record (retailer-defined types)
168
-
169
- - Individual records: e.g., LAST_ON_HAND, RESERVED, DELTA, SALE, CORRECTION (plus any custom IQ types)
170
- - Multiple quantities per product/location
171
- - Fields: locationRef, skuRef, qty, type, status, expectedOn (if applicable)
172
- - Used for: Detailed tracking, audit trails
173
-
174
- **InventoryPosition** = Aggregated on-hand calculation
175
-
176
- - One position per product/location
177
- - Calculated `onHand` from all associated quantities
178
- - Used for: Stock availability, reporting
179
-
180
- ## SDK Methods Used
181
-
182
- ```typescript
183
- import { Buffer } from 'node:buffer';
184
- import {
185
- createClient,
186
- UniversalMapper,
187
- S3DataSource,
188
- VersoriKVAdapter,
189
- CSVParserService,
190
- } from '@fluentcommerce/fc-connect-sdk';
191
-
192
- await createClient(ctx);
193
- await client.graphql({ query, variables, pagination });
194
- new VersoriKVAdapter(ctx.openKv(':project:'));
195
- new UniversalMapper(exportMapping);
196
- const csvParser = new CSVParserService({ includeHeaders: true });
197
- const csvContent = await csvParser.stringify(rows);
198
- await s3.uploadFile(key, Buffer.from(csvContent, 'utf8'), options);
199
- ```
200
-
201
- ## Activation Variables
202
-
203
- ```json
204
- {
205
- "retailerId": "your-retailer-id",
206
- "s3BucketName": "inventory-audit-exports",
207
- "awsAccessKeyId": "AKIAXXXXXXXXXXXX",
208
- "awsSecretAccessKey": "********",
209
- "awsRegion": "us-east-1",
210
- "s3Prefix": "inventory-quantities/daily/",
211
- "fileNamePrefix": "inventoryquantities",
212
- "catalogueRef": "DEFAULT_CATALOGUE",
213
- "pageSize": 200,
214
- "maxRecords": 100000,
215
- "extractionMode": "incremental",
216
- "fallbackStartDate": "2024-01-01T00:00:00Z",
217
- "overlapBufferSeconds": "60",
218
- "startDate": "",
219
- "endDate": ""
220
- }
221
- ```
222
-
223
- ### Variable Reference
224
-
225
- | Variable | Type | Required | Default | Description |
226
- |----------|------|----------|---------|-------------|
227
- | `retailerId` | string | Yes | - | Fluent Commerce retailer ID |
228
- | `s3BucketName` | string | Yes | - | S3 bucket for CSV export |
229
- | `awsAccessKeyId` | string | Yes | - | AWS access key with S3 write permissions |
230
- | `awsSecretAccessKey` | string | Yes | - | AWS secret access key |
231
- | `awsRegion` | string | Yes | - | AWS region (e.g., `us-east-1`) |
232
- | `s3Prefix` | string | No | `""` | S3 key prefix (e.g., `inventory-quantities/daily/`) |
233
- | `fileNamePrefix` | string | No | `"inventoryquantities"` | CSV filename prefix |
234
- | `catalogueRef` | string | No | - | Filter by catalogue reference (optional) |
235
- | `pageSize` | number | No | `200` | GraphQL page size (max 500) |
236
- | `maxRecords` | number | No | `100000` | Maximum records per extraction |
237
- | `extractionMode` | string | No | `"incremental"` | Extraction mode: `incremental`, `dateRange`, or `historical` |
238
- | `fallbackStartDate` | string | No | `"2024-01-01T00:00:00Z"` | Fallback date if no state exists |
239
- | `overlapBufferSeconds` | number | No | `60` | Overlap buffer to prevent missed records (seconds) |
240
- | `startDate` | string | No | - | Manual start date (for `dateRange`/`historical` modes) |
241
- | `endDate` | string | No | - | Manual end date (for `dateRange`/`historical` modes) |
242
-
243
- ### Extraction Mode Configuration
244
-
245
- **Mode 1: Incremental (default)**
246
-
247
- ```json
248
- {
249
- "extractionMode": "incremental",
250
- "fallbackStartDate": "2024-01-01T00:00:00Z"
251
- }
252
- ```
253
-
254
- Extracts quantities with `updatedOn > lastRunTime`. Ideal for daily syncs.
255
-
256
- **Mode 2: Date Range**
257
-
258
- ```json
259
- {
260
- "extractionMode": "dateRange",
261
- "startDate": "2025-01-01T00:00:00Z",
262
- "endDate": "2025-01-15T23:59:59Z"
263
- }
264
- ```
265
-
266
- Extracts quantities updated between `startDate` and `endDate`. Ideal for audits.
267
-
268
- **Mode 3: Historical**
269
-
270
- ```json
271
- {
272
- "extractionMode": "historical",
273
- "startDate": "2024-01-01T00:00:00Z",
274
- "endDate": "2024-12-31T23:59:59Z"
275
- }
276
- ```
277
-
278
- Extracts quantities created between `startDate` and `endDate` using `createdOn` filter.
279
-
280
- ## ⚠️ Production Safety & Guardrails
281
-
282
- ### Critical: Extraction Mode Selection
283
-
284
- **🟢 RECOMMENDED: Incremental Mode (Production)**
285
-
286
- - Safe for automated schedules
287
- - Natural rate limiting via timestamps
288
- - Predictable resource usage
289
- - **Use this for all production workflows**
290
-
291
- **🟡 CAUTION: Date Range Mode (Audit/Backfill)**
292
-
293
- - **Maximum 30-day window enforced**
294
- - Use for specific audit requests only
295
- - Run during off-peak hours
296
- - Monitor resource usage
297
-
298
- **🔴 DANGER: Historical Mode (One-Time Only)**
299
-
300
- - **Maximum 90-day window enforced**
301
- - **Requires explicit approval**
302
- - **Risk of platform overload**
303
- - Can fetch millions of records
304
- - Use multiple small incremental runs instead
305
- - Only for initial data migration
306
-
307
- ### Date Range Validation (Required)
308
-
309
- ```typescript
310
- // Validate date range limits to prevent platform overload
311
- function validateDateRange(mode, startDate, endDate) {
312
- if (mode === 'incremental') return { valid: true };
313
-
314
- if (!startDate || !endDate) {
315
- return {
316
- valid: false,
317
- error: `${mode} mode requires both startDate and endDate`,
318
- };
319
- }
320
-
321
- const start = new Date(startDate);
322
- const end = new Date(endDate);
323
- const daysDiff = (end - start) / (1000 * 60 * 60 * 24);
324
-
325
- // Guardrail: Maximum date ranges
326
- const maxDays = mode === 'dateRange' ? 30 : 90;
327
-
328
- if (daysDiff > maxDays) {
329
- return {
330
- valid: false,
331
- error: `${mode} mode: Maximum ${maxDays}-day window. Requested: ${Math.round(daysDiff)} days. Use multiple smaller extractions or incremental mode.`,
332
- recommendation: `Split into ${Math.ceil(daysDiff / maxDays)} separate extractions of ${maxDays} days each.`,
333
- };
334
- }
335
-
336
- if (daysDiff < 0) {
337
- return { valid: false, error: 'endDate must be after startDate' };
338
- }
339
-
340
- return { valid: true };
341
- }
342
- ```
343
-
344
- ### File Splitting Configuration
345
-
346
- Large extractions must split into multiple files to prevent memory issues and upload failures.
347
-
348
- ```json
349
- {
350
- "maxRecordsPerFile": 50000,
351
- "maxFileSizeMB": 100,
352
- "enableFileSplitting": true
353
- }
354
- ```
355
-
356
- **File Naming Pattern:**
357
-
358
- ```
359
- {prefix}inventory-quantities-{timestamp}-part-{sequence}.csv
360
-
361
- Examples:
362
- inventory-quantities/daily/inventory-quantities-2025-01-22T14-30-00Z-part-001.csv
363
- inventory-quantities/daily/inventory-quantities-2025-01-22T14-30-00Z-part-002.csv
364
- inventory-quantities/daily/inventory-quantities-2025-01-22T14-30-00Z-manifest.json
365
- ```
366
-
367
- **Manifest File (auto-generated):**
368
-
369
- ```json
370
- {
371
- "extractionId": "inventory-quantities-2025-01-22T14-30-00Z",
372
- "totalRecords": 127543,
373
- "totalFiles": 3,
374
- "files": [
375
- {
376
- "filename": "inventory-quantities-2025-01-22T14-30-00Z-part-001.csv",
377
- "recordCount": 50000,
378
- "s3Key": "inventory-quantities/daily/inventory-quantities-2025-01-22T14-30-00Z-part-001.csv"
379
- },
380
- {
381
- "filename": "inventory-quantities-2025-01-22T14-30-00Z-part-002.csv",
382
- "recordCount": 50000,
383
- "s3Key": "inventory-quantities/daily/inventory-quantities-2025-01-22T14-30-00Z-part-002.csv"
384
- },
385
- {
386
- "filename": "inventory-quantities-2025-01-22T14-30-00Z-part-003.csv",
387
- "recordCount": 27543,
388
- "s3Key": "inventory-quantities/daily/inventory-quantities-2025-01-22T14-30-00Z-part-003.csv"
389
- }
390
- ],
391
- "extractionMode": "dateRange",
392
- "dateRange": {
393
- "from": "2025-01-01T00:00:00Z",
394
- "to": "2025-01-31T23:59:59Z"
395
- },
396
- "completedAt": "2025-01-22T14:35:27Z"
397
- }
398
- ```
399
-
400
- ### Hard Limits (Enforced)
401
-
402
- ```typescript
403
- const SAFETY_LIMITS = {
404
- // Maximum records per single extraction
405
- MAX_RECORDS_TOTAL: 500000, // 500k hard limit
406
-
407
- // Maximum records per file before splitting
408
- MAX_RECORDS_PER_FILE: 50000, // 50k per file
409
-
410
- // Maximum file size before splitting
411
- MAX_FILE_SIZE_MB: 100, // 100MB per file
412
-
413
- // Date range limits
414
- MAX_DATE_RANGE_DAYS: 30, // dateRange mode
415
- MAX_HISTORICAL_DAYS: 90, // historical mode
416
-
417
- // Pagination limits
418
- MAX_PAGE_SIZE: 500, // Fluent API limit
419
- RECOMMENDED_PAGE_SIZE: 200, // Balance throughput/memory
420
-
421
- // Memory management
422
- CHUNK_SIZE: 10000, // Process in chunks
423
- };
424
- ```
425
-
426
- ### Memory-Safe Implementation Pattern
427
-
428
- ```typescript
429
- // Process large extractions in chunks to prevent OOM
430
- async function processLargeExtraction(edges, mapper, csvParser, s3, options) {
431
- const CHUNK_SIZE = 10000;
432
- const MAX_RECORDS_PER_FILE = options.maxRecordsPerFile || 50000;
433
-
434
- let fileSequence = 1;
435
- let currentFileRecords = [];
436
- const manifestFiles = [];
437
-
438
- for (let i = 0; i < edges.length; i += CHUNK_SIZE) {
439
- const chunk = edges.slice(i, i + CHUNK_SIZE);
440
-
441
- // Bulk mapping for chunk
442
- const chunkNodes = chunk.map(edge => edge.node);
443
- const mappingResult = await mapper.map(chunkNodes);
444
-
445
- if (!mappingResult.success) {
446
- const mappingErrors = mappingResult.errors || ['Unknown mapping failure'];
447
- log.error('Chunk mapping failed', {
448
- chunkIndex: i / CHUNK_SIZE,
449
- errorCount: mappingErrors.length,
450
- sampleErrors: mappingErrors.slice(0, 3),
451
- });
452
- throw new Error(`Mapping failed: ${mappingErrors[0] || 'Unknown error'}`);
453
- }
454
-
455
- const transformedChunk = mappingResult.data || [];
456
- const mappingErrors = mappingResult.errors || [];
457
-
458
- if (mappingErrors.length > 0) {
459
- log.warn('Some records in chunk failed transformation', {
460
- chunkIndex: i / CHUNK_SIZE,
461
- errorCount: mappingErrors.length,
462
- });
463
- }
464
-
465
- if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
466
- log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
467
- chunkIndex: i / CHUNK_SIZE,
468
- skippedFields: mappingResult.skippedFields,
469
- note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
470
- });
471
- }
472
-
473
- // Add to current file, handling splits
474
- for (const record of transformedChunk) {
475
- currentFileRecords.push(record);
476
-
477
- // Split file when limit reached
478
- if (currentFileRecords.length >= MAX_RECORDS_PER_FILE) {
479
- const fileInfo = await writeFileToS3(
480
- currentFileRecords,
481
- fileSequence++,
482
- csvParser,
483
- s3,
484
- options
485
- );
486
- manifestFiles.push(fileInfo);
487
- currentFileRecords = []; // Reset for next file
488
- }
489
- }
490
- }
491
-
492
- // Write remaining records
493
- if (currentFileRecords.length > 0) {
494
- const fileInfo = await writeFileToS3(
495
- currentFileRecords,
496
- fileSequence++,
497
- csvParser,
498
- s3,
499
- options
500
- );
501
- manifestFiles.push(fileInfo);
502
- }
503
-
504
- // Write manifest
505
- await writeManifest(manifestFiles, s3, options);
506
-
507
- return manifestFiles;
508
- }
509
- ```
510
-
511
- ### Enterprise Time Buffer Configuration
512
-
513
- ```json
514
- {
515
- "overlapBufferSeconds": "60"
516
- }
517
- ```
518
-
519
- **Default: 60 seconds (recommended for most deployments)**
520
-
521
- **Purpose**: Prevents missed records due to:
522
-
523
- - **Clock skew** between Fluent API servers (typically 1-5 seconds)
524
- - **Transaction timing** - records updated during query execution
525
- - **Race conditions** - records updated between extraction runs
526
-
527
- **How It Works**:
528
-
529
- - **Query**: Uses `updatedOn >= (lastRunTime - 60 seconds)`
530
- - **Save**: Stores `MAX(updatedOn)` WITHOUT buffer
531
- - **Result**: Records from the last minute of previous extraction are included again
532
-
533
- **Buffer Sizes by Deployment**:
534
-
535
- - `30` - Low-latency single-region (minimal clock skew expected)
536
- - `60` - **Standard production** (recommended default)
537
- - `300` - Cross-region deployments or high-latency networks
538
-
539
- **Duplicate Handling**: Downstream systems should upsert by `quantity_id` (idempotent). Duplicates are safe and expected.
540
-
541
- ### Timezone Handling
542
-
543
- **All timestamps are in ISO 8601 format (UTC)**:
544
-
545
- ```typescript
546
- // Input: ISO 8601 UTC timestamp
547
- const timestamp = '2025-01-22T14:30:00.000Z';
548
-
549
- // JavaScript Date operations preserve UTC
550
- new Date(timestamp).toISOString();
551
- // Returns: "2025-01-22T14:30:00.000Z" (same format)
552
-
553
- new Date(timestamp).getTime();
554
- // Returns: 1737558600000 (UTC epoch milliseconds)
555
-
556
- // Subtract 60 seconds for buffer
557
- const buffered = new Date(new Date(timestamp).getTime() - 60000).toISOString();
558
- // Returns: "2025-01-22T14:29:00.000Z"
559
- ```
560
-
561
- **Key Points**:
562
-
563
- - Fluent API returns all timestamps in UTC
564
- - `.getTime()` returns UTC epoch milliseconds
565
- - Buffer arithmetic is done in milliseconds
566
- - `.toISOString()` converts back to ISO 8601 UTC
567
- - No timezone conversion needed
568
-
569
- ## Export Mapping Configuration
570
-
571
- Create file: `./config/inventory-quantities.export.json`
572
-
573
- ```json
574
- {
575
- "name": "inventory-quantities.export",
576
- "version": "1.0.0",
577
- "description": "Fluent Inventory Quantities → CSV Export Mapping",
578
- "fields": {
579
- "quantity_id": { "source": "id", "required": true, "resolver": "sdk.trim" },
580
- "quantity_ref": { "source": "ref", "required": true, "resolver": "sdk.trim" },
581
- "catalogue_ref": { "source": "catalogue.ref", "required": false, "resolver": "sdk.trim" },
582
- "catalogue_name": { "source": "catalogue.name", "required": false, "resolver": "sdk.trim" },
583
- "location": { "source": "locationRef", "required": true, "resolver": "sdk.trim" },
584
- "sku": { "source": "skuRef", "required": true, "resolver": "sdk.trim" },
585
- "quantity": { "source": "qty", "required": true, "resolver": "sdk.parseInt" },
586
- "type": { "source": "type", "required": true, "resolver": "sdk.uppercase" },
587
- "status": { "source": "status", "required": true, "resolver": "sdk.uppercase" },
588
- "expected_on": { "source": "expectedOn", "resolver": "sdk.formatDate" },
589
- "created_on": { "source": "createdOn", "resolver": "sdk.formatDate" },
590
- "updated_on": { "source": "updatedOn", "required": true, "resolver": "sdk.formatDate" }
591
- }
592
- }
593
- ```
594
-
595
- ## Mapping & Resolvers Explained
596
-
597
- This section explains how the SDK transforms raw GraphQL data into your CSV export format using **UniversalMapper** and **SDK resolvers**.
598
-
599
- ### SDK Resolvers Used
600
-
601
- | Field | Resolver | Why? | Example Transformation |
602
- | ---------------- | ---------------- | ------------------------------------------ | ----------------------------------------------- |
603
- | `quantity_id` | `sdk.trim` | Clean quantity IDs from whitespace | `" Q001 "` → `"Q001"` |
604
- | `quantity_ref` | `sdk.trim` | Clean quantity references | `" QTY-REF-001 "` → `"QTY-REF-001"` |
605
- | `catalogue_ref` | `sdk.trim` | Clean catalogue references | `" DEFAULT_CATALOGUE "` → `"DEFAULT_CATALOGUE"` |
606
- | `catalogue_name` | `sdk.trim` | Clean catalogue names | `" Default Catalogue "` → `"Default Catalogue"` |
607
- | `location` | `sdk.trim` | Clean location references | `" DC01 "` → `"DC01"` |
608
- | `sku` | `sdk.trim` | Clean SKU references | `" SKU-001 "` → `"SKU-001"` |
609
- | `quantity` | `sdk.parseInt` | Parse quantity as integer for calculations | `"100"` → `100` |
610
- | `type` | `sdk.uppercase` | Normalize type codes | `"available"` → `"AVAILABLE"` |
611
- | `status` | `sdk.uppercase` | Normalize status codes | `"active"` → `"ACTIVE"` |
612
- | `expected_on` | `sdk.formatDate` | Format dates for CSV export | `"2025-01-30T00:00:00.000Z"` → `"2025-01-30"` |
613
- | `created_on` | `sdk.formatDate` | Format created timestamps | `"2025-01-15T10:00:00.000Z"` → `"2025-01-15"` |
614
- | `updated_on` | `sdk.formatDate` | Format updated timestamps for tracking | `"2025-01-22T08:30:00.000Z"` → `"2025-01-22"` |
615
-
616
- ### Transformation Flow
617
-
618
- ```typescript
619
- // 1. GraphQL Response (raw data from Fluent Commerce)
620
- const rawQuantity = {
621
- id: ' Q001 ',
622
- ref: ' QTY-REF-001 ',
623
- locationRef: ' DC01 ',
624
- skuRef: ' SKU-001 ',
625
- qty: '100',
626
- type: 'available',
627
- status: 'active',
628
- expectedOn: null,
629
- createdOn: '2025-01-15T10:00:00.000Z',
630
- updatedOn: '2025-01-22T08:30:00.000Z',
631
- catalogue: {
632
- ref: ' DEFAULT_CATALOGUE ',
633
- name: ' Default Catalogue ',
634
- },
635
- };
636
-
637
- // 2. UniversalMapper applies SDK resolvers
638
- const mapper = new UniversalMapper(inventoryQuantitiesExportMapping);
639
- const result = await mapper.map(rawQuantity);
640
-
641
- // 3. Transformed Output (clean, normalized for CSV)
642
- const transformedQuantity = {
643
- quantity_id: 'Q001',
644
- quantity_ref: 'QTY-REF-001',
645
- catalogue_ref: 'DEFAULT_CATALOGUE',
646
- catalogue_name: 'Default Catalogue',
647
- location: 'DC01',
648
- sku: 'SKU-001',
649
- quantity: 100,
650
- type: 'AVAILABLE',
651
- status: 'ACTIVE',
652
- expected_on: '', // null → empty string
653
- created_on: '2025-01-15',
654
- updated_on: '2025-01-22',
655
- };
656
- ```
657
-
658
- ### Custom Resolvers for Inventory Quantity-Specific Logic
659
-
660
- While the mapping above uses built-in SDK resolvers, you can extend with custom business logic:
661
-
662
- ```typescript
663
- const customResolvers = {
664
- /**
665
- * Validate that quantity values are positive
666
- */
667
- 'custom.validateQuantity': (qty: any) => {
668
- const parsed = parseInt(qty) || 0;
669
- return parsed >= 0 ? parsed : 0; // Ensure non-negative
670
- },
671
-
672
- /**
673
- * Add human-readable type descriptions for reporting
674
- */
675
- 'custom.enrichQuantityType': (type: string) => {
676
- const typeDescriptions: Record<string, string> = {
677
- LAST_ON_HAND: 'Last recorded on-hand quantity',
678
- RESERVED: 'Reserved against orders',
679
- DELTA: 'Incremental change (adjustment delta)',
680
- SALE: 'Quantity decreased due to sale',
681
- CORRECTION: 'Manual correction entry',
682
- };
683
- return typeDescriptions[(type || '').toUpperCase()] || type;
684
- },
685
-
686
- /**
687
- * Check if expected date is in the future
688
- */
689
- 'custom.isExpectedInFuture': (expectedOn: string) => {
690
- if (!expectedOn) return false;
691
- return new Date(expectedOn) > new Date();
692
- },
693
-
694
- /**
695
- * Calculate days until expected arrival
696
- */
697
- 'custom.calculateDaysUntilExpected': (expectedOn: string) => {
698
- if (!expectedOn) return null;
699
- const expected = new Date(expectedOn);
700
- const today = new Date();
701
- const diffMs = expected.getTime() - today.getTime();
702
- return Math.ceil(diffMs / (1000 * 60 * 60 * 24));
703
- },
704
-
705
- /**
706
- * Validate status-type combinations
707
- */
708
- 'custom.validateQuantityStatus': (_quantity: any) => {
709
- // Example placeholder – adapt rules to your retailer-defined IQ types
710
- return 'VALID';
711
- },
712
- };
713
-
714
- // Use custom resolvers with UniversalMapper
715
- const mapper = new UniversalMapper(inventoryQuantitiesExportMapping, {
716
- customResolvers,
717
- });
718
- ```
719
-
720
- ### Available SDK Resolvers
721
-
722
- The SDK provides these built-in resolvers (no custom code needed):
723
-
724
- **String Transformations:**
725
-
726
- - `sdk.trim` - Remove leading/trailing whitespace
727
- - `sdk.uppercase` - Convert to uppercase
728
- - `sdk.lowercase` - Convert to lowercase
729
- - `sdk.toString` - Convert to string
730
-
731
- **Number Parsing:**
732
-
733
- - `sdk.parseInt` - Parse as integer
734
- - `sdk.parseFloat` - Parse as decimal
735
- - `sdk.number` - Parse as number (auto-detect int/float)
736
-
737
- **Date Formatting:**
738
-
739
- - `sdk.formatDate` - ISO 8601 date only (YYYY-MM-DD)
740
- - `sdk.formatDateShort` - Short date format
741
- - `sdk.parseDate` - Parse various date formats
742
-
743
- **Type Conversions:**
744
-
745
- - `sdk.boolean` - Convert to boolean
746
- - `sdk.parseJson` - Parse JSON strings
747
- - `sdk.toJson` - Convert to JSON string
748
-
749
- **Utilities:**
750
-
751
- - `sdk.identity` - Return value unchanged
752
- - `sdk.coalesce` - Return first non-null value
753
-
754
- See [Universal Mapping Guide](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) for complete resolver documentation.
755
-
756
- ## GraphQL Query
757
-
758
- ```graphql
759
- query GetInventoryQuantities(
760
- $retailerId: ID!
761
- $updatedAfter: DateTime
762
- $createdAfter: DateTime
763
- $first: Int!
764
- $after: String
765
- ) {
766
- inventoryQuantities(
767
- retailerId: $retailerId
768
- updatedOn: { after: $updatedAfter }
769
- createdOn: { after: $createdAfter }
770
- first: $first
771
- after: $after
772
- ) {
773
- edges {
774
- node {
775
- id
776
- ref
777
- locationRef
778
- skuRef
779
- qty
780
- type
781
- status
782
- expectedOn
783
- createdOn
784
- updatedOn
785
- catalogue {
786
- ref
787
- name
788
- }
789
- }
790
- cursor
791
- }
792
- pageInfo {
793
- hasNextPage
794
- # Note: Fluent doesn't return endCursor/startCursor - cursors are in edges[].cursor
795
- }
796
- }
797
- }
798
- ```
799
-
800
- ## Guardrails Implementation (Required)
801
-
802
- ```typescript
803
- // Overlap buffer (safety window)
804
- const overlapBufferSeconds = parseInt(
805
- ctx.activation?.getVariable('overlapBufferSeconds') || '60',
806
- 10
807
- );
808
- const OVERLAP_BUFFER_MS = overlapBufferSeconds * 1000;
809
-
810
- // Read last successful run and apply buffer
811
- const kv = new VersoriKVAdapter(ctx.openKv(':project:'));
812
- const stateKey = ['extraction', 'inventory-quantities-csv', 'lastRunTime'];
813
- const lastRunState = await kv.get(stateKey);
814
- const rawLastRunTime = lastRunState?.value?.timestamp || fallbackStartDate;
815
- const bufferedLastRunTime = new Date(
816
- new Date(rawLastRunTime).getTime() - OVERLAP_BUFFER_MS
817
- ).toISOString();
818
-
819
- // Query WITH buffer
820
- const result = await client.graphql({
821
- query: INVENTORY_QUANTITIES_QUERY,
822
- variables: {
823
- retailerId,
824
- updatedAfter: bufferedLastRunTime,
825
- first: pageSize,
826
- },
827
- pagination: { maxRecords },
828
- });
829
-
830
- const edges = result.data?.inventoryQuantities?.edges || [];
831
-
832
- // 🛡️ GUARDRAIL: Validate extraction size limits
833
- const MAX_RECORDS_PER_RUN = 500000;
834
- const ESTIMATED_BYTES_PER_RECORD = 300; // Smaller than positions
835
- const estimatedSizeMB = (edges.length * ESTIMATED_BYTES_PER_RECORD) / (1024 * 1024);
836
- const MAX_CSV_SIZE_MB = 100;
837
-
838
- if (edges.length > MAX_RECORDS_PER_RUN) {
839
- log.error('Extraction limit exceeded', {
840
- recordCount: edges.length,
841
- maxAllowed: MAX_RECORDS_PER_RUN,
842
- });
843
- return {
844
- success: false,
845
- error: `Extraction limit exceeded: ${edges.length} records (max: ${MAX_RECORDS_PER_RUN})`,
846
- recommendation: `Split into smaller extractions or increase extraction frequency`,
847
- recordCount: edges.length,
848
- maxAllowed: MAX_RECORDS_PER_RUN,
849
- };
850
- }
851
-
852
- if (estimatedSizeMB > MAX_CSV_SIZE_MB) {
853
- log.warn('CSV size approaching limit', {
854
- estimatedSizeMB: estimatedSizeMB.toFixed(2),
855
- maxAllowed: MAX_CSV_SIZE_MB,
856
- });
857
- }
858
-
859
- log.info('Extraction limits validated', {
860
- recordCount: edges.length,
861
- estimatedSizeMB: estimatedSizeMB.toFixed(2),
862
- withinLimits: true,
863
- });
864
-
865
- // Transform with UniversalMapper
866
- const mapper = new UniversalMapper(inventoryQuantitiesExportMapping);
867
- const transformedRecords: any[] = [];
868
- for (const edge of edges) {
869
- const mapped = await mapper.map(edge.node);
870
- if (mapped.success) {
871
- transformedRecords.push(mapped.data);
872
- }
873
- }
874
-
875
- // Save state WITHOUT buffer (use MAX(updatedOn))
876
- const maxUpdatedOn = transformedRecords.reduce((max, r) => {
877
- const t = new Date(r.updated_on).getTime();
878
- return t > max ? t : max;
879
- }, new Date(rawLastRunTime).getTime());
880
-
881
- await kv.set(stateKey, {
882
- timestamp: new Date(maxUpdatedOn).toISOString(),
883
- recordCount: transformedRecords.length,
884
- extractedAt: new Date().toISOString(),
885
- overlapBufferSeconds,
886
- });
887
-
888
- // Date range guardrails (if you add dateRange/historical modes)
889
- function validateDateRange(mode: 'dateRange' | 'historical', from: string, to: string) {
890
- const start = new Date(from);
891
- const end = new Date(to);
892
- const daysDiff = (end.getTime() - start.getTime()) / (1000 * 60 * 60 * 24);
893
- const maxDays = mode === 'dateRange' ? 30 : 90;
894
- if (daysDiff > maxDays) {
895
- return {
896
- valid: false,
897
- error: `${mode} mode: Maximum ${maxDays}-day window. Requested: ${Math.round(daysDiff)} days.`,
898
- };
899
- }
900
- if (daysDiff < 0) return { valid: false, error: 'endDate must be after startDate' };
901
- return { valid: true };
902
- }
903
- ```
904
-
905
- ---
906
-
907
- ## Versori Workflows Structure
908
-
909
- **Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
910
-
911
- **Trigger Types:**
912
- - **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
913
- - **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
914
- - **`workflow()`** → Durable workflows (advanced, rarely used)
915
-
916
- **Execution Steps (chained to triggers):**
917
- - **`http()`** → External API calls (chained from schedule/webhook)
918
- - **`fn()`** → Internal processing (chained from schedule/webhook)
919
-
920
- ### Recommended Project Structure
921
-
922
- ```
923
- inventory-quantities-extraction/
924
- ├── index.ts # Entry point - exports all workflows
925
- └── src/
926
- ├── workflows/
927
- │ ├── scheduled/
928
- │ │ └── daily-inventory-quantities-extraction.ts # Scheduled: Daily extraction
929
- │ │
930
- │ └── webhook/
931
- │ ├── adhoc-inventory-quantities-extraction.ts # Webhook: Manual trigger
932
- │ └── job-status-check.ts # Webhook: Status query
933
-
934
- ├── services/
935
- │ └── inventory-quantities-extraction.service.ts # Shared orchestration logic (reusable)
936
-
937
- └── config/
938
- └── inventory-quantities.export.csv.json # Mapping configuration
939
- ```
940
-
941
- ---
942
-
943
- ````csv
944
- quantity_id,quantity_ref,catalogue_ref,catalogue_name,location,sku,quantity,type,status,expected_on,created_on,updated_on
945
- Q001,QTY-REF-001,DEFAULT_CATALOGUE,Default Catalogue,DC01,SKU-001,100,AVAILABLE,ACTIVE,,2025-01-15T10:00:00Z,2025-01-22T08:30:00Z
946
- Q002,QTY-REF-002,DEFAULT_CATALOGUE,Default Catalogue,DC01,SKU-001,50,RESERVED,ACTIVE,,2025-01-16T11:00:00Z,2025-01-22T09:15:00Z
947
- Q003,QTY-REF-003,DEFAULT_CATALOGUE,Default Catalogue,DC02,SKU-002,200,EXPECTED,CREATED,2025-01-30T00:00:00Z,2025-01-17T12:00:00Z,2025-01-22T10:00:00Z
948
- Q004,QTY-REF-004,DEFAULT_CATALOGUE,Default Catalogue,STORE-NYC,SKU-003,25,AVAILABLE,ACTIVE,,2025-01-18T13:00:00Z,2025-01-22T11:00:00Z
949
-
950
- ## Advanced Mapping Patterns
951
-
952
- ### Array Mapping (Preserving Nested Structure)
953
-
954
- For nested data structures, use `isArray: true` pattern:
955
-
956
- ```json
957
- {
958
- "fields": {
959
- "ref": { "source": "ref", "required": true },
960
- "relatedItems": {
961
- "source": "items",
962
- "isArray": true,
963
- "fields": {
964
- "itemRef": { "source": "ref", "required": true },
965
- "value": { "source": "value", "resolver": "sdk.parseFloat" }
966
- }
967
- }
968
- }
969
- }
970
- ````
971
-
972
- **When to use**:
973
-
974
- - **Flattened structure**: Simpler, easier for downstream systems
975
- - **Nested with arrays**: Complex data, preserves relationships
976
-
977
- ### Nested Object Mapping
978
-
979
- **Option 1: Flattened paths** (recommended):
980
-
981
- ```json
982
- {
983
- "fields": {
984
- "location_ref": { "source": "location.ref" },
985
- "location_name": { "source": "location.name" }
986
- }
987
- }
988
- ```
989
-
990
- **Option 2: Nested object definition**:
991
-
992
- ```json
993
- {
994
- "fields": {
995
- "location": {
996
- "fields": {
997
- "ref": { "source": "location.ref" },
998
- "name": { "source": "location.name" }
999
- }
1000
- }
1001
- }
1002
- }
1003
- ```
1004
-
1005
- ## Error Handling Strategies
1006
-
1007
- ### Handling Mapping Failures
1008
-
1009
- **Strategy 1: Fail-fast (strict)**:
1010
-
1011
- ```typescript
1012
- if (errors.length > 0) {
1013
- throw new Error(`${errors.length} records failed mapping validation`);
1014
- }
1015
- ```
1016
-
1017
- **Strategy 2: Threshold-based (recommended)**:
1018
-
1019
- ```typescript
1020
- const errorRate = errors.length / transformed.length;
1021
- if (errorRate > 0.05) {
1022
- // 5% threshold
1023
- throw new Error(`Error rate too high: ${(errorRate * 100).toFixed(1)}%`);
1024
- }
1025
- ```
1026
-
1027
- **Strategy 3: Upload error manifest**:
1028
-
1029
- ```typescript
1030
- if (errors.length > 0) {
1031
- const errorManifest = {
1032
- extractionTimestamp: new Date().toISOString(),
1033
- totalErrors: errors.length,
1034
- errors: errors.map(e => ({ record: e.record, errors: e.errors })),
1035
- };
1036
- // Upload to storage for review
1037
- }
1038
- ```
1039
-
1040
- ### State Management with Partial Failures
1041
-
1042
- **Recommended**: Only update state if extraction succeeded:
1043
-
1044
- ```typescript
1045
- if (errors.length === 0) {
1046
- await kv.set(stateKey, { timestamp: newTimestamp });
1047
- log.info('State updated - all records successful');
1048
- } else {
1049
- log.warn('State NOT updated - will retry next run', {
1050
- failedRecords: errors.length,
1051
- willRetryNextRun: true,
1052
- });
1053
- }
1054
- ```
1055
-
1056
- ## GraphQL Query Validation & Testing
1057
-
1058
- ### Schema Validation Workflow
1059
-
1060
- **Step 1: Introspect schema**
1061
-
1062
- ```bash
1063
- npx fc-connect introspect-schema \
1064
- --url https://your-instance.api.fluentcommerce.com/graphql \
1065
- --output fluent-schema.json
1066
- ```
1067
-
1068
- **Step 2: Validate mapping**
1069
-
1070
- ```bash
1071
- npx fc-connect validate-schema \
1072
- --mapping ./config/mapping.json \
1073
- --schema ./fluent-schema.json
1074
- ```
1075
-
1076
- **Step 3: Analyze coverage**
1077
-
1078
- ```bash
1079
- npx fc-connect analyze-coverage \
1080
- --mapping ./config/mapping.json \
1081
- --schema ./fluent-schema.json
1082
- ```
1083
-
1084
- ### GraphQL Pagination Explained
1085
-
1086
- The SDK handles pagination automatically:
1087
-
1088
- ```typescript
1089
- await client.graphql({
1090
- query: QUERY,
1091
- variables: { first: pageSize },
1092
- pagination: { maxRecords }, // SDK handles cursors automatically
1093
- });
1094
- ```
1095
-
1096
- ## Date Format Handling
1097
-
1098
- | Format | Resolver | Output | Use Case |
1099
- | -------- | --------------------- | -------------------------- | --------- |
1100
- | CSV/JSON | `sdk.formatDate` | `2025-01-22T14:30:00.000Z` | ISO 8601 |
1101
- | CSV/JSON | `sdk.formatDateShort` | `2025-01-22` | Date only |
1102
- | CSV/JSON | `sdk.toString` | Pass through | As-is |
1103
-
1104
- ## Monitoring & Alerting
1105
-
1106
- ### Key Metrics to Track
1107
-
1108
- ```typescript
1109
- const metrics = {
1110
- extractionDurationMs: Date.now() - startTime,
1111
- recordCount: edges.length,
1112
- transformedCount: transformed.length,
1113
- failedCount: errors.length,
1114
- errorRate: ((errors.length / edges.length) * 100).toFixed(2) + '%',
1115
- fileSizeMB: (buffer.length / (1024 * 1024)).toFixed(2),
1116
- lastRunTime: rawLastRunTime,
1117
- newTimestamp: newTimestamp,
1118
- };
1119
- log.info('Extraction complete', metrics);
1120
- ```
1121
-
1122
- ### Alert Thresholds
1123
-
1124
- ```typescript
1125
- const ALERTS = {
1126
- EXTRACTION_DURATION_MS: 5 * 60 * 1000, // 5 minutes
1127
- MAX_ERROR_RATE: 0.05, // 5%
1128
- MAX_FILE_SIZE_MB: 150, // 150MB
1129
- MAX_RECORDS_PER_RUN: 100000, // Adjust per entity
1130
- };
1131
- ```
1132
-
1133
- ## Testing Checklist
1134
-
1135
- **Before production deployment:**
1136
-
1137
- ### 1. Schema Validation
1138
-
1139
- - [ ] Run `npx fc-connect introspect-schema`
1140
- - [ ] Run `npx fc-connect validate-schema`
1141
- - [ ] Run `npx fc-connect analyze-coverage`
1142
- - [ ] Verify all `source` paths exist
1143
-
1144
- ### 2. Mapping Testing
1145
-
1146
- - [ ] Test with sample data (maxRecords=10)
1147
- - [ ] Verify required fields populated
1148
- - [ ] Verify SDK resolvers work correctly
1149
- - [ ] Test custom resolvers with edge cases
1150
-
1151
- ### 3. Error Handling
1152
-
1153
- - [ ] Test with invalid data
1154
- - [ ] Verify error collection
1155
- - [ ] Test error threshold logic
1156
-
1157
- ### 4. State Management
1158
-
1159
- - [ ] Verify overlap buffer prevents misses
1160
- - [ ] Test state recovery after failure
1161
- - [ ] Verify timestamp saved WITHOUT buffer
1162
-
1163
- ### 5. File Operations
1164
-
1165
- - [ ] Test connection and upload
1166
- - [ ] Verify file format validity
1167
- - [ ] Test with large files (>50MB)
1168
-
1169
- ### 6. Staging Environment
1170
-
1171
- - [ ] Run full extraction in staging
1172
- - [ ] Verify file format with downstream system
1173
- - [ ] Monitor duration and resource usage
1174
-
1175
- ## Troubleshooting Guide
1176
-
1177
- **Issue**: "Extraction timeout after 10 minutes"
1178
-
1179
- - **Cause**: Too many records
1180
- - **Fix**: Reduce maxRecords, increase frequency
1181
-
1182
- **Issue**: "Mapping errors for 50% of records"
1183
-
1184
- - **Cause**: Schema mismatch
1185
- - **Fix**: Run schema validation, check field names
1186
-
1187
- **Issue**: "State not updating"
1188
-
1189
- - **Cause**: KV write failure or intentional retry
1190
- - **Fix**: Check KV logs, verify state update code
1191
-
1192
- **Issue**: "First run exceeds limits"
1193
-
1194
- - **Cause**: No previous timestamp, fetches all
1195
- - **Fix**: Set fallbackStartDate close to current, apply filters
1196
-
1197
- **Issue**: "Excessive duplicates"
1198
-
1199
- - **Cause**: Overlap buffer (expected) or timestamp not saved
1200
- - **Fix**: Verify newTimestamp saved WITHOUT buffer
1201
-
1202
- ## Security Best Practices
1203
-
1204
- ### Credential Management
1205
-
1206
- **✅ DO**:
1207
-
1208
- - Store credentials in Versori activation variables
1209
- - Rotate credentials quarterly
1210
- - Use least-privilege accounts
1211
-
1212
- **❌ DON'T**:
1213
-
1214
- - Never log credentials
1215
- - Never commit to git
1216
- - Never share across environments
1217
-
1218
- ### Data Security
1219
-
1220
- - Enable encryption in transit and at rest
1221
- - Apply data retention policies
1222
- - Monitor access logs
1223
- - Use VPC/private networks for sensitive data
1224
-
1225
- ---
1226
-
1227
- ```
1228
-
1229
- ---
1230
-
1231
- **Pattern**: Enterprise incremental extraction with overlap buffer for inventory quantities
1232
- **⚠️ Sample Code**: For SDK demonstration only - **ONLY use incremental mode in production**
1233
- **Key Learning**: Use VersoriKVAdapter for state management with 60-second overlap buffer
1234
- **Critical**: Apply overlap buffer to prevent missed records due to clock skew (default: 60 seconds)
1235
- **Buffer Pattern**: Query WITH buffer (`updatedOn >= lastRunTime - 60s`), save WITHOUT buffer (`MAX(updatedOn)`)
1236
- **Timezone**: All timestamps are ISO 8601 UTC format - no conversion needed
1237
- ```
1238
-
1239
- ---
1240
-
1241
- ## 🔧 Complete Production Code
1242
-
1243
- ### 1. Entry Point (src/index.ts)
1244
-
1245
- ```typescript
1246
- /**
1247
- * Entry Point - Registers all workflows with Versori platform
1248
- *
1249
- * This file is the entry point for the Versori deployment.
1250
- * It imports and re-exports workflows from their respective files:
1251
- * 1. Scheduled extraction (runs automatically on cron schedule)
1252
- * 2. Ad hoc webhook (manual trigger with optional date override)
1253
- * 3. Job status webhook (query job progress)
1254
- *
1255
- * AI CUSTOMIZATION:
1256
- * - Add new workflows by importing from their respective files
1257
- * - Remove workflows by commenting out imports/exports
1258
- * - Organize workflows by type (scheduled vs webhook) for clarity
1259
- */
1260
-
1261
- import { scheduledInventoryQuantitiesExtraction } from './workflows/scheduled/daily-inventory-quantities-extraction';
1262
- import { adhocInventoryQuantitiesExtraction } from './workflows/webhook/adhoc-inventory-quantities-extraction';
1263
- import { inventoryQuantitiesJobStatus } from './workflows/webhook/job-status-check';
1264
-
1265
- // Register workflows with Versori platform
1266
- // The platform will expose webhooks as HTTP endpoints and run scheduled workflows on cron schedule
1267
-
1268
- export {
1269
- scheduledInventoryQuantitiesExtraction, // Cron-based auto-run (NOT exposed as HTTP endpoint)
1270
- adhocInventoryQuantitiesExtraction, // Manual webhook trigger (HTTP endpoint)
1271
- inventoryQuantitiesJobStatus, // Job status query (HTTP endpoint)
1272
- };
1273
- ```
1274
-
1275
- ---
1276
-
1277
- ### 2. Workflows
1278
-
1279
- #### src/workflows/scheduled/daily-inventory-quantities-extraction.ts
1280
-
1281
- ```typescript
1282
- /**
1283
- * WORKFLOW 1: Scheduled Extraction
1284
- *
1285
- * Purpose: Automated hourly extraction for incremental sync
1286
- * Trigger: Cron schedule (every hour at minute 0)
1287
- * State Update: Always updates lastSync timestamp
1288
- *
1289
- * AI CUSTOMIZATION:
1290
- * - Change schedule: Replace '0 * * * *' with your cron expression
1291
- * Examples:
1292
- * - Every 30 min: '*/30 * * * *'
1293
- * - Daily at 2 AM: '0 2 * * *'
1294
- * - Every 15 min: '*/15 * * * *'
1295
- */
1296
-
1297
- import { schedule, fn } from '@versori/run';
1298
- import { executeInventoryQuantityExtraction } from '../../services/extraction-orchestration';
1299
- import { generateJobId } from '../../utils/job-id-generator';
1300
-
1301
- /**
1302
- * WORKFLOW 1: Scheduled Extraction
1303
- *
1304
- * Purpose: Automated hourly extraction for incremental sync
1305
- * Trigger: Cron schedule (every hour at minute 0)
1306
- * State Update: Always updates lastSync timestamp
1307
- *
1308
- * AI CUSTOMIZATION:
1309
- * - Change schedule: Replace '0 * * * *' with your cron expression
1310
- * Examples:
1311
- * - Every 30 min: '*/30 * * * *'
1312
- * - Daily at 2 AM: '0 2 * * *'
1313
- * - Every 15 min: '*/15 * * * *'
1314
- */
1315
- export const scheduledInventoryQuantitiesExtraction = schedule(
1316
- 'inventory-quantities-scheduled',
1317
- '0 * * * *', // ← CUSTOMIZE: Cron expression
1318
- fn('execute-scheduled-extraction', async (ctx) => {
1319
- const { log, activation } = ctx;
1320
- const startTime = Date.now();
1321
-
1322
- // Generate unique job ID for tracking
1323
- // Format: SCHEDULED_IQ_YYYYMMDD_HHMMSS_random
1324
- const jobId = generateJobId('SCHEDULED', 'INVENTORY_QUANTITIES');
1325
-
1326
- log.info('🚀 [START] Scheduled extraction triggered', { jobId });
1327
-
1328
- try {
1329
- // Execute main workflow (extraction → transform → upload)
1330
- const result = await executeInventoryQuantityExtraction(ctx, {
1331
- jobId,
1332
- triggeredBy: 'schedule',
1333
- updateState: true, // Always update state for scheduled runs
1334
- });
1335
-
1336
- const durationMs = Date.now() - startTime;
1337
-
1338
- log.info('✅ [END] Scheduled extraction completed', {
1339
- jobId,
1340
- recordCount: result.recordsExtracted,
1341
- fileName: result.fileName,
1342
- durationMs,
1343
- durationSec: (durationMs / 1000).toFixed(2)
1344
- });
1345
-
1346
- return result;
1347
-
1348
- } catch (error: any) {
1349
- const durationMs = Date.now() - startTime;
1350
-
1351
- log.error('❌ [ERROR] Scheduled extraction failed', {
1352
- jobId,
1353
- message: error instanceof Error ? error.message : String(error),
1354
- stack: error instanceof Error ? error.stack : undefined,
1355
- errorType: error instanceof Error ? error.constructor.name : 'Error',
1356
- durationMs,
1357
- recommendation: 'Check Fluent API connectivity, S3 credentials, and date range configuration'
1358
- });
1359
- throw error;
1360
- }
1361
- }));
1362
- ```
1363
-
1364
- ---
1365
-
1366
- #### src/workflows/webhook/adhoc-inventory-quantities-extraction.ts
1367
-
1368
- ```typescript
1369
- /**
1370
- * WORKFLOW 2: Ad hoc Extraction (Manual Trigger)
1371
- *
1372
- * Purpose: Manual extraction with optional date range override
1373
- * Trigger: Webhook POST to /webhooks/inventory-quantities-adhoc
1374
- * State Update: Optional (controlled by request payload)
1375
- *
1376
- * WEBHOOK PAYLOAD EXAMPLES:
1377
- *
1378
- * 1. Incremental (use last sync timestamp):
1379
- * {}
1380
- *
1381
- * 2. Date range (manual override):
1382
- * {
1383
- * "fromDate": "2025-01-01T00:00:00Z",
1384
- * "toDate": "2025-01-31T23:59:59Z",
1385
- * "updateState": false
1386
- * }
1387
- *
1388
- * AI CUSTOMIZATION:
1389
- * - Add request validation
1390
- * - Add authentication check
1391
- * - Add custom filters from payload
1392
- */
1393
-
1394
- import { webhook, fn } from '@versori/run';
1395
- import { executeInventoryQuantityExtraction } from '../../services/extraction-orchestration';
1396
- import { generateJobId } from '../../utils/job-id-generator';
1397
-
1398
- export const adhocInventoryQuantitiesExtraction = webhook(
1399
- 'inventory-quantities-adhoc',
1400
- { connection: 'inventory-quantities-adhoc', response: { mode: 'sync' } },
1401
- fn('execute-adhoc-extraction', async (ctx) => {
1402
- const { data, log, connections, activation } = ctx;
1403
- const startTime = Date.now();
1404
-
1405
- // Generate unique job ID
1406
- const jobId = generateJobId('ADHOC', 'INVENTORY_QUANTITIES');
1407
-
1408
- // SECURITY: Authentication is enforced by Versori connection configuration
1409
- // Configure auth on the connection and reference it in webhook({ connection: '...' })
1410
-
1411
- // Extract optional date override from webhook payload
1412
- const fromDate = data.fromDate as string | undefined;
1413
- const toDate = data.toDate as string | undefined;
1414
- const updateState = data.updateState === true; // Default false; advance state only if explicitly true
1415
-
1416
- log.info('🌐 [START] Ad hoc extraction triggered via webhook', {
1417
- jobId,
1418
- hasDateOverride: !!fromDate,
1419
- fromDate: fromDate || 'not specified',
1420
- toDate: toDate || 'not specified',
1421
- updateState
1422
- });
1423
-
1424
- try {
1425
- // Execute main workflow with optional overrides
1426
- const result = await executeInventoryQuantityExtraction(ctx, {
1427
- jobId,
1428
- triggeredBy: 'webhook',
1429
- fromDate, // Optional: override start date
1430
- toDate, // Optional: override end date
1431
- updateState, // Optional: skip state update for historical queries
1432
- });
1433
-
1434
- const durationMs = Date.now() - startTime;
1435
-
1436
- log.info('✅ [END] Ad hoc extraction completed', {
1437
- jobId,
1438
- recordCount: result.recordsExtracted,
1439
- fileName: result.fileName,
1440
- isManualOverride: !!fromDate,
1441
- stateUpdated: result.stateUpdated,
1442
- durationMs,
1443
- durationSec: (durationMs / 1000).toFixed(2)
1444
- });
1445
-
1446
- // Return success with job details
1447
- return {
1448
- success: true,
1449
- jobId,
1450
- recordsExtracted: result.recordsExtracted,
1451
- fileName: result.fileName,
1452
- s3Path: result.s3Path,
1453
- statusUrl: `/webhooks/inventory-quantities-job-status?jobId=${jobId}`,
1454
- durationMs,
1455
- dateRange: fromDate ? {
1456
- from: fromDate,
1457
- to: toDate || 'not specified',
1458
- updateState
1459
- } : undefined
1460
- };
1461
-
1462
- } catch (error: any) {
1463
- const durationMs = Date.now() - startTime;
1464
-
1465
- log.error('❌ [ERROR] Ad hoc extraction failed', {
1466
- jobId,
1467
- message: error instanceof Error ? error.message : String(error),
1468
- stack: error instanceof Error ? error.stack : undefined,
1469
- errorType: error instanceof Error ? error.constructor.name : 'Error',
1470
- durationMs,
1471
- recommendation: 'Verify date range format (ISO 8601), check S3 credentials, and ensure Fluent API access'
1472
- });
1473
-
1474
- return {
1475
- success: false,
1476
- jobId,
1477
- message: error instanceof Error ? error.message : String(error),
1478
- stack: error instanceof Error ? error.stack : undefined,
1479
- errorType: error instanceof Error ? error.constructor.name : 'Error',
1480
- durationMs,
1481
- recommendation: 'Verify date range format (ISO 8601), check S3 credentials, and ensure Fluent API access'
1482
- };
1483
- }
1484
- }));
1485
- ```
1486
-
1487
- ---
1488
-
1489
- #### src/workflows/webhook/job-status-check.ts
1490
-
1491
- ```typescript
1492
- /**
1493
- * WORKFLOW 3: Job Status Query
1494
- *
1495
- * Purpose: Check job progress and status
1496
- * Trigger: Webhook GET/POST to /webhooks/inventory-quantities-job-status?jobId=xxx
1497
- * Returns: Current job status, stage, progress
1498
- *
1499
- * QUERY EXAMPLES:
1500
- *
1501
- * 1. HTTP GET:
1502
- * GET /webhooks/inventory-quantities-job-status?jobId=ADHOC_IQ_20251027_183045_abc123
1503
- *
1504
- * 2. HTTP POST:
1505
- * POST /webhooks/inventory-quantities-job-status
1506
- * { "jobId": "ADHOC_IQ_20251027_183045_abc123" }
1507
- */
1508
-
1509
- import { webhook, fn } from '@versori/run';
1510
- import { getJobStatus } from '../../services/extraction-orchestration';
1511
-
1512
- export const inventoryQuantitiesJobStatus = webhook(
1513
- 'inventory-quantities-job-status',
1514
- { connection: 'inventory-quantities-job-status', response: { mode: 'sync' } },
1515
- fn('query-job-status', async (ctx) => {
1516
- const { data, log, openKv, activation } = ctx;
1517
- const startTime = Date.now();
1518
-
1519
- // SECURITY: Authentication is enforced by Versori connection configuration
1520
- // Configure auth on the connection and reference it in webhook({ connection: '...' })
1521
-
1522
- // Get jobId from query param or POST body
1523
- const jobId = data.jobId as string;
1524
-
1525
- if (!jobId) {
1526
- log.error('❌ Job ID not provided in request');
1527
- return {
1528
- success: false,
1529
- error: 'Job ID is required. Provide jobId in query param or request body.'
1530
- };
1531
- }
1532
-
1533
- log.info('🔍 [START] Querying job status', { jobId });
1534
-
1535
- try {
1536
- // Query job status from KV store
1537
- const status = await getJobStatus(openKv(':project:'), jobId, log);
1538
-
1539
- const durationMs = Date.now() - startTime;
1540
-
1541
- if (!status) {
1542
- log.info('⚠️ Job not found', { jobId, durationMs });
1543
- return {
1544
- success: false,
1545
- error: 'Job not found',
1546
- jobId,
1547
- durationMs
1548
- };
1549
- }
1550
-
1551
- log.info('✅ [END] Job status retrieved', {
1552
- jobId,
1553
- status: status.status,
1554
- durationMs
1555
- });
1556
-
1557
- return {
1558
- success: true,
1559
- jobId,
1560
- ...status,
1561
- queryDurationMs: durationMs
1562
- };
1563
-
1564
- } catch (error: any) {
1565
- const durationMs = Date.now() - startTime;
1566
-
1567
- log.error('❌ [ERROR] Failed to query job status', {
1568
- jobId,
1569
- message: error instanceof Error ? error.message : String(error),
1570
- stack: error instanceof Error ? error.stack : undefined,
1571
- errorType: error instanceof Error ? error.constructor.name : 'Error',
1572
- durationMs,
1573
- recommendation: 'Verify KV store access and job ID format'
1574
- });
1575
-
1576
- return {
1577
- success: false,
1578
- jobId,
1579
- message: error instanceof Error ? error.message : String(error),
1580
- stack: error instanceof Error ? error.stack : undefined,
1581
- errorType: error instanceof Error ? error.constructor.name : 'Error',
1582
- durationMs,
1583
- recommendation: 'Verify KV store access and job ID format'
1584
- };
1585
- }
1586
- }));
1587
- ```
1588
-
1589
- ---
1590
-
1591
- ### 3. Main Orchestration Service (src/services/extraction-orchestration.ts)
1592
-
1593
- ```typescript
1594
- /**
1595
- * MAIN ORCHESTRATION SERVICE
1596
- *
1597
- * This is the heart of the extraction workflow. It coordinates all steps:
1598
- * 1. Initialize clients and services
1599
- * 2. Determine date range (incremental vs manual)
1600
- * 3. Extract data using ExtractionOrchestrator
1601
- * 4. Transform using UniversalMapper
1602
- * 5. Generate CSV using CSVParserService
1603
- * 6. Upload to S3
1604
- * 7. Track job progress with JobTracker
1605
- * 8. Update state for next run
1606
- *
1607
- * NAMING PATTERN (consistent across all use cases):
1608
- * - Interface: {Entity}ExtractionParams (e.g., InventoryQuantityExtractionParams)
1609
- * - Result: {Entity}ExtractionResult (e.g., InventoryQuantityExtractionResult)
1610
- * - Main function: execute{Entity}Extraction (e.g., executeInventoryQuantityExtraction)
1611
- *
1612
- * AI CUSTOMIZATION HINTS:
1613
- * - Change entity: Replace "InventoryQuantity" with "Order", "Product", etc.
1614
- * - Change output: Replace CSVParserService with XMLBuilder
1615
- * - Change destination: Replace S3DataSource with SftpDataSource
1616
- * - Add steps: Insert new service calls between existing steps
1617
- */
1618
-
1619
- import { Buffer } from 'node:buffer';
1620
- import {
1621
- createClient,
1622
- ExtractionOrchestrator,
1623
- JobTracker,
1624
- UniversalMapper,
1625
- CSVParserService,
1626
- S3DataSource,
1627
- } from '@fluentcommerce/fc-connect-sdk';
1628
-
1629
- import mappingConfig from '../../config/inventory-quantities.export.csv.json' with { type: 'json' };
1630
-
1631
- // ✅ VERSORI PLATFORM: Use native log from context, not LoggingService
1632
-
1633
- /**
1634
- * Parameters for extraction workflow
1635
- *
1636
- * NAMING: {Entity}ExtractionParams
1637
- */
1638
- export interface InventoryQuantityExtractionParams {
1639
- jobId: string;
1640
- triggeredBy: 'schedule' | 'webhook';
1641
- fromDate?: string; // Optional: manual date override
1642
- toDate?: string; // Optional: manual date override
1643
- updateState: boolean; // Whether to update lastSync timestamp
1644
-
1645
- // AI CUSTOMIZATION: Add filters specific to entity
1646
- quantityTypes?: string[]; // e.g., ['LAST_ON_HAND', 'RESERVED']
1647
- catalogueRef?: string; // e.g., 'DEFAULT_CATALOGUE'
1648
- }
1649
-
1650
- /**
1651
- * Result from extraction workflow
1652
- *
1653
- * NAMING: {Entity}ExtractionResult
1654
- */
1655
- export interface InventoryQuantityExtractionResult {
1656
- success: boolean;
1657
- jobId: string;
1658
- recordsExtracted: number;
1659
- fileName?: string;
1660
- s3Path?: string;
1661
- error?: string;
1662
- errors?: any[];
1663
- isManualOverride?: boolean;
1664
- stateUpdated?: boolean;
1665
- newTimestamp?: string;
1666
- }
1667
-
1668
- /**
1669
- * GraphQL Query for Inventory Quantities
1670
- *
1671
- * NAMING: {ENTITY}_EXTRACTION_QUERY (uppercase constant)
1672
- */
1673
- const INVENTORY_QUANTITIES_EXTRACTION_QUERY = `
1674
- query GetInventoryQuantities(
1675
- $catalogues: [InventoryCatalogueKey]
1676
- $dateRangeFilter: DateRange
1677
- $productRefs: [String!]
1678
- $types: [String!]
1679
- $first: Int!
1680
- $after: String
1681
- ) {
1682
- inventoryQuantities(
1683
- catalogues: $catalogues
1684
- updatedOn: $dateRangeFilter
1685
- productRef: $productRefs
1686
- type: $types
1687
- first: $first
1688
- after: $after
1689
- ) {
1690
- edges {
1691
- node {
1692
- id
1693
- ref
1694
- locationRef
1695
- productRef
1696
- qty
1697
- type
1698
- status
1699
- expectedOn
1700
- createdOn
1701
- updatedOn
1702
- catalogue {
1703
- ref
1704
- name
1705
- }
1706
- }
1707
- cursor
1708
- }
1709
- pageInfo {
1710
- hasNextPage
1711
- }
1712
- }
1713
- }
1714
- `;
1715
-
1716
- /**
1717
- * Query job status from KV store
1718
- *
1719
- * ✅ VERSORI PLATFORM: Uses native log from context (passed as parameter)
1720
- */
1721
- export async function getJobStatus(
1722
- kv: any, // ✅ Versori KV (compatible with JobTracker's KVAdapter interface)
1723
- jobId: string,
1724
- log: any // ✅ Native Versori log from context
1725
- ): Promise<any | undefined> {
1726
- try {
1727
- const tracker = new JobTracker(kv, log);
1728
- return await tracker.getJob(jobId);
1729
- } catch (error: any) {
1730
- log.error('Failed to get job status', { jobId, message: error instanceof Error ? error.message : String(error),
1731
- stack: error instanceof Error ? error.stack : undefined,
1732
- errorType: error instanceof Error ? error.constructor.name : 'Error', });
1733
- return undefined;
1734
- }
1735
- }
1736
-
1737
- /**
1738
- * MAIN ORCHESTRATION FUNCTION
1739
- *
1740
- * NAMING: execute{Entity}Extraction (e.g., executeInventoryQuantityExtraction)
1741
- *
1742
- * This function implements the complete workflow in steps.
1743
- * Each step is clearly commented for AI understanding.
1744
- */
1745
- export async function executeInventoryQuantityExtraction(
1746
- ctx: any,
1747
- params: InventoryQuantityExtractionParams
1748
- ): Promise<InventoryQuantityExtractionResult> {
1749
- // ✅ VERSORI PLATFORM: Extract native log from context
1750
- const { log, openKv, activation } = ctx;
1751
- const { jobId, triggeredBy, fromDate, toDate, updateState } = params;
1752
-
1753
- // Open KV store for state management and job tracking
1754
- // ✅ Pass raw Versori KV directly - it matches KVAdapter interface
1755
- // ✅ Pass native log to JobTracker
1756
- const kv = openKv(':project:');
1757
- const tracker = new JobTracker(kv, log);
1758
-
1759
- try {
1760
- // ═══════════════════════════════════════════════════════════
1761
- // STEP 1/8: Initialize Job Tracking
1762
- // ═══════════════════════════════════════════════════════════
1763
- log.info('📝 [STEP 1/8] Initializing job tracking', { jobId });
1764
-
1765
- await tracker.createJob(jobId, {
1766
- triggeredBy,
1767
- hasDateOverride: !!fromDate,
1768
- fromDate,
1769
- toDate,
1770
- updateStateAfterRun: updateState,
1771
- });
1772
-
1773
- // ═══════════════════════════════════════════════════════════
1774
- // STEP 2/8: Initialize Fluent Client
1775
- // ═══════════════════════════════════════════════════════════
1776
- log.info('📡 [STEP 2/8] Initializing Fluent Commerce client', { jobId });
1777
-
1778
- const client = await createClient(ctx, { validateConnection: true });
1779
-
1780
- if (!client) {
1781
- throw new Error('Failed to create Fluent Commerce client');
1782
- }
1783
-
1784
- log.info('✅ Fluent client initialized and connection validated', { jobId });
1785
-
1786
- // ═══════════════════════════════════════════════════════════
1787
- // STEP 3/8: Determine Date Range
1788
- // ═══════════════════════════════════════════════════════════
1789
- log.info('📅 [STEP 3/8] Determining date range for extraction', { jobId });
1790
-
1791
- // State key for incremental sync tracking
1792
- // NAMING: last{Entity}Sync (e.g., lastInventoryQuantitySync)
1793
- const STATE_KEY = 'lastInventoryQuantitySync';
1794
- const DEFAULT_FALLBACK = '2024-01-01T00:00:00Z';
1795
- const OVERLAP_BUFFER_SECONDS = parseInt(
1796
- activation.getVariable('overlapBufferSeconds') || '60',
1797
- 10
1798
- );
1799
-
1800
- let dateRangeFilter: { from?: string; to?: string } | null = null;
1801
- const isManualOverride = !!fromDate;
1802
-
1803
- if (isManualOverride) {
1804
- // Manual date override from webhook
1805
- dateRangeFilter = { from: fromDate, to: toDate };
1806
- log.info('Using manual date override', { fromDate, toDate });
1807
- } else {
1808
- // Incremental sync - get last sync timestamp
1809
- const rawLastRunTime = (await kv.get<string>(STATE_KEY)) || DEFAULT_FALLBACK;
1810
-
1811
- // Apply overlap buffer (prevents missed records)
1812
- const bufferedLastRunTime = new Date(
1813
- new Date(rawLastRunTime).getTime() - OVERLAP_BUFFER_SECONDS * 1000
1814
- ).toISOString();
1815
-
1816
- const effectiveEndTime = toDate || new Date().toISOString();
1817
-
1818
- dateRangeFilter = {
1819
- from: bufferedLastRunTime,
1820
- to: effectiveEndTime, // End of extraction window
1821
- };
1822
-
1823
- log.info('Using incremental sync with overlap buffer', {
1824
- rawLastRunTime,
1825
- bufferedLastRunTime,
1826
- effectiveEndTime,
1827
- overlapBufferSeconds: OVERLAP_BUFFER_SECONDS,
1828
- });
1829
- }
1830
-
1831
- // ═══════════════════════════════════════════════════════════
1832
- // STEP 4/8: Extract Data (ExtractionOrchestrator)
1833
- // ═══════════════════════════════════════════════════════════
1834
- log.info('🔄 [STEP 4/8] Extracting data from Fluent Commerce', { jobId });
1835
-
1836
- await tracker.updateJob(jobId, {
1837
- status: 'processing',
1838
- stage: 'extraction',
1839
- message: 'Extracting data with auto-pagination',
1840
- });
1841
-
1842
- // Build catalogues array from config
1843
- const catalogueRef = params.catalogueRef || activation.getVariable('catalogueRef');
1844
- const catalogues = catalogueRef ? [{ ref: catalogueRef }] : [];
1845
-
1846
- // Configure extraction
1847
- const pageSize = parseInt(activation.getVariable('pageSize') || '200', 10);
1848
- const maxRecords = parseInt(activation.getVariable('maxRecords') || '100000', 10);
1849
-
1850
- // Initialize ExtractionOrchestrator
1851
- const orchestrator = new ExtractionOrchestrator(client, log);
1852
-
1853
- // ? Enhanced: Extract context for progress logging
1854
- const dateRangeInfo = {
1855
- start: dateRangeFilter?.from || 'N/A',
1856
- end: dateRangeFilter?.to || 'N/A',
1857
- catalogues: catalogues.map((c: any) => c.ref).join(', ') || 'all',
1858
- types: params.quantityTypes?.join(', ') || 'all'
1859
- };
1860
-
1861
- // ? Enhanced: Start logging with context
1862
- log.info(`🔍 [ExtractionOrchestrator] Starting extraction`, {
1863
- query: 'inventoryQuantities',
1864
- pageSize,
1865
- maxRecords,
1866
- dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
1867
- catalogues: dateRangeInfo.catalogues,
1868
- quantityTypes: dateRangeInfo.types,
1869
- jobId
1870
- });
1871
-
1872
- // Execute extraction with auto-pagination
1873
- const extractionResult = await orchestrator.extract({
1874
- query: INVENTORY_QUANTITIES_EXTRACTION_QUERY,
1875
- resultPath: 'inventoryQuantities.edges.node',
1876
- variables: {
1877
- catalogues,
1878
- dateRangeFilter,
1879
- types: params.quantityTypes,
1880
- // Note: Don't include 'first' or 'after' here; orchestrator injects them
1881
- },
1882
- pageSize,
1883
- maxRecords,
1884
- // Optional: validate each record
1885
- validateItem: (item: any) => {
1886
- return !!(item.ref && item.productRef);
1887
- },
1888
- });
1889
-
1890
- const records = extractionResult.data || [];
1891
-
1892
- log.info('Extraction complete', {
1893
- totalRecords: extractionResult.stats.totalRecords,
1894
- totalPages: extractionResult.stats.totalPages,
1895
- validRecords: extractionResult.stats.validRecords ?? records.length,
1896
- failedValidations: extractionResult.stats.failedValidations,
1897
- errors: extractionResult.errors ? extractionResult.errors.length : 0,
1898
- });
1899
-
1900
- // ? Enhanced: Completion logging with summary
1901
- log.info(`✅ [ExtractionOrchestrator] Extraction completed`, {
1902
- totalRecords: extractionResult.stats.totalRecords,
1903
- totalPages: extractionResult.stats.totalPages,
1904
- validRecords: extractionResult.stats.validRecords ?? records.length,
1905
- failedValidations: extractionResult.stats.failedValidations,
1906
- truncated: extractionResult.stats.truncated,
1907
- truncationReason: extractionResult.stats.truncationReason,
1908
- dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
1909
- jobId
1910
- });
1911
-
1912
- if (extractionResult.errors && extractionResult.errors.length > 0) {
1913
- log.warn('Non-fatal extraction errors encountered', {
1914
- errorCount: extractionResult.errors.length,
1915
- sampleErrors: extractionResult.errors.slice(0, 3),
1916
- });
1917
- }
1918
-
1919
- // Handle empty result
1920
- if (records.length === 0) {
1921
- log.info('No records to process');
1922
-
1923
- // Update state even with no records (prevents re-querying empty window)
1924
- if (updateState && !isManualOverride) {
1925
- await kv.set(STATE_KEY, new Date().toISOString());
1926
- }
1927
-
1928
- await tracker.markCompleted(jobId, {
1929
- recordCount: 0,
1930
- message: 'No records to extract',
1931
- });
1932
-
1933
- return {
1934
- success: true,
1935
- jobId,
1936
- recordsExtracted: 0,
1937
- };
1938
- }
1939
-
1940
- // ═══════════════════════════════════════════════════════════
1941
- // STEP 5/8: Transform Data (UniversalMapper)
1942
- // ═══════════════════════════════════════════════════════════
1943
- log.info('🔧 [STEP 5/8] Transforming data with UniversalMapper', {
1944
- jobId,
1945
- recordCount: records.length,
1946
- });
1947
-
1948
- await tracker.updateJob(jobId, {
1949
- status: 'processing',
1950
- stage: 'transformation',
1951
- message: `Transforming ${records.length} records`,
1952
- });
1953
-
1954
- const mapper = new UniversalMapper(mappingConfig);
1955
- const mappingResult = await mapper.map(records);
1956
-
1957
- if (!mappingResult.success) {
1958
- const mappingErrors = mappingResult.errors || ['Unknown mapping failure'];
1959
- await tracker.markFailed(jobId, {
1960
- error: mappingErrors[0] || 'UniversalMapper returned unsuccessful result',
1961
- failedCount: mappingErrors.length,
1962
- });
1963
- return {
1964
- success: false,
1965
- error: `Transformation failed: ${mappingErrors[0] || 'Unknown error'}`,
1966
- jobId,
1967
- errors: mappingErrors,
1968
- };
1969
- }
1970
-
1971
- const transformedRecords = Array.isArray(mappingResult.data) ? mappingResult.data : [];
1972
- const mappingErrors = mappingResult.errors || [];
1973
-
1974
- if (mappingErrors.length > 0) {
1975
- log.warn('Some records failed transformation', {
1976
- jobId,
1977
- errorCount: mappingErrors.length,
1978
- sampleErrors: mappingErrors.slice(0, 3),
1979
- });
1980
- }
1981
-
1982
- if (transformedRecords.length === 0) {
1983
- await tracker.markFailed(jobId, {
1984
- error: 'All records failed mapping',
1985
- failedCount: mappingErrors.length,
1986
- errors: mappingErrors,
1987
- });
1988
- return {
1989
- success: false,
1990
- error: 'All records failed mapping',
1991
- jobId,
1992
- errors: mappingErrors,
1993
- };
1994
- }
1995
-
1996
- log.info('Transformation complete', {
1997
- successful: transformedRecords.length,
1998
- failed: mappingErrors.length,
1999
- skippedRecords: records.length - transformedRecords.length,
2000
- });
2001
-
2002
- // ═══════════════════════════════════════════════════════════
2003
- // STEP 6/8: Generate CSV (CSVParserService)
2004
- // ═══════════════════════════════════════════════════════════
2005
- log.info('📄 [STEP 6/8] Generating CSV file', { jobId });
2006
-
2007
- await tracker.updateJob(jobId, {
2008
- status: 'processing',
2009
- stage: 'csv_generation',
2010
- message: `Generating CSV for ${transformedRecords.length} records`,
2011
- });
2012
-
2013
- // Initialize CSVParserService
2014
- const csvParser = new CSVParserService({ includeHeaders: true });
2015
-
2016
- // Generate CSV content
2017
- const csvContent = await csvParser.stringify(transformedRecords);
2018
-
2019
- // Generate filename
2020
- const fileNamePrefix = activation.getVariable('fileNamePrefix') || 'inventoryquantities';
2021
- const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
2022
- const fileName = `${fileNamePrefix}-${timestamp}.csv`;
2023
-
2024
- log.info('CSV file generated', {
2025
- fileName,
2026
- sizeBytes: csvContent.length,
2027
- recordCount: transformedRecords.length,
2028
- });
2029
-
2030
- // ═══════════════════════════════════════════════════════════
2031
- // STEP 7/8: Upload to S3 (S3DataSource)
2032
- // ═══════════════════════════════════════════════════════════
2033
- log.info('☁️ [STEP 7/8] Uploading to S3', { jobId, fileName });
2034
-
2035
- await tracker.updateJob(jobId, {
2036
- status: 'processing',
2037
- stage: 's3_upload',
2038
- message: `Uploading ${fileName} to S3`,
2039
- });
2040
-
2041
- // Get S3 configuration from activation variables
2042
- const s3Config = {
2043
- bucket: activation.getVariable('s3BucketName'),
2044
- region: activation.getVariable('awsRegion') || 'us-east-1',
2045
- accessKeyId: activation.getVariable('awsAccessKeyId'),
2046
- secretAccessKey: activation.getVariable('awsSecretAccessKey'),
2047
- };
2048
- const s3Prefix = activation.getVariable('s3Prefix') || 'inventory-quantities/daily/';
2049
-
2050
- // Validate S3 config
2051
- if (!s3Config.bucket || !s3Config.accessKeyId || !s3Config.secretAccessKey) {
2052
- throw new Error(
2053
- 'S3 configuration incomplete: missing bucket, accessKeyId, or secretAccessKey'
2054
- );
2055
- }
2056
-
2057
- // Initialize S3 data source
2058
- // ✅ VERSORI PLATFORM: Pass native log from context
2059
- const s3 = new S3DataSource(
2060
- {
2061
- type: 'S3_CSV',
2062
- connectionId: 'inventory-quantities-s3',
2063
- name: 'Inventory Quantities S3 Upload',
2064
- s3Config,
2065
- },
2066
- log
2067
- );
2068
-
2069
- // Construct S3 key
2070
- const s3Key = `${s3Prefix}${fileName}`;
2071
-
2072
- // Upload with retry logic (built into S3DataSource)
2073
- await s3.uploadFile(s3Key, Buffer.from(csvContent, 'utf-8'), {
2074
- contentType: 'text/csv',
2075
- metadata: {
2076
- recordCount: String(transformedRecords.length),
2077
- extractedAt: new Date().toISOString(),
2078
- jobId,
2079
- mappingErrors: mappingErrors.length > 0 ? String(mappingErrors.length) : undefined,
2080
- },
2081
- });
2082
-
2083
- log.info('S3 upload successful', { fileName, s3Key });
2084
-
2085
- // ═══════════════════════════════════════════════════════════
2086
- // STEP 8/8: Update State & Complete Job
2087
- // ═══════════════════════════════════════════════════════════
2088
- log.info('💾 [STEP 8/8] Updating state and completing job', { jobId });
2089
-
2090
- // Calculate new timestamp for next incremental run
2091
- let newTimestamp: string | undefined;
2092
-
2093
- if (updateState && !isManualOverride) {
2094
- // Find max updatedOn from extracted records
2095
- const maxUpdatedOn = records.reduce(
2096
- (max, record) => {
2097
- const recordTime = new Date(record.updatedOn).getTime();
2098
- return recordTime > max ? recordTime : max;
2099
- },
2100
- new Date(dateRangeFilter?.from || DEFAULT_FALLBACK).getTime()
2101
- );
2102
-
2103
- newTimestamp = new Date(maxUpdatedOn).toISOString();
2104
-
2105
- // Store new timestamp (WITHOUT buffer - buffer only applied on read)
2106
- await kv.set(STATE_KEY, newTimestamp);
2107
-
2108
- log.info('State updated', {
2109
- oldTimestamp: dateRangeFilter?.from,
2110
- newTimestamp,
2111
- });
2112
- }
2113
-
2114
- // Mark job as completed
2115
- await tracker.markCompleted(jobId, {
2116
- recordCount: transformedRecords.length,
2117
- fileName,
2118
- s3Key,
2119
- errorCount: mappingErrors.length,
2120
- errors: mappingErrors,
2121
- isManualOverride,
2122
- stateUpdated: updateState,
2123
- newTimestamp,
2124
- });
2125
-
2126
- return {
2127
- success: true,
2128
- jobId,
2129
- recordsExtracted: transformedRecords.length,
2130
- fileName,
2131
- s3Path: s3Key,
2132
- isManualOverride,
2133
- stateUpdated: updateState,
2134
- newTimestamp,
2135
- errors: mappingErrors.length > 0 ? mappingErrors : undefined,
2136
- };
2137
- } catch (error: any) {
2138
- log.error('Extraction workflow failed', {
2139
- jobId,
2140
- message: error instanceof Error ? error.message : String(error),
2141
-
2142
- stack: error instanceof Error ? error.stack : undefined,
2143
-
2144
- errorType: error instanceof Error ? error.constructor.name : 'Error',
2145
- });
2146
-
2147
- // Mark job as failed
2148
- await tracker.markFailed(jobId, error);
2149
-
2150
- return {
2151
- success: false,
2152
- jobId,
2153
- recordsExtracted: 0,
2154
- message: error instanceof Error ? error.message : String(error),
2155
-
2156
- stack: error instanceof Error ? error.stack : undefined,
2157
-
2158
- errorType: error instanceof Error ? error.constructor.name : 'Error',
2159
- };
2160
- }
2161
- }
2162
- ```
2163
-
2164
- ---
2165
-
2166
- ### 4. Utility Functions (src/utils/job-id-generator.ts)
2167
-
2168
- ```typescript
2169
- /**
2170
- * Job ID Generator
2171
- *
2172
- * Generates unique job IDs for tracking extraction workflows
2173
- *
2174
- * FORMAT: {TYPE}_{ENTITY}_{YYYYMMDD}_{HHMMSS}_{RANDOM}
2175
- * Example: SCHEDULED_IQ_20251027_183045_a1b2c3
2176
- */
2177
-
2178
- /**
2179
- * Generate unique job ID
2180
- *
2181
- * @param type - Job trigger type (SCHEDULED, ADHOC, MANUAL)
2182
- * @param entity - Entity abbreviation (IQ=Inventory Quantities, IP, VP, ORD, PRD)
2183
- * @returns Unique job ID string
2184
- */
2185
- export function generateJobId(type: string, entity: string): string {
2186
- const now = new Date();
2187
-
2188
- // Format: YYYYMMDD
2189
- const dateStr = now.toISOString().slice(0, 10).replace(/-/g, '');
2190
-
2191
- // Format: HHMMSS
2192
- const timeStr = now.toISOString().slice(11, 19).replace(/:/g, '');
2193
-
2194
- // Random suffix (6 chars)
2195
- const randomStr = Math.random().toString(36).substring(2, 8);
2196
-
2197
- return `${type}_${entity}_${dateStr}_${timeStr}_${randomStr}`;
2198
- }
2199
-
2200
- /**
2201
- * Parse job ID components
2202
- */
2203
- export function parseJobId(jobId: string): {
2204
- type: string;
2205
- entity: string;
2206
- date: string;
2207
- time: string;
2208
- random: string;
2209
- } | null {
2210
- const parts = jobId.split('_');
2211
-
2212
- if (parts.length !== 5) {
2213
- return null;
2214
- }
2215
-
2216
- return {
2217
- type: parts[0],
2218
- entity: parts[1],
2219
- date: parts[2],
2220
- time: parts[3],
2221
- random: parts[4],
2222
- };
2223
- }
2224
- ```
2225
-
2226
- ---
2227
-
2228
- ### 5. Package Configuration
2229
-
2230
- #### package.json
2231
-
2232
- ```json
2233
- {
2234
- "name": "inventory-quantities-to-s3-csv",
2235
- "version": "1.0.0",
2236
- "description": "Extract inventory quantities from Fluent Commerce and export to S3 as CSV",
2237
- "type": "module",
2238
- "main": "src/index.ts",
2239
- "scripts": {
2240
- "dev": "versori dev",
2241
- "build": "versori build",
2242
- "deploy": "versori deploy"
2243
- },
2244
- "dependencies": {
2245
- "@fluentcommerce/fc-connect-sdk": "^0.1.39",
2246
- "@versori/run": "latest"
2247
- },
2248
- "devDependencies": {
2249
- "@types/node": "^20.0.0",
2250
- "typescript": "^5.0.0"
2251
- }
2252
- }
2253
- ```
2254
-
2255
- #### tsconfig.json
2256
-
2257
- ```json
2258
- {
2259
- "compilerOptions": {
2260
- "module": "ES2022",
2261
- "target": "ES2024",
2262
- "moduleResolution": "node"
2263
- }
2264
- }
2265
- ```
2266
-
2267
- ---
2268
-
2269
- ## 6. Deployment Instructions
2270
-
2271
- ### Deploy to Versori
2272
-
2273
- ```bash
2274
- # 1. Install dependencies
2275
- npm install
2276
-
2277
- # 2. Test locally (if using Versori CLI)
2278
- npm run dev
2279
-
2280
- # 3. Deploy to Versori platform
2281
- npm run deploy
2282
- ```
2283
-
2284
- ### Configure Activation Variables
2285
-
2286
- In Versori platform settings, configure:
2287
-
2288
- ```json
2289
- {
2290
- "catalogueRef": "DEFAULT_CATALOGUE",
2291
- "s3BucketName": "inventory-audit-exports",
2292
- "awsAccessKeyId": "AKIAXXXXXXXXXXXX",
2293
- "awsSecretAccessKey": "********",
2294
- "awsRegion": "us-east-1",
2295
- "s3Prefix": "inventory-quantities/daily/",
2296
- "fileNamePrefix": "inventoryquantities",
2297
- "pageSize": 200,
2298
- "maxRecords": 100000,
2299
- "overlapBufferSeconds": 60,
2300
- "webhookApiKey": "your-secure-api-key-here"
2301
- }
2302
- ```
2303
-
2304
- ---
2305
-
2306
- ## 7. Testing
2307
-
2308
- ### Test Scheduled Extraction
2309
-
2310
- The scheduled workflow runs automatically based on cron schedule.
2311
-
2312
- **Check logs:**
2313
-
2314
- ```
2315
- [STEP 1/8] Initializing job tracking
2316
- [STEP 2/8] Initializing Fluent Commerce client
2317
- [STEP 3/8] Determining date range for extraction
2318
- [STEP 4/8] Extracting data from Fluent Commerce
2319
- [STEP 5/8] Transforming data with UniversalMapper
2320
- [STEP 6/8] Generating CSV file
2321
- [STEP 7/8] Uploading to S3
2322
- [STEP 8/8] Updating state and completing job
2323
- ```
2324
-
2325
- ### Test Ad hoc Extraction
2326
-
2327
- ```bash
2328
- # Incremental (uses last sync timestamp)
2329
- curl -X POST https://api.versori.com/webhooks/inventory-quantities-adhoc \
2330
- -H "X-API-Key: your-api-key" \
2331
- -H "Content-Type: application/json" \
2332
- -d '{}'
2333
-
2334
- # Date range override
2335
- curl -X POST https://api.versori.com/webhooks/inventory-quantities-adhoc \
2336
- -H "X-API-Key: your-api-key" \
2337
- -H "Content-Type: application/json" \
2338
- -d '{
2339
- "fromDate": "2025-01-01T00:00:00Z",
2340
- "toDate": "2025-01-31T23:59:59Z",
2341
- "updateState": false
2342
- }'
2343
- ```
2344
-
2345
- ### Test Job Status Query
2346
-
2347
- ```bash
2348
- curl -X POST https://api.versori.com/webhooks/inventory-quantities-job-status \
2349
- -H "X-API-Key: your-api-key" \
2350
- -H "Content-Type: application/json" \
2351
- -d '{
2352
- "jobId": "ADHOC_IQ_20251027_183045_abc123"
2353
- }'
2354
- ```
2355
-
2356
- **Response:**
2357
-
2358
- ```json
2359
- {
2360
- "success": true,
2361
- "jobId": "ADHOC_IQ_20251027_183045_abc123",
2362
- "status": "processing",
2363
- "stage": "transformation",
2364
- "message": "Transforming 15000 records",
2365
- "createdAt": "2025-10-27T18:30:45.000Z",
2366
- "startedAt": "2025-10-27T18:30:46.000Z"
2367
- }
2368
- ```
2369
-
2370
- ---
2371
-
2372
- ## 8. Troubleshooting
2373
-
2374
- **Issue**: "No records extracted"
2375
-
2376
- - Check dateRange (manual override vs incremental)
2377
- - Check catalogueRef filter
2378
- - Verify quantity types filter
2379
-
2380
- **Issue**: "S3 upload failed"
2381
-
2382
- - Job fails; state not advanced
2383
- - Next run retries same window
2384
- - Check S3 credentials and bucket permissions
2385
-
2386
- **Issue**: "GraphQL pagination error"
2387
-
2388
- - Ensure edges.cursor and pageInfo.hasNextPage are in query
2389
-
2390
- **Issue**: "Memory pressure"
2391
-
2392
- - Lower pageSize or maxRecords
2393
- - Consider file splitting for large extractions
2394
-
2395
- **Issue**: "Transformation errors"
2396
-
2397
- - Check mapping config field paths
2398
- - Verify required fields are present in GraphQL response
2399
- - Review transformation error details in logs
2400
-
2401
- ---
2402
-
2403
- ## 9. Replication Checklist
2404
-
2405
- **To replicate this template for other entities/formats:**
2406
-
2407
- 1. **File Naming:** Replace `inventory-quantities`, `IQ`, `InventoryQuantity` with your entity name
2408
- 2. **GraphQL Query:** Update query constant and field selection to match your entity schema
2409
- 3. **Mapping Config:** Create new mapping file in `config/` with correct field paths
2410
- 4. **Workflows:** Rename workflow exports to match entity (e.g., `scheduledOrdersExtraction`)
2411
- 5. **Service Function:** Rename main function (e.g., `executeOrderExtraction`)
2412
- 6. **State Key:** Update KV key (e.g., `lastOrderSync`)
2413
- 7. **Output Format:** For XML use `XMLBuilder`, for JSON use `JSON.stringify()`, for CSV use `CSVParserService`
2414
- 8. **Upload Destination:** For SFTP replace `S3DataSource` with `SftpDataSource` (and add `dispose()` in finally block)
2415
- 9. **Job ID Entity Code:** Update entity abbreviation in generateJobId() (e.g., 'ORD' for orders)
2416
- 10. **Result Path:** Update `resultPath` in ExtractionOrchestrator (e.g., `'orders.edges.node'`)
2417
-
2418
- ---
2419
-
2420
- **Pattern**: Enterprise incremental extraction with overlap buffer for inventory quantities
2421
- **Key Learning**: Use ExtractionOrchestrator for auto-pagination, JobTracker for job status, CSVParserService for CSV generation
2422
- **Critical**: Apply 60-second overlap buffer to prevent missed records due to clock skew
2423
- **Buffer Pattern**: Query WITH buffer (`updatedOn >= lastRunTime - 60s`), save WITHOUT buffer (`MAX(updatedOn)`)
2424
- **SDK Services**: ExtractionOrchestrator, UniversalMapper, CSVParserService, S3DataSource, JobTracker
2425
- **Entity-Specific**: Query uses `inventoryQuantities`, resultPath is `'inventoryQuantities.edges.node'`, state key is `lastInventoryQuantitySync`
2426
-
2427
- ---
2428
-
2429
- ### Optional: Backward Pagination (Advanced)
2430
-
2431
- - Default: forward ($first/$after) + pageInfo.hasNextPage.
2432
- - Reverse: define $last/$before and include pageInfo.hasPreviousPage; set direction='backward'.
2433
-
2434
- GraphQL:
2435
-
2436
- ```graphql
2437
- query GetInventoryQuantitiesBackward($retailerId: ID!, $last: Int!, $before: String) {
2438
- inventoryQuantities(retailerId: $retailerId, last: $last, before: $before) {
2439
- edges {
2440
- cursor
2441
- node {
2442
- id
2443
- ref
2444
- updatedOn
2445
- }
2446
- }
2447
- pageInfo {
2448
- hasPreviousPage
2449
- }
2450
- }
2451
- }
2452
- ```
2453
-
2454
- SDK:
2455
-
2456
- ```typescript
2457
- await orchestrator.extract({
2458
- query: INVENTORY_QUANTITIES_BACKWARD_QUERY,
2459
- resultPath: 'inventoryQuantities.edges.node',
2460
- variables: { retailerId },
2461
- pageSize,
2462
- direction: 'backward',
2463
- });
2464
- ```
1
+ ---
2
+ template_id: tpl-extract-inventory-quantities-to-s3-csv
3
+ canonical_filename: template-extraction-inventory-quantities-to-s3-csv.md
4
+ version: 2.0.0
5
+ sdk_version: ^0.1.39
6
+ runtime: versori
7
+ direction: extraction
8
+ source: fluent-graphql
9
+ destination: s3-csv
10
+ entity: inventoryQuantities
11
+ format: csv
12
+ logging: versori
13
+ status: stable
14
+ features:
15
+ - memory-management
16
+ - enhanced-logging
17
+ - pagination-progress
18
+ ---
19
+
20
+ # Template: Extraction - Inventory Quantities to S3 CSV
21
+
22
+ **Template Version:** 2.0.0
23
+ **SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
24
+ **Last Updated:** 2025-01-24
25
+ **Deployment Target:** Versori Platform
26
+
27
+ **🆕 Version 2.0.0 Enhancements:**
28
+ - ✅ **Memory Management** - Clear large result sets after processing batches
29
+ - ✅ **Enhanced Logging** - Pagination progress tracking with emoji indicators (📊, 📥, ✅)
30
+ - ✅ **Pagination Progress** - Real-time page-by-page progress logging with metrics
31
+
32
+ ---
33
+
34
+ ## 📚 STEP 1: Load These Docs (Human Checklist)
35
+
36
+ 1. REQUIRED (load all)
37
+ - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
38
+ - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
39
+ - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
40
+ - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
41
+ - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
42
+ - [ ] fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
43
+
44
+ Copy-paste list (open these):
45
+ fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
46
+ fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
47
+ fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
48
+ fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
49
+ fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
50
+ fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
51
+
52
+ ---
53
+
54
+ ## 📋 STEP 2: Tell Your AI (Prompt)
55
+
56
+ Copy/paste this prompt into your AI tool after loading the documentation above:
57
+
58
+ ```
59
+ I need a Versori scheduled extractor that:
60
+
61
+ 1) Queries Fluent Commerce GraphQL for inventoryQuantities with auto-pagination
62
+ 2) Uses incremental mode with a 60-second overlap buffer stored in Versori KV
63
+ 3) Transforms results using UniversalMapper per mapping JSON
64
+ 4) Generates CSV with CSVParserService and uploads to S3
65
+ 5) Uses native Versori log (LoggingService removed - use native log)
66
+
67
+ Use the loaded docs for SDK specifics and best practices. Keep structure identical to the template.
68
+ ```
69
+
70
+ ---
71
+
72
+ ## 📦 SDK Imports (Verified - Versori Optimized)
73
+
74
+ ```typescript
75
+ import { Buffer } from 'node:buffer';
76
+ import {
77
+ createClient,
78
+ UniversalMapper,
79
+ S3DataSource,
80
+ CSVParserService,
81
+ } from '@fluentcommerce/fc-connect-sdk';
82
+
83
+ import { schedule, http } from '@versori/run';
84
+ ```
85
+
86
+ ---
87
+
88
+ # Versori Scheduled: Inventory Quantities Extraction to S3 CSV (Configurable)
89
+
90
+ **FC Connect SDK Use Case Guide**
91
+
92
+ > SDK: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
93
+ > Version: `npm install @fluentcommerce/fc-connect-sdk@^0.1.39`
94
+
95
+ Context: Scheduled Versori workflow that extracts inventory quantities (detailed quantity records) from Fluent Commerce via GraphQL query with **configurable extraction modes**, transforms with `UniversalMapper`, and writes CSV files to S3 for analytics and reporting.
96
+
97
+ **Pattern**: EXTRACTION (Fluent → S3 CSV)
98
+ **Entity**: inventoryQuantities
99
+ **Complexity**: High | Runtime: Versori Platform (Scheduled)
100
+
101
+ ---
102
+
103
+ ## ⚠️ IMPORTANT: Sample Code for SDK Demonstration Only
104
+
105
+ > **🔴 PRODUCTION WARNING**
106
+ >
107
+ > This guide demonstrates FC Connect SDK capabilities for **extraction and mapping workflows**. The multiple extraction modes (incremental, dateRange, historical) are included to show SDK flexibility and serve as **reference examples**.
108
+ >
109
+ > **✅ PRODUCTION RECOMMENDATION:**
110
+ >
111
+ > - **ONLY use INCREMENTAL mode with scheduled runs** (e.g., daily/hourly)
112
+ > - Incremental mode is safe, efficient, and production-ready
113
+ > - Uses overlap buffer to prevent missed records
114
+ > - Natural rate limiting via timestamps
115
+ >
116
+ > **🚫 DO NOT USE IN PRODUCTION:**
117
+ >
118
+ > - **dateRange mode** - High risk of platform overload with large date windows
119
+ > - **historical mode** - Extremely dangerous, can fetch millions of records
120
+ > - These modes are **demonstration only** to show SDK query patterns
121
+ > - Using these modes on large inventory datasets can crash your runtime and impact platform stability
122
+ >
123
+ > **📝 If you need historical data:**
124
+ >
125
+ > - Run multiple small incremental extractions (e.g., daily for past 30 days)
126
+ > - Use one-time migration scripts with proper monitoring (not scheduled workflows)
127
+ > - Always validate date ranges and implement file splitting
128
+ > - Get explicit approval before running large extractions
129
+ >
130
+ > **This sample code shows HOW to use the SDK - not WHAT to use in production.**
131
+
132
+ ---
133
+
134
+ ## What You'll Build
135
+
136
+ - **Three extraction modes**: Incremental, Date Range, or Historical
137
+ - **State management** with VersoriKVAdapter to track last successful run
138
+ - GraphQL query with auto-pagination
139
+ - UniversalMapper transformation for reporting schema
140
+ - CSV file generation with CSVParserService
141
+ - S3 upload to analytics system
142
+ - **Failure recovery** with timestamp tracking
143
+
144
+ ## Business Use Cases
145
+
146
+ **1. Incremental Daily Sync (Analytics)**
147
+
148
+ - Extract only changed inventory quantities since last run
149
+ - Run daily at 2 AM
150
+ - Minimize data transfer
151
+ - Track changes over time
152
+
153
+ **2. Date Range Extract (Audit)**
154
+
155
+ - Extract quantity changes within specific date window
156
+ - For audits, reconciliation, historical analysis
157
+ - Example: "Show all quantity changes between Jan 1-15"
158
+
159
+ **3. Historical Backfill**
160
+
161
+ - Extract all quantities created within date range
162
+ - For initial data warehouse load
163
+ - One-time backfill operation
164
+
165
+ ## Inventory Quantities vs Positions
166
+
167
+ **InventoryQuantity** = Specific quantity record (retailer-defined types)
168
+
169
+ - Individual records: e.g., LAST_ON_HAND, RESERVED, DELTA, SALE, CORRECTION (plus any custom IQ types)
170
+ - Multiple quantities per product/location
171
+ - Fields: locationRef, skuRef, qty, type, status, expectedOn (if applicable)
172
+ - Used for: Detailed tracking, audit trails
173
+
174
+ **InventoryPosition** = Aggregated on-hand calculation
175
+
176
+ - One position per product/location
177
+ - Calculated `onHand` from all associated quantities
178
+ - Used for: Stock availability, reporting
179
+
180
+ ## SDK Methods Used
181
+
182
+ ```typescript
183
+ import { Buffer } from 'node:buffer';
184
+ import {
185
+ createClient,
186
+ UniversalMapper,
187
+ S3DataSource,
188
+ VersoriKVAdapter,
189
+ CSVParserService,
190
+ } from '@fluentcommerce/fc-connect-sdk';
191
+
192
+ await createClient(ctx);
193
+ await client.graphql({ query, variables, pagination });
194
+ new VersoriKVAdapter(ctx.openKv(':project:'));
195
+ new UniversalMapper(exportMapping);
196
+ const csvParser = new CSVParserService({ includeHeaders: true });
197
+ const csvContent = await csvParser.stringify(rows);
198
+ await s3.uploadFile(key, Buffer.from(csvContent, 'utf8'), options);
199
+ ```
200
+
201
+ ## Activation Variables
202
+
203
+ ```json
204
+ {
205
+ "retailerId": "your-retailer-id",
206
+ "s3BucketName": "inventory-audit-exports",
207
+ "awsAccessKeyId": "AKIAXXXXXXXXXXXX",
208
+ "awsSecretAccessKey": "********",
209
+ "awsRegion": "us-east-1",
210
+ "s3Prefix": "inventory-quantities/daily/",
211
+ "fileNamePrefix": "inventoryquantities",
212
+ "catalogueRef": "DEFAULT_CATALOGUE",
213
+ "pageSize": 200,
214
+ "maxRecords": 100000,
215
+ "extractionMode": "incremental",
216
+ "fallbackStartDate": "2024-01-01T00:00:00Z",
217
+ "overlapBufferSeconds": "60",
218
+ "startDate": "",
219
+ "endDate": ""
220
+ }
221
+ ```
222
+
223
+ ### Variable Reference
224
+
225
+ | Variable | Type | Required | Default | Description |
226
+ |----------|------|----------|---------|-------------|
227
+ | `retailerId` | string | Yes | - | Fluent Commerce retailer ID |
228
+ | `s3BucketName` | string | Yes | - | S3 bucket for CSV export |
229
+ | `awsAccessKeyId` | string | Yes | - | AWS access key with S3 write permissions |
230
+ | `awsSecretAccessKey` | string | Yes | - | AWS secret access key |
231
+ | `awsRegion` | string | Yes | - | AWS region (e.g., `us-east-1`) |
232
+ | `s3Prefix` | string | No | `""` | S3 key prefix (e.g., `inventory-quantities/daily/`) |
233
+ | `fileNamePrefix` | string | No | `"inventoryquantities"` | CSV filename prefix |
234
+ | `catalogueRef` | string | No | - | Filter by catalogue reference (optional) |
235
+ | `pageSize` | number | No | `200` | GraphQL page size (max 500) |
236
+ | `maxRecords` | number | No | `100000` | Maximum records per extraction |
237
+ | `extractionMode` | string | No | `"incremental"` | Extraction mode: `incremental`, `dateRange`, or `historical` |
238
+ | `fallbackStartDate` | string | No | `"2024-01-01T00:00:00Z"` | Fallback date if no state exists |
239
+ | `overlapBufferSeconds` | number | No | `60` | Overlap buffer to prevent missed records (seconds) |
240
+ | `startDate` | string | No | - | Manual start date (for `dateRange`/`historical` modes) |
241
+ | `endDate` | string | No | - | Manual end date (for `dateRange`/`historical` modes) |
242
+
243
+ ### Extraction Mode Configuration
244
+
245
+ **Mode 1: Incremental (default)**
246
+
247
+ ```json
248
+ {
249
+ "extractionMode": "incremental",
250
+ "fallbackStartDate": "2024-01-01T00:00:00Z"
251
+ }
252
+ ```
253
+
254
+ Extracts quantities with `updatedOn > lastRunTime`. Ideal for daily syncs.
255
+
256
+ **Mode 2: Date Range**
257
+
258
+ ```json
259
+ {
260
+ "extractionMode": "dateRange",
261
+ "startDate": "2025-01-01T00:00:00Z",
262
+ "endDate": "2025-01-15T23:59:59Z"
263
+ }
264
+ ```
265
+
266
+ Extracts quantities updated between `startDate` and `endDate`. Ideal for audits.
267
+
268
+ **Mode 3: Historical**
269
+
270
+ ```json
271
+ {
272
+ "extractionMode": "historical",
273
+ "startDate": "2024-01-01T00:00:00Z",
274
+ "endDate": "2024-12-31T23:59:59Z"
275
+ }
276
+ ```
277
+
278
+ Extracts quantities created between `startDate` and `endDate` using `createdOn` filter.
279
+
280
+ ## ⚠️ Production Safety & Guardrails
281
+
282
+ ### Critical: Extraction Mode Selection
283
+
284
+ **🟢 RECOMMENDED: Incremental Mode (Production)**
285
+
286
+ - Safe for automated schedules
287
+ - Natural rate limiting via timestamps
288
+ - Predictable resource usage
289
+ - **Use this for all production workflows**
290
+
291
+ **🟡 CAUTION: Date Range Mode (Audit/Backfill)**
292
+
293
+ - **Maximum 30-day window enforced**
294
+ - Use for specific audit requests only
295
+ - Run during off-peak hours
296
+ - Monitor resource usage
297
+
298
+ **🔴 DANGER: Historical Mode (One-Time Only)**
299
+
300
+ - **Maximum 90-day window enforced**
301
+ - **Requires explicit approval**
302
+ - **Risk of platform overload**
303
+ - Can fetch millions of records
304
+ - Use multiple small incremental runs instead
305
+ - Only for initial data migration
306
+
307
+ ### Date Range Validation (Required)
308
+
309
+ ```typescript
310
+ // Validate date range limits to prevent platform overload
311
+ function validateDateRange(mode, startDate, endDate) {
312
+ if (mode === 'incremental') return { valid: true };
313
+
314
+ if (!startDate || !endDate) {
315
+ return {
316
+ valid: false,
317
+ error: `${mode} mode requires both startDate and endDate`,
318
+ };
319
+ }
320
+
321
+ const start = new Date(startDate);
322
+ const end = new Date(endDate);
323
+ const daysDiff = (end - start) / (1000 * 60 * 60 * 24);
324
+
325
+ // Guardrail: Maximum date ranges
326
+ const maxDays = mode === 'dateRange' ? 30 : 90;
327
+
328
+ if (daysDiff > maxDays) {
329
+ return {
330
+ valid: false,
331
+ error: `${mode} mode: Maximum ${maxDays}-day window. Requested: ${Math.round(daysDiff)} days. Use multiple smaller extractions or incremental mode.`,
332
+ recommendation: `Split into ${Math.ceil(daysDiff / maxDays)} separate extractions of ${maxDays} days each.`,
333
+ };
334
+ }
335
+
336
+ if (daysDiff < 0) {
337
+ return { valid: false, error: 'endDate must be after startDate' };
338
+ }
339
+
340
+ return { valid: true };
341
+ }
342
+ ```
343
+
344
+ ### File Splitting Configuration
345
+
346
+ Large extractions must split into multiple files to prevent memory issues and upload failures.
347
+
348
+ ```json
349
+ {
350
+ "maxRecordsPerFile": 50000,
351
+ "maxFileSizeMB": 100,
352
+ "enableFileSplitting": true
353
+ }
354
+ ```
355
+
356
+ **File Naming Pattern:**
357
+
358
+ ```
359
+ {prefix}inventory-quantities-{timestamp}-part-{sequence}.csv
360
+
361
+ Examples:
362
+ inventory-quantities/daily/inventory-quantities-2025-01-22T14-30-00Z-part-001.csv
363
+ inventory-quantities/daily/inventory-quantities-2025-01-22T14-30-00Z-part-002.csv
364
+ inventory-quantities/daily/inventory-quantities-2025-01-22T14-30-00Z-manifest.json
365
+ ```
366
+
367
+ **Manifest File (auto-generated):**
368
+
369
+ ```json
370
+ {
371
+ "extractionId": "inventory-quantities-2025-01-22T14-30-00Z",
372
+ "totalRecords": 127543,
373
+ "totalFiles": 3,
374
+ "files": [
375
+ {
376
+ "filename": "inventory-quantities-2025-01-22T14-30-00Z-part-001.csv",
377
+ "recordCount": 50000,
378
+ "s3Key": "inventory-quantities/daily/inventory-quantities-2025-01-22T14-30-00Z-part-001.csv"
379
+ },
380
+ {
381
+ "filename": "inventory-quantities-2025-01-22T14-30-00Z-part-002.csv",
382
+ "recordCount": 50000,
383
+ "s3Key": "inventory-quantities/daily/inventory-quantities-2025-01-22T14-30-00Z-part-002.csv"
384
+ },
385
+ {
386
+ "filename": "inventory-quantities-2025-01-22T14-30-00Z-part-003.csv",
387
+ "recordCount": 27543,
388
+ "s3Key": "inventory-quantities/daily/inventory-quantities-2025-01-22T14-30-00Z-part-003.csv"
389
+ }
390
+ ],
391
+ "extractionMode": "dateRange",
392
+ "dateRange": {
393
+ "from": "2025-01-01T00:00:00Z",
394
+ "to": "2025-01-31T23:59:59Z"
395
+ },
396
+ "completedAt": "2025-01-22T14:35:27Z"
397
+ }
398
+ ```
399
+
400
+ ### Hard Limits (Enforced)
401
+
402
+ ```typescript
403
+ const SAFETY_LIMITS = {
404
+ // Maximum records per single extraction
405
+ MAX_RECORDS_TOTAL: 500000, // 500k hard limit
406
+
407
+ // Maximum records per file before splitting
408
+ MAX_RECORDS_PER_FILE: 50000, // 50k per file
409
+
410
+ // Maximum file size before splitting
411
+ MAX_FILE_SIZE_MB: 100, // 100MB per file
412
+
413
+ // Date range limits
414
+ MAX_DATE_RANGE_DAYS: 30, // dateRange mode
415
+ MAX_HISTORICAL_DAYS: 90, // historical mode
416
+
417
+ // Pagination limits
418
+ MAX_PAGE_SIZE: 500, // Fluent API limit
419
+ RECOMMENDED_PAGE_SIZE: 200, // Balance throughput/memory
420
+
421
+ // Memory management
422
+ CHUNK_SIZE: 10000, // Process in chunks
423
+ };
424
+ ```
425
+
426
+ ### Memory-Safe Implementation Pattern
427
+
428
+ ```typescript
429
+ // Process large extractions in chunks to prevent OOM
430
+ async function processLargeExtraction(edges, mapper, csvParser, s3, options) {
431
+ const CHUNK_SIZE = 10000;
432
+ const MAX_RECORDS_PER_FILE = options.maxRecordsPerFile || 50000;
433
+
434
+ let fileSequence = 1;
435
+ let currentFileRecords = [];
436
+ const manifestFiles = [];
437
+
438
+ for (let i = 0; i < edges.length; i += CHUNK_SIZE) {
439
+ const chunk = edges.slice(i, i + CHUNK_SIZE);
440
+
441
+ // Bulk mapping for chunk
442
+ const chunkNodes = chunk.map(edge => edge.node);
443
+ const mappingResult = await mapper.map(chunkNodes);
444
+
445
+ if (!mappingResult.success) {
446
+ const mappingErrors = mappingResult.errors || ['Unknown mapping failure'];
447
+ log.error('Chunk mapping failed', {
448
+ chunkIndex: i / CHUNK_SIZE,
449
+ errorCount: mappingErrors.length,
450
+ sampleErrors: mappingErrors.slice(0, 3),
451
+ });
452
+ throw new Error(`Mapping failed: ${mappingErrors[0] || 'Unknown error'}`);
453
+ }
454
+
455
+ const transformedChunk = mappingResult.data || [];
456
+ const mappingErrors = mappingResult.errors || [];
457
+
458
+ if (mappingErrors.length > 0) {
459
+ log.warn('Some records in chunk failed transformation', {
460
+ chunkIndex: i / CHUNK_SIZE,
461
+ errorCount: mappingErrors.length,
462
+ });
463
+ }
464
+
465
+ if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
466
+ log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
467
+ chunkIndex: i / CHUNK_SIZE,
468
+ skippedFields: mappingResult.skippedFields,
469
+ note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
470
+ });
471
+ }
472
+
473
+ // Add to current file, handling splits
474
+ for (const record of transformedChunk) {
475
+ currentFileRecords.push(record);
476
+
477
+ // Split file when limit reached
478
+ if (currentFileRecords.length >= MAX_RECORDS_PER_FILE) {
479
+ const fileInfo = await writeFileToS3(
480
+ currentFileRecords,
481
+ fileSequence++,
482
+ csvParser,
483
+ s3,
484
+ options
485
+ );
486
+ manifestFiles.push(fileInfo);
487
+ currentFileRecords = []; // Reset for next file
488
+ }
489
+ }
490
+ }
491
+
492
+ // Write remaining records
493
+ if (currentFileRecords.length > 0) {
494
+ const fileInfo = await writeFileToS3(
495
+ currentFileRecords,
496
+ fileSequence++,
497
+ csvParser,
498
+ s3,
499
+ options
500
+ );
501
+ manifestFiles.push(fileInfo);
502
+ }
503
+
504
+ // Write manifest
505
+ await writeManifest(manifestFiles, s3, options);
506
+
507
+ return manifestFiles;
508
+ }
509
+ ```
510
+
511
+ ### Enterprise Time Buffer Configuration
512
+
513
+ ```json
514
+ {
515
+ "overlapBufferSeconds": "60"
516
+ }
517
+ ```
518
+
519
+ **Default: 60 seconds (recommended for most deployments)**
520
+
521
+ **Purpose**: Prevents missed records due to:
522
+
523
+ - **Clock skew** between Fluent API servers (typically 1-5 seconds)
524
+ - **Transaction timing** - records updated during query execution
525
+ - **Race conditions** - records updated between extraction runs
526
+
527
+ **How It Works**:
528
+
529
+ - **Query**: Uses `updatedOn >= (lastRunTime - 60 seconds)`
530
+ - **Save**: Stores `MAX(updatedOn)` WITHOUT buffer
531
+ - **Result**: Records from the last minute of previous extraction are included again
532
+
533
+ **Buffer Sizes by Deployment**:
534
+
535
+ - `30` - Low-latency single-region (minimal clock skew expected)
536
+ - `60` - **Standard production** (recommended default)
537
+ - `300` - Cross-region deployments or high-latency networks
538
+
539
+ **Duplicate Handling**: Downstream systems should upsert by `quantity_id` (idempotent). Duplicates are safe and expected.
540
+
541
+ ### Timezone Handling
542
+
543
+ **All timestamps are in ISO 8601 format (UTC)**:
544
+
545
+ ```typescript
546
+ // Input: ISO 8601 UTC timestamp
547
+ const timestamp = '2025-01-22T14:30:00.000Z';
548
+
549
+ // JavaScript Date operations preserve UTC
550
+ new Date(timestamp).toISOString();
551
+ // Returns: "2025-01-22T14:30:00.000Z" (same format)
552
+
553
+ new Date(timestamp).getTime();
554
+ // Returns: 1737558600000 (UTC epoch milliseconds)
555
+
556
+ // Subtract 60 seconds for buffer
557
+ const buffered = new Date(new Date(timestamp).getTime() - 60000).toISOString();
558
+ // Returns: "2025-01-22T14:29:00.000Z"
559
+ ```
560
+
561
+ **Key Points**:
562
+
563
+ - Fluent API returns all timestamps in UTC
564
+ - `.getTime()` returns UTC epoch milliseconds
565
+ - Buffer arithmetic is done in milliseconds
566
+ - `.toISOString()` converts back to ISO 8601 UTC
567
+ - No timezone conversion needed
568
+
569
+ ## Export Mapping Configuration
570
+
571
+ Create file: `./config/inventory-quantities.export.json`
572
+
573
+ ```json
574
+ {
575
+ "name": "inventory-quantities.export",
576
+ "version": "1.0.0",
577
+ "description": "Fluent Inventory Quantities → CSV Export Mapping",
578
+ "fields": {
579
+ "quantity_id": { "source": "id", "required": true, "resolver": "sdk.trim" },
580
+ "quantity_ref": { "source": "ref", "required": true, "resolver": "sdk.trim" },
581
+ "catalogue_ref": { "source": "catalogue.ref", "required": false, "resolver": "sdk.trim" },
582
+ "catalogue_name": { "source": "catalogue.name", "required": false, "resolver": "sdk.trim" },
583
+ "location": { "source": "locationRef", "required": true, "resolver": "sdk.trim" },
584
+ "sku": { "source": "skuRef", "required": true, "resolver": "sdk.trim" },
585
+ "quantity": { "source": "qty", "required": true, "resolver": "sdk.parseInt" },
586
+ "type": { "source": "type", "required": true, "resolver": "sdk.uppercase" },
587
+ "status": { "source": "status", "required": true, "resolver": "sdk.uppercase" },
588
+ "expected_on": { "source": "expectedOn", "resolver": "sdk.formatDate" },
589
+ "created_on": { "source": "createdOn", "resolver": "sdk.formatDate" },
590
+ "updated_on": { "source": "updatedOn", "required": true, "resolver": "sdk.formatDate" }
591
+ }
592
+ }
593
+ ```
594
+
595
+ ## Mapping & Resolvers Explained
596
+
597
+ This section explains how the SDK transforms raw GraphQL data into your CSV export format using **UniversalMapper** and **SDK resolvers**.
598
+
599
+ ### SDK Resolvers Used
600
+
601
+ | Field | Resolver | Why? | Example Transformation |
602
+ | ---------------- | ---------------- | ------------------------------------------ | ----------------------------------------------- |
603
+ | `quantity_id` | `sdk.trim` | Clean quantity IDs from whitespace | `" Q001 "` → `"Q001"` |
604
+ | `quantity_ref` | `sdk.trim` | Clean quantity references | `" QTY-REF-001 "` → `"QTY-REF-001"` |
605
+ | `catalogue_ref` | `sdk.trim` | Clean catalogue references | `" DEFAULT_CATALOGUE "` → `"DEFAULT_CATALOGUE"` |
606
+ | `catalogue_name` | `sdk.trim` | Clean catalogue names | `" Default Catalogue "` → `"Default Catalogue"` |
607
+ | `location` | `sdk.trim` | Clean location references | `" DC01 "` → `"DC01"` |
608
+ | `sku` | `sdk.trim` | Clean SKU references | `" SKU-001 "` → `"SKU-001"` |
609
+ | `quantity` | `sdk.parseInt` | Parse quantity as integer for calculations | `"100"` → `100` |
610
+ | `type` | `sdk.uppercase` | Normalize type codes | `"available"` → `"AVAILABLE"` |
611
+ | `status` | `sdk.uppercase` | Normalize status codes | `"active"` → `"ACTIVE"` |
612
+ | `expected_on` | `sdk.formatDate` | Format dates for CSV export | `"2025-01-30T00:00:00.000Z"` → `"2025-01-30"` |
613
+ | `created_on` | `sdk.formatDate` | Format created timestamps | `"2025-01-15T10:00:00.000Z"` → `"2025-01-15"` |
614
+ | `updated_on` | `sdk.formatDate` | Format updated timestamps for tracking | `"2025-01-22T08:30:00.000Z"` → `"2025-01-22"` |
615
+
616
+ ### Transformation Flow
617
+
618
+ ```typescript
619
+ // 1. GraphQL Response (raw data from Fluent Commerce)
620
+ const rawQuantity = {
621
+ id: ' Q001 ',
622
+ ref: ' QTY-REF-001 ',
623
+ locationRef: ' DC01 ',
624
+ skuRef: ' SKU-001 ',
625
+ qty: '100',
626
+ type: 'available',
627
+ status: 'active',
628
+ expectedOn: null,
629
+ createdOn: '2025-01-15T10:00:00.000Z',
630
+ updatedOn: '2025-01-22T08:30:00.000Z',
631
+ catalogue: {
632
+ ref: ' DEFAULT_CATALOGUE ',
633
+ name: ' Default Catalogue ',
634
+ },
635
+ };
636
+
637
+ // 2. UniversalMapper applies SDK resolvers
638
+ const mapper = new UniversalMapper(inventoryQuantitiesExportMapping);
639
+ const result = await mapper.map(rawQuantity);
640
+
641
+ // 3. Transformed Output (clean, normalized for CSV)
642
+ const transformedQuantity = {
643
+ quantity_id: 'Q001',
644
+ quantity_ref: 'QTY-REF-001',
645
+ catalogue_ref: 'DEFAULT_CATALOGUE',
646
+ catalogue_name: 'Default Catalogue',
647
+ location: 'DC01',
648
+ sku: 'SKU-001',
649
+ quantity: 100,
650
+ type: 'AVAILABLE',
651
+ status: 'ACTIVE',
652
+ expected_on: '', // null → empty string
653
+ created_on: '2025-01-15',
654
+ updated_on: '2025-01-22',
655
+ };
656
+ ```
657
+
658
+ ### Custom Resolvers for Inventory Quantity-Specific Logic
659
+
660
+ While the mapping above uses built-in SDK resolvers, you can extend with custom business logic:
661
+
662
+ ```typescript
663
+ const customResolvers = {
664
+ /**
665
+ * Validate that quantity values are positive
666
+ */
667
+ 'custom.validateQuantity': (qty: any) => {
668
+ const parsed = parseInt(qty) || 0;
669
+ return parsed >= 0 ? parsed : 0; // Ensure non-negative
670
+ },
671
+
672
+ /**
673
+ * Add human-readable type descriptions for reporting
674
+ */
675
+ 'custom.enrichQuantityType': (type: string) => {
676
+ const typeDescriptions: Record<string, string> = {
677
+ LAST_ON_HAND: 'Last recorded on-hand quantity',
678
+ RESERVED: 'Reserved against orders',
679
+ DELTA: 'Incremental change (adjustment delta)',
680
+ SALE: 'Quantity decreased due to sale',
681
+ CORRECTION: 'Manual correction entry',
682
+ };
683
+ return typeDescriptions[(type || '').toUpperCase()] || type;
684
+ },
685
+
686
+ /**
687
+ * Check if expected date is in the future
688
+ */
689
+ 'custom.isExpectedInFuture': (expectedOn: string) => {
690
+ if (!expectedOn) return false;
691
+ return new Date(expectedOn) > new Date();
692
+ },
693
+
694
+ /**
695
+ * Calculate days until expected arrival
696
+ */
697
+ 'custom.calculateDaysUntilExpected': (expectedOn: string) => {
698
+ if (!expectedOn) return null;
699
+ const expected = new Date(expectedOn);
700
+ const today = new Date();
701
+ const diffMs = expected.getTime() - today.getTime();
702
+ return Math.ceil(diffMs / (1000 * 60 * 60 * 24));
703
+ },
704
+
705
+ /**
706
+ * Validate status-type combinations
707
+ */
708
+ 'custom.validateQuantityStatus': (_quantity: any) => {
709
+ // Example placeholder – adapt rules to your retailer-defined IQ types
710
+ return 'VALID';
711
+ },
712
+ };
713
+
714
+ // Use custom resolvers with UniversalMapper
715
+ const mapper = new UniversalMapper(inventoryQuantitiesExportMapping, {
716
+ customResolvers,
717
+ });
718
+ ```
719
+
720
+ ### Available SDK Resolvers
721
+
722
+ The SDK provides these built-in resolvers (no custom code needed):
723
+
724
+ **String Transformations:**
725
+
726
+ - `sdk.trim` - Remove leading/trailing whitespace
727
+ - `sdk.uppercase` - Convert to uppercase
728
+ - `sdk.lowercase` - Convert to lowercase
729
+ - `sdk.toString` - Convert to string
730
+
731
+ **Number Parsing:**
732
+
733
+ - `sdk.parseInt` - Parse as integer
734
+ - `sdk.parseFloat` - Parse as decimal
735
+ - `sdk.number` - Parse as number (auto-detect int/float)
736
+
737
+ **Date Formatting:**
738
+
739
+ - `sdk.formatDate` - ISO 8601 date only (YYYY-MM-DD)
740
+ - `sdk.formatDateShort` - Short date format
741
+ - `sdk.parseDate` - Parse various date formats
742
+
743
+ **Type Conversions:**
744
+
745
+ - `sdk.boolean` - Convert to boolean
746
+ - `sdk.parseJson` - Parse JSON strings
747
+ - `sdk.toJson` - Convert to JSON string
748
+
749
+ **Utilities:**
750
+
751
+ - `sdk.identity` - Return value unchanged
752
+ - `sdk.coalesce` - Return first non-null value
753
+
754
+ See [Universal Mapping Guide](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) for complete resolver documentation.
755
+
756
+ ## GraphQL Query
757
+
758
+ ```graphql
759
+ query GetInventoryQuantities(
760
+ $retailerId: ID!
761
+ $updatedAfter: DateTime
762
+ $createdAfter: DateTime
763
+ $first: Int!
764
+ $after: String
765
+ ) {
766
+ inventoryQuantities(
767
+ retailerId: $retailerId
768
+ updatedOn: { after: $updatedAfter }
769
+ createdOn: { after: $createdAfter }
770
+ first: $first
771
+ after: $after
772
+ ) {
773
+ edges {
774
+ node {
775
+ id
776
+ ref
777
+ locationRef
778
+ skuRef
779
+ qty
780
+ type
781
+ status
782
+ expectedOn
783
+ createdOn
784
+ updatedOn
785
+ catalogue {
786
+ ref
787
+ name
788
+ }
789
+ }
790
+ cursor
791
+ }
792
+ pageInfo {
793
+ hasNextPage
794
+ # Note: Fluent doesn't return endCursor/startCursor - cursors are in edges[].cursor
795
+ }
796
+ }
797
+ }
798
+ ```
799
+
800
+ ## Guardrails Implementation (Required)
801
+
802
+ ```typescript
803
+ // Overlap buffer (safety window)
804
+ const overlapBufferSeconds = parseInt(
805
+ ctx.activation?.getVariable('overlapBufferSeconds') || '60',
806
+ 10
807
+ );
808
+ const OVERLAP_BUFFER_MS = overlapBufferSeconds * 1000;
809
+
810
+ // Read last successful run and apply buffer
811
+ const kv = new VersoriKVAdapter(ctx.openKv(':project:'));
812
+ const stateKey = ['extraction', 'inventory-quantities-csv', 'lastRunTime'];
813
+ const lastRunState = await kv.get(stateKey);
814
+ const rawLastRunTime = lastRunState?.value?.timestamp || fallbackStartDate;
815
+ const bufferedLastRunTime = new Date(
816
+ new Date(rawLastRunTime).getTime() - OVERLAP_BUFFER_MS
817
+ ).toISOString();
818
+
819
+ // Query WITH buffer
820
+ const result = await client.graphql({
821
+ query: INVENTORY_QUANTITIES_QUERY,
822
+ variables: {
823
+ retailerId,
824
+ updatedAfter: bufferedLastRunTime,
825
+ first: pageSize,
826
+ },
827
+ pagination: { maxRecords },
828
+ });
829
+
830
+ const edges = result.data?.inventoryQuantities?.edges || [];
831
+
832
+ // 🛡️ GUARDRAIL: Validate extraction size limits
833
+ const MAX_RECORDS_PER_RUN = 500000;
834
+ const ESTIMATED_BYTES_PER_RECORD = 300; // Smaller than positions
835
+ const estimatedSizeMB = (edges.length * ESTIMATED_BYTES_PER_RECORD) / (1024 * 1024);
836
+ const MAX_CSV_SIZE_MB = 100;
837
+
838
+ if (edges.length > MAX_RECORDS_PER_RUN) {
839
+ log.error('Extraction limit exceeded', {
840
+ recordCount: edges.length,
841
+ maxAllowed: MAX_RECORDS_PER_RUN,
842
+ });
843
+ return {
844
+ success: false,
845
+ error: `Extraction limit exceeded: ${edges.length} records (max: ${MAX_RECORDS_PER_RUN})`,
846
+ recommendation: `Split into smaller extractions or increase extraction frequency`,
847
+ recordCount: edges.length,
848
+ maxAllowed: MAX_RECORDS_PER_RUN,
849
+ };
850
+ }
851
+
852
+ if (estimatedSizeMB > MAX_CSV_SIZE_MB) {
853
+ log.warn('CSV size approaching limit', {
854
+ estimatedSizeMB: estimatedSizeMB.toFixed(2),
855
+ maxAllowed: MAX_CSV_SIZE_MB,
856
+ });
857
+ }
858
+
859
+ log.info('Extraction limits validated', {
860
+ recordCount: edges.length,
861
+ estimatedSizeMB: estimatedSizeMB.toFixed(2),
862
+ withinLimits: true,
863
+ });
864
+
865
+ // Transform with UniversalMapper
866
+ const mapper = new UniversalMapper(inventoryQuantitiesExportMapping);
867
+ const transformedRecords: any[] = [];
868
+ for (const edge of edges) {
869
+ const mapped = await mapper.map(edge.node);
870
+ if (mapped.success) {
871
+ transformedRecords.push(mapped.data);
872
+ }
873
+ }
874
+
875
+ // Save state WITHOUT buffer (use MAX(updatedOn))
876
+ const maxUpdatedOn = transformedRecords.reduce((max, r) => {
877
+ const t = new Date(r.updated_on).getTime();
878
+ return t > max ? t : max;
879
+ }, new Date(rawLastRunTime).getTime());
880
+
881
+ await kv.set(stateKey, {
882
+ timestamp: new Date(maxUpdatedOn).toISOString(),
883
+ recordCount: transformedRecords.length,
884
+ extractedAt: new Date().toISOString(),
885
+ overlapBufferSeconds,
886
+ });
887
+
888
+ // Date range guardrails (if you add dateRange/historical modes)
889
+ function validateDateRange(mode: 'dateRange' | 'historical', from: string, to: string) {
890
+ const start = new Date(from);
891
+ const end = new Date(to);
892
+ const daysDiff = (end.getTime() - start.getTime()) / (1000 * 60 * 60 * 24);
893
+ const maxDays = mode === 'dateRange' ? 30 : 90;
894
+ if (daysDiff > maxDays) {
895
+ return {
896
+ valid: false,
897
+ error: `${mode} mode: Maximum ${maxDays}-day window. Requested: ${Math.round(daysDiff)} days.`,
898
+ };
899
+ }
900
+ if (daysDiff < 0) return { valid: false, error: 'endDate must be after startDate' };
901
+ return { valid: true };
902
+ }
903
+ ```
904
+
905
+ ---
906
+
907
+ ## Versori Workflows Structure
908
+
909
+ **Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
910
+
911
+ **Trigger Types:**
912
+ - **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
913
+ - **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
914
+ - **`workflow()`** → Durable workflows (advanced, rarely used)
915
+
916
+ **Execution Steps (chained to triggers):**
917
+ - **`http()`** → External API calls (chained from schedule/webhook)
918
+ - **`fn()`** → Internal processing (chained from schedule/webhook)
919
+
920
+ ### Recommended Project Structure
921
+
922
+ ```
923
+ inventory-quantities-extraction/
924
+ ├── index.ts # Entry point - exports all workflows
925
+ └── src/
926
+ ├── workflows/
927
+ │ ├── scheduled/
928
+ │ │ └── daily-inventory-quantities-extraction.ts # Scheduled: Daily extraction
929
+ │ │
930
+ │ └── webhook/
931
+ │ ├── adhoc-inventory-quantities-extraction.ts # Webhook: Manual trigger
932
+ │ └── job-status-check.ts # Webhook: Status query
933
+
934
+ ├── services/
935
+ │ └── inventory-quantities-extraction.service.ts # Shared orchestration logic (reusable)
936
+
937
+ └── config/
938
+ └── inventory-quantities.export.csv.json # Mapping configuration
939
+ ```
940
+
941
+ ---
942
+
943
+ ````csv
944
+ quantity_id,quantity_ref,catalogue_ref,catalogue_name,location,sku,quantity,type,status,expected_on,created_on,updated_on
945
+ Q001,QTY-REF-001,DEFAULT_CATALOGUE,Default Catalogue,DC01,SKU-001,100,AVAILABLE,ACTIVE,,2025-01-15T10:00:00Z,2025-01-22T08:30:00Z
946
+ Q002,QTY-REF-002,DEFAULT_CATALOGUE,Default Catalogue,DC01,SKU-001,50,RESERVED,ACTIVE,,2025-01-16T11:00:00Z,2025-01-22T09:15:00Z
947
+ Q003,QTY-REF-003,DEFAULT_CATALOGUE,Default Catalogue,DC02,SKU-002,200,EXPECTED,CREATED,2025-01-30T00:00:00Z,2025-01-17T12:00:00Z,2025-01-22T10:00:00Z
948
+ Q004,QTY-REF-004,DEFAULT_CATALOGUE,Default Catalogue,STORE-NYC,SKU-003,25,AVAILABLE,ACTIVE,,2025-01-18T13:00:00Z,2025-01-22T11:00:00Z
949
+
950
+ ## Advanced Mapping Patterns
951
+
952
+ ### Array Mapping (Preserving Nested Structure)
953
+
954
+ For nested data structures, use `isArray: true` pattern:
955
+
956
+ ```json
957
+ {
958
+ "fields": {
959
+ "ref": { "source": "ref", "required": true },
960
+ "relatedItems": {
961
+ "source": "items",
962
+ "isArray": true,
963
+ "fields": {
964
+ "itemRef": { "source": "ref", "required": true },
965
+ "value": { "source": "value", "resolver": "sdk.parseFloat" }
966
+ }
967
+ }
968
+ }
969
+ }
970
+ ````
971
+
972
+ **When to use**:
973
+
974
+ - **Flattened structure**: Simpler, easier for downstream systems
975
+ - **Nested with arrays**: Complex data, preserves relationships
976
+
977
+ ### Nested Object Mapping
978
+
979
+ **Option 1: Flattened paths** (recommended):
980
+
981
+ ```json
982
+ {
983
+ "fields": {
984
+ "location_ref": { "source": "location.ref" },
985
+ "location_name": { "source": "location.name" }
986
+ }
987
+ }
988
+ ```
989
+
990
+ **Option 2: Nested object definition**:
991
+
992
+ ```json
993
+ {
994
+ "fields": {
995
+ "location": {
996
+ "fields": {
997
+ "ref": { "source": "location.ref" },
998
+ "name": { "source": "location.name" }
999
+ }
1000
+ }
1001
+ }
1002
+ }
1003
+ ```
1004
+
1005
+ ## Error Handling Strategies
1006
+
1007
+ ### Handling Mapping Failures
1008
+
1009
+ **Strategy 1: Fail-fast (strict)**:
1010
+
1011
+ ```typescript
1012
+ if (errors.length > 0) {
1013
+ throw new Error(`${errors.length} records failed mapping validation`);
1014
+ }
1015
+ ```
1016
+
1017
+ **Strategy 2: Threshold-based (recommended)**:
1018
+
1019
+ ```typescript
1020
+ const errorRate = errors.length / transformed.length;
1021
+ if (errorRate > 0.05) {
1022
+ // 5% threshold
1023
+ throw new Error(`Error rate too high: ${(errorRate * 100).toFixed(1)}%`);
1024
+ }
1025
+ ```
1026
+
1027
+ **Strategy 3: Upload error manifest**:
1028
+
1029
+ ```typescript
1030
+ if (errors.length > 0) {
1031
+ const errorManifest = {
1032
+ extractionTimestamp: new Date().toISOString(),
1033
+ totalErrors: errors.length,
1034
+ errors: errors.map(e => ({ record: e.record, errors: e.errors })),
1035
+ };
1036
+ // Upload to storage for review
1037
+ }
1038
+ ```
1039
+
1040
+ ### State Management with Partial Failures
1041
+
1042
+ **Recommended**: Only update state if extraction succeeded:
1043
+
1044
+ ```typescript
1045
+ if (errors.length === 0) {
1046
+ await kv.set(stateKey, { timestamp: newTimestamp });
1047
+ log.info('State updated - all records successful');
1048
+ } else {
1049
+ log.warn('State NOT updated - will retry next run', {
1050
+ failedRecords: errors.length,
1051
+ willRetryNextRun: true,
1052
+ });
1053
+ }
1054
+ ```
1055
+
1056
+ ## GraphQL Query Validation & Testing
1057
+
1058
+ ### Schema Validation Workflow
1059
+
1060
+ **Step 1: Introspect schema**
1061
+
1062
+ ```bash
1063
+ npx fc-connect introspect-schema \
1064
+ --url https://your-instance.api.fluentcommerce.com/graphql \
1065
+ --output fluent-schema.json
1066
+ ```
1067
+
1068
+ **Step 2: Validate mapping**
1069
+
1070
+ ```bash
1071
+ npx fc-connect validate-schema \
1072
+ --mapping ./config/mapping.json \
1073
+ --schema ./fluent-schema.json
1074
+ ```
1075
+
1076
+ **Step 3: Analyze coverage**
1077
+
1078
+ ```bash
1079
+ npx fc-connect analyze-coverage \
1080
+ --mapping ./config/mapping.json \
1081
+ --schema ./fluent-schema.json
1082
+ ```
1083
+
1084
+ ### GraphQL Pagination Explained
1085
+
1086
+ The SDK handles pagination automatically:
1087
+
1088
+ ```typescript
1089
+ await client.graphql({
1090
+ query: QUERY,
1091
+ variables: { first: pageSize },
1092
+ pagination: { maxRecords }, // SDK handles cursors automatically
1093
+ });
1094
+ ```
1095
+
1096
+ ## Date Format Handling
1097
+
1098
+ | Format | Resolver | Output | Use Case |
1099
+ | -------- | --------------------- | -------------------------- | --------- |
1100
+ | CSV/JSON | `sdk.formatDate` | `2025-01-22T14:30:00.000Z` | ISO 8601 |
1101
+ | CSV/JSON | `sdk.formatDateShort` | `2025-01-22` | Date only |
1102
+ | CSV/JSON | `sdk.toString` | Pass through | As-is |
1103
+
1104
+ ## Monitoring & Alerting
1105
+
1106
+ ### Key Metrics to Track
1107
+
1108
+ ```typescript
1109
+ const metrics = {
1110
+ extractionDurationMs: Date.now() - startTime,
1111
+ recordCount: edges.length,
1112
+ transformedCount: transformed.length,
1113
+ failedCount: errors.length,
1114
+ errorRate: ((errors.length / edges.length) * 100).toFixed(2) + '%',
1115
+ fileSizeMB: (buffer.length / (1024 * 1024)).toFixed(2),
1116
+ lastRunTime: rawLastRunTime,
1117
+ newTimestamp: newTimestamp,
1118
+ };
1119
+ log.info('Extraction complete', metrics);
1120
+ ```
1121
+
1122
+ ### Alert Thresholds
1123
+
1124
+ ```typescript
1125
+ const ALERTS = {
1126
+ EXTRACTION_DURATION_MS: 5 * 60 * 1000, // 5 minutes
1127
+ MAX_ERROR_RATE: 0.05, // 5%
1128
+ MAX_FILE_SIZE_MB: 150, // 150MB
1129
+ MAX_RECORDS_PER_RUN: 100000, // Adjust per entity
1130
+ };
1131
+ ```
1132
+
1133
+ ## Testing Checklist
1134
+
1135
+ **Before production deployment:**
1136
+
1137
+ ### 1. Schema Validation
1138
+
1139
+ - [ ] Run `npx fc-connect introspect-schema`
1140
+ - [ ] Run `npx fc-connect validate-schema`
1141
+ - [ ] Run `npx fc-connect analyze-coverage`
1142
+ - [ ] Verify all `source` paths exist
1143
+
1144
+ ### 2. Mapping Testing
1145
+
1146
+ - [ ] Test with sample data (maxRecords=10)
1147
+ - [ ] Verify required fields populated
1148
+ - [ ] Verify SDK resolvers work correctly
1149
+ - [ ] Test custom resolvers with edge cases
1150
+
1151
+ ### 3. Error Handling
1152
+
1153
+ - [ ] Test with invalid data
1154
+ - [ ] Verify error collection
1155
+ - [ ] Test error threshold logic
1156
+
1157
+ ### 4. State Management
1158
+
1159
+ - [ ] Verify overlap buffer prevents misses
1160
+ - [ ] Test state recovery after failure
1161
+ - [ ] Verify timestamp saved WITHOUT buffer
1162
+
1163
+ ### 5. File Operations
1164
+
1165
+ - [ ] Test connection and upload
1166
+ - [ ] Verify file format validity
1167
+ - [ ] Test with large files (>50MB)
1168
+
1169
+ ### 6. Staging Environment
1170
+
1171
+ - [ ] Run full extraction in staging
1172
+ - [ ] Verify file format with downstream system
1173
+ - [ ] Monitor duration and resource usage
1174
+
1175
+ ## Troubleshooting Guide
1176
+
1177
+ **Issue**: "Extraction timeout after 10 minutes"
1178
+
1179
+ - **Cause**: Too many records
1180
+ - **Fix**: Reduce maxRecords, increase frequency
1181
+
1182
+ **Issue**: "Mapping errors for 50% of records"
1183
+
1184
+ - **Cause**: Schema mismatch
1185
+ - **Fix**: Run schema validation, check field names
1186
+
1187
+ **Issue**: "State not updating"
1188
+
1189
+ - **Cause**: KV write failure or intentional retry
1190
+ - **Fix**: Check KV logs, verify state update code
1191
+
1192
+ **Issue**: "First run exceeds limits"
1193
+
1194
+ - **Cause**: No previous timestamp, fetches all
1195
+ - **Fix**: Set fallbackStartDate close to current, apply filters
1196
+
1197
+ **Issue**: "Excessive duplicates"
1198
+
1199
+ - **Cause**: Overlap buffer (expected) or timestamp not saved
1200
+ - **Fix**: Verify newTimestamp saved WITHOUT buffer
1201
+
1202
+ ## Security Best Practices
1203
+
1204
+ ### Credential Management
1205
+
1206
+ **✅ DO**:
1207
+
1208
+ - Store credentials in Versori activation variables
1209
+ - Rotate credentials quarterly
1210
+ - Use least-privilege accounts
1211
+
1212
+ **❌ DON'T**:
1213
+
1214
+ - Never log credentials
1215
+ - Never commit to git
1216
+ - Never share across environments
1217
+
1218
+ ### Data Security
1219
+
1220
+ - Enable encryption in transit and at rest
1221
+ - Apply data retention policies
1222
+ - Monitor access logs
1223
+ - Use VPC/private networks for sensitive data
1224
+
1225
+ ---
1226
+
1227
+ ```
1228
+
1229
+ ---
1230
+
1231
+ **Pattern**: Enterprise incremental extraction with overlap buffer for inventory quantities
1232
+ **⚠️ Sample Code**: For SDK demonstration only - **ONLY use incremental mode in production**
1233
+ **Key Learning**: Use VersoriKVAdapter for state management with 60-second overlap buffer
1234
+ **Critical**: Apply overlap buffer to prevent missed records due to clock skew (default: 60 seconds)
1235
+ **Buffer Pattern**: Query WITH buffer (`updatedOn >= lastRunTime - 60s`), save WITHOUT buffer (`MAX(updatedOn)`)
1236
+ **Timezone**: All timestamps are ISO 8601 UTC format - no conversion needed
1237
+ ```
1238
+
1239
+ ---
1240
+
1241
+ ## 🔧 Complete Production Code
1242
+
1243
+ ### 1. Entry Point (src/index.ts)
1244
+
1245
+ ```typescript
1246
+ /**
1247
+ * Entry Point - Registers all workflows with Versori platform
1248
+ *
1249
+ * This file is the entry point for the Versori deployment.
1250
+ * It imports and re-exports workflows from their respective files:
1251
+ * 1. Scheduled extraction (runs automatically on cron schedule)
1252
+ * 2. Ad hoc webhook (manual trigger with optional date override)
1253
+ * 3. Job status webhook (query job progress)
1254
+ *
1255
+ * AI CUSTOMIZATION:
1256
+ * - Add new workflows by importing from their respective files
1257
+ * - Remove workflows by commenting out imports/exports
1258
+ * - Organize workflows by type (scheduled vs webhook) for clarity
1259
+ */
1260
+
1261
+ import { scheduledInventoryQuantitiesExtraction } from './workflows/scheduled/daily-inventory-quantities-extraction';
1262
+ import { adhocInventoryQuantitiesExtraction } from './workflows/webhook/adhoc-inventory-quantities-extraction';
1263
+ import { inventoryQuantitiesJobStatus } from './workflows/webhook/job-status-check';
1264
+
1265
+ // Register workflows with Versori platform
1266
+ // The platform will expose webhooks as HTTP endpoints and run scheduled workflows on cron schedule
1267
+
1268
+ export {
1269
+ scheduledInventoryQuantitiesExtraction, // Cron-based auto-run (NOT exposed as HTTP endpoint)
1270
+ adhocInventoryQuantitiesExtraction, // Manual webhook trigger (HTTP endpoint)
1271
+ inventoryQuantitiesJobStatus, // Job status query (HTTP endpoint)
1272
+ };
1273
+ ```
1274
+
1275
+ ---
1276
+
1277
+ ### 2. Workflows
1278
+
1279
+ #### src/workflows/scheduled/daily-inventory-quantities-extraction.ts
1280
+
1281
+ ```typescript
1282
+ /**
1283
+ * WORKFLOW 1: Scheduled Extraction
1284
+ *
1285
+ * Purpose: Automated hourly extraction for incremental sync
1286
+ * Trigger: Cron schedule (every hour at minute 0)
1287
+ * State Update: Always updates lastSync timestamp
1288
+ *
1289
+ * AI CUSTOMIZATION:
1290
+ * - Change schedule: Replace '0 * * * *' with your cron expression
1291
+ * Examples:
1292
+ * - Every 30 min: '*/30 * * * *'
1293
+ * - Daily at 2 AM: '0 2 * * *'
1294
+ * - Every 15 min: '*/15 * * * *'
1295
+ */
1296
+
1297
+ import { schedule, fn } from '@versori/run';
1298
+ import { executeInventoryQuantityExtraction } from '../../services/extraction-orchestration';
1299
+ import { generateJobId } from '../../utils/job-id-generator';
1300
+
1301
+ /**
1302
+ * WORKFLOW 1: Scheduled Extraction
1303
+ *
1304
+ * Purpose: Automated hourly extraction for incremental sync
1305
+ * Trigger: Cron schedule (every hour at minute 0)
1306
+ * State Update: Always updates lastSync timestamp
1307
+ *
1308
+ * AI CUSTOMIZATION:
1309
+ * - Change schedule: Replace '0 * * * *' with your cron expression
1310
+ * Examples:
1311
+ * - Every 30 min: '*/30 * * * *'
1312
+ * - Daily at 2 AM: '0 2 * * *'
1313
+ * - Every 15 min: '*/15 * * * *'
1314
+ */
1315
+ export const scheduledInventoryQuantitiesExtraction = schedule(
1316
+ 'inventory-quantities-scheduled',
1317
+ '0 * * * *', // ← CUSTOMIZE: Cron expression
1318
+ fn('execute-scheduled-extraction', async (ctx) => {
1319
+ const { log, activation } = ctx;
1320
+ const startTime = Date.now();
1321
+
1322
+ // Generate unique job ID for tracking
1323
+ // Format: SCHEDULED_IQ_YYYYMMDD_HHMMSS_random
1324
+ const jobId = generateJobId('SCHEDULED', 'INVENTORY_QUANTITIES');
1325
+
1326
+ log.info('🚀 [START] Scheduled extraction triggered', { jobId });
1327
+
1328
+ try {
1329
+ // Execute main workflow (extraction → transform → upload)
1330
+ const result = await executeInventoryQuantityExtraction(ctx, {
1331
+ jobId,
1332
+ triggeredBy: 'schedule',
1333
+ updateState: true, // Always update state for scheduled runs
1334
+ });
1335
+
1336
+ const durationMs = Date.now() - startTime;
1337
+
1338
+ log.info('✅ [END] Scheduled extraction completed', {
1339
+ jobId,
1340
+ recordCount: result.recordsExtracted,
1341
+ fileName: result.fileName,
1342
+ durationMs,
1343
+ durationSec: (durationMs / 1000).toFixed(2)
1344
+ });
1345
+
1346
+ return result;
1347
+
1348
+ } catch (error: any) {
1349
+ const durationMs = Date.now() - startTime;
1350
+
1351
+ log.error('❌ [ERROR] Scheduled extraction failed', {
1352
+ jobId,
1353
+ message: error instanceof Error ? error.message : String(error),
1354
+ stack: error instanceof Error ? error.stack : undefined,
1355
+ errorType: error instanceof Error ? error.constructor.name : 'Error',
1356
+ durationMs,
1357
+ recommendation: 'Check Fluent API connectivity, S3 credentials, and date range configuration'
1358
+ });
1359
+ throw error;
1360
+ }
1361
+ }));
1362
+ ```
1363
+
1364
+ ---
1365
+
1366
+ #### src/workflows/webhook/adhoc-inventory-quantities-extraction.ts
1367
+
1368
+ ```typescript
1369
+ /**
1370
+ * WORKFLOW 2: Ad hoc Extraction (Manual Trigger)
1371
+ *
1372
+ * Purpose: Manual extraction with optional date range override
1373
+ * Trigger: Webhook POST to /webhooks/inventory-quantities-adhoc
1374
+ * State Update: Optional (controlled by request payload)
1375
+ *
1376
+ * WEBHOOK PAYLOAD EXAMPLES:
1377
+ *
1378
+ * 1. Incremental (use last sync timestamp):
1379
+ * {}
1380
+ *
1381
+ * 2. Date range (manual override):
1382
+ * {
1383
+ * "fromDate": "2025-01-01T00:00:00Z",
1384
+ * "toDate": "2025-01-31T23:59:59Z",
1385
+ * "updateState": false
1386
+ * }
1387
+ *
1388
+ * AI CUSTOMIZATION:
1389
+ * - Add request validation
1390
+ * - Add authentication check
1391
+ * - Add custom filters from payload
1392
+ */
1393
+
1394
+ import { webhook, fn } from '@versori/run';
1395
+ import { executeInventoryQuantityExtraction } from '../../services/extraction-orchestration';
1396
+ import { generateJobId } from '../../utils/job-id-generator';
1397
+
1398
+ export const adhocInventoryQuantitiesExtraction = webhook(
1399
+ 'inventory-quantities-adhoc',
1400
+ { connection: 'inventory-quantities-adhoc', response: { mode: 'sync' } },
1401
+ fn('execute-adhoc-extraction', async (ctx) => {
1402
+ const { data, log, connections, activation } = ctx;
1403
+ const startTime = Date.now();
1404
+
1405
+ // Generate unique job ID
1406
+ const jobId = generateJobId('ADHOC', 'INVENTORY_QUANTITIES');
1407
+
1408
+ // SECURITY: Authentication is enforced by Versori connection configuration
1409
+ // Configure auth on the connection and reference it in webhook({ connection: '...' })
1410
+
1411
+ // Extract optional date override from webhook payload
1412
+ const fromDate = data.fromDate as string | undefined;
1413
+ const toDate = data.toDate as string | undefined;
1414
+ const updateState = data.updateState === true; // Default false; advance state only if explicitly true
1415
+
1416
+ log.info('🌐 [START] Ad hoc extraction triggered via webhook', {
1417
+ jobId,
1418
+ hasDateOverride: !!fromDate,
1419
+ fromDate: fromDate || 'not specified',
1420
+ toDate: toDate || 'not specified',
1421
+ updateState
1422
+ });
1423
+
1424
+ try {
1425
+ // Execute main workflow with optional overrides
1426
+ const result = await executeInventoryQuantityExtraction(ctx, {
1427
+ jobId,
1428
+ triggeredBy: 'webhook',
1429
+ fromDate, // Optional: override start date
1430
+ toDate, // Optional: override end date
1431
+ updateState, // Optional: skip state update for historical queries
1432
+ });
1433
+
1434
+ const durationMs = Date.now() - startTime;
1435
+
1436
+ log.info('✅ [END] Ad hoc extraction completed', {
1437
+ jobId,
1438
+ recordCount: result.recordsExtracted,
1439
+ fileName: result.fileName,
1440
+ isManualOverride: !!fromDate,
1441
+ stateUpdated: result.stateUpdated,
1442
+ durationMs,
1443
+ durationSec: (durationMs / 1000).toFixed(2)
1444
+ });
1445
+
1446
+ // Return success with job details
1447
+ return {
1448
+ success: true,
1449
+ jobId,
1450
+ recordsExtracted: result.recordsExtracted,
1451
+ fileName: result.fileName,
1452
+ s3Path: result.s3Path,
1453
+ statusUrl: `/webhooks/inventory-quantities-job-status?jobId=${jobId}`,
1454
+ durationMs,
1455
+ dateRange: fromDate ? {
1456
+ from: fromDate,
1457
+ to: toDate || 'not specified',
1458
+ updateState
1459
+ } : undefined
1460
+ };
1461
+
1462
+ } catch (error: any) {
1463
+ const durationMs = Date.now() - startTime;
1464
+
1465
+ log.error('❌ [ERROR] Ad hoc extraction failed', {
1466
+ jobId,
1467
+ message: error instanceof Error ? error.message : String(error),
1468
+ stack: error instanceof Error ? error.stack : undefined,
1469
+ errorType: error instanceof Error ? error.constructor.name : 'Error',
1470
+ durationMs,
1471
+ recommendation: 'Verify date range format (ISO 8601), check S3 credentials, and ensure Fluent API access'
1472
+ });
1473
+
1474
+ return {
1475
+ success: false,
1476
+ jobId,
1477
+ message: error instanceof Error ? error.message : String(error),
1478
+ stack: error instanceof Error ? error.stack : undefined,
1479
+ errorType: error instanceof Error ? error.constructor.name : 'Error',
1480
+ durationMs,
1481
+ recommendation: 'Verify date range format (ISO 8601), check S3 credentials, and ensure Fluent API access'
1482
+ };
1483
+ }
1484
+ }));
1485
+ ```
1486
+
1487
+ ---
1488
+
1489
+ #### src/workflows/webhook/job-status-check.ts
1490
+
1491
+ ```typescript
1492
+ /**
1493
+ * WORKFLOW 3: Job Status Query
1494
+ *
1495
+ * Purpose: Check job progress and status
1496
+ * Trigger: Webhook GET/POST to /webhooks/inventory-quantities-job-status?jobId=xxx
1497
+ * Returns: Current job status, stage, progress
1498
+ *
1499
+ * QUERY EXAMPLES:
1500
+ *
1501
+ * 1. HTTP GET:
1502
+ * GET /webhooks/inventory-quantities-job-status?jobId=ADHOC_IQ_20251027_183045_abc123
1503
+ *
1504
+ * 2. HTTP POST:
1505
+ * POST /webhooks/inventory-quantities-job-status
1506
+ * { "jobId": "ADHOC_IQ_20251027_183045_abc123" }
1507
+ */
1508
+
1509
+ import { webhook, fn } from '@versori/run';
1510
+ import { getJobStatus } from '../../services/extraction-orchestration';
1511
+
1512
+ export const inventoryQuantitiesJobStatus = webhook(
1513
+ 'inventory-quantities-job-status',
1514
+ { connection: 'inventory-quantities-job-status', response: { mode: 'sync' } },
1515
+ fn('query-job-status', async (ctx) => {
1516
+ const { data, log, openKv, activation } = ctx;
1517
+ const startTime = Date.now();
1518
+
1519
+ // SECURITY: Authentication is enforced by Versori connection configuration
1520
+ // Configure auth on the connection and reference it in webhook({ connection: '...' })
1521
+
1522
+ // Get jobId from query param or POST body
1523
+ const jobId = data.jobId as string;
1524
+
1525
+ if (!jobId) {
1526
+ log.error('❌ Job ID not provided in request');
1527
+ return {
1528
+ success: false,
1529
+ error: 'Job ID is required. Provide jobId in query param or request body.'
1530
+ };
1531
+ }
1532
+
1533
+ log.info('🔍 [START] Querying job status', { jobId });
1534
+
1535
+ try {
1536
+ // Query job status from KV store
1537
+ const status = await getJobStatus(openKv(':project:'), jobId, log);
1538
+
1539
+ const durationMs = Date.now() - startTime;
1540
+
1541
+ if (!status) {
1542
+ log.info('⚠️ Job not found', { jobId, durationMs });
1543
+ return {
1544
+ success: false,
1545
+ error: 'Job not found',
1546
+ jobId,
1547
+ durationMs
1548
+ };
1549
+ }
1550
+
1551
+ log.info('✅ [END] Job status retrieved', {
1552
+ jobId,
1553
+ status: status.status,
1554
+ durationMs
1555
+ });
1556
+
1557
+ return {
1558
+ success: true,
1559
+ jobId,
1560
+ ...status,
1561
+ queryDurationMs: durationMs
1562
+ };
1563
+
1564
+ } catch (error: any) {
1565
+ const durationMs = Date.now() - startTime;
1566
+
1567
+ log.error('❌ [ERROR] Failed to query job status', {
1568
+ jobId,
1569
+ message: error instanceof Error ? error.message : String(error),
1570
+ stack: error instanceof Error ? error.stack : undefined,
1571
+ errorType: error instanceof Error ? error.constructor.name : 'Error',
1572
+ durationMs,
1573
+ recommendation: 'Verify KV store access and job ID format'
1574
+ });
1575
+
1576
+ return {
1577
+ success: false,
1578
+ jobId,
1579
+ message: error instanceof Error ? error.message : String(error),
1580
+ stack: error instanceof Error ? error.stack : undefined,
1581
+ errorType: error instanceof Error ? error.constructor.name : 'Error',
1582
+ durationMs,
1583
+ recommendation: 'Verify KV store access and job ID format'
1584
+ };
1585
+ }
1586
+ }));
1587
+ ```
1588
+
1589
+ ---
1590
+
1591
+ ### 3. Main Orchestration Service (src/services/extraction-orchestration.ts)
1592
+
1593
+ ```typescript
1594
+ /**
1595
+ * MAIN ORCHESTRATION SERVICE
1596
+ *
1597
+ * This is the heart of the extraction workflow. It coordinates all steps:
1598
+ * 1. Initialize clients and services
1599
+ * 2. Determine date range (incremental vs manual)
1600
+ * 3. Extract data using ExtractionOrchestrator
1601
+ * 4. Transform using UniversalMapper
1602
+ * 5. Generate CSV using CSVParserService
1603
+ * 6. Upload to S3
1604
+ * 7. Track job progress with JobTracker
1605
+ * 8. Update state for next run
1606
+ *
1607
+ * NAMING PATTERN (consistent across all use cases):
1608
+ * - Interface: {Entity}ExtractionParams (e.g., InventoryQuantityExtractionParams)
1609
+ * - Result: {Entity}ExtractionResult (e.g., InventoryQuantityExtractionResult)
1610
+ * - Main function: execute{Entity}Extraction (e.g., executeInventoryQuantityExtraction)
1611
+ *
1612
+ * AI CUSTOMIZATION HINTS:
1613
+ * - Change entity: Replace "InventoryQuantity" with "Order", "Product", etc.
1614
+ * - Change output: Replace CSVParserService with XMLBuilder
1615
+ * - Change destination: Replace S3DataSource with SftpDataSource
1616
+ * - Add steps: Insert new service calls between existing steps
1617
+ */
1618
+
1619
+ import { Buffer } from 'node:buffer';
1620
+ import {
1621
+ createClient,
1622
+ ExtractionOrchestrator,
1623
+ JobTracker,
1624
+ UniversalMapper,
1625
+ CSVParserService,
1626
+ S3DataSource,
1627
+ } from '@fluentcommerce/fc-connect-sdk';
1628
+
1629
+ import mappingConfig from '../../config/inventory-quantities.export.csv.json' with { type: 'json' };
1630
+
1631
+ // ✅ VERSORI PLATFORM: Use native log from context, not LoggingService
1632
+
1633
+ /**
1634
+ * Parameters for extraction workflow
1635
+ *
1636
+ * NAMING: {Entity}ExtractionParams
1637
+ */
1638
+ export interface InventoryQuantityExtractionParams {
1639
+ jobId: string;
1640
+ triggeredBy: 'schedule' | 'webhook';
1641
+ fromDate?: string; // Optional: manual date override
1642
+ toDate?: string; // Optional: manual date override
1643
+ updateState: boolean; // Whether to update lastSync timestamp
1644
+
1645
+ // AI CUSTOMIZATION: Add filters specific to entity
1646
+ quantityTypes?: string[]; // e.g., ['LAST_ON_HAND', 'RESERVED']
1647
+ catalogueRef?: string; // e.g., 'DEFAULT_CATALOGUE'
1648
+ }
1649
+
1650
+ /**
1651
+ * Result from extraction workflow
1652
+ *
1653
+ * NAMING: {Entity}ExtractionResult
1654
+ */
1655
+ export interface InventoryQuantityExtractionResult {
1656
+ success: boolean;
1657
+ jobId: string;
1658
+ recordsExtracted: number;
1659
+ fileName?: string;
1660
+ s3Path?: string;
1661
+ error?: string;
1662
+ errors?: any[];
1663
+ isManualOverride?: boolean;
1664
+ stateUpdated?: boolean;
1665
+ newTimestamp?: string;
1666
+ }
1667
+
1668
+ /**
1669
+ * GraphQL Query for Inventory Quantities
1670
+ *
1671
+ * NAMING: {ENTITY}_EXTRACTION_QUERY (uppercase constant)
1672
+ */
1673
+ const INVENTORY_QUANTITIES_EXTRACTION_QUERY = `
1674
+ query GetInventoryQuantities(
1675
+ $catalogues: [InventoryCatalogueKey]
1676
+ $dateRangeFilter: DateRange
1677
+ $productRefs: [String!]
1678
+ $types: [String!]
1679
+ $first: Int!
1680
+ $after: String
1681
+ ) {
1682
+ inventoryQuantities(
1683
+ catalogues: $catalogues
1684
+ updatedOn: $dateRangeFilter
1685
+ productRef: $productRefs
1686
+ type: $types
1687
+ first: $first
1688
+ after: $after
1689
+ ) {
1690
+ edges {
1691
+ node {
1692
+ id
1693
+ ref
1694
+ locationRef
1695
+ productRef
1696
+ qty
1697
+ type
1698
+ status
1699
+ expectedOn
1700
+ createdOn
1701
+ updatedOn
1702
+ catalogue {
1703
+ ref
1704
+ name
1705
+ }
1706
+ }
1707
+ cursor
1708
+ }
1709
+ pageInfo {
1710
+ hasNextPage
1711
+ }
1712
+ }
1713
+ }
1714
+ `;
1715
+
1716
+ /**
1717
+ * Query job status from KV store
1718
+ *
1719
+ * ✅ VERSORI PLATFORM: Uses native log from context (passed as parameter)
1720
+ */
1721
+ export async function getJobStatus(
1722
+ kv: any, // ✅ Versori KV (compatible with JobTracker's KVAdapter interface)
1723
+ jobId: string,
1724
+ log: any // ✅ Native Versori log from context
1725
+ ): Promise<any | undefined> {
1726
+ try {
1727
+ const tracker = new JobTracker(kv, log);
1728
+ return await tracker.getJob(jobId);
1729
+ } catch (error: any) {
1730
+ log.error('Failed to get job status', { jobId, message: error instanceof Error ? error.message : String(error),
1731
+ stack: error instanceof Error ? error.stack : undefined,
1732
+ errorType: error instanceof Error ? error.constructor.name : 'Error', });
1733
+ return undefined;
1734
+ }
1735
+ }
1736
+
1737
+ /**
1738
+ * MAIN ORCHESTRATION FUNCTION
1739
+ *
1740
+ * NAMING: execute{Entity}Extraction (e.g., executeInventoryQuantityExtraction)
1741
+ *
1742
+ * This function implements the complete workflow in steps.
1743
+ * Each step is clearly commented for AI understanding.
1744
+ */
1745
+ export async function executeInventoryQuantityExtraction(
1746
+ ctx: any,
1747
+ params: InventoryQuantityExtractionParams
1748
+ ): Promise<InventoryQuantityExtractionResult> {
1749
+ // ✅ VERSORI PLATFORM: Extract native log from context
1750
+ const { log, openKv, activation } = ctx;
1751
+ const { jobId, triggeredBy, fromDate, toDate, updateState } = params;
1752
+
1753
+ // Open KV store for state management and job tracking
1754
+ // ✅ Pass raw Versori KV directly - it matches KVAdapter interface
1755
+ // ✅ Pass native log to JobTracker
1756
+ const kv = openKv(':project:');
1757
+ const tracker = new JobTracker(kv, log);
1758
+
1759
+ try {
1760
+ // ═══════════════════════════════════════════════════════════
1761
+ // STEP 1/8: Initialize Job Tracking
1762
+ // ═══════════════════════════════════════════════════════════
1763
+ log.info('📝 [STEP 1/8] Initializing job tracking', { jobId });
1764
+
1765
+ await tracker.createJob(jobId, {
1766
+ triggeredBy,
1767
+ hasDateOverride: !!fromDate,
1768
+ fromDate,
1769
+ toDate,
1770
+ updateStateAfterRun: updateState,
1771
+ });
1772
+
1773
+ // ═══════════════════════════════════════════════════════════
1774
+ // STEP 2/8: Initialize Fluent Client
1775
+ // ═══════════════════════════════════════════════════════════
1776
+ log.info('📡 [STEP 2/8] Initializing Fluent Commerce client', { jobId });
1777
+
1778
+ const client = await createClient(ctx, { validateConnection: true });
1779
+
1780
+ if (!client) {
1781
+ throw new Error('Failed to create Fluent Commerce client');
1782
+ }
1783
+
1784
+ log.info('✅ Fluent client initialized and connection validated', { jobId });
1785
+
1786
+ // ═══════════════════════════════════════════════════════════
1787
+ // STEP 3/8: Determine Date Range
1788
+ // ═══════════════════════════════════════════════════════════
1789
+ log.info('📅 [STEP 3/8] Determining date range for extraction', { jobId });
1790
+
1791
+ // State key for incremental sync tracking
1792
+ // NAMING: last{Entity}Sync (e.g., lastInventoryQuantitySync)
1793
+ const STATE_KEY = 'lastInventoryQuantitySync';
1794
+ const DEFAULT_FALLBACK = '2024-01-01T00:00:00Z';
1795
+ const OVERLAP_BUFFER_SECONDS = parseInt(
1796
+ activation.getVariable('overlapBufferSeconds') || '60',
1797
+ 10
1798
+ );
1799
+
1800
+ let dateRangeFilter: { from?: string; to?: string } | null = null;
1801
+ const isManualOverride = !!fromDate;
1802
+
1803
+ if (isManualOverride) {
1804
+ // Manual date override from webhook
1805
+ dateRangeFilter = { from: fromDate, to: toDate };
1806
+ log.info('Using manual date override', { fromDate, toDate });
1807
+ } else {
1808
+ // Incremental sync - get last sync timestamp
1809
+ const rawLastRunTime = (await kv.get<string>(STATE_KEY)) || DEFAULT_FALLBACK;
1810
+
1811
+ // Apply overlap buffer (prevents missed records)
1812
+ const bufferedLastRunTime = new Date(
1813
+ new Date(rawLastRunTime).getTime() - OVERLAP_BUFFER_SECONDS * 1000
1814
+ ).toISOString();
1815
+
1816
+ const effectiveEndTime = toDate || new Date().toISOString();
1817
+
1818
+ dateRangeFilter = {
1819
+ from: bufferedLastRunTime,
1820
+ to: effectiveEndTime, // End of extraction window
1821
+ };
1822
+
1823
+ log.info('Using incremental sync with overlap buffer', {
1824
+ rawLastRunTime,
1825
+ bufferedLastRunTime,
1826
+ effectiveEndTime,
1827
+ overlapBufferSeconds: OVERLAP_BUFFER_SECONDS,
1828
+ });
1829
+ }
1830
+
1831
+ // ═══════════════════════════════════════════════════════════
1832
+ // STEP 4/8: Extract Data (ExtractionOrchestrator)
1833
+ // ═══════════════════════════════════════════════════════════
1834
+ log.info('🔄 [STEP 4/8] Extracting data from Fluent Commerce', { jobId });
1835
+
1836
+ await tracker.updateJob(jobId, {
1837
+ status: 'processing',
1838
+ stage: 'extraction',
1839
+ message: 'Extracting data with auto-pagination',
1840
+ });
1841
+
1842
+ // Build catalogues array from config
1843
+ const catalogueRef = params.catalogueRef || activation.getVariable('catalogueRef');
1844
+ const catalogues = catalogueRef ? [{ ref: catalogueRef }] : [];
1845
+
1846
+ // Configure extraction
1847
+ const pageSize = parseInt(activation.getVariable('pageSize') || '200', 10);
1848
+ const maxRecords = parseInt(activation.getVariable('maxRecords') || '100000', 10);
1849
+
1850
+ // Initialize ExtractionOrchestrator
1851
+ const orchestrator = new ExtractionOrchestrator(client, log);
1852
+
1853
+ // ? Enhanced: Extract context for progress logging
1854
+ const dateRangeInfo = {
1855
+ start: dateRangeFilter?.from || 'N/A',
1856
+ end: dateRangeFilter?.to || 'N/A',
1857
+ catalogues: catalogues.map((c: any) => c.ref).join(', ') || 'all',
1858
+ types: params.quantityTypes?.join(', ') || 'all'
1859
+ };
1860
+
1861
+ // ? Enhanced: Start logging with context
1862
+ log.info(`🔍 [ExtractionOrchestrator] Starting extraction`, {
1863
+ query: 'inventoryQuantities',
1864
+ pageSize,
1865
+ maxRecords,
1866
+ dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
1867
+ catalogues: dateRangeInfo.catalogues,
1868
+ quantityTypes: dateRangeInfo.types,
1869
+ jobId
1870
+ });
1871
+
1872
+ // Execute extraction with auto-pagination
1873
+ const extractionResult = await orchestrator.extract({
1874
+ query: INVENTORY_QUANTITIES_EXTRACTION_QUERY,
1875
+ resultPath: 'inventoryQuantities.edges.node',
1876
+ variables: {
1877
+ catalogues,
1878
+ dateRangeFilter,
1879
+ types: params.quantityTypes,
1880
+ // Note: Don't include 'first' or 'after' here; orchestrator injects them
1881
+ },
1882
+ pageSize,
1883
+ maxRecords,
1884
+ // Optional: validate each record
1885
+ validateItem: (item: any) => {
1886
+ return !!(item.ref && item.productRef);
1887
+ },
1888
+ });
1889
+
1890
+ const records = extractionResult.data || [];
1891
+
1892
+ log.info('Extraction complete', {
1893
+ totalRecords: extractionResult.stats.totalRecords,
1894
+ totalPages: extractionResult.stats.totalPages,
1895
+ validRecords: extractionResult.stats.validRecords ?? records.length,
1896
+ failedValidations: extractionResult.stats.failedValidations,
1897
+ errors: extractionResult.errors ? extractionResult.errors.length : 0,
1898
+ });
1899
+
1900
+ // ? Enhanced: Completion logging with summary
1901
+ log.info(`✅ [ExtractionOrchestrator] Extraction completed`, {
1902
+ totalRecords: extractionResult.stats.totalRecords,
1903
+ totalPages: extractionResult.stats.totalPages,
1904
+ validRecords: extractionResult.stats.validRecords ?? records.length,
1905
+ failedValidations: extractionResult.stats.failedValidations,
1906
+ truncated: extractionResult.stats.truncated,
1907
+ truncationReason: extractionResult.stats.truncationReason,
1908
+ dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
1909
+ jobId
1910
+ });
1911
+
1912
+ if (extractionResult.errors && extractionResult.errors.length > 0) {
1913
+ log.warn('Non-fatal extraction errors encountered', {
1914
+ errorCount: extractionResult.errors.length,
1915
+ sampleErrors: extractionResult.errors.slice(0, 3),
1916
+ });
1917
+ }
1918
+
1919
+ // Handle empty result
1920
+ if (records.length === 0) {
1921
+ log.info('No records to process');
1922
+
1923
+ // Update state even with no records (prevents re-querying empty window)
1924
+ if (updateState && !isManualOverride) {
1925
+ await kv.set(STATE_KEY, new Date().toISOString());
1926
+ }
1927
+
1928
+ await tracker.markCompleted(jobId, {
1929
+ recordCount: 0,
1930
+ message: 'No records to extract',
1931
+ });
1932
+
1933
+ return {
1934
+ success: true,
1935
+ jobId,
1936
+ recordsExtracted: 0,
1937
+ };
1938
+ }
1939
+
1940
+ // ═══════════════════════════════════════════════════════════
1941
+ // STEP 5/8: Transform Data (UniversalMapper)
1942
+ // ═══════════════════════════════════════════════════════════
1943
+ log.info('🔧 [STEP 5/8] Transforming data with UniversalMapper', {
1944
+ jobId,
1945
+ recordCount: records.length,
1946
+ });
1947
+
1948
+ await tracker.updateJob(jobId, {
1949
+ status: 'processing',
1950
+ stage: 'transformation',
1951
+ message: `Transforming ${records.length} records`,
1952
+ });
1953
+
1954
+ const mapper = new UniversalMapper(mappingConfig);
1955
+ const mappingResult = await mapper.map(records);
1956
+
1957
+ if (!mappingResult.success) {
1958
+ const mappingErrors = mappingResult.errors || ['Unknown mapping failure'];
1959
+ await tracker.markFailed(jobId, {
1960
+ error: mappingErrors[0] || 'UniversalMapper returned unsuccessful result',
1961
+ failedCount: mappingErrors.length,
1962
+ });
1963
+ return {
1964
+ success: false,
1965
+ error: `Transformation failed: ${mappingErrors[0] || 'Unknown error'}`,
1966
+ jobId,
1967
+ errors: mappingErrors,
1968
+ };
1969
+ }
1970
+
1971
+ const transformedRecords = Array.isArray(mappingResult.data) ? mappingResult.data : [];
1972
+ const mappingErrors = mappingResult.errors || [];
1973
+
1974
+ if (mappingErrors.length > 0) {
1975
+ log.warn('Some records failed transformation', {
1976
+ jobId,
1977
+ errorCount: mappingErrors.length,
1978
+ sampleErrors: mappingErrors.slice(0, 3),
1979
+ });
1980
+ }
1981
+
1982
+ if (transformedRecords.length === 0) {
1983
+ await tracker.markFailed(jobId, {
1984
+ error: 'All records failed mapping',
1985
+ failedCount: mappingErrors.length,
1986
+ errors: mappingErrors,
1987
+ });
1988
+ return {
1989
+ success: false,
1990
+ error: 'All records failed mapping',
1991
+ jobId,
1992
+ errors: mappingErrors,
1993
+ };
1994
+ }
1995
+
1996
+ log.info('Transformation complete', {
1997
+ successful: transformedRecords.length,
1998
+ failed: mappingErrors.length,
1999
+ skippedRecords: records.length - transformedRecords.length,
2000
+ });
2001
+
2002
+ // ═══════════════════════════════════════════════════════════
2003
+ // STEP 6/8: Generate CSV (CSVParserService)
2004
+ // ═══════════════════════════════════════════════════════════
2005
+ log.info('📄 [STEP 6/8] Generating CSV file', { jobId });
2006
+
2007
+ await tracker.updateJob(jobId, {
2008
+ status: 'processing',
2009
+ stage: 'csv_generation',
2010
+ message: `Generating CSV for ${transformedRecords.length} records`,
2011
+ });
2012
+
2013
+ // Initialize CSVParserService
2014
+ const csvParser = new CSVParserService({ includeHeaders: true });
2015
+
2016
+ // Generate CSV content
2017
+ const csvContent = await csvParser.stringify(transformedRecords);
2018
+
2019
+ // Generate filename
2020
+ const fileNamePrefix = activation.getVariable('fileNamePrefix') || 'inventoryquantities';
2021
+ const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
2022
+ const fileName = `${fileNamePrefix}-${timestamp}.csv`;
2023
+
2024
+ log.info('CSV file generated', {
2025
+ fileName,
2026
+ sizeBytes: csvContent.length,
2027
+ recordCount: transformedRecords.length,
2028
+ });
2029
+
2030
+ // ═══════════════════════════════════════════════════════════
2031
+ // STEP 7/8: Upload to S3 (S3DataSource)
2032
+ // ═══════════════════════════════════════════════════════════
2033
+ log.info('☁️ [STEP 7/8] Uploading to S3', { jobId, fileName });
2034
+
2035
+ await tracker.updateJob(jobId, {
2036
+ status: 'processing',
2037
+ stage: 's3_upload',
2038
+ message: `Uploading ${fileName} to S3`,
2039
+ });
2040
+
2041
+ // Get S3 configuration from activation variables
2042
+ const s3Config = {
2043
+ bucket: activation.getVariable('s3BucketName'),
2044
+ region: activation.getVariable('awsRegion') || 'us-east-1',
2045
+ accessKeyId: activation.getVariable('awsAccessKeyId'),
2046
+ secretAccessKey: activation.getVariable('awsSecretAccessKey'),
2047
+ };
2048
+ const s3Prefix = activation.getVariable('s3Prefix') || 'inventory-quantities/daily/';
2049
+
2050
+ // Validate S3 config
2051
+ if (!s3Config.bucket || !s3Config.accessKeyId || !s3Config.secretAccessKey) {
2052
+ throw new Error(
2053
+ 'S3 configuration incomplete: missing bucket, accessKeyId, or secretAccessKey'
2054
+ );
2055
+ }
2056
+
2057
+ // Initialize S3 data source
2058
+ // ✅ VERSORI PLATFORM: Pass native log from context
2059
+ const s3 = new S3DataSource(
2060
+ {
2061
+ type: 'S3_CSV',
2062
+ connectionId: 'inventory-quantities-s3',
2063
+ name: 'Inventory Quantities S3 Upload',
2064
+ s3Config,
2065
+ },
2066
+ log
2067
+ );
2068
+
2069
+ // Construct S3 key
2070
+ const s3Key = `${s3Prefix}${fileName}`;
2071
+
2072
+ // Upload with retry logic (built into S3DataSource)
2073
+ await s3.uploadFile(s3Key, Buffer.from(csvContent, 'utf-8'), {
2074
+ contentType: 'text/csv',
2075
+ metadata: {
2076
+ recordCount: String(transformedRecords.length),
2077
+ extractedAt: new Date().toISOString(),
2078
+ jobId,
2079
+ mappingErrors: mappingErrors.length > 0 ? String(mappingErrors.length) : undefined,
2080
+ },
2081
+ });
2082
+
2083
+ log.info('S3 upload successful', { fileName, s3Key });
2084
+
2085
+ // ═══════════════════════════════════════════════════════════
2086
+ // STEP 8/8: Update State & Complete Job
2087
+ // ═══════════════════════════════════════════════════════════
2088
+ log.info('💾 [STEP 8/8] Updating state and completing job', { jobId });
2089
+
2090
+ // Calculate new timestamp for next incremental run
2091
+ let newTimestamp: string | undefined;
2092
+
2093
+ if (updateState && !isManualOverride) {
2094
+ // Find max updatedOn from extracted records
2095
+ const maxUpdatedOn = records.reduce(
2096
+ (max, record) => {
2097
+ const recordTime = new Date(record.updatedOn).getTime();
2098
+ return recordTime > max ? recordTime : max;
2099
+ },
2100
+ new Date(dateRangeFilter?.from || DEFAULT_FALLBACK).getTime()
2101
+ );
2102
+
2103
+ newTimestamp = new Date(maxUpdatedOn).toISOString();
2104
+
2105
+ // Store new timestamp (WITHOUT buffer - buffer only applied on read)
2106
+ await kv.set(STATE_KEY, newTimestamp);
2107
+
2108
+ log.info('State updated', {
2109
+ oldTimestamp: dateRangeFilter?.from,
2110
+ newTimestamp,
2111
+ });
2112
+ }
2113
+
2114
+ // Mark job as completed
2115
+ await tracker.markCompleted(jobId, {
2116
+ recordCount: transformedRecords.length,
2117
+ fileName,
2118
+ s3Key,
2119
+ errorCount: mappingErrors.length,
2120
+ errors: mappingErrors,
2121
+ isManualOverride,
2122
+ stateUpdated: updateState,
2123
+ newTimestamp,
2124
+ });
2125
+
2126
+ return {
2127
+ success: true,
2128
+ jobId,
2129
+ recordsExtracted: transformedRecords.length,
2130
+ fileName,
2131
+ s3Path: s3Key,
2132
+ isManualOverride,
2133
+ stateUpdated: updateState,
2134
+ newTimestamp,
2135
+ errors: mappingErrors.length > 0 ? mappingErrors : undefined,
2136
+ };
2137
+ } catch (error: any) {
2138
+ log.error('Extraction workflow failed', {
2139
+ jobId,
2140
+ message: error instanceof Error ? error.message : String(error),
2141
+
2142
+ stack: error instanceof Error ? error.stack : undefined,
2143
+
2144
+ errorType: error instanceof Error ? error.constructor.name : 'Error',
2145
+ });
2146
+
2147
+ // Mark job as failed
2148
+ await tracker.markFailed(jobId, error);
2149
+
2150
+ return {
2151
+ success: false,
2152
+ jobId,
2153
+ recordsExtracted: 0,
2154
+ message: error instanceof Error ? error.message : String(error),
2155
+
2156
+ stack: error instanceof Error ? error.stack : undefined,
2157
+
2158
+ errorType: error instanceof Error ? error.constructor.name : 'Error',
2159
+ };
2160
+ }
2161
+ }
2162
+ ```
2163
+
2164
+ ---
2165
+
2166
+ ### 4. Utility Functions (src/utils/job-id-generator.ts)
2167
+
2168
+ ```typescript
2169
+ /**
2170
+ * Job ID Generator
2171
+ *
2172
+ * Generates unique job IDs for tracking extraction workflows
2173
+ *
2174
+ * FORMAT: {TYPE}_{ENTITY}_{YYYYMMDD}_{HHMMSS}_{RANDOM}
2175
+ * Example: SCHEDULED_IQ_20251027_183045_a1b2c3
2176
+ */
2177
+
2178
+ /**
2179
+ * Generate unique job ID
2180
+ *
2181
+ * @param type - Job trigger type (SCHEDULED, ADHOC, MANUAL)
2182
+ * @param entity - Entity abbreviation (IQ=Inventory Quantities, IP, VP, ORD, PRD)
2183
+ * @returns Unique job ID string
2184
+ */
2185
+ export function generateJobId(type: string, entity: string): string {
2186
+ const now = new Date();
2187
+
2188
+ // Format: YYYYMMDD
2189
+ const dateStr = now.toISOString().slice(0, 10).replace(/-/g, '');
2190
+
2191
+ // Format: HHMMSS
2192
+ const timeStr = now.toISOString().slice(11, 19).replace(/:/g, '');
2193
+
2194
+ // Random suffix (6 chars)
2195
+ const randomStr = Math.random().toString(36).substring(2, 8);
2196
+
2197
+ return `${type}_${entity}_${dateStr}_${timeStr}_${randomStr}`;
2198
+ }
2199
+
2200
+ /**
2201
+ * Parse job ID components
2202
+ */
2203
+ export function parseJobId(jobId: string): {
2204
+ type: string;
2205
+ entity: string;
2206
+ date: string;
2207
+ time: string;
2208
+ random: string;
2209
+ } | null {
2210
+ const parts = jobId.split('_');
2211
+
2212
+ if (parts.length !== 5) {
2213
+ return null;
2214
+ }
2215
+
2216
+ return {
2217
+ type: parts[0],
2218
+ entity: parts[1],
2219
+ date: parts[2],
2220
+ time: parts[3],
2221
+ random: parts[4],
2222
+ };
2223
+ }
2224
+ ```
2225
+
2226
+ ---
2227
+
2228
+ ### 5. Package Configuration
2229
+
2230
+ #### package.json
2231
+
2232
+ ```json
2233
+ {
2234
+ "name": "inventory-quantities-to-s3-csv",
2235
+ "version": "1.0.0",
2236
+ "description": "Extract inventory quantities from Fluent Commerce and export to S3 as CSV",
2237
+ "type": "module",
2238
+ "main": "src/index.ts",
2239
+ "scripts": {
2240
+ "dev": "versori dev",
2241
+ "build": "versori build",
2242
+ "deploy": "versori deploy"
2243
+ },
2244
+ "dependencies": {
2245
+ "@fluentcommerce/fc-connect-sdk": "^0.1.39",
2246
+ "@versori/run": "latest"
2247
+ },
2248
+ "devDependencies": {
2249
+ "@types/node": "^20.0.0",
2250
+ "typescript": "^5.0.0"
2251
+ }
2252
+ }
2253
+ ```
2254
+
2255
+ #### tsconfig.json
2256
+
2257
+ ```json
2258
+ {
2259
+ "compilerOptions": {
2260
+ "module": "ES2022",
2261
+ "target": "ES2024",
2262
+ "moduleResolution": "node"
2263
+ }
2264
+ }
2265
+ ```
2266
+
2267
+ ---
2268
+
2269
+ ## 6. Deployment Instructions
2270
+
2271
+ ### Deploy to Versori
2272
+
2273
+ ```bash
2274
+ # 1. Install dependencies
2275
+ npm install
2276
+
2277
+ # 2. Test locally (if using Versori CLI)
2278
+ npm run dev
2279
+
2280
+ # 3. Deploy to Versori platform
2281
+ npm run deploy
2282
+ ```
2283
+
2284
+ ### Configure Activation Variables
2285
+
2286
+ In Versori platform settings, configure:
2287
+
2288
+ ```json
2289
+ {
2290
+ "catalogueRef": "DEFAULT_CATALOGUE",
2291
+ "s3BucketName": "inventory-audit-exports",
2292
+ "awsAccessKeyId": "AKIAXXXXXXXXXXXX",
2293
+ "awsSecretAccessKey": "********",
2294
+ "awsRegion": "us-east-1",
2295
+ "s3Prefix": "inventory-quantities/daily/",
2296
+ "fileNamePrefix": "inventoryquantities",
2297
+ "pageSize": 200,
2298
+ "maxRecords": 100000,
2299
+ "overlapBufferSeconds": 60,
2300
+ "webhookApiKey": "your-secure-api-key-here"
2301
+ }
2302
+ ```
2303
+
2304
+ ---
2305
+
2306
+ ## 7. Testing
2307
+
2308
+ ### Test Scheduled Extraction
2309
+
2310
+ The scheduled workflow runs automatically based on cron schedule.
2311
+
2312
+ **Check logs:**
2313
+
2314
+ ```
2315
+ [STEP 1/8] Initializing job tracking
2316
+ [STEP 2/8] Initializing Fluent Commerce client
2317
+ [STEP 3/8] Determining date range for extraction
2318
+ [STEP 4/8] Extracting data from Fluent Commerce
2319
+ [STEP 5/8] Transforming data with UniversalMapper
2320
+ [STEP 6/8] Generating CSV file
2321
+ [STEP 7/8] Uploading to S3
2322
+ [STEP 8/8] Updating state and completing job
2323
+ ```
2324
+
2325
+ ### Test Ad hoc Extraction
2326
+
2327
+ ```bash
2328
+ # Incremental (uses last sync timestamp)
2329
+ curl -X POST https://api.versori.com/webhooks/inventory-quantities-adhoc \
2330
+ -H "X-API-Key: your-api-key" \
2331
+ -H "Content-Type: application/json" \
2332
+ -d '{}'
2333
+
2334
+ # Date range override
2335
+ curl -X POST https://api.versori.com/webhooks/inventory-quantities-adhoc \
2336
+ -H "X-API-Key: your-api-key" \
2337
+ -H "Content-Type: application/json" \
2338
+ -d '{
2339
+ "fromDate": "2025-01-01T00:00:00Z",
2340
+ "toDate": "2025-01-31T23:59:59Z",
2341
+ "updateState": false
2342
+ }'
2343
+ ```
2344
+
2345
+ ### Test Job Status Query
2346
+
2347
+ ```bash
2348
+ curl -X POST https://api.versori.com/webhooks/inventory-quantities-job-status \
2349
+ -H "X-API-Key: your-api-key" \
2350
+ -H "Content-Type: application/json" \
2351
+ -d '{
2352
+ "jobId": "ADHOC_IQ_20251027_183045_abc123"
2353
+ }'
2354
+ ```
2355
+
2356
+ **Response:**
2357
+
2358
+ ```json
2359
+ {
2360
+ "success": true,
2361
+ "jobId": "ADHOC_IQ_20251027_183045_abc123",
2362
+ "status": "processing",
2363
+ "stage": "transformation",
2364
+ "message": "Transforming 15000 records",
2365
+ "createdAt": "2025-10-27T18:30:45.000Z",
2366
+ "startedAt": "2025-10-27T18:30:46.000Z"
2367
+ }
2368
+ ```
2369
+
2370
+ ---
2371
+
2372
+ ## 8. Troubleshooting
2373
+
2374
+ **Issue**: "No records extracted"
2375
+
2376
+ - Check dateRange (manual override vs incremental)
2377
+ - Check catalogueRef filter
2378
+ - Verify quantity types filter
2379
+
2380
+ **Issue**: "S3 upload failed"
2381
+
2382
+ - Job fails; state not advanced
2383
+ - Next run retries same window
2384
+ - Check S3 credentials and bucket permissions
2385
+
2386
+ **Issue**: "GraphQL pagination error"
2387
+
2388
+ - Ensure edges.cursor and pageInfo.hasNextPage are in query
2389
+
2390
+ **Issue**: "Memory pressure"
2391
+
2392
+ - Lower pageSize or maxRecords
2393
+ - Consider file splitting for large extractions
2394
+
2395
+ **Issue**: "Transformation errors"
2396
+
2397
+ - Check mapping config field paths
2398
+ - Verify required fields are present in GraphQL response
2399
+ - Review transformation error details in logs
2400
+
2401
+ ---
2402
+
2403
+ ## 9. Replication Checklist
2404
+
2405
+ **To replicate this template for other entities/formats:**
2406
+
2407
+ 1. **File Naming:** Replace `inventory-quantities`, `IQ`, `InventoryQuantity` with your entity name
2408
+ 2. **GraphQL Query:** Update query constant and field selection to match your entity schema
2409
+ 3. **Mapping Config:** Create new mapping file in `config/` with correct field paths
2410
+ 4. **Workflows:** Rename workflow exports to match entity (e.g., `scheduledOrdersExtraction`)
2411
+ 5. **Service Function:** Rename main function (e.g., `executeOrderExtraction`)
2412
+ 6. **State Key:** Update KV key (e.g., `lastOrderSync`)
2413
+ 7. **Output Format:** For XML use `XMLBuilder`, for JSON use `JSON.stringify()`, for CSV use `CSVParserService`
2414
+ 8. **Upload Destination:** For SFTP replace `S3DataSource` with `SftpDataSource` (and add `dispose()` in finally block)
2415
+ 9. **Job ID Entity Code:** Update entity abbreviation in generateJobId() (e.g., 'ORD' for orders)
2416
+ 10. **Result Path:** Update `resultPath` in ExtractionOrchestrator (e.g., `'orders.edges.node'`)
2417
+
2418
+ ---
2419
+
2420
+ **Pattern**: Enterprise incremental extraction with overlap buffer for inventory quantities
2421
+ **Key Learning**: Use ExtractionOrchestrator for auto-pagination, JobTracker for job status, CSVParserService for CSV generation
2422
+ **Critical**: Apply 60-second overlap buffer to prevent missed records due to clock skew
2423
+ **Buffer Pattern**: Query WITH buffer (`updatedOn >= lastRunTime - 60s`), save WITHOUT buffer (`MAX(updatedOn)`)
2424
+ **SDK Services**: ExtractionOrchestrator, UniversalMapper, CSVParserService, S3DataSource, JobTracker
2425
+ **Entity-Specific**: Query uses `inventoryQuantities`, resultPath is `'inventoryQuantities.edges.node'`, state key is `lastInventoryQuantitySync`
2426
+
2427
+ ---
2428
+
2429
+ ### Optional: Backward Pagination (Advanced)
2430
+
2431
+ - Default: forward ($first/$after) + pageInfo.hasNextPage.
2432
+ - Reverse: define $last/$before and include pageInfo.hasPreviousPage; set direction='backward'.
2433
+
2434
+ GraphQL:
2435
+
2436
+ ```graphql
2437
+ query GetInventoryQuantitiesBackward($retailerId: ID!, $last: Int!, $before: String) {
2438
+ inventoryQuantities(retailerId: $retailerId, last: $last, before: $before) {
2439
+ edges {
2440
+ cursor
2441
+ node {
2442
+ id
2443
+ ref
2444
+ updatedOn
2445
+ }
2446
+ }
2447
+ pageInfo {
2448
+ hasPreviousPage
2449
+ }
2450
+ }
2451
+ }
2452
+ ```
2453
+
2454
+ SDK:
2455
+
2456
+ ```typescript
2457
+ await orchestrator.extract({
2458
+ query: INVENTORY_QUANTITIES_BACKWARD_QUERY,
2459
+ resultPath: 'inventoryQuantities.edges.node',
2460
+ variables: { retailerId },
2461
+ pageSize,
2462
+ direction: 'backward',
2463
+ });
2464
+ ```