@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.56
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +12 -0
- package/README.md +11 -0
- package/dist/cjs/clients/fluent-client.js +13 -6
- package/dist/cjs/utils/pagination-helpers.js +38 -2
- package/dist/cjs/versori/fluent-versori-client.js +11 -5
- package/dist/esm/clients/fluent-client.js +13 -6
- package/dist/esm/utils/pagination-helpers.js +38 -2
- package/dist/esm/versori/fluent-versori-client.js +11 -5
- package/dist/tsconfig.esm.tsbuildinfo +1 -1
- package/dist/tsconfig.tsbuildinfo +1 -1
- package/dist/tsconfig.types.tsbuildinfo +1 -1
- package/docs/00-START-HERE/EXPORT-VALIDATION.md +158 -158
- package/docs/00-START-HERE/cli-analyze-source-structure-guide.md +655 -655
- package/docs/00-START-HERE/cli-documentation-index.md +202 -202
- package/docs/00-START-HERE/cli-quick-reference.md +252 -252
- package/docs/00-START-HERE/decision-tree.md +552 -552
- package/docs/00-START-HERE/getting-started.md +1070 -1070
- package/docs/00-START-HERE/mapper-quick-decision-guide.md +235 -235
- package/docs/00-START-HERE/readme.md +237 -237
- package/docs/00-START-HERE/retailerid-configuration.md +404 -404
- package/docs/00-START-HERE/sdk-philosophy.md +794 -794
- package/docs/00-START-HERE/troubleshooting-quick-reference.md +1086 -1086
- package/docs/01-TEMPLATES/faq.md +686 -686
- package/docs/01-TEMPLATES/patterns/pattern-templates-guide.md +68 -68
- package/docs/01-TEMPLATES/patterns/patterns-csv-schema-validation-and-rejection-report.md +233 -233
- package/docs/01-TEMPLATES/patterns/patterns-custom-resolvers.md +407 -407
- package/docs/01-TEMPLATES/patterns/patterns-error-handling-retry.md +511 -511
- package/docs/01-TEMPLATES/patterns/patterns-field-mapping-universal.md +701 -701
- package/docs/01-TEMPLATES/patterns/patterns-large-file-splitting.md +1430 -1430
- package/docs/01-TEMPLATES/patterns/patterns-master-data-etl.md +2399 -2399
- package/docs/01-TEMPLATES/patterns/patterns-pagination-streaming.md +447 -447
- package/docs/01-TEMPLATES/patterns/patterns-state-duplicate-prevention.md +385 -385
- package/docs/01-TEMPLATES/readme.md +957 -957
- package/docs/01-TEMPLATES/standalone/standalone-asn-inbound-processing.md +1209 -1209
- package/docs/01-TEMPLATES/standalone/standalone-graphql-query-export.md +1140 -1140
- package/docs/01-TEMPLATES/standalone/standalone-graphql-to-parquet-partitioned-s3.md +432 -432
- package/docs/01-TEMPLATES/standalone/standalone-multi-channel-inventory-sync.md +1185 -1185
- package/docs/01-TEMPLATES/standalone/standalone-multi-source-aggregation.md +1462 -1462
- package/docs/01-TEMPLATES/standalone/standalone-s3-csv-batch-api.md +1390 -1390
- package/docs/01-TEMPLATES/standalone/standalone-s3-csv-inventory-to-batch.md +330 -330
- package/docs/01-TEMPLATES/standalone/standalone-scripts-guide.md +87 -87
- package/docs/01-TEMPLATES/standalone/standalone-sftp-xml-graphql.md +1444 -1444
- package/docs/01-TEMPLATES/standalone/standalone-webhook-payload-processing.md +688 -688
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-dropship-order-routing.md +193 -193
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-graphql-parquet-extraction.md +518 -518
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-inter-location-transfers.md +2162 -2162
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-pre-order-allocation.md +2226 -2226
- package/docs/01-TEMPLATES/versori/business-examples/business-scenarios-guide.md +87 -87
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-connection-validation-pattern.md +656 -656
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-dual-workflow-connector.md +835 -835
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-guide.md +108 -108
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-kv-state-management.md +1533 -1533
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-xml-response-patterns.md +1160 -1160
- package/docs/01-TEMPLATES/versori/versori-platform-guide.md +201 -201
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-asn-purchase-order.md +1906 -1906
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-dropship-routing.md +1074 -1074
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-flash-sale-reserve.md +1395 -1395
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-generic-xml-order.md +888 -888
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-payment-gateway-integration.md +2478 -2478
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-rma-returns-comprehensive.md +2240 -2240
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-xml-order-ingestion.md +2029 -2029
- package/docs/01-TEMPLATES/versori/webhooks/webhook-templates-guide.md +140 -140
- package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/inventory-mapping.json +20 -20
- package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/products_2025-01-22.csv +11 -11
- package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/sample-data-guide.md +34 -34
- package/docs/01-TEMPLATES/versori/workflows/_examples/workflow-examples-guide.md +36 -36
- package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-modes-guide.md +1038 -1038
- package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-workflows-guide.md +138 -138
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/graphql-extraction-guide.md +63 -63
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-csv.md +2062 -2062
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-xml.md +2294 -2294
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-s3-csv.md +2461 -2461
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-sftp-xml.md +2529 -2529
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-csv.md +2464 -2464
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-json.md +1959 -1959
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-s3-csv.md +1953 -1953
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-sftp-xml.md +2541 -2541
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-s3-json.md +2384 -2384
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-sftp-xml.md +2445 -2445
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-csv.md +2355 -2355
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-json.md +2042 -2042
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-sftp-xml.md +2726 -2726
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/batch-api-guide.md +206 -206
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-cycle-count-reconciliation.md +2030 -2030
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-multi-channel-inventory-sync.md +1882 -1882
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-csv-inventory-batch.md +2827 -2827
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-json-inventory-batch.md +1952 -1952
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-xml-inventory-batch.md +3289 -3289
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-csv-inventory-batch.md +3064 -3064
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-json-inventory-batch.md +3238 -3238
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-xml-inventory-batch.md +2977 -2977
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/event-api-guide.md +321 -321
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-json-order-cancel-event.md +959 -959
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-xml-order-cancel-event.md +1170 -1170
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-csv-product-event.md +2312 -2312
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-json-product-event.md +2999 -2999
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-parquet-product-event.md +2836 -2836
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-xml-product-event.md +2395 -2395
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-csv-product-event.md +2295 -2295
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-json-product-event.md +2602 -2602
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-parquet-product-event.md +2589 -2589
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-xml-product-event.md +3578 -3578
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/graphql-mutations-guide.md +93 -93
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-json-order-update-graphql.md +1260 -1260
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-xml-order-update-graphql.md +1472 -1472
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-control-graphql.md +2417 -2417
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-location-graphql.md +2811 -2811
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-price-graphql.md +2619 -2619
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-json-location-graphql.md +2807 -2807
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-xml-location-graphql.md +2373 -2373
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-control-graphql.md +2740 -2740
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-location-graphql.md +2760 -2760
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-json-location-graphql.md +1710 -1710
- package/docs/01-TEMPLATES/versori/workflows/ingestion/ingestion-workflows-guide.md +136 -136
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/rubix-webhooks-guide.md +520 -520
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-inline.md +1418 -1418
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-universal-mapper.md +1785 -1785
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-order-attribute-update.md +824 -824
- package/docs/01-TEMPLATES/versori/workflows/workflows-overview-guide.md +646 -646
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-batch-archival.md +724 -724
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-job-tracker.md +627 -627
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-partial-batch-recovery.md +561 -561
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-quick-reference.md +367 -367
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-readme.md +407 -407
- package/docs/02-CORE-GUIDES/advanced-services/readme.md +49 -49
- package/docs/02-CORE-GUIDES/api-reference/api-reference-quick-reference.md +548 -548
- package/docs/02-CORE-GUIDES/api-reference/event-api-input-output-reference.md +702 -1171
- package/docs/02-CORE-GUIDES/api-reference/examples/client-initialization.ts +286 -286
- package/docs/02-CORE-GUIDES/api-reference/graphql-error-classification.md +337 -337
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-01-client-api.md +399 -520
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-03-authentication.md +199 -199
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-04-graphql-mapping.md +925 -925
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-05-services.md +1198 -1198
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-06-data-sources.md +1083 -1083
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-07-parsers.md +1097 -1097
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-pagination.md +513 -513
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-types.md +545 -597
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-error-handling.md +527 -527
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-webhook-validation.md +514 -514
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-extraction.md +557 -557
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-utilities.md +412 -412
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-cli-tools.md +423 -423
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-error-handling.md +716 -716
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-analyze-source-structure.md +518 -518
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-partial-responses.md +212 -212
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-testing.md +300 -300
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-13-resolver-builder.md +322 -322
- package/docs/02-CORE-GUIDES/api-reference/readme.md +279 -279
- package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-quick-reference.md +351 -351
- package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-readme.md +277 -277
- package/docs/02-CORE-GUIDES/auto-pagination/examples/auto-pagination-readme.md +178 -178
- package/docs/02-CORE-GUIDES/auto-pagination/examples/common-patterns.ts +351 -351
- package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-products.ts +384 -384
- package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-virtual-positions.ts +308 -308
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-01-foundations.md +470 -470
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-02-quick-start.md +713 -713
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-03-configuration.md +754 -754
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-04-advanced-patterns.md +732 -732
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-05-sdk-integration.md +847 -847
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-06-troubleshooting.md +359 -359
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-07-api-reference.md +462 -462
- package/docs/02-CORE-GUIDES/auto-pagination/readme.md +54 -54
- package/docs/02-CORE-GUIDES/data-sources/data-sources-file-operations-error-handling.md +1487 -1487
- package/docs/02-CORE-GUIDES/data-sources/data-sources-quick-reference.md +836 -836
- package/docs/02-CORE-GUIDES/data-sources/data-sources-readme.md +276 -276
- package/docs/02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md +553 -553
- package/docs/02-CORE-GUIDES/data-sources/examples/common-patterns.ts +409 -409
- package/docs/02-CORE-GUIDES/data-sources/examples/data-sources-readme.md +178 -178
- package/docs/02-CORE-GUIDES/data-sources/examples/s3-operations.ts +308 -308
- package/docs/02-CORE-GUIDES/data-sources/examples/sftp-operations.ts +371 -371
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-01-foundations.md +735 -735
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-02-s3-operations.md +1302 -1302
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-03-sftp-operations.md +1379 -1379
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-04-file-patterns.md +941 -941
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-05-advanced-topics.md +813 -813
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-06-integration-patterns.md +486 -486
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-07-troubleshooting.md +387 -387
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-08-api-reference.md +417 -417
- package/docs/02-CORE-GUIDES/data-sources/readme.md +77 -77
- package/docs/02-CORE-GUIDES/error-handling-guide.md +936 -936
- package/docs/02-CORE-GUIDES/extraction/examples/02-core-guides-extraction-readme.md +116 -116
- package/docs/02-CORE-GUIDES/extraction/examples/common-patterns.ts +428 -428
- package/docs/02-CORE-GUIDES/extraction/examples/extract-inventory-basic.ts +187 -187
- package/docs/02-CORE-GUIDES/extraction/extraction-quick-reference.md +596 -596
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-01-foundations.md +514 -514
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-02-basic-extraction.md +823 -823
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-03-parquet-processing.md +507 -507
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-04-data-enrichment.md +546 -546
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-05-transformation.md +494 -494
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-export-formats.md +458 -458
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-performance.md +138 -138
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-api-reference.md +148 -148
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-optimization.md +692 -692
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-08-extraction-orchestrator.md +1008 -1008
- package/docs/02-CORE-GUIDES/extraction/readme.md +151 -151
- package/docs/02-CORE-GUIDES/ingestion/examples/_simple-kv-store.ts +40 -40
- package/docs/02-CORE-GUIDES/ingestion/examples/error-recovery.ts +728 -728
- package/docs/02-CORE-GUIDES/ingestion/examples/event-driven.ts +501 -501
- package/docs/02-CORE-GUIDES/ingestion/examples/local-file-ingestion.ts +88 -88
- package/docs/02-CORE-GUIDES/ingestion/examples/parquet-ingestion.ts +117 -117
- package/docs/02-CORE-GUIDES/ingestion/examples/performance-optimized.ts +647 -647
- package/docs/02-CORE-GUIDES/ingestion/examples/s3-csv-ingestion.ts +169 -169
- package/docs/02-CORE-GUIDES/ingestion/examples/sftp-csv-ingestion.ts +134 -134
- package/docs/02-CORE-GUIDES/ingestion/ingestion-quick-reference.md +546 -546
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-01-introduction.md +626 -626
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-02-quick-start.md +658 -658
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-03-data-sources.md +1052 -1052
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-04-field-mapping.md +763 -763
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-05-advanced-parsers.md +676 -676
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-06-batch-api.md +1295 -1295
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-api-reference.md +138 -138
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-state-management.md +1037 -1037
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-08-performance-optimization.md +1349 -1349
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-09-best-practices.md +1893 -1893
- package/docs/02-CORE-GUIDES/ingestion/readme.md +160 -160
- package/docs/02-CORE-GUIDES/logging-guide.md +585 -585
- package/docs/02-CORE-GUIDES/mapping/error-handling-patterns.md +401 -401
- package/docs/02-CORE-GUIDES/mapping/examples/02-core-guides-mapping-readme.md +128 -128
- package/docs/02-CORE-GUIDES/mapping/examples/common-patterns.ts +273 -273
- package/docs/02-CORE-GUIDES/mapping/examples/csv-location-ingestion.json +36 -36
- package/docs/02-CORE-GUIDES/mapping/examples/csv-mapping.ts +242 -242
- package/docs/02-CORE-GUIDES/mapping/examples/graphql-to-parquet-extraction.json +36 -36
- package/docs/02-CORE-GUIDES/mapping/examples/json-mapping.ts +213 -213
- package/docs/02-CORE-GUIDES/mapping/examples/json-product-to-mutation.json +48 -48
- package/docs/02-CORE-GUIDES/mapping/examples/xml-mapping.ts +291 -291
- package/docs/02-CORE-GUIDES/mapping/examples/xml-order-to-mutation.json +45 -45
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-quick-reference.md +463 -463
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-readme.md +227 -227
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-01-introduction.md +222 -222
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-02-quick-start.md +351 -351
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-03-schema-validation.md +569 -569
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-04-mapping-patterns.md +471 -471
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-05-configuration-reference.md +611 -611
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-advanced-xpath.md +148 -148
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-path-syntax.md +464 -464
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-api-reference.md +94 -94
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-array-handling.md +307 -307
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-08-custom-resolvers.md +544 -544
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-09-advanced-patterns.md +427 -427
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-10-hooks-and-variables.md +336 -336
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-11-error-handling.md +488 -488
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-12-arguments-vs-nodes.md +383 -383
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-13-best-practices.md +477 -477
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/readme.md +62 -62
- package/docs/02-CORE-GUIDES/mapping/mapping-format-decision-tree.md +480 -480
- package/docs/02-CORE-GUIDES/mapping/mapping-graphql-alias-batching-guide.md +820 -820
- package/docs/02-CORE-GUIDES/mapping/mapping-javascript-objects.md +2369 -2369
- package/docs/02-CORE-GUIDES/mapping/mapping-mapper-comparison-guide.md +682 -682
- package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-07-api-reference.md +1327 -1327
- package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-08-error-handling.md +1142 -1142
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-04-use-cases.md +891 -891
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-helpers-resolvers.md +1126 -1126
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-sdk-resolvers.md +199 -199
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-07-api-reference.md +1319 -1319
- package/docs/02-CORE-GUIDES/mapping/readme.md +178 -178
- package/docs/02-CORE-GUIDES/mapping/resolver-registration.md +410 -410
- package/docs/02-CORE-GUIDES/mapping/resolvers/examples/common-patterns.ts +226 -226
- package/docs/02-CORE-GUIDES/mapping/resolvers/examples/custom-resolvers.ts +227 -227
- package/docs/02-CORE-GUIDES/mapping/resolvers/examples/sdk-resolvers-usage.ts +203 -203
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-readme.md +274 -274
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-api-reference.md +679 -679
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-cookbook.md +826 -826
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-guide.md +1330 -1330
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-helpers-reference.md +1437 -1437
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-parameters-reference.md +553 -553
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-troubleshooting.md +854 -854
- package/docs/02-CORE-GUIDES/mapping/resolvers/readme.md +75 -75
- package/docs/02-CORE-GUIDES/parsers/examples/02-core-guides-parsers-readme.md +161 -161
- package/docs/02-CORE-GUIDES/parsers/examples/csv-parser-examples.ts +110 -110
- package/docs/02-CORE-GUIDES/parsers/examples/json-parser-examples.ts +33 -33
- package/docs/02-CORE-GUIDES/parsers/examples/parquet-parser-examples.ts +47 -47
- package/docs/02-CORE-GUIDES/parsers/examples/xml-parser-examples.ts +38 -38
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-01-foundations.md +355 -355
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-02-csv-parser.md +772 -772
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-03-json-parser.md +789 -789
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-04-xml-parser.md +857 -857
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-05-parquet-parser.md +603 -603
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-integration-patterns.md +702 -702
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-streaming.md +121 -121
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-api-reference.md +89 -89
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-troubleshooting.md +727 -727
- package/docs/02-CORE-GUIDES/parsers/parsers-quick-reference.md +482 -482
- package/docs/02-CORE-GUIDES/parsers/parsers-readme.md +258 -258
- package/docs/02-CORE-GUIDES/parsers/readme.md +65 -65
- package/docs/02-CORE-GUIDES/readme.md +194 -194
- package/docs/02-CORE-GUIDES/webhook-validation/examples/basic-validation.ts +108 -108
- package/docs/02-CORE-GUIDES/webhook-validation/examples/common-patterns.ts +316 -316
- package/docs/02-CORE-GUIDES/webhook-validation/examples/webhook-validation-readme.md +61 -61
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-01-foundations.md +440 -440
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-02-quick-start.md +525 -525
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-03-versori-integration.md +741 -741
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-04-platform-integration.md +629 -629
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-05-configuration.md +535 -535
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-error-handling.md +611 -611
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-troubleshooting.md +124 -124
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-07-api-reference.md +511 -511
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-08-rubix-webhooks.md +590 -590
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-09-rubix-event-vs-http-call.md +432 -432
- package/docs/02-CORE-GUIDES/webhook-validation/readme.md +239 -239
- package/docs/02-CORE-GUIDES/webhook-validation/webhook-validation-quick-reference.md +392 -392
- package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-quick-reference.md +498 -498
- package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-readme.md +313 -313
- package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/common-patterns.ts +612 -612
- package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/connector-scenarios-readme.md +253 -253
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-01-foundations.md +452 -452
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-02-simple-scenarios.md +681 -681
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-03-intermediate-scenarios.md +637 -637
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-04-advanced-scenarios.md +650 -650
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-05-bidirectional-sync.md +233 -233
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-06-production-patterns.md +442 -442
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-07-reference.md +445 -445
- package/docs/03-PATTERN-GUIDES/connector-scenarios/readme.md +31 -31
- package/docs/03-PATTERN-GUIDES/enterprise-integration-patterns.md +1528 -1528
- package/docs/03-PATTERN-GUIDES/error-handling/comprehensive-error-handling-guide.md +1437 -1437
- package/docs/03-PATTERN-GUIDES/error-handling/error-handling-quick-reference.md +390 -390
- package/docs/03-PATTERN-GUIDES/error-handling/examples/common-patterns.ts +438 -438
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-01-foundations.md +362 -362
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-02-error-types.md +850 -850
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-03-utf8-handling.md +456 -456
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-04-error-scenarios.md +658 -658
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-05-calling-patterns.md +671 -671
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-06-retry-strategies.md +1034 -1034
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-07-monitoring.md +653 -653
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-08-api-reference.md +847 -847
- package/docs/03-PATTERN-GUIDES/error-handling/readme.md +36 -36
- package/docs/03-PATTERN-GUIDES/examples/__tests__/readme.md +40 -40
- package/docs/03-PATTERN-GUIDES/examples/__tests__/resolver-examples.test.js +282 -282
- package/docs/03-PATTERN-GUIDES/examples/test-data/03-pattern-guides-readme.md +110 -110
- package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-inventory.json +123 -123
- package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-order.json +171 -171
- package/docs/03-PATTERN-GUIDES/examples/test-data/readme.md +28 -28
- package/docs/03-PATTERN-GUIDES/extraction/extraction-readme.md +15 -15
- package/docs/03-PATTERN-GUIDES/extraction/readme.md +25 -25
- package/docs/03-PATTERN-GUIDES/file-operations/examples/common-patterns.ts +407 -407
- package/docs/03-PATTERN-GUIDES/file-operations/examples/file-operations-readme.md +142 -142
- package/docs/03-PATTERN-GUIDES/file-operations/file-operations-quick-reference.md +462 -462
- package/docs/03-PATTERN-GUIDES/file-operations/file-operations-readme.md +379 -379
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-01-foundations.md +430 -430
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-02-quick-start.md +484 -484
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-03-s3-operations.md +507 -507
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-04-sftp-operations.md +963 -963
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-05-streaming-performance.md +503 -503
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-archive-patterns.md +386 -386
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-error-handling.md +117 -117
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-api-reference.md +78 -78
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-testing-troubleshooting.md +567 -567
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-08-api-reference.md +1055 -1055
- package/docs/03-PATTERN-GUIDES/file-operations/readme.md +32 -32
- package/docs/03-PATTERN-GUIDES/ingestion/ingestion-readme.md +15 -15
- package/docs/03-PATTERN-GUIDES/ingestion/readme.md +25 -25
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/batch-processing.ts +130 -130
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/common-patterns.ts +360 -360
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/delta-sync.ts +130 -130
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/integration-patterns-readme.md +100 -100
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/real-time-webhook.ts +398 -398
- package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-quick-reference.md +962 -962
- package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-readme.md +134 -134
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-01-real-time-processing.md +991 -991
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-02-batch-processing.md +1547 -1547
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-03-delta-sync.md +1108 -1108
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-04-webhook-patterns.md +1181 -1181
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-05-error-handling.md +1061 -1061
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-advanced-integration-services.md +1547 -1547
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-performance.md +109 -109
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-07-api-reference.md +34 -34
- package/docs/03-PATTERN-GUIDES/integration-patterns/readme.md +30 -30
- package/docs/03-PATTERN-GUIDES/logging-minimal-mode.md +128 -128
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/common-patterns.ts +380 -380
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/multiple-connections-readme.md +139 -139
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/parallel-root-connections.ts +149 -149
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/real-world-scenarios.ts +405 -405
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-01-foundations.md +378 -378
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-02-quick-start.md +566 -566
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-03-targeting-connections.md +659 -659
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-04-parallel-queries.md +656 -656
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-05-best-practices.md +624 -624
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-api-reference.md +824 -824
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-versori.md +119 -119
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-07-api-reference.md +87 -87
- package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-quick-reference.md +353 -353
- package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-readme.md +270 -270
- package/docs/03-PATTERN-GUIDES/multiple-connections/readme.md +30 -30
- package/docs/03-PATTERN-GUIDES/pagination/pagination-readme.md +14 -14
- package/docs/03-PATTERN-GUIDES/pagination/readme.md +24 -24
- package/docs/03-PATTERN-GUIDES/parquet/examples/common-patterns.ts +180 -180
- package/docs/03-PATTERN-GUIDES/parquet/examples/read-parquet.ts +48 -48
- package/docs/03-PATTERN-GUIDES/parquet/examples/write-parquet.ts +65 -65
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-01-introduction.md +393 -393
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-02-quick-start.md +572 -572
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-03-reading-parquet.md +525 -525
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-04-writing-parquet.md +554 -554
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-05-graphql-extraction.md +405 -405
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-performance.md +104 -104
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-s3-integration.md +511 -511
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-api-reference.md +90 -90
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-performance-optimization.md +525 -525
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-08-best-practices.md +712 -712
- package/docs/03-PATTERN-GUIDES/parquet/parquet-quick-reference.md +683 -683
- package/docs/03-PATTERN-GUIDES/parquet/parquet-readme.md +248 -248
- package/docs/03-PATTERN-GUIDES/parquet/readme.md +32 -32
- package/docs/03-PATTERN-GUIDES/parsers/parsers-readme.md +12 -12
- package/docs/03-PATTERN-GUIDES/parsers/readme.md +24 -24
- package/docs/03-PATTERN-GUIDES/readme.md +159 -159
- package/docs/03-PATTERN-GUIDES/webhooks/readme.md +24 -24
- package/docs/03-PATTERN-GUIDES/webhooks/webhooks-readme.md +8 -8
- package/docs/04-REFERENCE/architecture/architecture-01-overview.md +427 -427
- package/docs/04-REFERENCE/architecture/architecture-02-client-architecture.md +424 -424
- package/docs/04-REFERENCE/architecture/architecture-03-data-flow.md +690 -690
- package/docs/04-REFERENCE/architecture/architecture-04-service-layer.md +834 -834
- package/docs/04-REFERENCE/architecture/architecture-05-integration-architecture.md +655 -655
- package/docs/04-REFERENCE/architecture/architecture-06-state-management.md +653 -653
- package/docs/04-REFERENCE/architecture/architecture-adding-new-data-sources.md +686 -686
- package/docs/04-REFERENCE/architecture/readme.md +279 -279
- package/docs/04-REFERENCE/platforms/deno/readme.md +117 -117
- package/docs/04-REFERENCE/platforms/nodejs/readme.md +146 -146
- package/docs/04-REFERENCE/platforms/readme.md +135 -135
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-01-introduction.md +398 -398
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-02-quick-start.md +560 -560
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-03-authentication.md +757 -757
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-04-workflows.md +2476 -2476
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-05-connections.md +1167 -1167
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-kv-storage.md +990 -990
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-state-management.md +121 -121
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-api-reference.md +68 -68
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-deployment.md +731 -731
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-08-best-practices.md +1111 -1111
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-09-signature-reference.md +766 -766
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-readme.md +299 -299
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-s3-sftp-configuration-guide.md +1425 -1425
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-api-key-security.md +816 -816
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-connection-security.md +681 -681
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-workflow-task-types.md +708 -708
- package/docs/04-REFERENCE/platforms/versori/readme.md +108 -108
- package/docs/04-REFERENCE/readme.md +148 -148
- package/docs/04-REFERENCE/resolver-signature/examples/advanced-resolvers.ts +482 -482
- package/docs/04-REFERENCE/resolver-signature/examples/async-resolvers.ts +496 -496
- package/docs/04-REFERENCE/resolver-signature/examples/basic-resolvers.ts +343 -343
- package/docs/04-REFERENCE/resolver-signature/examples/resolver-signature-readme.md +188 -188
- package/docs/04-REFERENCE/resolver-signature/examples/testing-resolvers.ts +463 -463
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-01-foundations.md +286 -286
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-02-parameter-reference.md +643 -643
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-03-basic-examples.md +521 -521
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-04-advanced-patterns.md +739 -739
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-05-sdk-resolvers.md +531 -531
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-migration-guide.md +650 -650
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-testing.md +125 -125
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-07-api-reference.md +794 -794
- package/docs/04-REFERENCE/resolver-signature/readme.md +64 -64
- package/docs/04-REFERENCE/resolver-signature/resolver-signature-quick-reference.md +270 -270
- package/docs/04-REFERENCE/resolver-signature/resolver-signature-readme.md +351 -351
- package/docs/04-REFERENCE/schema/fluent-commerce-schema.json +764 -764
- package/docs/04-REFERENCE/schema/readme.md +141 -141
- package/docs/04-REFERENCE/testing/examples/04-reference-testing-readme.md +158 -158
- package/docs/04-REFERENCE/testing/examples/fluent-testing.ts +62 -62
- package/docs/04-REFERENCE/testing/examples/health-check.ts +155 -155
- package/docs/04-REFERENCE/testing/examples/integration-test.ts +119 -119
- package/docs/04-REFERENCE/testing/examples/performance-test.ts +183 -183
- package/docs/04-REFERENCE/testing/examples/s3-testing.ts +127 -127
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-01-foundations.md +267 -267
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-02-s3-testing.md +599 -599
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-03-fluent-testing.md +589 -589
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-04-integration-testing.md +699 -699
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-05-debugging.md +478 -478
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-cicd-integration.md +463 -463
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-preflight-validation.md +131 -131
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-best-practices.md +499 -499
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-coverage-ci.md +165 -165
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-08-api-reference.md +634 -634
- package/docs/04-REFERENCE/testing/readme.md +86 -86
- package/docs/04-REFERENCE/testing/testing-quick-reference.md +667 -667
- package/docs/04-REFERENCE/testing/testing-readme.md +286 -286
- package/docs/04-REFERENCE/troubleshooting/readme.md +144 -144
- package/docs/04-REFERENCE/troubleshooting/troubleshooting-deno-sftp-compatibility.md +392 -392
- package/docs/template-loading-matrix.md +242 -242
- package/package.json +5 -3
- package/docs/02-CORE-GUIDES/api-reference/cli-profile-integration.md +0 -377
|
@@ -1,1390 +1,1390 @@
|
|
|
1
|
-
# Standalone: S3 CSV → Fluent Batch API
|
|
2
|
-
|
|
3
|
-
**FC Connect SDK Use Case Guide**
|
|
4
|
-
|
|
5
|
-
> **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
|
|
6
|
-
> **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
|
|
7
|
-
|
|
8
|
-
**Context**: Node.js script that reads CSV inventory files from S3 and updates Fluent Commerce inventory via Batch API
|
|
9
|
-
|
|
10
|
-
**Complexity**: Medium
|
|
11
|
-
|
|
12
|
-
**Runtime**: Node.js ≥18 / Deno
|
|
13
|
-
|
|
14
|
-
**Estimated Lines**: ~500 lines
|
|
15
|
-
|
|
16
|
-
## What You'll Build
|
|
17
|
-
|
|
18
|
-
- Standalone Node.js/Deno script (no Versori)
|
|
19
|
-
- OAuth2 authentication with Fluent Commerce
|
|
20
|
-
- S3 file listing and download with AWS SDK
|
|
21
|
-
- CSV parsing with validation
|
|
22
|
-
- UniversalMapper for field transformations
|
|
23
|
-
- Batch API processing with job management
|
|
24
|
-
- Error handling and logging
|
|
25
|
-
- Optional: Schedule with cron or AWS EventBridge
|
|
26
|
-
|
|
27
|
-
## SDK Methods Used
|
|
28
|
-
|
|
29
|
-
- `createClient({ config: { baseUrl, clientId, clientSecret, retailerId } })` - OAuth2 client
|
|
30
|
-
- `S3DataSource(config, logger)` - S3 operations
|
|
31
|
-
- `CSVParserService` - CSV parsing
|
|
32
|
-
- `UniversalMapper(mappingConfig)` - Field mapping
|
|
33
|
-
- `client.createJob({ name, retailerId })` - Create Batch job
|
|
34
|
-
- `client.sendBatch(jobId, { entities })` - Send inventory batch
|
|
35
|
-
- `client.getJobStatus(jobId)` - Check status
|
|
36
|
-
|
|
37
|
-
---
|
|
38
|
-
|
|
39
|
-
## Complete Working Implementation
|
|
40
|
-
|
|
41
|
-
### 1. Project Setup
|
|
42
|
-
|
|
43
|
-
```bash
|
|
44
|
-
# Create project directory
|
|
45
|
-
mkdir inventory-sync-standalone
|
|
46
|
-
cd inventory-sync-standalone
|
|
47
|
-
|
|
48
|
-
# Initialize Node.js project
|
|
49
|
-
npm init -y
|
|
50
|
-
|
|
51
|
-
# Install dependencies
|
|
52
|
-
npm install @fluentcommerce/fc-connect-sdk dotenv
|
|
53
|
-
```
|
|
54
|
-
|
|
55
|
-
### Method selection (SDK)
|
|
56
|
-
|
|
57
|
-
- Queries → `client.graphql`
|
|
58
|
-
- Mutations → `client.graphql`
|
|
59
|
-
- Advanced needs (pagination/telemetry) → `client.graphql` with pagination options
|
|
60
|
-
|
|
61
|
-
```typescript
|
|
62
|
-
// Query (use graphql for both queries and mutations)
|
|
63
|
-
const products = await client.graphql({ query, variables });
|
|
64
|
-
|
|
65
|
-
// Mutation (use graphql for both queries and mutations)
|
|
66
|
-
const result = await client.graphql({ query: mutation, variables });
|
|
67
|
-
|
|
68
|
-
// Advanced pagination
|
|
69
|
-
const res = await client.graphql({ query, variables, pagination: { maxPages: 50 } });
|
|
70
|
-
```
|
|
71
|
-
|
|
72
|
-
### 2. Environment Configuration
|
|
73
|
-
|
|
74
|
-
Create `.env` file:
|
|
75
|
-
|
|
76
|
-
```env
|
|
77
|
-
# Fluent Commerce Configuration
|
|
78
|
-
FLUENT_BASE_URL=https://api.fluentcommerce.com
|
|
79
|
-
FLUENT_CLIENT_ID=your-oauth2-client-id
|
|
80
|
-
FLUENT_CLIENT_SECRET=your-oauth2-client-secret
|
|
81
|
-
FLUENT_RETAILER_ID=your-retailer-id
|
|
82
|
-
|
|
83
|
-
# AWS S3 Configuration
|
|
84
|
-
AWS_ACCESS_KEY_ID=your-aws-access-key
|
|
85
|
-
AWS_SECRET_ACCESS_KEY=your-aws-secret-key
|
|
86
|
-
AWS_REGION=us-east-1
|
|
87
|
-
AWS_BUCKET_NAME=your-inventory-bucket
|
|
88
|
-
AWS_FILE_PREFIX=inventory/updates/
|
|
89
|
-
|
|
90
|
-
# Processing Configuration
|
|
91
|
-
BATCH_SIZE=100
|
|
92
|
-
JOB_NAME_PREFIX=S3-CSV-Inventory-Update
|
|
93
|
-
ARCHIVE_PREFIX=inventory/archive/
|
|
94
|
-
ERROR_PREFIX=inventory/errors/
|
|
95
|
-
|
|
96
|
-
# Optional: Scheduling
|
|
97
|
-
ENABLE_SCHEDULING=false
|
|
98
|
-
CRON_SCHEDULE=0 */4 * * *
|
|
99
|
-
```
|
|
100
|
-
|
|
101
|
-
### 3. Main Script Implementation
|
|
102
|
-
|
|
103
|
-
Create `src/inventory-sync.ts`:
|
|
104
|
-
|
|
105
|
-
```typescript
|
|
106
|
-
import 'dotenv/config';
|
|
107
|
-
// FC Connect SDK+
|
|
108
|
-
// Install: npm install @fluentcommerce/fc-connect-sdk@latest
|
|
109
|
-
// Docs: https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk
|
|
110
|
-
// GitHub: https://github.com/fluentcommerce/fc-connect-sdk
|
|
111
|
-
import { createClient } from '@fluentcommerce/fc-connect-sdk';
|
|
112
|
-
import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
|
|
113
|
-
import { CSVParserService } from '@fluentcommerce/fc-connect-sdk';
|
|
114
|
-
import { UniversalMapper } from '@fluentcommerce/fc-connect-sdk';
|
|
115
|
-
|
|
116
|
-
/**
|
|
117
|
-
* ARCHITECTURE:
|
|
118
|
-
*
|
|
119
|
-
* This standalone script implements a complete inventory sync workflow:
|
|
120
|
-
* 1. List CSV files from S3 bucket
|
|
121
|
-
* 2. Download and parse CSV files
|
|
122
|
-
* 3. Map CSV fields to Fluent inventory format
|
|
123
|
-
* 4. Create Batch API job
|
|
124
|
-
* 5. Send inventory updates in batches
|
|
125
|
-
* 6. Poll job status until complete
|
|
126
|
-
* 7. Archive successful files, move errors
|
|
127
|
-
*
|
|
128
|
-
* ERROR HANDLING STRATEGY:
|
|
129
|
-
* - File-level errors: Move to error folder, continue with other files
|
|
130
|
-
* - Batch errors: Log and continue (Fluent API provides detailed error reports)
|
|
131
|
-
* - Fatal errors: Exit with error code for monitoring/alerting
|
|
132
|
-
*
|
|
133
|
-
* STATE MANAGEMENT:
|
|
134
|
-
* - No state tracking in this basic version (processes all files)
|
|
135
|
-
* - Advanced: Use S3 metadata or external DB to track processed files
|
|
136
|
-
*/
|
|
137
|
-
|
|
138
|
-
// ============================================================================
|
|
139
|
-
// CONFIGURATION & SETUP
|
|
140
|
-
// ============================================================================
|
|
141
|
-
|
|
142
|
-
interface SyncConfig {
|
|
143
|
-
fluent: {
|
|
144
|
-
baseUrl: string;
|
|
145
|
-
clientId: string;
|
|
146
|
-
clientSecret: string;
|
|
147
|
-
retailerId: string;
|
|
148
|
-
};
|
|
149
|
-
s3: {
|
|
150
|
-
accessKeyId: string;
|
|
151
|
-
secretAccessKey: string;
|
|
152
|
-
region: string;
|
|
153
|
-
bucket: string;
|
|
154
|
-
prefix: string;
|
|
155
|
-
};
|
|
156
|
-
processing: {
|
|
157
|
-
batchSize: number;
|
|
158
|
-
jobNamePrefix: string;
|
|
159
|
-
archivePrefix: string;
|
|
160
|
-
errorPrefix: string;
|
|
161
|
-
};
|
|
162
|
-
}
|
|
163
|
-
|
|
164
|
-
/**
|
|
165
|
-
* Load configuration from environment variables
|
|
166
|
-
* Validates all required config before proceeding
|
|
167
|
-
*/
|
|
168
|
-
function loadConfig(): SyncConfig {
|
|
169
|
-
const requiredEnvVars = [
|
|
170
|
-
'FLUENT_BASE_URL',
|
|
171
|
-
'FLUENT_CLIENT_ID',
|
|
172
|
-
'FLUENT_CLIENT_SECRET',
|
|
173
|
-
'FLUENT_RETAILER_ID',
|
|
174
|
-
'AWS_ACCESS_KEY_ID',
|
|
175
|
-
'AWS_SECRET_ACCESS_KEY',
|
|
176
|
-
'AWS_REGION',
|
|
177
|
-
'AWS_BUCKET_NAME',
|
|
178
|
-
];
|
|
179
|
-
|
|
180
|
-
// Validate all required environment variables are present
|
|
181
|
-
const missing = requiredEnvVars.filter(v => !process.env[v]);
|
|
182
|
-
if (missing.length > 0) {
|
|
183
|
-
throw new Error(`Missing required environment variables: ${missing.join(', ')}`);
|
|
184
|
-
}
|
|
185
|
-
|
|
186
|
-
return {
|
|
187
|
-
fluent: {
|
|
188
|
-
baseUrl: process.env.FLUENT_BASE_URL!,
|
|
189
|
-
clientId: process.env.FLUENT_CLIENT_ID!,
|
|
190
|
-
clientSecret: process.env.FLUENT_CLIENT_SECRET!,
|
|
191
|
-
retailerId: process.env.FLUENT_RETAILER_ID!,
|
|
192
|
-
},
|
|
193
|
-
s3: {
|
|
194
|
-
accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
|
|
195
|
-
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
|
|
196
|
-
region: process.env.AWS_REGION!,
|
|
197
|
-
bucket: process.env.AWS_BUCKET_NAME!,
|
|
198
|
-
prefix: process.env.AWS_FILE_PREFIX || 'inventory/updates/',
|
|
199
|
-
},
|
|
200
|
-
processing: {
|
|
201
|
-
batchSize: parseInt(process.env.BATCH_SIZE || '100'),
|
|
202
|
-
jobNamePrefix: process.env.JOB_NAME_PREFIX || 'S3-CSV-Inventory-Update',
|
|
203
|
-
archivePrefix: process.env.ARCHIVE_PREFIX || 'inventory/archive/',
|
|
204
|
-
errorPrefix: process.env.ERROR_PREFIX || 'inventory/errors/',
|
|
205
|
-
},
|
|
206
|
-
};
|
|
207
|
-
}
|
|
208
|
-
|
|
209
|
-
// ============================================================================
|
|
210
|
-
// FIELD MAPPING CONFIGURATION
|
|
211
|
-
// ============================================================================
|
|
212
|
-
|
|
213
|
-
/**
|
|
214
|
-
* UniversalMapper configuration for CSV → Fluent Batch API transformation
|
|
215
|
-
*
|
|
216
|
-
* FIELD MAPPING RULES:
|
|
217
|
-
* - ref: Product SKU (required) - uppercase transformation
|
|
218
|
-
* - locationRef: Location code (required) - prefix with "LOC:"
|
|
219
|
-
* - qty: Quantity (required) - parse as integer, default to 0
|
|
220
|
-
* - type: Inventory type (required) - defaults to "ADJUSTMENT"
|
|
221
|
-
* - status: Inventory status (optional) - defaults to "ACTIVE"
|
|
222
|
-
*
|
|
223
|
-
* SDK RESOLVERS USED:
|
|
224
|
-
* - sdk.uppercase: Convert SKU to uppercase for consistency
|
|
225
|
-
* - sdk.parseInt: Parse quantity string to integer
|
|
226
|
-
* - sdk.trim: Remove whitespace from location codes
|
|
227
|
-
* - sdk.defaultTo: Provide fallback values
|
|
228
|
-
*/
|
|
229
|
-
const inventoryMappingConfig = {
|
|
230
|
-
version: '1.0.0',
|
|
231
|
-
description: 'S3 CSV to Fluent Commerce Inventory Mapping',
|
|
232
|
-
fields: {
|
|
233
|
-
// Product SKU (required field)
|
|
234
|
-
ref: {
|
|
235
|
-
source: 'sku',
|
|
236
|
-
resolver: 'sdk.uppercase', // Normalize to uppercase
|
|
237
|
-
required: true,
|
|
238
|
-
},
|
|
239
|
-
// Location reference (required field)
|
|
240
|
-
locationRef: {
|
|
241
|
-
source: 'location',
|
|
242
|
-
resolver: 'sdk.trim', // Remove whitespace
|
|
243
|
-
required: true,
|
|
244
|
-
},
|
|
245
|
-
// Quantity (required field)
|
|
246
|
-
qty: {
|
|
247
|
-
source: 'quantity',
|
|
248
|
-
resolver: 'sdk.parseInt', // Parse as integer
|
|
249
|
-
required: true,
|
|
250
|
-
},
|
|
251
|
-
// Inventory type (defaults to ADJUSTMENT)
|
|
252
|
-
type: {
|
|
253
|
-
value: 'ADJUSTMENT', // Static value
|
|
254
|
-
},
|
|
255
|
-
// Inventory status (optional, with default)
|
|
256
|
-
status: {
|
|
257
|
-
source: 'status',
|
|
258
|
-
resolver: 'sdk.uppercase',
|
|
259
|
-
defaultValue: 'ACTIVE', // Fallback if not provided
|
|
260
|
-
},
|
|
261
|
-
// Expected date (optional)
|
|
262
|
-
expectedOn: {
|
|
263
|
-
source: 'expected_date',
|
|
264
|
-
resolver: 'sdk.formatDate', // Convert to ISO date format
|
|
265
|
-
},
|
|
266
|
-
},
|
|
267
|
-
};
|
|
268
|
-
|
|
269
|
-
// ============================================================================
|
|
270
|
-
// LOGGER IMPLEMENTATION
|
|
271
|
-
// ============================================================================
|
|
272
|
-
|
|
273
|
-
/**
|
|
274
|
-
* Simple console logger with timestamp and log levels
|
|
275
|
-
* Production: Replace with Winston, Bunyan, or Pino
|
|
276
|
-
*/
|
|
277
|
-
const logger = {
|
|
278
|
-
debug: (message: string, meta?: any) => {
|
|
279
|
-
if (process.env.LOG_LEVEL === 'debug') {
|
|
280
|
-
console.log(`[${new Date().toISOString()}] DEBUG: ${message}`, meta || '');
|
|
281
|
-
}
|
|
282
|
-
},
|
|
283
|
-
info: (message: string, meta?: any) => {
|
|
284
|
-
console.log(`[${new Date().toISOString()}] INFO: ${message}`, meta || '');
|
|
285
|
-
},
|
|
286
|
-
warn: (message: string, meta?: any) => {
|
|
287
|
-
console.warn(`[${new Date().toISOString()}] WARN: ${message}`, meta || '');
|
|
288
|
-
},
|
|
289
|
-
error: (message: string, error?: Error, meta?: any) => {
|
|
290
|
-
console.error(
|
|
291
|
-
`[${new Date().toISOString()}] ERROR: ${message}`,
|
|
292
|
-
error?.message || '',
|
|
293
|
-
meta || ''
|
|
294
|
-
);
|
|
295
|
-
},
|
|
296
|
-
};
|
|
297
|
-
|
|
298
|
-
// ============================================================================
|
|
299
|
-
// MAIN SYNC ORCHESTRATION
|
|
300
|
-
// ============================================================================
|
|
301
|
-
|
|
302
|
-
/**
|
|
303
|
-
* Main inventory sync function
|
|
304
|
-
*
|
|
305
|
-
* ALGORITHM:
|
|
306
|
-
* 1. Initialize SDK clients (Fluent, S3, CSV parser, mapper)
|
|
307
|
-
* 2. List CSV files from S3 prefix
|
|
308
|
-
* 3. For each file:
|
|
309
|
-
* a. Download file content
|
|
310
|
-
* b. Parse CSV to records
|
|
311
|
-
* c. Validate records (skip invalid)
|
|
312
|
-
* d. Map records using UniversalMapper
|
|
313
|
-
* e. Create Batch API job
|
|
314
|
-
* f. Send batches (chunked by batch size)
|
|
315
|
-
* g. Poll job status until complete
|
|
316
|
-
* h. Archive successful files OR move to error folder
|
|
317
|
-
* 4. Return summary statistics
|
|
318
|
-
*/
|
|
319
|
-
async function syncInventory(): Promise<{
|
|
320
|
-
filesProcessed: number;
|
|
321
|
-
filesSucceeded: number;
|
|
322
|
-
filesFailed: number;
|
|
323
|
-
totalRecordsProcessed: number;
|
|
324
|
-
totalRecordsFailed: number;
|
|
325
|
-
}> {
|
|
326
|
-
const startTime = Date.now();
|
|
327
|
-
logger.info('='.repeat(80));
|
|
328
|
-
logger.info('Starting S3 CSV → Fluent Batch API inventory sync');
|
|
329
|
-
logger.info('='.repeat(80));
|
|
330
|
-
|
|
331
|
-
// Load configuration
|
|
332
|
-
const config = loadConfig();
|
|
333
|
-
logger.info('Configuration loaded', {
|
|
334
|
-
s3Bucket: config.s3.bucket,
|
|
335
|
-
s3Prefix: config.s3.prefix,
|
|
336
|
-
batchSize: config.processing.batchSize,
|
|
337
|
-
});
|
|
338
|
-
|
|
339
|
-
// ───────────────────────────────────────────────────────────────────────────
|
|
340
|
-
// STEP 1: Initialize SDK Clients
|
|
341
|
-
// ───────────────────────────────────────────────────────────────────────────
|
|
342
|
-
|
|
343
|
-
// Initialize Fluent Client with OAuth2
|
|
344
|
-
logger.info('Initializing Fluent Commerce client...');
|
|
345
|
-
const fluentClient = await createClient({
|
|
346
|
-
config: {
|
|
347
|
-
baseUrl: config.fluent.baseUrl,
|
|
348
|
-
clientId: config.fluent.clientId,
|
|
349
|
-
clientSecret: config.fluent.clientSecret,
|
|
350
|
-
retailerId: config.fluent.retailerId,
|
|
351
|
-
},
|
|
352
|
-
});
|
|
353
|
-
|
|
354
|
-
// Initialize S3 Data Source
|
|
355
|
-
logger.info('Initializing S3 data source...');
|
|
356
|
-
const s3Source = new S3DataSource(
|
|
357
|
-
{
|
|
358
|
-
type: 'S3_CSV',
|
|
359
|
-
connectionId: 'inventory-s3',
|
|
360
|
-
name: 'Inventory S3 Source',
|
|
361
|
-
s3Config: {
|
|
362
|
-
bucket: config.s3.bucket,
|
|
363
|
-
region: config.s3.region,
|
|
364
|
-
accessKeyId: config.s3.accessKeyId,
|
|
365
|
-
secretAccessKey: config.s3.secretAccessKey,
|
|
366
|
-
},
|
|
367
|
-
},
|
|
368
|
-
logger
|
|
369
|
-
);
|
|
370
|
-
|
|
371
|
-
// Initialize CSV Parser
|
|
372
|
-
const csvParser = new CSVParserService();
|
|
373
|
-
|
|
374
|
-
// Initialize Universal Mapper
|
|
375
|
-
const mapper = new UniversalMapper(inventoryMappingConfig, {
|
|
376
|
-
logger,
|
|
377
|
-
fluentClient,
|
|
378
|
-
});
|
|
379
|
-
|
|
380
|
-
// Validate S3 connection
|
|
381
|
-
logger.info('Validating S3 connection...');
|
|
382
|
-
const s3Connected = await s3Source.validateConnection();
|
|
383
|
-
if (!s3Connected) {
|
|
384
|
-
throw new Error('Failed to connect to S3 bucket');
|
|
385
|
-
}
|
|
386
|
-
logger.info('S3 connection validated successfully');
|
|
387
|
-
|
|
388
|
-
// ───────────────────────────────────────────────────────────────────────────
|
|
389
|
-
// STEP 2: List CSV Files from S3
|
|
390
|
-
// ───────────────────────────────────────────────────────────────────────────
|
|
391
|
-
|
|
392
|
-
logger.info(`Listing CSV files from S3 prefix: ${config.s3.prefix}`);
|
|
393
|
-
const files = await s3Source.listFiles({
|
|
394
|
-
prefix: config.s3.prefix,
|
|
395
|
-
maxKeys: 1000,
|
|
396
|
-
});
|
|
397
|
-
|
|
398
|
-
// Filter for CSV files only
|
|
399
|
-
// NOTE: S3 doesn't support glob patterns, so manual filtering is required
|
|
400
|
-
// IMPORTANT: Use .endsWith('.csv') NOT .endsWith('*.csv')
|
|
401
|
-
// ✓ Correct: f.name.endsWith('.csv')
|
|
402
|
-
// ✗ Wrong: f.name.endsWith('*.csv') // Would never match!
|
|
403
|
-
const csvFiles = files.filter(f => f.name.toLowerCase().endsWith('.csv'));
|
|
404
|
-
|
|
405
|
-
logger.info(`Found ${csvFiles.length} CSV files to process`, {
|
|
406
|
-
totalFiles: files.length,
|
|
407
|
-
csvFiles: csvFiles.length,
|
|
408
|
-
});
|
|
409
|
-
|
|
410
|
-
if (csvFiles.length === 0) {
|
|
411
|
-
logger.info('No CSV files found. Exiting.');
|
|
412
|
-
return {
|
|
413
|
-
filesProcessed: 0,
|
|
414
|
-
filesSucceeded: 0,
|
|
415
|
-
filesFailed: 0,
|
|
416
|
-
totalRecordsProcessed: 0,
|
|
417
|
-
totalRecordsFailed: 0,
|
|
418
|
-
};
|
|
419
|
-
}
|
|
420
|
-
|
|
421
|
-
// ───────────────────────────────────────────────────────────────────────────
|
|
422
|
-
// STEP 3: Process Each CSV File
|
|
423
|
-
// ───────────────────────────────────────────────────────────────────────────
|
|
424
|
-
|
|
425
|
-
let filesSucceeded = 0;
|
|
426
|
-
let filesFailed = 0;
|
|
427
|
-
let totalRecordsProcessed = 0;
|
|
428
|
-
let totalRecordsFailed = 0;
|
|
429
|
-
|
|
430
|
-
for (const file of csvFiles) {
|
|
431
|
-
logger.info('-'.repeat(80));
|
|
432
|
-
logger.info(`Processing file: ${file.name}`, {
|
|
433
|
-
size: file.size,
|
|
434
|
-
lastModified: file.lastModified,
|
|
435
|
-
});
|
|
436
|
-
|
|
437
|
-
try {
|
|
438
|
-
// ─────────────────────────────────────────────────────────────────────
|
|
439
|
-
// STEP 3a: Download CSV File
|
|
440
|
-
// ─────────────────────────────────────────────────────────────────────
|
|
441
|
-
logger.info(`Downloading file: ${file.path}`);
|
|
442
|
-
const csvContent = await s3Source.downloadFile(file.path);
|
|
443
|
-
logger.info(
|
|
444
|
-
`Downloaded ${typeof csvContent === 'string' ? csvContent.length : csvContent.length} bytes`
|
|
445
|
-
);
|
|
446
|
-
|
|
447
|
-
// ─────────────────────────────────────────────────────────────────────
|
|
448
|
-
// STEP 3b: Parse CSV to Records
|
|
449
|
-
// ─────────────────────────────────────────────────────────────────────
|
|
450
|
-
logger.info('Parsing CSV content...');
|
|
451
|
-
const records = await csvParser.parse(csvContent as string);
|
|
452
|
-
logger.info(`Parsed ${records.length} records from CSV`);
|
|
453
|
-
|
|
454
|
-
if (records.length === 0) {
|
|
455
|
-
logger.warn('CSV file is empty, skipping');
|
|
456
|
-
continue;
|
|
457
|
-
}
|
|
458
|
-
|
|
459
|
-
// ─────────────────────────────────────────────────────────────────────
|
|
460
|
-
// STEP 3c: Validate Records (Basic Validation)
|
|
461
|
-
// ─────────────────────────────────────────────────────────────────────
|
|
462
|
-
logger.info('Validating records...');
|
|
463
|
-
const validRecords = records.filter((record: any, index: number) => {
|
|
464
|
-
// Check for required fields
|
|
465
|
-
if (!record.sku || !record.location || record.quantity === undefined) {
|
|
466
|
-
logger.warn(`Record ${index + 1} missing required fields, skipping`, {
|
|
467
|
-
record,
|
|
468
|
-
});
|
|
469
|
-
totalRecordsFailed++;
|
|
470
|
-
return false;
|
|
471
|
-
}
|
|
472
|
-
|
|
473
|
-
// Validate quantity is numeric
|
|
474
|
-
if (isNaN(Number(record.quantity))) {
|
|
475
|
-
logger.warn(`Record ${index + 1} has invalid quantity, skipping`, {
|
|
476
|
-
record,
|
|
477
|
-
});
|
|
478
|
-
totalRecordsFailed++;
|
|
479
|
-
return false;
|
|
480
|
-
}
|
|
481
|
-
|
|
482
|
-
return true;
|
|
483
|
-
});
|
|
484
|
-
|
|
485
|
-
logger.info(
|
|
486
|
-
`Validation complete: ${validRecords.length} valid, ${records.length - validRecords.length} invalid`
|
|
487
|
-
);
|
|
488
|
-
|
|
489
|
-
if (validRecords.length === 0) {
|
|
490
|
-
logger.warn('No valid records found in file, skipping');
|
|
491
|
-
continue;
|
|
492
|
-
}
|
|
493
|
-
|
|
494
|
-
// ─────────────────────────────────────────────────────────────────────
|
|
495
|
-
// STEP 3d: Map Records Using UniversalMapper
|
|
496
|
-
// ─────────────────────────────────────────────────────────────────────
|
|
497
|
-
logger.info('Mapping records using UniversalMapper...');
|
|
498
|
-
const mappedRecords: any[] = [];
|
|
499
|
-
const mappingErrors: string[] = [];
|
|
500
|
-
|
|
501
|
-
for (let i = 0; i < validRecords.length; i++) {
|
|
502
|
-
const record = validRecords[i];
|
|
503
|
-
const result = await mapper.map(record);
|
|
504
|
-
|
|
505
|
-
if (result.success) {
|
|
506
|
-
mappedRecords.push(result.data);
|
|
507
|
-
} else {
|
|
508
|
-
logger.warn(`Record ${i + 1} mapping failed`, {
|
|
509
|
-
errors: result.errors,
|
|
510
|
-
record,
|
|
511
|
-
});
|
|
512
|
-
mappingErrors.push(`Record ${i + 1}: ${result.errors?.join(', ')}`);
|
|
513
|
-
totalRecordsFailed++;
|
|
514
|
-
}
|
|
515
|
-
}
|
|
516
|
-
|
|
517
|
-
logger.info(
|
|
518
|
-
`Mapping complete: ${mappedRecords.length} mapped, ${mappingErrors.length} failed`
|
|
519
|
-
);
|
|
520
|
-
|
|
521
|
-
if (mappedRecords.length === 0) {
|
|
522
|
-
logger.warn('No records successfully mapped, skipping file');
|
|
523
|
-
continue;
|
|
524
|
-
}
|
|
525
|
-
|
|
526
|
-
// ─────────────────────────────────────────────────────────────────────
|
|
527
|
-
// STEP 3e: Create Batch API Job
|
|
528
|
-
// ─────────────────────────────────────────────────────────────────────
|
|
529
|
-
const jobName = `${config.processing.jobNamePrefix} - ${file.name} - ${new Date().toISOString()}`;
|
|
530
|
-
logger.info(`Creating Batch API job: ${jobName}`);
|
|
531
|
-
|
|
532
|
-
const job = await fluentClient.createJob({
|
|
533
|
-
name: jobName,
|
|
534
|
-
retailerId: config.fluent.retailerId,
|
|
535
|
-
});
|
|
536
|
-
|
|
537
|
-
logger.info(`Job created successfully`, {
|
|
538
|
-
jobId: job.id,
|
|
539
|
-
status: job.status,
|
|
540
|
-
});
|
|
541
|
-
|
|
542
|
-
// ─────────────────────────────────────────────────────────────────────
|
|
543
|
-
// STEP 3f: Send Batches (Chunked by Batch Size)
|
|
544
|
-
// ─────────────────────────────────────────────────────────────────────
|
|
545
|
-
logger.info(
|
|
546
|
-
`Sending ${mappedRecords.length} records in batches of ${config.processing.batchSize}`
|
|
547
|
-
);
|
|
548
|
-
const batches = chunk(mappedRecords, config.processing.batchSize);
|
|
549
|
-
|
|
550
|
-
for (let i = 0; i < batches.length; i++) {
|
|
551
|
-
const batch = batches[i];
|
|
552
|
-
logger.info(`Sending batch ${i + 1}/${batches.length} (${batch.length} records)`);
|
|
553
|
-
|
|
554
|
-
const batchResponse = await fluentClient.sendBatch(job.id, {
|
|
555
|
-
entities: batch,
|
|
556
|
-
});
|
|
557
|
-
|
|
558
|
-
logger.info(`Batch ${i + 1} sent successfully`, {
|
|
559
|
-
batchId: batchResponse.id,
|
|
560
|
-
status: batchResponse.status,
|
|
561
|
-
});
|
|
562
|
-
}
|
|
563
|
-
|
|
564
|
-
// ─────────────────────────────────────────────────────────────────────
|
|
565
|
-
// STEP 3g: Poll Job Status Until Complete
|
|
566
|
-
// ─────────────────────────────────────────────────────────────────────
|
|
567
|
-
logger.info('Polling job status...');
|
|
568
|
-
const finalStatus = await pollJobStatus(fluentClient, job.id, logger);
|
|
569
|
-
|
|
570
|
-
logger.info(`Job completed with status: ${finalStatus.status}`, {
|
|
571
|
-
jobId: job.id,
|
|
572
|
-
status: finalStatus.status,
|
|
573
|
-
});
|
|
574
|
-
|
|
575
|
-
// Check if job succeeded
|
|
576
|
-
if (finalStatus.status === 'COMPLETE') {
|
|
577
|
-
totalRecordsProcessed += mappedRecords.length;
|
|
578
|
-
filesSucceeded++;
|
|
579
|
-
|
|
580
|
-
// ───────────────────────────────────────────────────────────────────
|
|
581
|
-
// STEP 3h: Archive Successful File
|
|
582
|
-
// ───────────────────────────────────────────────────────────────────
|
|
583
|
-
const archivePath = `${config.processing.archivePrefix}${new Date().toISOString().split('T')[0]}/${file.name}`;
|
|
584
|
-
logger.info(`Archiving file to: ${archivePath}`);
|
|
585
|
-
await s3Source.moveFile(file.path, archivePath);
|
|
586
|
-
logger.info('File archived successfully');
|
|
587
|
-
} else {
|
|
588
|
-
// Job failed or has errors
|
|
589
|
-
logger.error('Job failed or has errors', undefined, {
|
|
590
|
-
jobId: job.id,
|
|
591
|
-
status: finalStatus.status,
|
|
592
|
-
});
|
|
593
|
-
|
|
594
|
-
// Move to error folder
|
|
595
|
-
const errorPath = `${config.processing.errorPrefix}${new Date().toISOString().split('T')[0]}/${file.name}`;
|
|
596
|
-
logger.info(`Moving file to error folder: ${errorPath}`);
|
|
597
|
-
await s3Source.moveFile(file.path, errorPath);
|
|
598
|
-
logger.info('File moved to error folder');
|
|
599
|
-
|
|
600
|
-
filesFailed++;
|
|
601
|
-
}
|
|
602
|
-
} catch (error) {
|
|
603
|
-
// File processing error
|
|
604
|
-
logger.error(`Failed to process file: ${file.name}`, error as Error);
|
|
605
|
-
filesFailed++;
|
|
606
|
-
|
|
607
|
-
try {
|
|
608
|
-
// Move to error folder
|
|
609
|
-
const errorPath = `${config.processing.errorPrefix}${new Date().toISOString().split('T')[0]}/${file.name}`;
|
|
610
|
-
await s3Source.moveFile(file.path, errorPath);
|
|
611
|
-
logger.info('File moved to error folder after processing error');
|
|
612
|
-
} catch (moveError) {
|
|
613
|
-
logger.error('Failed to move file to error folder', moveError as Error);
|
|
614
|
-
}
|
|
615
|
-
}
|
|
616
|
-
}
|
|
617
|
-
|
|
618
|
-
// ───────────────────────────────────────────────────────────────────────────
|
|
619
|
-
// STEP 4: Summary Statistics
|
|
620
|
-
// ───────────────────────────────────────────────────────────────────────────
|
|
621
|
-
|
|
622
|
-
const duration = Date.now() - startTime;
|
|
623
|
-
logger.info('='.repeat(80));
|
|
624
|
-
logger.info('Inventory sync completed', {
|
|
625
|
-
duration: `${(duration / 1000).toFixed(2)}s`,
|
|
626
|
-
filesProcessed: csvFiles.length,
|
|
627
|
-
filesSucceeded,
|
|
628
|
-
filesFailed,
|
|
629
|
-
totalRecordsProcessed,
|
|
630
|
-
totalRecordsFailed,
|
|
631
|
-
});
|
|
632
|
-
logger.info('='.repeat(80));
|
|
633
|
-
|
|
634
|
-
return {
|
|
635
|
-
filesProcessed: csvFiles.length,
|
|
636
|
-
filesSucceeded,
|
|
637
|
-
filesFailed,
|
|
638
|
-
totalRecordsProcessed,
|
|
639
|
-
totalRecordsFailed,
|
|
640
|
-
};
|
|
641
|
-
}
|
|
642
|
-
|
|
643
|
-
// ============================================================================
|
|
644
|
-
// HELPER FUNCTIONS
|
|
645
|
-
// ============================================================================
|
|
646
|
-
|
|
647
|
-
/**
|
|
648
|
-
* Poll Batch API job status until completion
|
|
649
|
-
*
|
|
650
|
-
* POLLING STRATEGY:
|
|
651
|
-
* - Initial delay: 2 seconds
|
|
652
|
-
* - Max delay: 30 seconds
|
|
653
|
-
* - Exponential backoff with 1.5x multiplier
|
|
654
|
-
* - Timeout: 30 minutes
|
|
655
|
-
*
|
|
656
|
-
* TERMINAL STATES:
|
|
657
|
-
* - COMPLETE: Job succeeded
|
|
658
|
-
* - FAILED: Job failed (API error, validation errors, etc.)
|
|
659
|
-
* - Other statuses: Continue polling
|
|
660
|
-
*/
|
|
661
|
-
async function pollJobStatus(client: any, jobId: string, logger: any): Promise<any> {
|
|
662
|
-
let delay = 2000; // Start with 2 second delay
|
|
663
|
-
const maxDelay = 30000; // Cap at 30 seconds
|
|
664
|
-
const timeout = 30 * 60 * 1000; // 30 minutes
|
|
665
|
-
const startTime = Date.now();
|
|
666
|
-
|
|
667
|
-
while (true) {
|
|
668
|
-
// Check timeout
|
|
669
|
-
if (Date.now() - startTime > timeout) {
|
|
670
|
-
throw new Error('Job status polling timeout (30 minutes)');
|
|
671
|
-
}
|
|
672
|
-
|
|
673
|
-
// Get job status
|
|
674
|
-
const status = await client.getJobStatus(jobId);
|
|
675
|
-
logger.debug(`Job status: ${status.status}`, {
|
|
676
|
-
jobId,
|
|
677
|
-
status: status.status,
|
|
678
|
-
});
|
|
679
|
-
|
|
680
|
-
// Check terminal states
|
|
681
|
-
if (status.status === 'COMPLETE' || status.status === 'FAILED') {
|
|
682
|
-
return status;
|
|
683
|
-
}
|
|
684
|
-
|
|
685
|
-
// Wait before next poll
|
|
686
|
-
await sleep(delay);
|
|
687
|
-
|
|
688
|
-
// Increase delay with exponential backoff (1.5x multiplier)
|
|
689
|
-
delay = Math.min(delay * 1.5, maxDelay);
|
|
690
|
-
}
|
|
691
|
-
}
|
|
692
|
-
|
|
693
|
-
/**
|
|
694
|
-
* Sleep helper function
|
|
695
|
-
*/
|
|
696
|
-
function sleep(ms: number): Promise<void> {
|
|
697
|
-
return new Promise(resolve => setTimeout(resolve, ms));
|
|
698
|
-
}
|
|
699
|
-
|
|
700
|
-
/**
|
|
701
|
-
* Chunk array into smaller arrays
|
|
702
|
-
*/
|
|
703
|
-
function chunk<T>(array: T[], size: number): T[][] {
|
|
704
|
-
const chunks: T[][] = [];
|
|
705
|
-
for (let i = 0; i < array.length; i += size) {
|
|
706
|
-
chunks.push(array.slice(i, i + size));
|
|
707
|
-
}
|
|
708
|
-
return chunks;
|
|
709
|
-
}
|
|
710
|
-
|
|
711
|
-
// ============================================================================
|
|
712
|
-
// ENTRY POINT
|
|
713
|
-
// ============================================================================
|
|
714
|
-
|
|
715
|
-
/**
|
|
716
|
-
* Main entry point with error handling
|
|
717
|
-
*/
|
|
718
|
-
async function main() {
|
|
719
|
-
try {
|
|
720
|
-
const result = await syncInventory();
|
|
721
|
-
|
|
722
|
-
// Exit with success code if no files failed
|
|
723
|
-
if (result.filesFailed === 0) {
|
|
724
|
-
logger.info('All files processed successfully');
|
|
725
|
-
process.exit(0);
|
|
726
|
-
} else {
|
|
727
|
-
logger.warn('Some files failed to process');
|
|
728
|
-
process.exit(1);
|
|
729
|
-
}
|
|
730
|
-
} catch (error) {
|
|
731
|
-
logger.error('Fatal error during inventory sync', error as Error);
|
|
732
|
-
process.exit(1);
|
|
733
|
-
}
|
|
734
|
-
}
|
|
735
|
-
|
|
736
|
-
// Run the sync
|
|
737
|
-
main();
|
|
738
|
-
```
|
|
739
|
-
|
|
740
|
-
### 4. Package.json Configuration
|
|
741
|
-
|
|
742
|
-
```json
|
|
743
|
-
{
|
|
744
|
-
"name": "inventory-sync-standalone",
|
|
745
|
-
"version": "1.0.0",
|
|
746
|
-
"description": "Standalone S3 CSV to Fluent Commerce inventory sync",
|
|
747
|
-
"main": "dist/inventory-sync.js",
|
|
748
|
-
"type": "module",
|
|
749
|
-
"scripts": {
|
|
750
|
-
"build": "tsc",
|
|
751
|
-
"start": "node dist/inventory-sync.js",
|
|
752
|
-
"dev": "tsx src/inventory-sync.ts",
|
|
753
|
-
"sync": "npm run dev",
|
|
754
|
-
"lint": "eslint src --ext .ts",
|
|
755
|
-
"type-check": "tsc --noEmit"
|
|
756
|
-
},
|
|
757
|
-
"keywords": ["fluent-commerce", "inventory", "s3", "batch-api"],
|
|
758
|
-
"author": "",
|
|
759
|
-
"license": "MIT",
|
|
760
|
-
"dependencies": {
|
|
761
|
-
"@fluentcommerce/fc-connect-sdk": "^0.1.39",
|
|
762
|
-
"dotenv": "^16.0.0"
|
|
763
|
-
},
|
|
764
|
-
"devDependencies": {
|
|
765
|
-
"@types/node": "^18.0.0",
|
|
766
|
-
"tsx": "^4.0.0",
|
|
767
|
-
"typescript": "^5.0.0"
|
|
768
|
-
}
|
|
769
|
-
}
|
|
770
|
-
```
|
|
771
|
-
|
|
772
|
-
### 5. TypeScript Configuration
|
|
773
|
-
|
|
774
|
-
Create `tsconfig.json`:
|
|
775
|
-
|
|
776
|
-
```json
|
|
777
|
-
{
|
|
778
|
-
"compilerOptions": {
|
|
779
|
-
"module": "ES2022",
|
|
780
|
-
"target": "ES2024",
|
|
781
|
-
"moduleResolution": "node"
|
|
782
|
-
}
|
|
783
|
-
}
|
|
784
|
-
```
|
|
785
|
-
|
|
786
|
-
---
|
|
787
|
-
|
|
788
|
-
## Key Patterns Explained
|
|
789
|
-
|
|
790
|
-
### Pattern 1: OAuth2 Authentication
|
|
791
|
-
|
|
792
|
-
**Purpose**: Authenticate with Fluent Commerce without Versori connection
|
|
793
|
-
|
|
794
|
-
**Implementation**:
|
|
795
|
-
|
|
796
|
-
```typescript
|
|
797
|
-
const fluentClient = await createClient({
|
|
798
|
-
config: {
|
|
799
|
-
baseUrl: 'https://api.fluentcommerce.com',
|
|
800
|
-
clientId: process.env.FLUENT_CLIENT_ID,
|
|
801
|
-
clientSecret: process.env.FLUENT_CLIENT_SECRET,
|
|
802
|
-
retailerId: process.env.FLUENT_RETAILER_ID,
|
|
803
|
-
},
|
|
804
|
-
});
|
|
805
|
-
```
|
|
806
|
-
|
|
807
|
-
**Key Points**:
|
|
808
|
-
|
|
809
|
-
- Uses client credentials OAuth2 grant type
|
|
810
|
-
- Token automatically cached and refreshed by SDK
|
|
811
|
-
- No manual token management needed
|
|
812
|
-
- Token expires after ~1 hour, SDK handles refresh transparently
|
|
813
|
-
|
|
814
|
-
**When to Use**:
|
|
815
|
-
|
|
816
|
-
- Standalone Node.js/Deno scripts
|
|
817
|
-
- Background jobs/cron tasks
|
|
818
|
-
- CI/CD pipelines
|
|
819
|
-
- Local development
|
|
820
|
-
|
|
821
|
-
### Pattern 2: S3 File Operations
|
|
822
|
-
|
|
823
|
-
**Purpose**: List, download, and move files in S3 bucket
|
|
824
|
-
|
|
825
|
-
**Implementation**:
|
|
826
|
-
|
|
827
|
-
```typescript
|
|
828
|
-
// Initialize S3 Data Source
|
|
829
|
-
const s3Source = new S3DataSource(
|
|
830
|
-
{
|
|
831
|
-
type: 'S3_CSV',
|
|
832
|
-
connectionId: 'inventory-s3',
|
|
833
|
-
name: 'Inventory S3 Source',
|
|
834
|
-
s3Config: {
|
|
835
|
-
bucket: 'my-bucket',
|
|
836
|
-
region: 'us-east-1',
|
|
837
|
-
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
|
838
|
-
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
|
|
839
|
-
},
|
|
840
|
-
},
|
|
841
|
-
logger
|
|
842
|
-
);
|
|
843
|
-
|
|
844
|
-
// List files with prefix filter
|
|
845
|
-
const files = await s3Source.listFiles({
|
|
846
|
-
prefix: 'inventory/updates/',
|
|
847
|
-
maxKeys: 1000,
|
|
848
|
-
});
|
|
849
|
-
|
|
850
|
-
// Download file content
|
|
851
|
-
const content = await s3Source.downloadFile(file.path);
|
|
852
|
-
|
|
853
|
-
// Move file (copy + delete)
|
|
854
|
-
await s3Source.moveFile(sourcePath, destPath);
|
|
855
|
-
```
|
|
856
|
-
|
|
857
|
-
**Key Points**:
|
|
858
|
-
|
|
859
|
-
- Uses presigned URLs (no AWS SDK required)
|
|
860
|
-
- Automatic retry logic for transient failures
|
|
861
|
-
- Streaming support for large files
|
|
862
|
-
- Cross-bucket move support
|
|
863
|
-
|
|
864
|
-
**Best Practices**:
|
|
865
|
-
|
|
866
|
-
- Always use prefix filtering to limit results
|
|
867
|
-
- Move files to archive/error folders (don't delete)
|
|
868
|
-
- Use date-based folder structure for archiving
|
|
869
|
-
- Validate S3 connection before processing
|
|
870
|
-
|
|
871
|
-
### Pattern 3: CSV Parsing & Validation
|
|
872
|
-
|
|
873
|
-
**Purpose**: Parse CSV files and validate data quality
|
|
874
|
-
|
|
875
|
-
**Implementation**:
|
|
876
|
-
|
|
877
|
-
```typescript
|
|
878
|
-
// Initialize CSV Parser
|
|
879
|
-
const csvParser = new CSVParserService();
|
|
880
|
-
|
|
881
|
-
// Parse CSV content (returns array of objects)
|
|
882
|
-
const records = await csvParser.parse(csvContent);
|
|
883
|
-
|
|
884
|
-
// Validate records
|
|
885
|
-
const validRecords = records.filter(record => {
|
|
886
|
-
// Required field validation
|
|
887
|
-
if (!record.sku || !record.location || record.quantity === undefined) {
|
|
888
|
-
logger.warn('Missing required fields', { record });
|
|
889
|
-
return false;
|
|
890
|
-
}
|
|
891
|
-
|
|
892
|
-
// Type validation
|
|
893
|
-
if (isNaN(Number(record.quantity))) {
|
|
894
|
-
logger.warn('Invalid quantity', { record });
|
|
895
|
-
return false;
|
|
896
|
-
}
|
|
897
|
-
|
|
898
|
-
return true;
|
|
899
|
-
});
|
|
900
|
-
```
|
|
901
|
-
|
|
902
|
-
**Key Points**:
|
|
903
|
-
|
|
904
|
-
- CSV parser auto-detects headers by default
|
|
905
|
-
- Trims whitespace and skips empty lines
|
|
906
|
-
- Returns array of objects (header → value mapping)
|
|
907
|
-
- Validation happens before mapping (fail fast)
|
|
908
|
-
|
|
909
|
-
**Validation Strategies**:
|
|
910
|
-
|
|
911
|
-
1. **Required Fields**: Check for null/undefined/empty
|
|
912
|
-
2. **Type Validation**: Ensure numeric fields are numbers
|
|
913
|
-
3. **Format Validation**: Regex for SKU, email, phone, etc.
|
|
914
|
-
4. **Business Rules**: Range checks, valid enum values
|
|
915
|
-
5. **Cross-field Validation**: Relationships between fields
|
|
916
|
-
|
|
917
|
-
### Pattern 4: Batch API Workflow
|
|
918
|
-
|
|
919
|
-
**Purpose**: Send large datasets to Fluent Commerce efficiently
|
|
920
|
-
|
|
921
|
-
**Implementation**:
|
|
922
|
-
|
|
923
|
-
```typescript
|
|
924
|
-
// STEP 1: Create Job
|
|
925
|
-
const job = await fluentClient.createJob({
|
|
926
|
-
name: 'Inventory Update - file.csv',
|
|
927
|
-
retailerId: 'my-retailer',
|
|
928
|
-
});
|
|
929
|
-
|
|
930
|
-
// STEP 2: Send Batches (chunked)
|
|
931
|
-
const batchSize = 100;
|
|
932
|
-
for (let i = 0; i < records.length; i += batchSize) {
|
|
933
|
-
const batch = records.slice(i, i + batchSize);
|
|
934
|
-
await fluentClient.sendBatch(job.id, {
|
|
935
|
-
entities: batch,
|
|
936
|
-
});
|
|
937
|
-
}
|
|
938
|
-
|
|
939
|
-
// STEP 3: Poll Job Status
|
|
940
|
-
let status = await fluentClient.getJobStatus(job.id);
|
|
941
|
-
while (status.status !== 'COMPLETE' && status.status !== 'FAILED') {
|
|
942
|
-
await sleep(5000); // Wait 5 seconds
|
|
943
|
-
status = await fluentClient.getJobStatus(job.id);
|
|
944
|
-
}
|
|
945
|
-
|
|
946
|
-
// STEP 4: Check Results
|
|
947
|
-
if (status.status === 'COMPLETE') {
|
|
948
|
-
console.log('Job succeeded!');
|
|
949
|
-
} else {
|
|
950
|
-
console.error('Job failed:', status);
|
|
951
|
-
}
|
|
952
|
-
```
|
|
953
|
-
|
|
954
|
-
**Key Points**:
|
|
955
|
-
|
|
956
|
-
- One job per file (easier tracking)
|
|
957
|
-
- Batch size 50-500 records (100 recommended)
|
|
958
|
-
- Poll with exponential backoff
|
|
959
|
-
- Handle both COMPLETE and FAILED states
|
|
960
|
-
|
|
961
|
-
**Batch Size Guidelines**:
|
|
962
|
-
|
|
963
|
-
- Small records (<10 fields): 200-500 per batch
|
|
964
|
-
- Medium records (10-30 fields): 100-200 per batch
|
|
965
|
-
- Large records (>30 fields): 50-100 per batch
|
|
966
|
-
- API limit: Usually 1000 records per batch
|
|
967
|
-
|
|
968
|
-
### Pattern 5: Error Handling & Retry
|
|
969
|
-
|
|
970
|
-
**Purpose**: Graceful degradation and recovery from failures
|
|
971
|
-
|
|
972
|
-
**Implementation**:
|
|
973
|
-
|
|
974
|
-
```typescript
|
|
975
|
-
// File-level try-catch
|
|
976
|
-
for (const file of files) {
|
|
977
|
-
try {
|
|
978
|
-
await processFile(file);
|
|
979
|
-
filesSucceeded++;
|
|
980
|
-
} catch (error) {
|
|
981
|
-
logger.error('File processing failed', error);
|
|
982
|
-
filesFailed++;
|
|
983
|
-
|
|
984
|
-
// Move to error folder
|
|
985
|
-
try {
|
|
986
|
-
await s3Source.moveFile(file.path, errorPath);
|
|
987
|
-
} catch (moveError) {
|
|
988
|
-
logger.error('Failed to move error file', moveError);
|
|
989
|
-
}
|
|
990
|
-
}
|
|
991
|
-
}
|
|
992
|
-
|
|
993
|
-
// Automatic retry with exponential backoff
|
|
994
|
-
async function retryWithBackoff<T>(fn: () => Promise<T>, maxRetries: number = 3): Promise<T> {
|
|
995
|
-
let lastError: Error;
|
|
996
|
-
for (let attempt = 0; attempt < maxRetries; attempt++) {
|
|
997
|
-
try {
|
|
998
|
-
return await fn();
|
|
999
|
-
} catch (error) {
|
|
1000
|
-
lastError = error as Error;
|
|
1001
|
-
if (attempt < maxRetries - 1) {
|
|
1002
|
-
const delay = Math.pow(2, attempt) * 1000;
|
|
1003
|
-
await sleep(delay);
|
|
1004
|
-
}
|
|
1005
|
-
}
|
|
1006
|
-
}
|
|
1007
|
-
throw lastError!;
|
|
1008
|
-
}
|
|
1009
|
-
```
|
|
1010
|
-
|
|
1011
|
-
**Error Categories**:
|
|
1012
|
-
|
|
1013
|
-
1. **Network Errors**: Retry with exponential backoff
|
|
1014
|
-
2. **4xx Client Errors**: Don't retry (bad request)
|
|
1015
|
-
3. **5xx Server Errors**: Retry (transient failure)
|
|
1016
|
-
4. **Validation Errors**: Skip record, continue processing
|
|
1017
|
-
5. **Fatal Errors**: Exit with error code
|
|
1018
|
-
|
|
1019
|
-
**Retry Best Practices**:
|
|
1020
|
-
|
|
1021
|
-
- Max 3-5 retry attempts
|
|
1022
|
-
- Exponential backoff (1s, 2s, 4s, 8s, ...)
|
|
1023
|
-
- Jitter to prevent thundering herd
|
|
1024
|
-
- Log each retry attempt
|
|
1025
|
-
- Give up after max retries
|
|
1026
|
-
|
|
1027
|
-
---
|
|
1028
|
-
|
|
1029
|
-
## Deployment Options
|
|
1030
|
-
|
|
1031
|
-
### Option 1: Local Execution
|
|
1032
|
-
|
|
1033
|
-
```bash
|
|
1034
|
-
# One-time sync
|
|
1035
|
-
npm run sync
|
|
1036
|
-
|
|
1037
|
-
# With debug logging
|
|
1038
|
-
LOG_LEVEL=debug npm run sync
|
|
1039
|
-
```
|
|
1040
|
-
|
|
1041
|
-
### Option 2: Docker Container
|
|
1042
|
-
|
|
1043
|
-
Create `Dockerfile`:
|
|
1044
|
-
|
|
1045
|
-
```dockerfile
|
|
1046
|
-
FROM node:18-alpine
|
|
1047
|
-
WORKDIR /app
|
|
1048
|
-
|
|
1049
|
-
# Copy package files
|
|
1050
|
-
COPY package*.json ./
|
|
1051
|
-
|
|
1052
|
-
# Install dependencies
|
|
1053
|
-
RUN npm ci --only=production
|
|
1054
|
-
|
|
1055
|
-
# Copy application code
|
|
1056
|
-
COPY dist ./dist
|
|
1057
|
-
|
|
1058
|
-
# Run the sync
|
|
1059
|
-
CMD ["node", "dist/inventory-sync.js"]
|
|
1060
|
-
```
|
|
1061
|
-
|
|
1062
|
-
Build and run:
|
|
1063
|
-
|
|
1064
|
-
```bash
|
|
1065
|
-
# Build image
|
|
1066
|
-
docker build -t inventory-sync .
|
|
1067
|
-
|
|
1068
|
-
# Run container
|
|
1069
|
-
docker run --env-file .env inventory-sync
|
|
1070
|
-
```
|
|
1071
|
-
|
|
1072
|
-
### Option 3: AWS Lambda
|
|
1073
|
-
|
|
1074
|
-
Create `lambda-handler.ts`:
|
|
1075
|
-
|
|
1076
|
-
```typescript
|
|
1077
|
-
import { syncInventory } from './inventory-sync';
|
|
1078
|
-
|
|
1079
|
-
export const handler = async (event: any, context: any) => {
|
|
1080
|
-
try {
|
|
1081
|
-
const result = await syncInventory();
|
|
1082
|
-
return {
|
|
1083
|
-
statusCode: 200,
|
|
1084
|
-
body: JSON.stringify(result),
|
|
1085
|
-
};
|
|
1086
|
-
} catch (error) {
|
|
1087
|
-
console.error('Lambda error:', error);
|
|
1088
|
-
return {
|
|
1089
|
-
statusCode: 500,
|
|
1090
|
-
body: JSON.stringify({ error: (error as Error).message }),
|
|
1091
|
-
};
|
|
1092
|
-
}
|
|
1093
|
-
};
|
|
1094
|
-
```
|
|
1095
|
-
|
|
1096
|
-
Deploy:
|
|
1097
|
-
|
|
1098
|
-
```bash
|
|
1099
|
-
# Package for Lambda
|
|
1100
|
-
npm run build
|
|
1101
|
-
zip -r function.zip dist node_modules
|
|
1102
|
-
|
|
1103
|
-
# Deploy with AWS CLI
|
|
1104
|
-
aws lambda create-function \
|
|
1105
|
-
--function-name inventory-sync \
|
|
1106
|
-
--runtime nodejs18.x \
|
|
1107
|
-
--handler dist/lambda-handler.handler \
|
|
1108
|
-
--zip-file fileb://function.zip \
|
|
1109
|
-
--role arn:aws:iam::123456789:role/lambda-execution \
|
|
1110
|
-
--timeout 900 \
|
|
1111
|
-
--memory-size 512
|
|
1112
|
-
```
|
|
1113
|
-
|
|
1114
|
-
### Option 4: Cron Scheduling
|
|
1115
|
-
|
|
1116
|
-
**Linux/Mac cron**:
|
|
1117
|
-
|
|
1118
|
-
```bash
|
|
1119
|
-
# Edit crontab
|
|
1120
|
-
crontab -e
|
|
1121
|
-
|
|
1122
|
-
# Run every 4 hours
|
|
1123
|
-
0 */4 * * * cd /path/to/inventory-sync && npm run sync >> /var/log/inventory-sync.log 2>&1
|
|
1124
|
-
```
|
|
1125
|
-
|
|
1126
|
-
**Node.js cron**:
|
|
1127
|
-
|
|
1128
|
-
Install: `npm install node-cron`
|
|
1129
|
-
|
|
1130
|
-
```typescript
|
|
1131
|
-
import cron from 'node-cron';
|
|
1132
|
-
|
|
1133
|
-
// Run every 4 hours
|
|
1134
|
-
cron.schedule('0 */4 * * *', async () => {
|
|
1135
|
-
logger.info('Starting scheduled inventory sync');
|
|
1136
|
-
await syncInventory();
|
|
1137
|
-
});
|
|
1138
|
-
|
|
1139
|
-
logger.info('Cron scheduler started (every 4 hours)');
|
|
1140
|
-
```
|
|
1141
|
-
|
|
1142
|
-
**AWS EventBridge**:
|
|
1143
|
-
|
|
1144
|
-
```bash
|
|
1145
|
-
# Create rule
|
|
1146
|
-
aws events put-rule \
|
|
1147
|
-
--name inventory-sync-schedule \
|
|
1148
|
-
--schedule-expression "rate(4 hours)"
|
|
1149
|
-
|
|
1150
|
-
# Add Lambda target
|
|
1151
|
-
aws events put-targets \
|
|
1152
|
-
--rule inventory-sync-schedule \
|
|
1153
|
-
--targets "Id"="1","Arn"="arn:aws:lambda:us-east-1:123:function:inventory-sync"
|
|
1154
|
-
```
|
|
1155
|
-
|
|
1156
|
-
---
|
|
1157
|
-
|
|
1158
|
-
## Testing
|
|
1159
|
-
|
|
1160
|
-
### Local Testing
|
|
1161
|
-
|
|
1162
|
-
```bash
|
|
1163
|
-
# 1. Create test CSV file
|
|
1164
|
-
cat > test-inventory.csv << EOF
|
|
1165
|
-
sku,location,quantity,status,expected_date
|
|
1166
|
-
TEST-SKU-001,WAREHOUSE-A,100,ACTIVE,2024-01-15
|
|
1167
|
-
TEST-SKU-002,WAREHOUSE-B,50,ACTIVE,2024-01-15
|
|
1168
|
-
TEST-SKU-003,STORE-001,25,ACTIVE,2024-01-16
|
|
1169
|
-
EOF
|
|
1170
|
-
|
|
1171
|
-
# 2. Upload to S3
|
|
1172
|
-
aws s3 cp test-inventory.csv s3://your-bucket/inventory/updates/
|
|
1173
|
-
|
|
1174
|
-
# 3. Run sync
|
|
1175
|
-
npm run sync
|
|
1176
|
-
|
|
1177
|
-
# 4. Check logs
|
|
1178
|
-
cat logs/inventory-sync.log
|
|
1179
|
-
|
|
1180
|
-
# 5. Verify in Fluent Commerce
|
|
1181
|
-
# - Check job status in admin portal
|
|
1182
|
-
# - Query inventory positions via GraphQL
|
|
1183
|
-
```
|
|
1184
|
-
|
|
1185
|
-
### Integration Testing
|
|
1186
|
-
|
|
1187
|
-
Create `tests/integration.test.ts`:
|
|
1188
|
-
|
|
1189
|
-
```typescript
|
|
1190
|
-
import { describe, it, expect, beforeAll } from '@jest/globals';
|
|
1191
|
-
import { syncInventory } from '../src/inventory-sync';
|
|
1192
|
-
|
|
1193
|
-
describe('Inventory Sync Integration', () => {
|
|
1194
|
-
beforeAll(async () => {
|
|
1195
|
-
// Setup test environment
|
|
1196
|
-
});
|
|
1197
|
-
|
|
1198
|
-
it('should process valid CSV file', async () => {
|
|
1199
|
-
// Upload test file to S3
|
|
1200
|
-
// Run sync
|
|
1201
|
-
const result = await syncInventory();
|
|
1202
|
-
|
|
1203
|
-
// Assertions
|
|
1204
|
-
expect(result.filesSucceeded).toBe(1);
|
|
1205
|
-
expect(result.filesFailed).toBe(0);
|
|
1206
|
-
expect(result.totalRecordsProcessed).toBeGreaterThan(0);
|
|
1207
|
-
});
|
|
1208
|
-
|
|
1209
|
-
it('should handle invalid records gracefully', async () => {
|
|
1210
|
-
// Upload CSV with invalid records
|
|
1211
|
-
// Run sync
|
|
1212
|
-
const result = await syncInventory();
|
|
1213
|
-
|
|
1214
|
-
// Should still succeed but skip invalid records
|
|
1215
|
-
expect(result.filesSucceeded).toBe(1);
|
|
1216
|
-
expect(result.totalRecordsFailed).toBeGreaterThan(0);
|
|
1217
|
-
});
|
|
1218
|
-
|
|
1219
|
-
it('should move failed files to error folder', async () => {
|
|
1220
|
-
// Upload malformed CSV
|
|
1221
|
-
// Run sync
|
|
1222
|
-
const result = await syncInventory();
|
|
1223
|
-
|
|
1224
|
-
// File should be in error folder
|
|
1225
|
-
expect(result.filesFailed).toBe(1);
|
|
1226
|
-
// Check S3 error folder
|
|
1227
|
-
});
|
|
1228
|
-
});
|
|
1229
|
-
```
|
|
1230
|
-
|
|
1231
|
-
Run tests:
|
|
1232
|
-
|
|
1233
|
-
```bash
|
|
1234
|
-
npm install --save-dev jest @jest/globals @types/jest ts-jest
|
|
1235
|
-
npm test
|
|
1236
|
-
```
|
|
1237
|
-
|
|
1238
|
-
---
|
|
1239
|
-
|
|
1240
|
-
## Common Issues
|
|
1241
|
-
|
|
1242
|
-
### Issue 1: OAuth2 Token Expired
|
|
1243
|
-
|
|
1244
|
-
**Symptom**: `401 Unauthorized` errors
|
|
1245
|
-
|
|
1246
|
-
**Cause**: Token expired and SDK failed to refresh
|
|
1247
|
-
|
|
1248
|
-
**Solution**:
|
|
1249
|
-
|
|
1250
|
-
```typescript
|
|
1251
|
-
// SDK handles refresh automatically, but you can force refresh
|
|
1252
|
-
const client = await createClient({
|
|
1253
|
-
config: {
|
|
1254
|
-
baseUrl: 'https://api.fluentcommerce.com',
|
|
1255
|
-
clientId: process.env.FLUENT_CLIENT_ID,
|
|
1256
|
-
clientSecret: process.env.FLUENT_CLIENT_SECRET,
|
|
1257
|
-
retailerId: process.env.FLUENT_RETAILER_ID,
|
|
1258
|
-
retryAttempts: 3,
|
|
1259
|
-
},
|
|
1260
|
-
});
|
|
1261
|
-
```
|
|
1262
|
-
|
|
1263
|
-
### Issue 2: S3 Permission Denied
|
|
1264
|
-
|
|
1265
|
-
**Symptom**: `AccessDenied` errors when listing/downloading files
|
|
1266
|
-
|
|
1267
|
-
**Cause**: IAM policy missing required permissions
|
|
1268
|
-
|
|
1269
|
-
**Solution**:
|
|
1270
|
-
|
|
1271
|
-
```json
|
|
1272
|
-
{
|
|
1273
|
-
"Version": "2012-10-17",
|
|
1274
|
-
"Statement": [
|
|
1275
|
-
{
|
|
1276
|
-
"Effect": "Allow",
|
|
1277
|
-
"Action": ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
|
|
1278
|
-
"Resource": ["arn:aws:s3:::your-bucket", "arn:aws:s3:::your-bucket/*"]
|
|
1279
|
-
}
|
|
1280
|
-
]
|
|
1281
|
-
}
|
|
1282
|
-
```
|
|
1283
|
-
|
|
1284
|
-
### Issue 3: CSV Parsing Errors
|
|
1285
|
-
|
|
1286
|
-
**Symptom**: `CSV parse error: Invalid record`
|
|
1287
|
-
|
|
1288
|
-
**Cause**: Malformed CSV (unescaped quotes, inconsistent columns)
|
|
1289
|
-
|
|
1290
|
-
**Solution**:
|
|
1291
|
-
|
|
1292
|
-
```typescript
|
|
1293
|
-
// Use more lenient CSV parser options
|
|
1294
|
-
const csvParser = CSVParserService.create({
|
|
1295
|
-
columns: true,
|
|
1296
|
-
skip_empty_lines: true,
|
|
1297
|
-
trim: true,
|
|
1298
|
-
relax_column_count: true, // Allow inconsistent column counts
|
|
1299
|
-
quote: '"',
|
|
1300
|
-
escape: '"',
|
|
1301
|
-
delimiter: ',',
|
|
1302
|
-
});
|
|
1303
|
-
```
|
|
1304
|
-
|
|
1305
|
-
### Issue 4: Batch API Timeout
|
|
1306
|
-
|
|
1307
|
-
**Symptom**: Job stays in `PENDING` state indefinitely
|
|
1308
|
-
|
|
1309
|
-
**Cause**: Fluent API processing delay or large batch size
|
|
1310
|
-
|
|
1311
|
-
**Solution**:
|
|
1312
|
-
|
|
1313
|
-
```typescript
|
|
1314
|
-
// Reduce batch size
|
|
1315
|
-
const batchSize = 50; // Instead of 100
|
|
1316
|
-
|
|
1317
|
-
// Increase polling timeout
|
|
1318
|
-
const timeout = 60 * 60 * 1000; // 1 hour instead of 30 minutes
|
|
1319
|
-
|
|
1320
|
-
// Add more aggressive polling
|
|
1321
|
-
let delay = 1000; // Start with 1 second
|
|
1322
|
-
const maxDelay = 15000; // Cap at 15 seconds
|
|
1323
|
-
```
|
|
1324
|
-
|
|
1325
|
-
### Issue 5: Memory Exhaustion
|
|
1326
|
-
|
|
1327
|
-
**Symptom**: `JavaScript heap out of memory` error
|
|
1328
|
-
|
|
1329
|
-
**Cause**: Loading entire large CSV file into memory
|
|
1330
|
-
|
|
1331
|
-
**Solution**:
|
|
1332
|
-
|
|
1333
|
-
```typescript
|
|
1334
|
-
// Use streaming CSV parsing instead
|
|
1335
|
-
for await (const record of csvParser.parseStreaming(csvContent, {}, 100)) {
|
|
1336
|
-
// Process records in batches of 100
|
|
1337
|
-
const mappedBatch = await Promise.all(record.map(r => mapper.map(r)));
|
|
1338
|
-
|
|
1339
|
-
// Send batch immediately (don't accumulate)
|
|
1340
|
-
await fluentClient.sendBatch(jobId, {
|
|
1341
|
-
entities: mappedBatch.filter(r => r.success).map(r => r.data),
|
|
1342
|
-
});
|
|
1343
|
-
}
|
|
1344
|
-
```
|
|
1345
|
-
|
|
1346
|
-
---
|
|
1347
|
-
|
|
1348
|
-
## Related Guides
|
|
1349
|
-
|
|
1350
|
-
- **SDK Reference**: `../../readme.md` - Core SDK documentation
|
|
1351
|
-
- **Universal Mapping Guide**: `../../02-CORE-GUIDES/mapping/readme.md` - Field mapping patterns
|
|
1352
|
-
- **Batch API Guide**: `../../02-CORE-GUIDES/api-reference/modules/03-batch-processing.md` - Batch API details
|
|
1353
|
-
- **S3 Data Source**: `../../02-CORE-GUIDES/data-sources/readme.md` - S3 operations
|
|
1354
|
-
- **CSV Parser**: `../../02-CORE-GUIDES/parsers/readme.md` - CSV parsing options
|
|
1355
|
-
- **Error Handling**: `../../03-PATTERN-GUIDES/error-handling/readme.md` - Error patterns
|
|
1356
|
-
|
|
1357
|
-
---
|
|
1358
|
-
|
|
1359
|
-
## Production Checklist
|
|
1360
|
-
|
|
1361
|
-
Before deploying to production:
|
|
1362
|
-
|
|
1363
|
-
- [ ] Environment variables secured (AWS Secrets Manager, vault, etc.)
|
|
1364
|
-
- [ ] Logging configured (Winston, CloudWatch, Datadog, etc.)
|
|
1365
|
-
- [ ] Error alerting setup (PagerDuty, Slack, email)
|
|
1366
|
-
- [ ] Monitoring dashboard (metrics, job success rate, duration)
|
|
1367
|
-
- [ ] Retry logic tested (network failures, API errors)
|
|
1368
|
-
- [ ] Batch size optimized (tested with production file sizes)
|
|
1369
|
-
- [ ] Memory profiling done (no leaks)
|
|
1370
|
-
- [ ] S3 lifecycle policies configured (archive old files)
|
|
1371
|
-
- [ ] IAM permissions minimized (least privilege)
|
|
1372
|
-
- [ ] Health check endpoint added (for orchestration tools)
|
|
1373
|
-
- [ ] Documentation updated (runbook, troubleshooting)
|
|
1374
|
-
- [ ] Disaster recovery tested (restore from archive)
|
|
1375
|
-
|
|
1376
|
-
---
|
|
1377
|
-
|
|
1378
|
-
**Next Steps**:
|
|
1379
|
-
|
|
1380
|
-
1. Customize field mappings for your inventory data structure
|
|
1381
|
-
2. Add custom validation rules for your business logic
|
|
1382
|
-
3. Implement state tracking to prevent duplicate processing
|
|
1383
|
-
4. Set up monitoring and alerting
|
|
1384
|
-
5. Deploy to your preferred environment (Lambda, ECS, K8s, etc.)
|
|
1385
|
-
|
|
1386
|
-
**Support**:
|
|
1387
|
-
|
|
1388
|
-
- SDK Issues: https://github.com/fluentcommerce/fc-connect-sdk/issues
|
|
1389
|
-
- Fluent Commerce API: https://docs.fluentcommerce.com
|
|
1390
|
-
- Community: https://community.fluentcommerce.com
|
|
1
|
+
# Standalone: S3 CSV → Fluent Batch API
|
|
2
|
+
|
|
3
|
+
**FC Connect SDK Use Case Guide**
|
|
4
|
+
|
|
5
|
+
> **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
|
|
6
|
+
> **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
|
|
7
|
+
|
|
8
|
+
**Context**: Node.js script that reads CSV inventory files from S3 and updates Fluent Commerce inventory via Batch API
|
|
9
|
+
|
|
10
|
+
**Complexity**: Medium
|
|
11
|
+
|
|
12
|
+
**Runtime**: Node.js ≥18 / Deno
|
|
13
|
+
|
|
14
|
+
**Estimated Lines**: ~500 lines
|
|
15
|
+
|
|
16
|
+
## What You'll Build
|
|
17
|
+
|
|
18
|
+
- Standalone Node.js/Deno script (no Versori)
|
|
19
|
+
- OAuth2 authentication with Fluent Commerce
|
|
20
|
+
- S3 file listing and download with AWS SDK
|
|
21
|
+
- CSV parsing with validation
|
|
22
|
+
- UniversalMapper for field transformations
|
|
23
|
+
- Batch API processing with job management
|
|
24
|
+
- Error handling and logging
|
|
25
|
+
- Optional: Schedule with cron or AWS EventBridge
|
|
26
|
+
|
|
27
|
+
## SDK Methods Used
|
|
28
|
+
|
|
29
|
+
- `createClient({ config: { baseUrl, clientId, clientSecret, retailerId } })` - OAuth2 client
|
|
30
|
+
- `S3DataSource(config, logger)` - S3 operations
|
|
31
|
+
- `CSVParserService` - CSV parsing
|
|
32
|
+
- `UniversalMapper(mappingConfig)` - Field mapping
|
|
33
|
+
- `client.createJob({ name, retailerId })` - Create Batch job
|
|
34
|
+
- `client.sendBatch(jobId, { entities })` - Send inventory batch
|
|
35
|
+
- `client.getJobStatus(jobId)` - Check status
|
|
36
|
+
|
|
37
|
+
---
|
|
38
|
+
|
|
39
|
+
## Complete Working Implementation
|
|
40
|
+
|
|
41
|
+
### 1. Project Setup
|
|
42
|
+
|
|
43
|
+
```bash
|
|
44
|
+
# Create project directory
|
|
45
|
+
mkdir inventory-sync-standalone
|
|
46
|
+
cd inventory-sync-standalone
|
|
47
|
+
|
|
48
|
+
# Initialize Node.js project
|
|
49
|
+
npm init -y
|
|
50
|
+
|
|
51
|
+
# Install dependencies
|
|
52
|
+
npm install @fluentcommerce/fc-connect-sdk dotenv
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
### Method selection (SDK)
|
|
56
|
+
|
|
57
|
+
- Queries → `client.graphql`
|
|
58
|
+
- Mutations → `client.graphql`
|
|
59
|
+
- Advanced needs (pagination/telemetry) → `client.graphql` with pagination options
|
|
60
|
+
|
|
61
|
+
```typescript
|
|
62
|
+
// Query (use graphql for both queries and mutations)
|
|
63
|
+
const products = await client.graphql({ query, variables });
|
|
64
|
+
|
|
65
|
+
// Mutation (use graphql for both queries and mutations)
|
|
66
|
+
const result = await client.graphql({ query: mutation, variables });
|
|
67
|
+
|
|
68
|
+
// Advanced pagination
|
|
69
|
+
const res = await client.graphql({ query, variables, pagination: { maxPages: 50 } });
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
### 2. Environment Configuration
|
|
73
|
+
|
|
74
|
+
Create `.env` file:
|
|
75
|
+
|
|
76
|
+
```env
|
|
77
|
+
# Fluent Commerce Configuration
|
|
78
|
+
FLUENT_BASE_URL=https://api.fluentcommerce.com
|
|
79
|
+
FLUENT_CLIENT_ID=your-oauth2-client-id
|
|
80
|
+
FLUENT_CLIENT_SECRET=your-oauth2-client-secret
|
|
81
|
+
FLUENT_RETAILER_ID=your-retailer-id
|
|
82
|
+
|
|
83
|
+
# AWS S3 Configuration
|
|
84
|
+
AWS_ACCESS_KEY_ID=your-aws-access-key
|
|
85
|
+
AWS_SECRET_ACCESS_KEY=your-aws-secret-key
|
|
86
|
+
AWS_REGION=us-east-1
|
|
87
|
+
AWS_BUCKET_NAME=your-inventory-bucket
|
|
88
|
+
AWS_FILE_PREFIX=inventory/updates/
|
|
89
|
+
|
|
90
|
+
# Processing Configuration
|
|
91
|
+
BATCH_SIZE=100
|
|
92
|
+
JOB_NAME_PREFIX=S3-CSV-Inventory-Update
|
|
93
|
+
ARCHIVE_PREFIX=inventory/archive/
|
|
94
|
+
ERROR_PREFIX=inventory/errors/
|
|
95
|
+
|
|
96
|
+
# Optional: Scheduling
|
|
97
|
+
ENABLE_SCHEDULING=false
|
|
98
|
+
CRON_SCHEDULE=0 */4 * * *
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
### 3. Main Script Implementation
|
|
102
|
+
|
|
103
|
+
Create `src/inventory-sync.ts`:
|
|
104
|
+
|
|
105
|
+
```typescript
|
|
106
|
+
import 'dotenv/config';
|
|
107
|
+
// FC Connect SDK+
|
|
108
|
+
// Install: npm install @fluentcommerce/fc-connect-sdk@latest
|
|
109
|
+
// Docs: https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk
|
|
110
|
+
// GitHub: https://github.com/fluentcommerce/fc-connect-sdk
|
|
111
|
+
import { createClient } from '@fluentcommerce/fc-connect-sdk';
|
|
112
|
+
import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
|
|
113
|
+
import { CSVParserService } from '@fluentcommerce/fc-connect-sdk';
|
|
114
|
+
import { UniversalMapper } from '@fluentcommerce/fc-connect-sdk';
|
|
115
|
+
|
|
116
|
+
/**
|
|
117
|
+
* ARCHITECTURE:
|
|
118
|
+
*
|
|
119
|
+
* This standalone script implements a complete inventory sync workflow:
|
|
120
|
+
* 1. List CSV files from S3 bucket
|
|
121
|
+
* 2. Download and parse CSV files
|
|
122
|
+
* 3. Map CSV fields to Fluent inventory format
|
|
123
|
+
* 4. Create Batch API job
|
|
124
|
+
* 5. Send inventory updates in batches
|
|
125
|
+
* 6. Poll job status until complete
|
|
126
|
+
* 7. Archive successful files, move errors
|
|
127
|
+
*
|
|
128
|
+
* ERROR HANDLING STRATEGY:
|
|
129
|
+
* - File-level errors: Move to error folder, continue with other files
|
|
130
|
+
* - Batch errors: Log and continue (Fluent API provides detailed error reports)
|
|
131
|
+
* - Fatal errors: Exit with error code for monitoring/alerting
|
|
132
|
+
*
|
|
133
|
+
* STATE MANAGEMENT:
|
|
134
|
+
* - No state tracking in this basic version (processes all files)
|
|
135
|
+
* - Advanced: Use S3 metadata or external DB to track processed files
|
|
136
|
+
*/
|
|
137
|
+
|
|
138
|
+
// ============================================================================
|
|
139
|
+
// CONFIGURATION & SETUP
|
|
140
|
+
// ============================================================================
|
|
141
|
+
|
|
142
|
+
interface SyncConfig {
|
|
143
|
+
fluent: {
|
|
144
|
+
baseUrl: string;
|
|
145
|
+
clientId: string;
|
|
146
|
+
clientSecret: string;
|
|
147
|
+
retailerId: string;
|
|
148
|
+
};
|
|
149
|
+
s3: {
|
|
150
|
+
accessKeyId: string;
|
|
151
|
+
secretAccessKey: string;
|
|
152
|
+
region: string;
|
|
153
|
+
bucket: string;
|
|
154
|
+
prefix: string;
|
|
155
|
+
};
|
|
156
|
+
processing: {
|
|
157
|
+
batchSize: number;
|
|
158
|
+
jobNamePrefix: string;
|
|
159
|
+
archivePrefix: string;
|
|
160
|
+
errorPrefix: string;
|
|
161
|
+
};
|
|
162
|
+
}
|
|
163
|
+
|
|
164
|
+
/**
|
|
165
|
+
* Load configuration from environment variables
|
|
166
|
+
* Validates all required config before proceeding
|
|
167
|
+
*/
|
|
168
|
+
function loadConfig(): SyncConfig {
|
|
169
|
+
const requiredEnvVars = [
|
|
170
|
+
'FLUENT_BASE_URL',
|
|
171
|
+
'FLUENT_CLIENT_ID',
|
|
172
|
+
'FLUENT_CLIENT_SECRET',
|
|
173
|
+
'FLUENT_RETAILER_ID',
|
|
174
|
+
'AWS_ACCESS_KEY_ID',
|
|
175
|
+
'AWS_SECRET_ACCESS_KEY',
|
|
176
|
+
'AWS_REGION',
|
|
177
|
+
'AWS_BUCKET_NAME',
|
|
178
|
+
];
|
|
179
|
+
|
|
180
|
+
// Validate all required environment variables are present
|
|
181
|
+
const missing = requiredEnvVars.filter(v => !process.env[v]);
|
|
182
|
+
if (missing.length > 0) {
|
|
183
|
+
throw new Error(`Missing required environment variables: ${missing.join(', ')}`);
|
|
184
|
+
}
|
|
185
|
+
|
|
186
|
+
return {
|
|
187
|
+
fluent: {
|
|
188
|
+
baseUrl: process.env.FLUENT_BASE_URL!,
|
|
189
|
+
clientId: process.env.FLUENT_CLIENT_ID!,
|
|
190
|
+
clientSecret: process.env.FLUENT_CLIENT_SECRET!,
|
|
191
|
+
retailerId: process.env.FLUENT_RETAILER_ID!,
|
|
192
|
+
},
|
|
193
|
+
s3: {
|
|
194
|
+
accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
|
|
195
|
+
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
|
|
196
|
+
region: process.env.AWS_REGION!,
|
|
197
|
+
bucket: process.env.AWS_BUCKET_NAME!,
|
|
198
|
+
prefix: process.env.AWS_FILE_PREFIX || 'inventory/updates/',
|
|
199
|
+
},
|
|
200
|
+
processing: {
|
|
201
|
+
batchSize: parseInt(process.env.BATCH_SIZE || '100'),
|
|
202
|
+
jobNamePrefix: process.env.JOB_NAME_PREFIX || 'S3-CSV-Inventory-Update',
|
|
203
|
+
archivePrefix: process.env.ARCHIVE_PREFIX || 'inventory/archive/',
|
|
204
|
+
errorPrefix: process.env.ERROR_PREFIX || 'inventory/errors/',
|
|
205
|
+
},
|
|
206
|
+
};
|
|
207
|
+
}
|
|
208
|
+
|
|
209
|
+
// ============================================================================
|
|
210
|
+
// FIELD MAPPING CONFIGURATION
|
|
211
|
+
// ============================================================================
|
|
212
|
+
|
|
213
|
+
/**
|
|
214
|
+
* UniversalMapper configuration for CSV → Fluent Batch API transformation
|
|
215
|
+
*
|
|
216
|
+
* FIELD MAPPING RULES:
|
|
217
|
+
* - ref: Product SKU (required) - uppercase transformation
|
|
218
|
+
* - locationRef: Location code (required) - prefix with "LOC:"
|
|
219
|
+
* - qty: Quantity (required) - parse as integer, default to 0
|
|
220
|
+
* - type: Inventory type (required) - defaults to "ADJUSTMENT"
|
|
221
|
+
* - status: Inventory status (optional) - defaults to "ACTIVE"
|
|
222
|
+
*
|
|
223
|
+
* SDK RESOLVERS USED:
|
|
224
|
+
* - sdk.uppercase: Convert SKU to uppercase for consistency
|
|
225
|
+
* - sdk.parseInt: Parse quantity string to integer
|
|
226
|
+
* - sdk.trim: Remove whitespace from location codes
|
|
227
|
+
* - sdk.defaultTo: Provide fallback values
|
|
228
|
+
*/
|
|
229
|
+
const inventoryMappingConfig = {
|
|
230
|
+
version: '1.0.0',
|
|
231
|
+
description: 'S3 CSV to Fluent Commerce Inventory Mapping',
|
|
232
|
+
fields: {
|
|
233
|
+
// Product SKU (required field)
|
|
234
|
+
ref: {
|
|
235
|
+
source: 'sku',
|
|
236
|
+
resolver: 'sdk.uppercase', // Normalize to uppercase
|
|
237
|
+
required: true,
|
|
238
|
+
},
|
|
239
|
+
// Location reference (required field)
|
|
240
|
+
locationRef: {
|
|
241
|
+
source: 'location',
|
|
242
|
+
resolver: 'sdk.trim', // Remove whitespace
|
|
243
|
+
required: true,
|
|
244
|
+
},
|
|
245
|
+
// Quantity (required field)
|
|
246
|
+
qty: {
|
|
247
|
+
source: 'quantity',
|
|
248
|
+
resolver: 'sdk.parseInt', // Parse as integer
|
|
249
|
+
required: true,
|
|
250
|
+
},
|
|
251
|
+
// Inventory type (defaults to ADJUSTMENT)
|
|
252
|
+
type: {
|
|
253
|
+
value: 'ADJUSTMENT', // Static value
|
|
254
|
+
},
|
|
255
|
+
// Inventory status (optional, with default)
|
|
256
|
+
status: {
|
|
257
|
+
source: 'status',
|
|
258
|
+
resolver: 'sdk.uppercase',
|
|
259
|
+
defaultValue: 'ACTIVE', // Fallback if not provided
|
|
260
|
+
},
|
|
261
|
+
// Expected date (optional)
|
|
262
|
+
expectedOn: {
|
|
263
|
+
source: 'expected_date',
|
|
264
|
+
resolver: 'sdk.formatDate', // Convert to ISO date format
|
|
265
|
+
},
|
|
266
|
+
},
|
|
267
|
+
};
|
|
268
|
+
|
|
269
|
+
// ============================================================================
|
|
270
|
+
// LOGGER IMPLEMENTATION
|
|
271
|
+
// ============================================================================
|
|
272
|
+
|
|
273
|
+
/**
|
|
274
|
+
* Simple console logger with timestamp and log levels
|
|
275
|
+
* Production: Replace with Winston, Bunyan, or Pino
|
|
276
|
+
*/
|
|
277
|
+
const logger = {
|
|
278
|
+
debug: (message: string, meta?: any) => {
|
|
279
|
+
if (process.env.LOG_LEVEL === 'debug') {
|
|
280
|
+
console.log(`[${new Date().toISOString()}] DEBUG: ${message}`, meta || '');
|
|
281
|
+
}
|
|
282
|
+
},
|
|
283
|
+
info: (message: string, meta?: any) => {
|
|
284
|
+
console.log(`[${new Date().toISOString()}] INFO: ${message}`, meta || '');
|
|
285
|
+
},
|
|
286
|
+
warn: (message: string, meta?: any) => {
|
|
287
|
+
console.warn(`[${new Date().toISOString()}] WARN: ${message}`, meta || '');
|
|
288
|
+
},
|
|
289
|
+
error: (message: string, error?: Error, meta?: any) => {
|
|
290
|
+
console.error(
|
|
291
|
+
`[${new Date().toISOString()}] ERROR: ${message}`,
|
|
292
|
+
error?.message || '',
|
|
293
|
+
meta || ''
|
|
294
|
+
);
|
|
295
|
+
},
|
|
296
|
+
};
|
|
297
|
+
|
|
298
|
+
// ============================================================================
|
|
299
|
+
// MAIN SYNC ORCHESTRATION
|
|
300
|
+
// ============================================================================
|
|
301
|
+
|
|
302
|
+
/**
|
|
303
|
+
* Main inventory sync function
|
|
304
|
+
*
|
|
305
|
+
* ALGORITHM:
|
|
306
|
+
* 1. Initialize SDK clients (Fluent, S3, CSV parser, mapper)
|
|
307
|
+
* 2. List CSV files from S3 prefix
|
|
308
|
+
* 3. For each file:
|
|
309
|
+
* a. Download file content
|
|
310
|
+
* b. Parse CSV to records
|
|
311
|
+
* c. Validate records (skip invalid)
|
|
312
|
+
* d. Map records using UniversalMapper
|
|
313
|
+
* e. Create Batch API job
|
|
314
|
+
* f. Send batches (chunked by batch size)
|
|
315
|
+
* g. Poll job status until complete
|
|
316
|
+
* h. Archive successful files OR move to error folder
|
|
317
|
+
* 4. Return summary statistics
|
|
318
|
+
*/
|
|
319
|
+
async function syncInventory(): Promise<{
|
|
320
|
+
filesProcessed: number;
|
|
321
|
+
filesSucceeded: number;
|
|
322
|
+
filesFailed: number;
|
|
323
|
+
totalRecordsProcessed: number;
|
|
324
|
+
totalRecordsFailed: number;
|
|
325
|
+
}> {
|
|
326
|
+
const startTime = Date.now();
|
|
327
|
+
logger.info('='.repeat(80));
|
|
328
|
+
logger.info('Starting S3 CSV → Fluent Batch API inventory sync');
|
|
329
|
+
logger.info('='.repeat(80));
|
|
330
|
+
|
|
331
|
+
// Load configuration
|
|
332
|
+
const config = loadConfig();
|
|
333
|
+
logger.info('Configuration loaded', {
|
|
334
|
+
s3Bucket: config.s3.bucket,
|
|
335
|
+
s3Prefix: config.s3.prefix,
|
|
336
|
+
batchSize: config.processing.batchSize,
|
|
337
|
+
});
|
|
338
|
+
|
|
339
|
+
// ───────────────────────────────────────────────────────────────────────────
|
|
340
|
+
// STEP 1: Initialize SDK Clients
|
|
341
|
+
// ───────────────────────────────────────────────────────────────────────────
|
|
342
|
+
|
|
343
|
+
// Initialize Fluent Client with OAuth2
|
|
344
|
+
logger.info('Initializing Fluent Commerce client...');
|
|
345
|
+
const fluentClient = await createClient({
|
|
346
|
+
config: {
|
|
347
|
+
baseUrl: config.fluent.baseUrl,
|
|
348
|
+
clientId: config.fluent.clientId,
|
|
349
|
+
clientSecret: config.fluent.clientSecret,
|
|
350
|
+
retailerId: config.fluent.retailerId,
|
|
351
|
+
},
|
|
352
|
+
});
|
|
353
|
+
|
|
354
|
+
// Initialize S3 Data Source
|
|
355
|
+
logger.info('Initializing S3 data source...');
|
|
356
|
+
const s3Source = new S3DataSource(
|
|
357
|
+
{
|
|
358
|
+
type: 'S3_CSV',
|
|
359
|
+
connectionId: 'inventory-s3',
|
|
360
|
+
name: 'Inventory S3 Source',
|
|
361
|
+
s3Config: {
|
|
362
|
+
bucket: config.s3.bucket,
|
|
363
|
+
region: config.s3.region,
|
|
364
|
+
accessKeyId: config.s3.accessKeyId,
|
|
365
|
+
secretAccessKey: config.s3.secretAccessKey,
|
|
366
|
+
},
|
|
367
|
+
},
|
|
368
|
+
logger
|
|
369
|
+
);
|
|
370
|
+
|
|
371
|
+
// Initialize CSV Parser
|
|
372
|
+
const csvParser = new CSVParserService();
|
|
373
|
+
|
|
374
|
+
// Initialize Universal Mapper
|
|
375
|
+
const mapper = new UniversalMapper(inventoryMappingConfig, {
|
|
376
|
+
logger,
|
|
377
|
+
fluentClient,
|
|
378
|
+
});
|
|
379
|
+
|
|
380
|
+
// Validate S3 connection
|
|
381
|
+
logger.info('Validating S3 connection...');
|
|
382
|
+
const s3Connected = await s3Source.validateConnection();
|
|
383
|
+
if (!s3Connected) {
|
|
384
|
+
throw new Error('Failed to connect to S3 bucket');
|
|
385
|
+
}
|
|
386
|
+
logger.info('S3 connection validated successfully');
|
|
387
|
+
|
|
388
|
+
// ───────────────────────────────────────────────────────────────────────────
|
|
389
|
+
// STEP 2: List CSV Files from S3
|
|
390
|
+
// ───────────────────────────────────────────────────────────────────────────
|
|
391
|
+
|
|
392
|
+
logger.info(`Listing CSV files from S3 prefix: ${config.s3.prefix}`);
|
|
393
|
+
const files = await s3Source.listFiles({
|
|
394
|
+
prefix: config.s3.prefix,
|
|
395
|
+
maxKeys: 1000,
|
|
396
|
+
});
|
|
397
|
+
|
|
398
|
+
// Filter for CSV files only
|
|
399
|
+
// NOTE: S3 doesn't support glob patterns, so manual filtering is required
|
|
400
|
+
// IMPORTANT: Use .endsWith('.csv') NOT .endsWith('*.csv')
|
|
401
|
+
// ✓ Correct: f.name.endsWith('.csv')
|
|
402
|
+
// ✗ Wrong: f.name.endsWith('*.csv') // Would never match!
|
|
403
|
+
const csvFiles = files.filter(f => f.name.toLowerCase().endsWith('.csv'));
|
|
404
|
+
|
|
405
|
+
logger.info(`Found ${csvFiles.length} CSV files to process`, {
|
|
406
|
+
totalFiles: files.length,
|
|
407
|
+
csvFiles: csvFiles.length,
|
|
408
|
+
});
|
|
409
|
+
|
|
410
|
+
if (csvFiles.length === 0) {
|
|
411
|
+
logger.info('No CSV files found. Exiting.');
|
|
412
|
+
return {
|
|
413
|
+
filesProcessed: 0,
|
|
414
|
+
filesSucceeded: 0,
|
|
415
|
+
filesFailed: 0,
|
|
416
|
+
totalRecordsProcessed: 0,
|
|
417
|
+
totalRecordsFailed: 0,
|
|
418
|
+
};
|
|
419
|
+
}
|
|
420
|
+
|
|
421
|
+
// ───────────────────────────────────────────────────────────────────────────
|
|
422
|
+
// STEP 3: Process Each CSV File
|
|
423
|
+
// ───────────────────────────────────────────────────────────────────────────
|
|
424
|
+
|
|
425
|
+
let filesSucceeded = 0;
|
|
426
|
+
let filesFailed = 0;
|
|
427
|
+
let totalRecordsProcessed = 0;
|
|
428
|
+
let totalRecordsFailed = 0;
|
|
429
|
+
|
|
430
|
+
for (const file of csvFiles) {
|
|
431
|
+
logger.info('-'.repeat(80));
|
|
432
|
+
logger.info(`Processing file: ${file.name}`, {
|
|
433
|
+
size: file.size,
|
|
434
|
+
lastModified: file.lastModified,
|
|
435
|
+
});
|
|
436
|
+
|
|
437
|
+
try {
|
|
438
|
+
// ─────────────────────────────────────────────────────────────────────
|
|
439
|
+
// STEP 3a: Download CSV File
|
|
440
|
+
// ─────────────────────────────────────────────────────────────────────
|
|
441
|
+
logger.info(`Downloading file: ${file.path}`);
|
|
442
|
+
const csvContent = await s3Source.downloadFile(file.path);
|
|
443
|
+
logger.info(
|
|
444
|
+
`Downloaded ${typeof csvContent === 'string' ? csvContent.length : csvContent.length} bytes`
|
|
445
|
+
);
|
|
446
|
+
|
|
447
|
+
// ─────────────────────────────────────────────────────────────────────
|
|
448
|
+
// STEP 3b: Parse CSV to Records
|
|
449
|
+
// ─────────────────────────────────────────────────────────────────────
|
|
450
|
+
logger.info('Parsing CSV content...');
|
|
451
|
+
const records = await csvParser.parse(csvContent as string);
|
|
452
|
+
logger.info(`Parsed ${records.length} records from CSV`);
|
|
453
|
+
|
|
454
|
+
if (records.length === 0) {
|
|
455
|
+
logger.warn('CSV file is empty, skipping');
|
|
456
|
+
continue;
|
|
457
|
+
}
|
|
458
|
+
|
|
459
|
+
// ─────────────────────────────────────────────────────────────────────
|
|
460
|
+
// STEP 3c: Validate Records (Basic Validation)
|
|
461
|
+
// ─────────────────────────────────────────────────────────────────────
|
|
462
|
+
logger.info('Validating records...');
|
|
463
|
+
const validRecords = records.filter((record: any, index: number) => {
|
|
464
|
+
// Check for required fields
|
|
465
|
+
if (!record.sku || !record.location || record.quantity === undefined) {
|
|
466
|
+
logger.warn(`Record ${index + 1} missing required fields, skipping`, {
|
|
467
|
+
record,
|
|
468
|
+
});
|
|
469
|
+
totalRecordsFailed++;
|
|
470
|
+
return false;
|
|
471
|
+
}
|
|
472
|
+
|
|
473
|
+
// Validate quantity is numeric
|
|
474
|
+
if (isNaN(Number(record.quantity))) {
|
|
475
|
+
logger.warn(`Record ${index + 1} has invalid quantity, skipping`, {
|
|
476
|
+
record,
|
|
477
|
+
});
|
|
478
|
+
totalRecordsFailed++;
|
|
479
|
+
return false;
|
|
480
|
+
}
|
|
481
|
+
|
|
482
|
+
return true;
|
|
483
|
+
});
|
|
484
|
+
|
|
485
|
+
logger.info(
|
|
486
|
+
`Validation complete: ${validRecords.length} valid, ${records.length - validRecords.length} invalid`
|
|
487
|
+
);
|
|
488
|
+
|
|
489
|
+
if (validRecords.length === 0) {
|
|
490
|
+
logger.warn('No valid records found in file, skipping');
|
|
491
|
+
continue;
|
|
492
|
+
}
|
|
493
|
+
|
|
494
|
+
// ─────────────────────────────────────────────────────────────────────
|
|
495
|
+
// STEP 3d: Map Records Using UniversalMapper
|
|
496
|
+
// ─────────────────────────────────────────────────────────────────────
|
|
497
|
+
logger.info('Mapping records using UniversalMapper...');
|
|
498
|
+
const mappedRecords: any[] = [];
|
|
499
|
+
const mappingErrors: string[] = [];
|
|
500
|
+
|
|
501
|
+
for (let i = 0; i < validRecords.length; i++) {
|
|
502
|
+
const record = validRecords[i];
|
|
503
|
+
const result = await mapper.map(record);
|
|
504
|
+
|
|
505
|
+
if (result.success) {
|
|
506
|
+
mappedRecords.push(result.data);
|
|
507
|
+
} else {
|
|
508
|
+
logger.warn(`Record ${i + 1} mapping failed`, {
|
|
509
|
+
errors: result.errors,
|
|
510
|
+
record,
|
|
511
|
+
});
|
|
512
|
+
mappingErrors.push(`Record ${i + 1}: ${result.errors?.join(', ')}`);
|
|
513
|
+
totalRecordsFailed++;
|
|
514
|
+
}
|
|
515
|
+
}
|
|
516
|
+
|
|
517
|
+
logger.info(
|
|
518
|
+
`Mapping complete: ${mappedRecords.length} mapped, ${mappingErrors.length} failed`
|
|
519
|
+
);
|
|
520
|
+
|
|
521
|
+
if (mappedRecords.length === 0) {
|
|
522
|
+
logger.warn('No records successfully mapped, skipping file');
|
|
523
|
+
continue;
|
|
524
|
+
}
|
|
525
|
+
|
|
526
|
+
// ─────────────────────────────────────────────────────────────────────
|
|
527
|
+
// STEP 3e: Create Batch API Job
|
|
528
|
+
// ─────────────────────────────────────────────────────────────────────
|
|
529
|
+
const jobName = `${config.processing.jobNamePrefix} - ${file.name} - ${new Date().toISOString()}`;
|
|
530
|
+
logger.info(`Creating Batch API job: ${jobName}`);
|
|
531
|
+
|
|
532
|
+
const job = await fluentClient.createJob({
|
|
533
|
+
name: jobName,
|
|
534
|
+
retailerId: config.fluent.retailerId,
|
|
535
|
+
});
|
|
536
|
+
|
|
537
|
+
logger.info(`Job created successfully`, {
|
|
538
|
+
jobId: job.id,
|
|
539
|
+
status: job.status,
|
|
540
|
+
});
|
|
541
|
+
|
|
542
|
+
// ─────────────────────────────────────────────────────────────────────
|
|
543
|
+
// STEP 3f: Send Batches (Chunked by Batch Size)
|
|
544
|
+
// ─────────────────────────────────────────────────────────────────────
|
|
545
|
+
logger.info(
|
|
546
|
+
`Sending ${mappedRecords.length} records in batches of ${config.processing.batchSize}`
|
|
547
|
+
);
|
|
548
|
+
const batches = chunk(mappedRecords, config.processing.batchSize);
|
|
549
|
+
|
|
550
|
+
for (let i = 0; i < batches.length; i++) {
|
|
551
|
+
const batch = batches[i];
|
|
552
|
+
logger.info(`Sending batch ${i + 1}/${batches.length} (${batch.length} records)`);
|
|
553
|
+
|
|
554
|
+
const batchResponse = await fluentClient.sendBatch(job.id, {
|
|
555
|
+
entities: batch,
|
|
556
|
+
});
|
|
557
|
+
|
|
558
|
+
logger.info(`Batch ${i + 1} sent successfully`, {
|
|
559
|
+
batchId: batchResponse.id,
|
|
560
|
+
status: batchResponse.status,
|
|
561
|
+
});
|
|
562
|
+
}
|
|
563
|
+
|
|
564
|
+
// ─────────────────────────────────────────────────────────────────────
|
|
565
|
+
// STEP 3g: Poll Job Status Until Complete
|
|
566
|
+
// ─────────────────────────────────────────────────────────────────────
|
|
567
|
+
logger.info('Polling job status...');
|
|
568
|
+
const finalStatus = await pollJobStatus(fluentClient, job.id, logger);
|
|
569
|
+
|
|
570
|
+
logger.info(`Job completed with status: ${finalStatus.status}`, {
|
|
571
|
+
jobId: job.id,
|
|
572
|
+
status: finalStatus.status,
|
|
573
|
+
});
|
|
574
|
+
|
|
575
|
+
// Check if job succeeded
|
|
576
|
+
if (finalStatus.status === 'COMPLETE') {
|
|
577
|
+
totalRecordsProcessed += mappedRecords.length;
|
|
578
|
+
filesSucceeded++;
|
|
579
|
+
|
|
580
|
+
// ───────────────────────────────────────────────────────────────────
|
|
581
|
+
// STEP 3h: Archive Successful File
|
|
582
|
+
// ───────────────────────────────────────────────────────────────────
|
|
583
|
+
const archivePath = `${config.processing.archivePrefix}${new Date().toISOString().split('T')[0]}/${file.name}`;
|
|
584
|
+
logger.info(`Archiving file to: ${archivePath}`);
|
|
585
|
+
await s3Source.moveFile(file.path, archivePath);
|
|
586
|
+
logger.info('File archived successfully');
|
|
587
|
+
} else {
|
|
588
|
+
// Job failed or has errors
|
|
589
|
+
logger.error('Job failed or has errors', undefined, {
|
|
590
|
+
jobId: job.id,
|
|
591
|
+
status: finalStatus.status,
|
|
592
|
+
});
|
|
593
|
+
|
|
594
|
+
// Move to error folder
|
|
595
|
+
const errorPath = `${config.processing.errorPrefix}${new Date().toISOString().split('T')[0]}/${file.name}`;
|
|
596
|
+
logger.info(`Moving file to error folder: ${errorPath}`);
|
|
597
|
+
await s3Source.moveFile(file.path, errorPath);
|
|
598
|
+
logger.info('File moved to error folder');
|
|
599
|
+
|
|
600
|
+
filesFailed++;
|
|
601
|
+
}
|
|
602
|
+
} catch (error) {
|
|
603
|
+
// File processing error
|
|
604
|
+
logger.error(`Failed to process file: ${file.name}`, error as Error);
|
|
605
|
+
filesFailed++;
|
|
606
|
+
|
|
607
|
+
try {
|
|
608
|
+
// Move to error folder
|
|
609
|
+
const errorPath = `${config.processing.errorPrefix}${new Date().toISOString().split('T')[0]}/${file.name}`;
|
|
610
|
+
await s3Source.moveFile(file.path, errorPath);
|
|
611
|
+
logger.info('File moved to error folder after processing error');
|
|
612
|
+
} catch (moveError) {
|
|
613
|
+
logger.error('Failed to move file to error folder', moveError as Error);
|
|
614
|
+
}
|
|
615
|
+
}
|
|
616
|
+
}
|
|
617
|
+
|
|
618
|
+
// ───────────────────────────────────────────────────────────────────────────
|
|
619
|
+
// STEP 4: Summary Statistics
|
|
620
|
+
// ───────────────────────────────────────────────────────────────────────────
|
|
621
|
+
|
|
622
|
+
const duration = Date.now() - startTime;
|
|
623
|
+
logger.info('='.repeat(80));
|
|
624
|
+
logger.info('Inventory sync completed', {
|
|
625
|
+
duration: `${(duration / 1000).toFixed(2)}s`,
|
|
626
|
+
filesProcessed: csvFiles.length,
|
|
627
|
+
filesSucceeded,
|
|
628
|
+
filesFailed,
|
|
629
|
+
totalRecordsProcessed,
|
|
630
|
+
totalRecordsFailed,
|
|
631
|
+
});
|
|
632
|
+
logger.info('='.repeat(80));
|
|
633
|
+
|
|
634
|
+
return {
|
|
635
|
+
filesProcessed: csvFiles.length,
|
|
636
|
+
filesSucceeded,
|
|
637
|
+
filesFailed,
|
|
638
|
+
totalRecordsProcessed,
|
|
639
|
+
totalRecordsFailed,
|
|
640
|
+
};
|
|
641
|
+
}
|
|
642
|
+
|
|
643
|
+
// ============================================================================
|
|
644
|
+
// HELPER FUNCTIONS
|
|
645
|
+
// ============================================================================
|
|
646
|
+
|
|
647
|
+
/**
|
|
648
|
+
* Poll Batch API job status until completion
|
|
649
|
+
*
|
|
650
|
+
* POLLING STRATEGY:
|
|
651
|
+
* - Initial delay: 2 seconds
|
|
652
|
+
* - Max delay: 30 seconds
|
|
653
|
+
* - Exponential backoff with 1.5x multiplier
|
|
654
|
+
* - Timeout: 30 minutes
|
|
655
|
+
*
|
|
656
|
+
* TERMINAL STATES:
|
|
657
|
+
* - COMPLETE: Job succeeded
|
|
658
|
+
* - FAILED: Job failed (API error, validation errors, etc.)
|
|
659
|
+
* - Other statuses: Continue polling
|
|
660
|
+
*/
|
|
661
|
+
async function pollJobStatus(client: any, jobId: string, logger: any): Promise<any> {
|
|
662
|
+
let delay = 2000; // Start with 2 second delay
|
|
663
|
+
const maxDelay = 30000; // Cap at 30 seconds
|
|
664
|
+
const timeout = 30 * 60 * 1000; // 30 minutes
|
|
665
|
+
const startTime = Date.now();
|
|
666
|
+
|
|
667
|
+
while (true) {
|
|
668
|
+
// Check timeout
|
|
669
|
+
if (Date.now() - startTime > timeout) {
|
|
670
|
+
throw new Error('Job status polling timeout (30 minutes)');
|
|
671
|
+
}
|
|
672
|
+
|
|
673
|
+
// Get job status
|
|
674
|
+
const status = await client.getJobStatus(jobId);
|
|
675
|
+
logger.debug(`Job status: ${status.status}`, {
|
|
676
|
+
jobId,
|
|
677
|
+
status: status.status,
|
|
678
|
+
});
|
|
679
|
+
|
|
680
|
+
// Check terminal states
|
|
681
|
+
if (status.status === 'COMPLETE' || status.status === 'FAILED') {
|
|
682
|
+
return status;
|
|
683
|
+
}
|
|
684
|
+
|
|
685
|
+
// Wait before next poll
|
|
686
|
+
await sleep(delay);
|
|
687
|
+
|
|
688
|
+
// Increase delay with exponential backoff (1.5x multiplier)
|
|
689
|
+
delay = Math.min(delay * 1.5, maxDelay);
|
|
690
|
+
}
|
|
691
|
+
}
|
|
692
|
+
|
|
693
|
+
/**
|
|
694
|
+
* Sleep helper function
|
|
695
|
+
*/
|
|
696
|
+
function sleep(ms: number): Promise<void> {
|
|
697
|
+
return new Promise(resolve => setTimeout(resolve, ms));
|
|
698
|
+
}
|
|
699
|
+
|
|
700
|
+
/**
|
|
701
|
+
* Chunk array into smaller arrays
|
|
702
|
+
*/
|
|
703
|
+
function chunk<T>(array: T[], size: number): T[][] {
|
|
704
|
+
const chunks: T[][] = [];
|
|
705
|
+
for (let i = 0; i < array.length; i += size) {
|
|
706
|
+
chunks.push(array.slice(i, i + size));
|
|
707
|
+
}
|
|
708
|
+
return chunks;
|
|
709
|
+
}
|
|
710
|
+
|
|
711
|
+
// ============================================================================
|
|
712
|
+
// ENTRY POINT
|
|
713
|
+
// ============================================================================
|
|
714
|
+
|
|
715
|
+
/**
|
|
716
|
+
* Main entry point with error handling
|
|
717
|
+
*/
|
|
718
|
+
async function main() {
|
|
719
|
+
try {
|
|
720
|
+
const result = await syncInventory();
|
|
721
|
+
|
|
722
|
+
// Exit with success code if no files failed
|
|
723
|
+
if (result.filesFailed === 0) {
|
|
724
|
+
logger.info('All files processed successfully');
|
|
725
|
+
process.exit(0);
|
|
726
|
+
} else {
|
|
727
|
+
logger.warn('Some files failed to process');
|
|
728
|
+
process.exit(1);
|
|
729
|
+
}
|
|
730
|
+
} catch (error) {
|
|
731
|
+
logger.error('Fatal error during inventory sync', error as Error);
|
|
732
|
+
process.exit(1);
|
|
733
|
+
}
|
|
734
|
+
}
|
|
735
|
+
|
|
736
|
+
// Run the sync
|
|
737
|
+
main();
|
|
738
|
+
```
|
|
739
|
+
|
|
740
|
+
### 4. Package.json Configuration
|
|
741
|
+
|
|
742
|
+
```json
|
|
743
|
+
{
|
|
744
|
+
"name": "inventory-sync-standalone",
|
|
745
|
+
"version": "1.0.0",
|
|
746
|
+
"description": "Standalone S3 CSV to Fluent Commerce inventory sync",
|
|
747
|
+
"main": "dist/inventory-sync.js",
|
|
748
|
+
"type": "module",
|
|
749
|
+
"scripts": {
|
|
750
|
+
"build": "tsc",
|
|
751
|
+
"start": "node dist/inventory-sync.js",
|
|
752
|
+
"dev": "tsx src/inventory-sync.ts",
|
|
753
|
+
"sync": "npm run dev",
|
|
754
|
+
"lint": "eslint src --ext .ts",
|
|
755
|
+
"type-check": "tsc --noEmit"
|
|
756
|
+
},
|
|
757
|
+
"keywords": ["fluent-commerce", "inventory", "s3", "batch-api"],
|
|
758
|
+
"author": "",
|
|
759
|
+
"license": "MIT",
|
|
760
|
+
"dependencies": {
|
|
761
|
+
"@fluentcommerce/fc-connect-sdk": "^0.1.39",
|
|
762
|
+
"dotenv": "^16.0.0"
|
|
763
|
+
},
|
|
764
|
+
"devDependencies": {
|
|
765
|
+
"@types/node": "^18.0.0",
|
|
766
|
+
"tsx": "^4.0.0",
|
|
767
|
+
"typescript": "^5.0.0"
|
|
768
|
+
}
|
|
769
|
+
}
|
|
770
|
+
```
|
|
771
|
+
|
|
772
|
+
### 5. TypeScript Configuration
|
|
773
|
+
|
|
774
|
+
Create `tsconfig.json`:
|
|
775
|
+
|
|
776
|
+
```json
|
|
777
|
+
{
|
|
778
|
+
"compilerOptions": {
|
|
779
|
+
"module": "ES2022",
|
|
780
|
+
"target": "ES2024",
|
|
781
|
+
"moduleResolution": "node"
|
|
782
|
+
}
|
|
783
|
+
}
|
|
784
|
+
```
|
|
785
|
+
|
|
786
|
+
---
|
|
787
|
+
|
|
788
|
+
## Key Patterns Explained
|
|
789
|
+
|
|
790
|
+
### Pattern 1: OAuth2 Authentication
|
|
791
|
+
|
|
792
|
+
**Purpose**: Authenticate with Fluent Commerce without Versori connection
|
|
793
|
+
|
|
794
|
+
**Implementation**:
|
|
795
|
+
|
|
796
|
+
```typescript
|
|
797
|
+
const fluentClient = await createClient({
|
|
798
|
+
config: {
|
|
799
|
+
baseUrl: 'https://api.fluentcommerce.com',
|
|
800
|
+
clientId: process.env.FLUENT_CLIENT_ID,
|
|
801
|
+
clientSecret: process.env.FLUENT_CLIENT_SECRET,
|
|
802
|
+
retailerId: process.env.FLUENT_RETAILER_ID,
|
|
803
|
+
},
|
|
804
|
+
});
|
|
805
|
+
```
|
|
806
|
+
|
|
807
|
+
**Key Points**:
|
|
808
|
+
|
|
809
|
+
- Uses client credentials OAuth2 grant type
|
|
810
|
+
- Token automatically cached and refreshed by SDK
|
|
811
|
+
- No manual token management needed
|
|
812
|
+
- Token expires after ~1 hour, SDK handles refresh transparently
|
|
813
|
+
|
|
814
|
+
**When to Use**:
|
|
815
|
+
|
|
816
|
+
- Standalone Node.js/Deno scripts
|
|
817
|
+
- Background jobs/cron tasks
|
|
818
|
+
- CI/CD pipelines
|
|
819
|
+
- Local development
|
|
820
|
+
|
|
821
|
+
### Pattern 2: S3 File Operations
|
|
822
|
+
|
|
823
|
+
**Purpose**: List, download, and move files in S3 bucket
|
|
824
|
+
|
|
825
|
+
**Implementation**:
|
|
826
|
+
|
|
827
|
+
```typescript
|
|
828
|
+
// Initialize S3 Data Source
|
|
829
|
+
const s3Source = new S3DataSource(
|
|
830
|
+
{
|
|
831
|
+
type: 'S3_CSV',
|
|
832
|
+
connectionId: 'inventory-s3',
|
|
833
|
+
name: 'Inventory S3 Source',
|
|
834
|
+
s3Config: {
|
|
835
|
+
bucket: 'my-bucket',
|
|
836
|
+
region: 'us-east-1',
|
|
837
|
+
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
|
838
|
+
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
|
|
839
|
+
},
|
|
840
|
+
},
|
|
841
|
+
logger
|
|
842
|
+
);
|
|
843
|
+
|
|
844
|
+
// List files with prefix filter
|
|
845
|
+
const files = await s3Source.listFiles({
|
|
846
|
+
prefix: 'inventory/updates/',
|
|
847
|
+
maxKeys: 1000,
|
|
848
|
+
});
|
|
849
|
+
|
|
850
|
+
// Download file content
|
|
851
|
+
const content = await s3Source.downloadFile(file.path);
|
|
852
|
+
|
|
853
|
+
// Move file (copy + delete)
|
|
854
|
+
await s3Source.moveFile(sourcePath, destPath);
|
|
855
|
+
```
|
|
856
|
+
|
|
857
|
+
**Key Points**:
|
|
858
|
+
|
|
859
|
+
- Uses presigned URLs (no AWS SDK required)
|
|
860
|
+
- Automatic retry logic for transient failures
|
|
861
|
+
- Streaming support for large files
|
|
862
|
+
- Cross-bucket move support
|
|
863
|
+
|
|
864
|
+
**Best Practices**:
|
|
865
|
+
|
|
866
|
+
- Always use prefix filtering to limit results
|
|
867
|
+
- Move files to archive/error folders (don't delete)
|
|
868
|
+
- Use date-based folder structure for archiving
|
|
869
|
+
- Validate S3 connection before processing
|
|
870
|
+
|
|
871
|
+
### Pattern 3: CSV Parsing & Validation
|
|
872
|
+
|
|
873
|
+
**Purpose**: Parse CSV files and validate data quality
|
|
874
|
+
|
|
875
|
+
**Implementation**:
|
|
876
|
+
|
|
877
|
+
```typescript
|
|
878
|
+
// Initialize CSV Parser
|
|
879
|
+
const csvParser = new CSVParserService();
|
|
880
|
+
|
|
881
|
+
// Parse CSV content (returns array of objects)
|
|
882
|
+
const records = await csvParser.parse(csvContent);
|
|
883
|
+
|
|
884
|
+
// Validate records
|
|
885
|
+
const validRecords = records.filter(record => {
|
|
886
|
+
// Required field validation
|
|
887
|
+
if (!record.sku || !record.location || record.quantity === undefined) {
|
|
888
|
+
logger.warn('Missing required fields', { record });
|
|
889
|
+
return false;
|
|
890
|
+
}
|
|
891
|
+
|
|
892
|
+
// Type validation
|
|
893
|
+
if (isNaN(Number(record.quantity))) {
|
|
894
|
+
logger.warn('Invalid quantity', { record });
|
|
895
|
+
return false;
|
|
896
|
+
}
|
|
897
|
+
|
|
898
|
+
return true;
|
|
899
|
+
});
|
|
900
|
+
```
|
|
901
|
+
|
|
902
|
+
**Key Points**:
|
|
903
|
+
|
|
904
|
+
- CSV parser auto-detects headers by default
|
|
905
|
+
- Trims whitespace and skips empty lines
|
|
906
|
+
- Returns array of objects (header → value mapping)
|
|
907
|
+
- Validation happens before mapping (fail fast)
|
|
908
|
+
|
|
909
|
+
**Validation Strategies**:
|
|
910
|
+
|
|
911
|
+
1. **Required Fields**: Check for null/undefined/empty
|
|
912
|
+
2. **Type Validation**: Ensure numeric fields are numbers
|
|
913
|
+
3. **Format Validation**: Regex for SKU, email, phone, etc.
|
|
914
|
+
4. **Business Rules**: Range checks, valid enum values
|
|
915
|
+
5. **Cross-field Validation**: Relationships between fields
|
|
916
|
+
|
|
917
|
+
### Pattern 4: Batch API Workflow
|
|
918
|
+
|
|
919
|
+
**Purpose**: Send large datasets to Fluent Commerce efficiently
|
|
920
|
+
|
|
921
|
+
**Implementation**:
|
|
922
|
+
|
|
923
|
+
```typescript
|
|
924
|
+
// STEP 1: Create Job
|
|
925
|
+
const job = await fluentClient.createJob({
|
|
926
|
+
name: 'Inventory Update - file.csv',
|
|
927
|
+
retailerId: 'my-retailer',
|
|
928
|
+
});
|
|
929
|
+
|
|
930
|
+
// STEP 2: Send Batches (chunked)
|
|
931
|
+
const batchSize = 100;
|
|
932
|
+
for (let i = 0; i < records.length; i += batchSize) {
|
|
933
|
+
const batch = records.slice(i, i + batchSize);
|
|
934
|
+
await fluentClient.sendBatch(job.id, {
|
|
935
|
+
entities: batch,
|
|
936
|
+
});
|
|
937
|
+
}
|
|
938
|
+
|
|
939
|
+
// STEP 3: Poll Job Status
|
|
940
|
+
let status = await fluentClient.getJobStatus(job.id);
|
|
941
|
+
while (status.status !== 'COMPLETE' && status.status !== 'FAILED') {
|
|
942
|
+
await sleep(5000); // Wait 5 seconds
|
|
943
|
+
status = await fluentClient.getJobStatus(job.id);
|
|
944
|
+
}
|
|
945
|
+
|
|
946
|
+
// STEP 4: Check Results
|
|
947
|
+
if (status.status === 'COMPLETE') {
|
|
948
|
+
console.log('Job succeeded!');
|
|
949
|
+
} else {
|
|
950
|
+
console.error('Job failed:', status);
|
|
951
|
+
}
|
|
952
|
+
```
|
|
953
|
+
|
|
954
|
+
**Key Points**:
|
|
955
|
+
|
|
956
|
+
- One job per file (easier tracking)
|
|
957
|
+
- Batch size 50-500 records (100 recommended)
|
|
958
|
+
- Poll with exponential backoff
|
|
959
|
+
- Handle both COMPLETE and FAILED states
|
|
960
|
+
|
|
961
|
+
**Batch Size Guidelines**:
|
|
962
|
+
|
|
963
|
+
- Small records (<10 fields): 200-500 per batch
|
|
964
|
+
- Medium records (10-30 fields): 100-200 per batch
|
|
965
|
+
- Large records (>30 fields): 50-100 per batch
|
|
966
|
+
- API limit: Usually 1000 records per batch
|
|
967
|
+
|
|
968
|
+
### Pattern 5: Error Handling & Retry
|
|
969
|
+
|
|
970
|
+
**Purpose**: Graceful degradation and recovery from failures
|
|
971
|
+
|
|
972
|
+
**Implementation**:
|
|
973
|
+
|
|
974
|
+
```typescript
|
|
975
|
+
// File-level try-catch
|
|
976
|
+
for (const file of files) {
|
|
977
|
+
try {
|
|
978
|
+
await processFile(file);
|
|
979
|
+
filesSucceeded++;
|
|
980
|
+
} catch (error) {
|
|
981
|
+
logger.error('File processing failed', error);
|
|
982
|
+
filesFailed++;
|
|
983
|
+
|
|
984
|
+
// Move to error folder
|
|
985
|
+
try {
|
|
986
|
+
await s3Source.moveFile(file.path, errorPath);
|
|
987
|
+
} catch (moveError) {
|
|
988
|
+
logger.error('Failed to move error file', moveError);
|
|
989
|
+
}
|
|
990
|
+
}
|
|
991
|
+
}
|
|
992
|
+
|
|
993
|
+
// Automatic retry with exponential backoff
|
|
994
|
+
async function retryWithBackoff<T>(fn: () => Promise<T>, maxRetries: number = 3): Promise<T> {
|
|
995
|
+
let lastError: Error;
|
|
996
|
+
for (let attempt = 0; attempt < maxRetries; attempt++) {
|
|
997
|
+
try {
|
|
998
|
+
return await fn();
|
|
999
|
+
} catch (error) {
|
|
1000
|
+
lastError = error as Error;
|
|
1001
|
+
if (attempt < maxRetries - 1) {
|
|
1002
|
+
const delay = Math.pow(2, attempt) * 1000;
|
|
1003
|
+
await sleep(delay);
|
|
1004
|
+
}
|
|
1005
|
+
}
|
|
1006
|
+
}
|
|
1007
|
+
throw lastError!;
|
|
1008
|
+
}
|
|
1009
|
+
```
|
|
1010
|
+
|
|
1011
|
+
**Error Categories**:
|
|
1012
|
+
|
|
1013
|
+
1. **Network Errors**: Retry with exponential backoff
|
|
1014
|
+
2. **4xx Client Errors**: Don't retry (bad request)
|
|
1015
|
+
3. **5xx Server Errors**: Retry (transient failure)
|
|
1016
|
+
4. **Validation Errors**: Skip record, continue processing
|
|
1017
|
+
5. **Fatal Errors**: Exit with error code
|
|
1018
|
+
|
|
1019
|
+
**Retry Best Practices**:
|
|
1020
|
+
|
|
1021
|
+
- Max 3-5 retry attempts
|
|
1022
|
+
- Exponential backoff (1s, 2s, 4s, 8s, ...)
|
|
1023
|
+
- Jitter to prevent thundering herd
|
|
1024
|
+
- Log each retry attempt
|
|
1025
|
+
- Give up after max retries
|
|
1026
|
+
|
|
1027
|
+
---
|
|
1028
|
+
|
|
1029
|
+
## Deployment Options
|
|
1030
|
+
|
|
1031
|
+
### Option 1: Local Execution
|
|
1032
|
+
|
|
1033
|
+
```bash
|
|
1034
|
+
# One-time sync
|
|
1035
|
+
npm run sync
|
|
1036
|
+
|
|
1037
|
+
# With debug logging
|
|
1038
|
+
LOG_LEVEL=debug npm run sync
|
|
1039
|
+
```
|
|
1040
|
+
|
|
1041
|
+
### Option 2: Docker Container
|
|
1042
|
+
|
|
1043
|
+
Create `Dockerfile`:
|
|
1044
|
+
|
|
1045
|
+
```dockerfile
|
|
1046
|
+
FROM node:18-alpine
|
|
1047
|
+
WORKDIR /app
|
|
1048
|
+
|
|
1049
|
+
# Copy package files
|
|
1050
|
+
COPY package*.json ./
|
|
1051
|
+
|
|
1052
|
+
# Install dependencies
|
|
1053
|
+
RUN npm ci --only=production
|
|
1054
|
+
|
|
1055
|
+
# Copy application code
|
|
1056
|
+
COPY dist ./dist
|
|
1057
|
+
|
|
1058
|
+
# Run the sync
|
|
1059
|
+
CMD ["node", "dist/inventory-sync.js"]
|
|
1060
|
+
```
|
|
1061
|
+
|
|
1062
|
+
Build and run:
|
|
1063
|
+
|
|
1064
|
+
```bash
|
|
1065
|
+
# Build image
|
|
1066
|
+
docker build -t inventory-sync .
|
|
1067
|
+
|
|
1068
|
+
# Run container
|
|
1069
|
+
docker run --env-file .env inventory-sync
|
|
1070
|
+
```
|
|
1071
|
+
|
|
1072
|
+
### Option 3: AWS Lambda
|
|
1073
|
+
|
|
1074
|
+
Create `lambda-handler.ts`:
|
|
1075
|
+
|
|
1076
|
+
```typescript
|
|
1077
|
+
import { syncInventory } from './inventory-sync';
|
|
1078
|
+
|
|
1079
|
+
export const handler = async (event: any, context: any) => {
|
|
1080
|
+
try {
|
|
1081
|
+
const result = await syncInventory();
|
|
1082
|
+
return {
|
|
1083
|
+
statusCode: 200,
|
|
1084
|
+
body: JSON.stringify(result),
|
|
1085
|
+
};
|
|
1086
|
+
} catch (error) {
|
|
1087
|
+
console.error('Lambda error:', error);
|
|
1088
|
+
return {
|
|
1089
|
+
statusCode: 500,
|
|
1090
|
+
body: JSON.stringify({ error: (error as Error).message }),
|
|
1091
|
+
};
|
|
1092
|
+
}
|
|
1093
|
+
};
|
|
1094
|
+
```
|
|
1095
|
+
|
|
1096
|
+
Deploy:
|
|
1097
|
+
|
|
1098
|
+
```bash
|
|
1099
|
+
# Package for Lambda
|
|
1100
|
+
npm run build
|
|
1101
|
+
zip -r function.zip dist node_modules
|
|
1102
|
+
|
|
1103
|
+
# Deploy with AWS CLI
|
|
1104
|
+
aws lambda create-function \
|
|
1105
|
+
--function-name inventory-sync \
|
|
1106
|
+
--runtime nodejs18.x \
|
|
1107
|
+
--handler dist/lambda-handler.handler \
|
|
1108
|
+
--zip-file fileb://function.zip \
|
|
1109
|
+
--role arn:aws:iam::123456789:role/lambda-execution \
|
|
1110
|
+
--timeout 900 \
|
|
1111
|
+
--memory-size 512
|
|
1112
|
+
```
|
|
1113
|
+
|
|
1114
|
+
### Option 4: Cron Scheduling
|
|
1115
|
+
|
|
1116
|
+
**Linux/Mac cron**:
|
|
1117
|
+
|
|
1118
|
+
```bash
|
|
1119
|
+
# Edit crontab
|
|
1120
|
+
crontab -e
|
|
1121
|
+
|
|
1122
|
+
# Run every 4 hours
|
|
1123
|
+
0 */4 * * * cd /path/to/inventory-sync && npm run sync >> /var/log/inventory-sync.log 2>&1
|
|
1124
|
+
```
|
|
1125
|
+
|
|
1126
|
+
**Node.js cron**:
|
|
1127
|
+
|
|
1128
|
+
Install: `npm install node-cron`
|
|
1129
|
+
|
|
1130
|
+
```typescript
|
|
1131
|
+
import cron from 'node-cron';
|
|
1132
|
+
|
|
1133
|
+
// Run every 4 hours
|
|
1134
|
+
cron.schedule('0 */4 * * *', async () => {
|
|
1135
|
+
logger.info('Starting scheduled inventory sync');
|
|
1136
|
+
await syncInventory();
|
|
1137
|
+
});
|
|
1138
|
+
|
|
1139
|
+
logger.info('Cron scheduler started (every 4 hours)');
|
|
1140
|
+
```
|
|
1141
|
+
|
|
1142
|
+
**AWS EventBridge**:
|
|
1143
|
+
|
|
1144
|
+
```bash
|
|
1145
|
+
# Create rule
|
|
1146
|
+
aws events put-rule \
|
|
1147
|
+
--name inventory-sync-schedule \
|
|
1148
|
+
--schedule-expression "rate(4 hours)"
|
|
1149
|
+
|
|
1150
|
+
# Add Lambda target
|
|
1151
|
+
aws events put-targets \
|
|
1152
|
+
--rule inventory-sync-schedule \
|
|
1153
|
+
--targets "Id"="1","Arn"="arn:aws:lambda:us-east-1:123:function:inventory-sync"
|
|
1154
|
+
```
|
|
1155
|
+
|
|
1156
|
+
---
|
|
1157
|
+
|
|
1158
|
+
## Testing
|
|
1159
|
+
|
|
1160
|
+
### Local Testing
|
|
1161
|
+
|
|
1162
|
+
```bash
|
|
1163
|
+
# 1. Create test CSV file
|
|
1164
|
+
cat > test-inventory.csv << EOF
|
|
1165
|
+
sku,location,quantity,status,expected_date
|
|
1166
|
+
TEST-SKU-001,WAREHOUSE-A,100,ACTIVE,2024-01-15
|
|
1167
|
+
TEST-SKU-002,WAREHOUSE-B,50,ACTIVE,2024-01-15
|
|
1168
|
+
TEST-SKU-003,STORE-001,25,ACTIVE,2024-01-16
|
|
1169
|
+
EOF
|
|
1170
|
+
|
|
1171
|
+
# 2. Upload to S3
|
|
1172
|
+
aws s3 cp test-inventory.csv s3://your-bucket/inventory/updates/
|
|
1173
|
+
|
|
1174
|
+
# 3. Run sync
|
|
1175
|
+
npm run sync
|
|
1176
|
+
|
|
1177
|
+
# 4. Check logs
|
|
1178
|
+
cat logs/inventory-sync.log
|
|
1179
|
+
|
|
1180
|
+
# 5. Verify in Fluent Commerce
|
|
1181
|
+
# - Check job status in admin portal
|
|
1182
|
+
# - Query inventory positions via GraphQL
|
|
1183
|
+
```
|
|
1184
|
+
|
|
1185
|
+
### Integration Testing
|
|
1186
|
+
|
|
1187
|
+
Create `tests/integration.test.ts`:
|
|
1188
|
+
|
|
1189
|
+
```typescript
|
|
1190
|
+
import { describe, it, expect, beforeAll } from '@jest/globals';
|
|
1191
|
+
import { syncInventory } from '../src/inventory-sync';
|
|
1192
|
+
|
|
1193
|
+
describe('Inventory Sync Integration', () => {
|
|
1194
|
+
beforeAll(async () => {
|
|
1195
|
+
// Setup test environment
|
|
1196
|
+
});
|
|
1197
|
+
|
|
1198
|
+
it('should process valid CSV file', async () => {
|
|
1199
|
+
// Upload test file to S3
|
|
1200
|
+
// Run sync
|
|
1201
|
+
const result = await syncInventory();
|
|
1202
|
+
|
|
1203
|
+
// Assertions
|
|
1204
|
+
expect(result.filesSucceeded).toBe(1);
|
|
1205
|
+
expect(result.filesFailed).toBe(0);
|
|
1206
|
+
expect(result.totalRecordsProcessed).toBeGreaterThan(0);
|
|
1207
|
+
});
|
|
1208
|
+
|
|
1209
|
+
it('should handle invalid records gracefully', async () => {
|
|
1210
|
+
// Upload CSV with invalid records
|
|
1211
|
+
// Run sync
|
|
1212
|
+
const result = await syncInventory();
|
|
1213
|
+
|
|
1214
|
+
// Should still succeed but skip invalid records
|
|
1215
|
+
expect(result.filesSucceeded).toBe(1);
|
|
1216
|
+
expect(result.totalRecordsFailed).toBeGreaterThan(0);
|
|
1217
|
+
});
|
|
1218
|
+
|
|
1219
|
+
it('should move failed files to error folder', async () => {
|
|
1220
|
+
// Upload malformed CSV
|
|
1221
|
+
// Run sync
|
|
1222
|
+
const result = await syncInventory();
|
|
1223
|
+
|
|
1224
|
+
// File should be in error folder
|
|
1225
|
+
expect(result.filesFailed).toBe(1);
|
|
1226
|
+
// Check S3 error folder
|
|
1227
|
+
});
|
|
1228
|
+
});
|
|
1229
|
+
```
|
|
1230
|
+
|
|
1231
|
+
Run tests:
|
|
1232
|
+
|
|
1233
|
+
```bash
|
|
1234
|
+
npm install --save-dev jest @jest/globals @types/jest ts-jest
|
|
1235
|
+
npm test
|
|
1236
|
+
```
|
|
1237
|
+
|
|
1238
|
+
---
|
|
1239
|
+
|
|
1240
|
+
## Common Issues
|
|
1241
|
+
|
|
1242
|
+
### Issue 1: OAuth2 Token Expired
|
|
1243
|
+
|
|
1244
|
+
**Symptom**: `401 Unauthorized` errors
|
|
1245
|
+
|
|
1246
|
+
**Cause**: Token expired and SDK failed to refresh
|
|
1247
|
+
|
|
1248
|
+
**Solution**:
|
|
1249
|
+
|
|
1250
|
+
```typescript
|
|
1251
|
+
// SDK handles refresh automatically, but you can force refresh
|
|
1252
|
+
const client = await createClient({
|
|
1253
|
+
config: {
|
|
1254
|
+
baseUrl: 'https://api.fluentcommerce.com',
|
|
1255
|
+
clientId: process.env.FLUENT_CLIENT_ID,
|
|
1256
|
+
clientSecret: process.env.FLUENT_CLIENT_SECRET,
|
|
1257
|
+
retailerId: process.env.FLUENT_RETAILER_ID,
|
|
1258
|
+
retryAttempts: 3,
|
|
1259
|
+
},
|
|
1260
|
+
});
|
|
1261
|
+
```
|
|
1262
|
+
|
|
1263
|
+
### Issue 2: S3 Permission Denied
|
|
1264
|
+
|
|
1265
|
+
**Symptom**: `AccessDenied` errors when listing/downloading files
|
|
1266
|
+
|
|
1267
|
+
**Cause**: IAM policy missing required permissions
|
|
1268
|
+
|
|
1269
|
+
**Solution**:
|
|
1270
|
+
|
|
1271
|
+
```json
|
|
1272
|
+
{
|
|
1273
|
+
"Version": "2012-10-17",
|
|
1274
|
+
"Statement": [
|
|
1275
|
+
{
|
|
1276
|
+
"Effect": "Allow",
|
|
1277
|
+
"Action": ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
|
|
1278
|
+
"Resource": ["arn:aws:s3:::your-bucket", "arn:aws:s3:::your-bucket/*"]
|
|
1279
|
+
}
|
|
1280
|
+
]
|
|
1281
|
+
}
|
|
1282
|
+
```
|
|
1283
|
+
|
|
1284
|
+
### Issue 3: CSV Parsing Errors
|
|
1285
|
+
|
|
1286
|
+
**Symptom**: `CSV parse error: Invalid record`
|
|
1287
|
+
|
|
1288
|
+
**Cause**: Malformed CSV (unescaped quotes, inconsistent columns)
|
|
1289
|
+
|
|
1290
|
+
**Solution**:
|
|
1291
|
+
|
|
1292
|
+
```typescript
|
|
1293
|
+
// Use more lenient CSV parser options
|
|
1294
|
+
const csvParser = CSVParserService.create({
|
|
1295
|
+
columns: true,
|
|
1296
|
+
skip_empty_lines: true,
|
|
1297
|
+
trim: true,
|
|
1298
|
+
relax_column_count: true, // Allow inconsistent column counts
|
|
1299
|
+
quote: '"',
|
|
1300
|
+
escape: '"',
|
|
1301
|
+
delimiter: ',',
|
|
1302
|
+
});
|
|
1303
|
+
```
|
|
1304
|
+
|
|
1305
|
+
### Issue 4: Batch API Timeout
|
|
1306
|
+
|
|
1307
|
+
**Symptom**: Job stays in `PENDING` state indefinitely
|
|
1308
|
+
|
|
1309
|
+
**Cause**: Fluent API processing delay or large batch size
|
|
1310
|
+
|
|
1311
|
+
**Solution**:
|
|
1312
|
+
|
|
1313
|
+
```typescript
|
|
1314
|
+
// Reduce batch size
|
|
1315
|
+
const batchSize = 50; // Instead of 100
|
|
1316
|
+
|
|
1317
|
+
// Increase polling timeout
|
|
1318
|
+
const timeout = 60 * 60 * 1000; // 1 hour instead of 30 minutes
|
|
1319
|
+
|
|
1320
|
+
// Add more aggressive polling
|
|
1321
|
+
let delay = 1000; // Start with 1 second
|
|
1322
|
+
const maxDelay = 15000; // Cap at 15 seconds
|
|
1323
|
+
```
|
|
1324
|
+
|
|
1325
|
+
### Issue 5: Memory Exhaustion
|
|
1326
|
+
|
|
1327
|
+
**Symptom**: `JavaScript heap out of memory` error
|
|
1328
|
+
|
|
1329
|
+
**Cause**: Loading entire large CSV file into memory
|
|
1330
|
+
|
|
1331
|
+
**Solution**:
|
|
1332
|
+
|
|
1333
|
+
```typescript
|
|
1334
|
+
// Use streaming CSV parsing instead
|
|
1335
|
+
for await (const record of csvParser.parseStreaming(csvContent, {}, 100)) {
|
|
1336
|
+
// Process records in batches of 100
|
|
1337
|
+
const mappedBatch = await Promise.all(record.map(r => mapper.map(r)));
|
|
1338
|
+
|
|
1339
|
+
// Send batch immediately (don't accumulate)
|
|
1340
|
+
await fluentClient.sendBatch(jobId, {
|
|
1341
|
+
entities: mappedBatch.filter(r => r.success).map(r => r.data),
|
|
1342
|
+
});
|
|
1343
|
+
}
|
|
1344
|
+
```
|
|
1345
|
+
|
|
1346
|
+
---
|
|
1347
|
+
|
|
1348
|
+
## Related Guides
|
|
1349
|
+
|
|
1350
|
+
- **SDK Reference**: `../../readme.md` - Core SDK documentation
|
|
1351
|
+
- **Universal Mapping Guide**: `../../02-CORE-GUIDES/mapping/readme.md` - Field mapping patterns
|
|
1352
|
+
- **Batch API Guide**: `../../02-CORE-GUIDES/api-reference/modules/03-batch-processing.md` - Batch API details
|
|
1353
|
+
- **S3 Data Source**: `../../02-CORE-GUIDES/data-sources/readme.md` - S3 operations
|
|
1354
|
+
- **CSV Parser**: `../../02-CORE-GUIDES/parsers/readme.md` - CSV parsing options
|
|
1355
|
+
- **Error Handling**: `../../03-PATTERN-GUIDES/error-handling/readme.md` - Error patterns
|
|
1356
|
+
|
|
1357
|
+
---
|
|
1358
|
+
|
|
1359
|
+
## Production Checklist
|
|
1360
|
+
|
|
1361
|
+
Before deploying to production:
|
|
1362
|
+
|
|
1363
|
+
- [ ] Environment variables secured (AWS Secrets Manager, vault, etc.)
|
|
1364
|
+
- [ ] Logging configured (Winston, CloudWatch, Datadog, etc.)
|
|
1365
|
+
- [ ] Error alerting setup (PagerDuty, Slack, email)
|
|
1366
|
+
- [ ] Monitoring dashboard (metrics, job success rate, duration)
|
|
1367
|
+
- [ ] Retry logic tested (network failures, API errors)
|
|
1368
|
+
- [ ] Batch size optimized (tested with production file sizes)
|
|
1369
|
+
- [ ] Memory profiling done (no leaks)
|
|
1370
|
+
- [ ] S3 lifecycle policies configured (archive old files)
|
|
1371
|
+
- [ ] IAM permissions minimized (least privilege)
|
|
1372
|
+
- [ ] Health check endpoint added (for orchestration tools)
|
|
1373
|
+
- [ ] Documentation updated (runbook, troubleshooting)
|
|
1374
|
+
- [ ] Disaster recovery tested (restore from archive)
|
|
1375
|
+
|
|
1376
|
+
---
|
|
1377
|
+
|
|
1378
|
+
**Next Steps**:
|
|
1379
|
+
|
|
1380
|
+
1. Customize field mappings for your inventory data structure
|
|
1381
|
+
2. Add custom validation rules for your business logic
|
|
1382
|
+
3. Implement state tracking to prevent duplicate processing
|
|
1383
|
+
4. Set up monitoring and alerting
|
|
1384
|
+
5. Deploy to your preferred environment (Lambda, ECS, K8s, etc.)
|
|
1385
|
+
|
|
1386
|
+
**Support**:
|
|
1387
|
+
|
|
1388
|
+
- SDK Issues: https://github.com/fluentcommerce/fc-connect-sdk/issues
|
|
1389
|
+
- Fluent Commerce API: https://docs.fluentcommerce.com
|
|
1390
|
+
- Community: https://community.fluentcommerce.com
|