@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +30 -2
- package/README.md +39 -0
- package/dist/cjs/auth/index.d.ts +3 -0
- package/dist/cjs/auth/index.js +13 -0
- package/dist/cjs/auth/profile-loader.d.ts +18 -0
- package/dist/cjs/auth/profile-loader.js +208 -0
- package/dist/cjs/client-factory.d.ts +4 -0
- package/dist/cjs/client-factory.js +10 -0
- package/dist/cjs/clients/fluent-client.js +13 -6
- package/dist/cjs/index.d.ts +3 -1
- package/dist/cjs/index.js +8 -2
- package/dist/cjs/utils/pagination-helpers.js +38 -2
- package/dist/cjs/versori/fluent-versori-client.js +11 -5
- package/dist/esm/auth/index.d.ts +3 -0
- package/dist/esm/auth/index.js +2 -0
- package/dist/esm/auth/profile-loader.d.ts +18 -0
- package/dist/esm/auth/profile-loader.js +169 -0
- package/dist/esm/client-factory.d.ts +4 -0
- package/dist/esm/client-factory.js +9 -0
- package/dist/esm/clients/fluent-client.js +13 -6
- package/dist/esm/index.d.ts +3 -1
- package/dist/esm/index.js +2 -1
- package/dist/esm/utils/pagination-helpers.js +38 -2
- package/dist/esm/versori/fluent-versori-client.js +11 -5
- package/dist/tsconfig.esm.tsbuildinfo +1 -1
- package/dist/tsconfig.tsbuildinfo +1 -1
- package/dist/tsconfig.types.tsbuildinfo +1 -1
- package/dist/types/auth/index.d.ts +3 -0
- package/dist/types/auth/profile-loader.d.ts +18 -0
- package/dist/types/client-factory.d.ts +4 -0
- package/dist/types/index.d.ts +3 -1
- package/docs/00-START-HERE/EXPORT-VALIDATION.md +158 -158
- package/docs/00-START-HERE/cli-analyze-source-structure-guide.md +655 -655
- package/docs/00-START-HERE/cli-documentation-index.md +202 -202
- package/docs/00-START-HERE/cli-quick-reference.md +252 -252
- package/docs/00-START-HERE/decision-tree.md +552 -552
- package/docs/00-START-HERE/getting-started.md +1070 -1070
- package/docs/00-START-HERE/mapper-quick-decision-guide.md +235 -235
- package/docs/00-START-HERE/readme.md +237 -237
- package/docs/00-START-HERE/retailerid-configuration.md +404 -404
- package/docs/00-START-HERE/sdk-philosophy.md +794 -794
- package/docs/00-START-HERE/troubleshooting-quick-reference.md +1086 -1086
- package/docs/01-TEMPLATES/faq.md +686 -686
- package/docs/01-TEMPLATES/patterns/pattern-templates-guide.md +68 -68
- package/docs/01-TEMPLATES/patterns/patterns-csv-schema-validation-and-rejection-report.md +233 -233
- package/docs/01-TEMPLATES/patterns/patterns-custom-resolvers.md +407 -407
- package/docs/01-TEMPLATES/patterns/patterns-error-handling-retry.md +511 -511
- package/docs/01-TEMPLATES/patterns/patterns-field-mapping-universal.md +701 -701
- package/docs/01-TEMPLATES/patterns/patterns-large-file-splitting.md +1430 -1430
- package/docs/01-TEMPLATES/patterns/patterns-master-data-etl.md +2399 -2399
- package/docs/01-TEMPLATES/patterns/patterns-pagination-streaming.md +447 -447
- package/docs/01-TEMPLATES/patterns/patterns-state-duplicate-prevention.md +385 -385
- package/docs/01-TEMPLATES/readme.md +957 -957
- package/docs/01-TEMPLATES/standalone/standalone-asn-inbound-processing.md +1209 -1209
- package/docs/01-TEMPLATES/standalone/standalone-graphql-query-export.md +1140 -1140
- package/docs/01-TEMPLATES/standalone/standalone-graphql-to-parquet-partitioned-s3.md +432 -432
- package/docs/01-TEMPLATES/standalone/standalone-multi-channel-inventory-sync.md +1185 -1185
- package/docs/01-TEMPLATES/standalone/standalone-multi-source-aggregation.md +1462 -1462
- package/docs/01-TEMPLATES/standalone/standalone-s3-csv-batch-api.md +1390 -1390
- package/docs/01-TEMPLATES/standalone/standalone-s3-csv-inventory-to-batch.md +330 -330
- package/docs/01-TEMPLATES/standalone/standalone-scripts-guide.md +87 -87
- package/docs/01-TEMPLATES/standalone/standalone-sftp-xml-graphql.md +1444 -1444
- package/docs/01-TEMPLATES/standalone/standalone-webhook-payload-processing.md +688 -688
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-dropship-order-routing.md +193 -193
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-graphql-parquet-extraction.md +518 -518
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-inter-location-transfers.md +2162 -2162
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-pre-order-allocation.md +2226 -2226
- package/docs/01-TEMPLATES/versori/business-examples/business-scenarios-guide.md +87 -87
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-connection-validation-pattern.md +656 -656
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-dual-workflow-connector.md +835 -835
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-guide.md +108 -108
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-kv-state-management.md +1533 -1533
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-xml-response-patterns.md +1160 -1160
- package/docs/01-TEMPLATES/versori/versori-platform-guide.md +201 -201
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-asn-purchase-order.md +1906 -1906
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-dropship-routing.md +1074 -1074
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-flash-sale-reserve.md +1395 -1395
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-generic-xml-order.md +888 -888
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-payment-gateway-integration.md +2478 -2478
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-rma-returns-comprehensive.md +2240 -2240
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-xml-order-ingestion.md +2029 -2029
- package/docs/01-TEMPLATES/versori/webhooks/webhook-templates-guide.md +140 -140
- package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/inventory-mapping.json +20 -20
- package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/products_2025-01-22.csv +11 -11
- package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/sample-data-guide.md +34 -34
- package/docs/01-TEMPLATES/versori/workflows/_examples/workflow-examples-guide.md +36 -36
- package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-modes-guide.md +1038 -1038
- package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-workflows-guide.md +138 -138
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/graphql-extraction-guide.md +63 -63
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-csv.md +2062 -2062
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-xml.md +2294 -2294
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-s3-csv.md +2461 -2461
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-sftp-xml.md +2529 -2529
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-csv.md +2464 -2464
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-json.md +1959 -1959
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-s3-csv.md +1953 -1953
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-sftp-xml.md +2541 -2541
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-s3-json.md +2384 -2384
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-sftp-xml.md +2445 -2445
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-csv.md +2355 -2355
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-json.md +2042 -2042
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-sftp-xml.md +2726 -2726
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/batch-api-guide.md +206 -206
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-cycle-count-reconciliation.md +2030 -2030
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-multi-channel-inventory-sync.md +1882 -1882
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-csv-inventory-batch.md +2827 -2827
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-json-inventory-batch.md +1952 -1952
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-xml-inventory-batch.md +3289 -3289
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-csv-inventory-batch.md +3064 -3064
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-json-inventory-batch.md +3238 -3238
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-xml-inventory-batch.md +2977 -2977
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/event-api-guide.md +321 -321
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-json-order-cancel-event.md +959 -959
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-xml-order-cancel-event.md +1170 -1170
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-csv-product-event.md +2312 -2312
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-json-product-event.md +2999 -2999
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-parquet-product-event.md +2836 -2836
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-xml-product-event.md +2395 -2395
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-csv-product-event.md +2295 -2295
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-json-product-event.md +2602 -2602
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-parquet-product-event.md +2589 -2589
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-xml-product-event.md +3578 -3578
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/graphql-mutations-guide.md +93 -93
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-json-order-update-graphql.md +1260 -1260
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-xml-order-update-graphql.md +1472 -1472
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-control-graphql.md +2417 -2417
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-location-graphql.md +2811 -2811
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-price-graphql.md +2619 -2619
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-json-location-graphql.md +2807 -2807
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-xml-location-graphql.md +2373 -2373
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-control-graphql.md +2740 -2740
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-location-graphql.md +2760 -2760
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-json-location-graphql.md +1710 -1710
- package/docs/01-TEMPLATES/versori/workflows/ingestion/ingestion-workflows-guide.md +136 -136
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/rubix-webhooks-guide.md +520 -520
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-inline.md +1418 -1418
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-universal-mapper.md +1785 -1785
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-order-attribute-update.md +824 -824
- package/docs/01-TEMPLATES/versori/workflows/workflows-overview-guide.md +646 -646
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-batch-archival.md +724 -724
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-job-tracker.md +627 -627
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-partial-batch-recovery.md +561 -561
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-quick-reference.md +367 -367
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-readme.md +407 -407
- package/docs/02-CORE-GUIDES/advanced-services/readme.md +49 -49
- package/docs/02-CORE-GUIDES/api-reference/api-reference-quick-reference.md +548 -548
- package/docs/02-CORE-GUIDES/api-reference/event-api-input-output-reference.md +702 -1171
- package/docs/02-CORE-GUIDES/api-reference/examples/client-initialization.ts +286 -286
- package/docs/02-CORE-GUIDES/api-reference/graphql-error-classification.md +337 -337
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-01-client-api.md +399 -482
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-03-authentication.md +199 -199
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-04-graphql-mapping.md +925 -925
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-05-services.md +1198 -1198
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-06-data-sources.md +1083 -1083
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-07-parsers.md +1097 -1097
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-pagination.md +513 -513
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-types.md +545 -597
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-error-handling.md +527 -527
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-webhook-validation.md +514 -514
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-extraction.md +557 -557
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-utilities.md +412 -412
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-cli-tools.md +423 -423
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-error-handling.md +716 -716
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-analyze-source-structure.md +518 -518
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-partial-responses.md +212 -212
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-testing.md +300 -300
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-13-resolver-builder.md +322 -322
- package/docs/02-CORE-GUIDES/api-reference/readme.md +279 -279
- package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-quick-reference.md +351 -351
- package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-readme.md +277 -277
- package/docs/02-CORE-GUIDES/auto-pagination/examples/auto-pagination-readme.md +178 -178
- package/docs/02-CORE-GUIDES/auto-pagination/examples/common-patterns.ts +351 -351
- package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-products.ts +384 -384
- package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-virtual-positions.ts +308 -308
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-01-foundations.md +470 -470
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-02-quick-start.md +713 -713
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-03-configuration.md +754 -754
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-04-advanced-patterns.md +732 -732
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-05-sdk-integration.md +847 -847
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-06-troubleshooting.md +359 -359
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-07-api-reference.md +462 -462
- package/docs/02-CORE-GUIDES/auto-pagination/readme.md +54 -54
- package/docs/02-CORE-GUIDES/data-sources/data-sources-file-operations-error-handling.md +1487 -1487
- package/docs/02-CORE-GUIDES/data-sources/data-sources-quick-reference.md +836 -836
- package/docs/02-CORE-GUIDES/data-sources/data-sources-readme.md +276 -276
- package/docs/02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md +553 -553
- package/docs/02-CORE-GUIDES/data-sources/examples/common-patterns.ts +409 -409
- package/docs/02-CORE-GUIDES/data-sources/examples/data-sources-readme.md +178 -178
- package/docs/02-CORE-GUIDES/data-sources/examples/s3-operations.ts +308 -308
- package/docs/02-CORE-GUIDES/data-sources/examples/sftp-operations.ts +371 -371
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-01-foundations.md +735 -735
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-02-s3-operations.md +1302 -1302
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-03-sftp-operations.md +1379 -1379
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-04-file-patterns.md +941 -941
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-05-advanced-topics.md +813 -813
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-06-integration-patterns.md +486 -486
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-07-troubleshooting.md +387 -387
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-08-api-reference.md +417 -417
- package/docs/02-CORE-GUIDES/data-sources/readme.md +77 -77
- package/docs/02-CORE-GUIDES/error-handling-guide.md +936 -936
- package/docs/02-CORE-GUIDES/extraction/examples/02-core-guides-extraction-readme.md +116 -116
- package/docs/02-CORE-GUIDES/extraction/examples/common-patterns.ts +428 -428
- package/docs/02-CORE-GUIDES/extraction/examples/extract-inventory-basic.ts +187 -187
- package/docs/02-CORE-GUIDES/extraction/extraction-quick-reference.md +596 -596
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-01-foundations.md +514 -514
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-02-basic-extraction.md +823 -823
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-03-parquet-processing.md +507 -507
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-04-data-enrichment.md +546 -546
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-05-transformation.md +494 -494
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-export-formats.md +458 -458
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-performance.md +138 -138
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-api-reference.md +148 -148
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-optimization.md +692 -692
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-08-extraction-orchestrator.md +1008 -1008
- package/docs/02-CORE-GUIDES/extraction/readme.md +151 -151
- package/docs/02-CORE-GUIDES/ingestion/examples/_simple-kv-store.ts +40 -40
- package/docs/02-CORE-GUIDES/ingestion/examples/error-recovery.ts +728 -728
- package/docs/02-CORE-GUIDES/ingestion/examples/event-driven.ts +501 -501
- package/docs/02-CORE-GUIDES/ingestion/examples/local-file-ingestion.ts +88 -88
- package/docs/02-CORE-GUIDES/ingestion/examples/parquet-ingestion.ts +117 -117
- package/docs/02-CORE-GUIDES/ingestion/examples/performance-optimized.ts +647 -647
- package/docs/02-CORE-GUIDES/ingestion/examples/s3-csv-ingestion.ts +169 -169
- package/docs/02-CORE-GUIDES/ingestion/examples/sftp-csv-ingestion.ts +134 -134
- package/docs/02-CORE-GUIDES/ingestion/ingestion-quick-reference.md +546 -546
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-01-introduction.md +626 -626
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-02-quick-start.md +658 -658
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-03-data-sources.md +1052 -1052
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-04-field-mapping.md +763 -763
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-05-advanced-parsers.md +676 -676
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-06-batch-api.md +1295 -1295
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-api-reference.md +138 -138
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-state-management.md +1037 -1037
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-08-performance-optimization.md +1349 -1349
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-09-best-practices.md +1893 -1893
- package/docs/02-CORE-GUIDES/ingestion/readme.md +160 -160
- package/docs/02-CORE-GUIDES/logging-guide.md +585 -585
- package/docs/02-CORE-GUIDES/mapping/error-handling-patterns.md +401 -401
- package/docs/02-CORE-GUIDES/mapping/examples/02-core-guides-mapping-readme.md +128 -128
- package/docs/02-CORE-GUIDES/mapping/examples/common-patterns.ts +273 -273
- package/docs/02-CORE-GUIDES/mapping/examples/csv-location-ingestion.json +36 -36
- package/docs/02-CORE-GUIDES/mapping/examples/csv-mapping.ts +242 -242
- package/docs/02-CORE-GUIDES/mapping/examples/graphql-to-parquet-extraction.json +36 -36
- package/docs/02-CORE-GUIDES/mapping/examples/json-mapping.ts +213 -213
- package/docs/02-CORE-GUIDES/mapping/examples/json-product-to-mutation.json +48 -48
- package/docs/02-CORE-GUIDES/mapping/examples/xml-mapping.ts +291 -291
- package/docs/02-CORE-GUIDES/mapping/examples/xml-order-to-mutation.json +45 -45
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-quick-reference.md +463 -463
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-readme.md +227 -227
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-01-introduction.md +222 -222
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-02-quick-start.md +351 -351
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-03-schema-validation.md +569 -569
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-04-mapping-patterns.md +471 -471
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-05-configuration-reference.md +611 -611
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-advanced-xpath.md +148 -148
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-path-syntax.md +464 -464
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-api-reference.md +94 -94
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-array-handling.md +307 -307
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-08-custom-resolvers.md +544 -544
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-09-advanced-patterns.md +427 -427
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-10-hooks-and-variables.md +336 -336
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-11-error-handling.md +488 -488
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-12-arguments-vs-nodes.md +383 -383
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-13-best-practices.md +477 -477
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/readme.md +62 -62
- package/docs/02-CORE-GUIDES/mapping/mapping-format-decision-tree.md +480 -480
- package/docs/02-CORE-GUIDES/mapping/mapping-graphql-alias-batching-guide.md +820 -820
- package/docs/02-CORE-GUIDES/mapping/mapping-javascript-objects.md +2369 -2369
- package/docs/02-CORE-GUIDES/mapping/mapping-mapper-comparison-guide.md +682 -682
- package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-07-api-reference.md +1327 -1327
- package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-08-error-handling.md +1142 -1142
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-04-use-cases.md +891 -891
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-helpers-resolvers.md +1126 -1126
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-sdk-resolvers.md +199 -199
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-07-api-reference.md +1319 -1319
- package/docs/02-CORE-GUIDES/mapping/readme.md +178 -178
- package/docs/02-CORE-GUIDES/mapping/resolver-registration.md +410 -410
- package/docs/02-CORE-GUIDES/mapping/resolvers/examples/common-patterns.ts +226 -226
- package/docs/02-CORE-GUIDES/mapping/resolvers/examples/custom-resolvers.ts +227 -227
- package/docs/02-CORE-GUIDES/mapping/resolvers/examples/sdk-resolvers-usage.ts +203 -203
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-readme.md +274 -274
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-api-reference.md +679 -679
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-cookbook.md +826 -826
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-guide.md +1330 -1330
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-helpers-reference.md +1437 -1437
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-parameters-reference.md +553 -553
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-troubleshooting.md +854 -854
- package/docs/02-CORE-GUIDES/mapping/resolvers/readme.md +75 -75
- package/docs/02-CORE-GUIDES/parsers/examples/02-core-guides-parsers-readme.md +161 -161
- package/docs/02-CORE-GUIDES/parsers/examples/csv-parser-examples.ts +110 -110
- package/docs/02-CORE-GUIDES/parsers/examples/json-parser-examples.ts +33 -33
- package/docs/02-CORE-GUIDES/parsers/examples/parquet-parser-examples.ts +47 -47
- package/docs/02-CORE-GUIDES/parsers/examples/xml-parser-examples.ts +38 -38
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-01-foundations.md +355 -355
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-02-csv-parser.md +772 -772
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-03-json-parser.md +789 -789
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-04-xml-parser.md +857 -857
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-05-parquet-parser.md +603 -603
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-integration-patterns.md +702 -702
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-streaming.md +121 -121
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-api-reference.md +89 -89
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-troubleshooting.md +727 -727
- package/docs/02-CORE-GUIDES/parsers/parsers-quick-reference.md +482 -482
- package/docs/02-CORE-GUIDES/parsers/parsers-readme.md +258 -258
- package/docs/02-CORE-GUIDES/parsers/readme.md +65 -65
- package/docs/02-CORE-GUIDES/readme.md +194 -194
- package/docs/02-CORE-GUIDES/webhook-validation/examples/basic-validation.ts +108 -108
- package/docs/02-CORE-GUIDES/webhook-validation/examples/common-patterns.ts +316 -316
- package/docs/02-CORE-GUIDES/webhook-validation/examples/webhook-validation-readme.md +61 -61
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-01-foundations.md +440 -440
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-02-quick-start.md +525 -525
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-03-versori-integration.md +741 -741
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-04-platform-integration.md +629 -629
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-05-configuration.md +535 -535
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-error-handling.md +611 -611
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-troubleshooting.md +124 -124
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-07-api-reference.md +511 -511
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-08-rubix-webhooks.md +590 -590
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-09-rubix-event-vs-http-call.md +432 -432
- package/docs/02-CORE-GUIDES/webhook-validation/readme.md +239 -239
- package/docs/02-CORE-GUIDES/webhook-validation/webhook-validation-quick-reference.md +392 -392
- package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-quick-reference.md +498 -498
- package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-readme.md +313 -313
- package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/common-patterns.ts +612 -612
- package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/connector-scenarios-readme.md +253 -253
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-01-foundations.md +452 -452
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-02-simple-scenarios.md +681 -681
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-03-intermediate-scenarios.md +637 -637
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-04-advanced-scenarios.md +650 -650
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-05-bidirectional-sync.md +233 -233
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-06-production-patterns.md +442 -442
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-07-reference.md +445 -445
- package/docs/03-PATTERN-GUIDES/connector-scenarios/readme.md +31 -31
- package/docs/03-PATTERN-GUIDES/enterprise-integration-patterns.md +1528 -1528
- package/docs/03-PATTERN-GUIDES/error-handling/comprehensive-error-handling-guide.md +1437 -1437
- package/docs/03-PATTERN-GUIDES/error-handling/error-handling-quick-reference.md +390 -390
- package/docs/03-PATTERN-GUIDES/error-handling/examples/common-patterns.ts +438 -438
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-01-foundations.md +362 -362
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-02-error-types.md +850 -850
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-03-utf8-handling.md +456 -456
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-04-error-scenarios.md +658 -658
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-05-calling-patterns.md +671 -671
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-06-retry-strategies.md +1034 -1034
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-07-monitoring.md +653 -653
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-08-api-reference.md +847 -847
- package/docs/03-PATTERN-GUIDES/error-handling/readme.md +36 -36
- package/docs/03-PATTERN-GUIDES/examples/__tests__/readme.md +40 -40
- package/docs/03-PATTERN-GUIDES/examples/__tests__/resolver-examples.test.js +282 -282
- package/docs/03-PATTERN-GUIDES/examples/test-data/03-pattern-guides-readme.md +110 -110
- package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-inventory.json +123 -123
- package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-order.json +171 -171
- package/docs/03-PATTERN-GUIDES/examples/test-data/readme.md +28 -28
- package/docs/03-PATTERN-GUIDES/extraction/extraction-readme.md +15 -15
- package/docs/03-PATTERN-GUIDES/extraction/readme.md +25 -25
- package/docs/03-PATTERN-GUIDES/file-operations/examples/common-patterns.ts +407 -407
- package/docs/03-PATTERN-GUIDES/file-operations/examples/file-operations-readme.md +142 -142
- package/docs/03-PATTERN-GUIDES/file-operations/file-operations-quick-reference.md +462 -462
- package/docs/03-PATTERN-GUIDES/file-operations/file-operations-readme.md +379 -379
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-01-foundations.md +430 -430
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-02-quick-start.md +484 -484
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-03-s3-operations.md +507 -507
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-04-sftp-operations.md +963 -963
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-05-streaming-performance.md +503 -503
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-archive-patterns.md +386 -386
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-error-handling.md +117 -117
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-api-reference.md +78 -78
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-testing-troubleshooting.md +567 -567
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-08-api-reference.md +1055 -1055
- package/docs/03-PATTERN-GUIDES/file-operations/readme.md +32 -32
- package/docs/03-PATTERN-GUIDES/ingestion/ingestion-readme.md +15 -15
- package/docs/03-PATTERN-GUIDES/ingestion/readme.md +25 -25
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/batch-processing.ts +130 -130
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/common-patterns.ts +360 -360
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/delta-sync.ts +130 -130
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/integration-patterns-readme.md +100 -100
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/real-time-webhook.ts +398 -398
- package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-quick-reference.md +962 -962
- package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-readme.md +134 -134
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-01-real-time-processing.md +991 -991
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-02-batch-processing.md +1547 -1547
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-03-delta-sync.md +1108 -1108
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-04-webhook-patterns.md +1181 -1181
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-05-error-handling.md +1061 -1061
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-advanced-integration-services.md +1547 -1547
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-performance.md +109 -109
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-07-api-reference.md +34 -34
- package/docs/03-PATTERN-GUIDES/integration-patterns/readme.md +30 -30
- package/docs/03-PATTERN-GUIDES/logging-minimal-mode.md +128 -128
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/common-patterns.ts +380 -380
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/multiple-connections-readme.md +139 -139
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/parallel-root-connections.ts +149 -149
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/real-world-scenarios.ts +405 -405
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-01-foundations.md +378 -378
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-02-quick-start.md +566 -566
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-03-targeting-connections.md +659 -659
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-04-parallel-queries.md +656 -656
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-05-best-practices.md +624 -624
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-api-reference.md +824 -824
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-versori.md +119 -119
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-07-api-reference.md +87 -87
- package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-quick-reference.md +353 -353
- package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-readme.md +270 -270
- package/docs/03-PATTERN-GUIDES/multiple-connections/readme.md +30 -30
- package/docs/03-PATTERN-GUIDES/pagination/pagination-readme.md +14 -14
- package/docs/03-PATTERN-GUIDES/pagination/readme.md +24 -24
- package/docs/03-PATTERN-GUIDES/parquet/examples/common-patterns.ts +180 -180
- package/docs/03-PATTERN-GUIDES/parquet/examples/read-parquet.ts +48 -48
- package/docs/03-PATTERN-GUIDES/parquet/examples/write-parquet.ts +65 -65
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-01-introduction.md +393 -393
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-02-quick-start.md +572 -572
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-03-reading-parquet.md +525 -525
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-04-writing-parquet.md +554 -554
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-05-graphql-extraction.md +405 -405
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-performance.md +104 -104
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-s3-integration.md +511 -511
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-api-reference.md +90 -90
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-performance-optimization.md +525 -525
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-08-best-practices.md +712 -712
- package/docs/03-PATTERN-GUIDES/parquet/parquet-quick-reference.md +683 -683
- package/docs/03-PATTERN-GUIDES/parquet/parquet-readme.md +248 -248
- package/docs/03-PATTERN-GUIDES/parquet/readme.md +32 -32
- package/docs/03-PATTERN-GUIDES/parsers/parsers-readme.md +12 -12
- package/docs/03-PATTERN-GUIDES/parsers/readme.md +24 -24
- package/docs/03-PATTERN-GUIDES/readme.md +159 -159
- package/docs/03-PATTERN-GUIDES/webhooks/readme.md +24 -24
- package/docs/03-PATTERN-GUIDES/webhooks/webhooks-readme.md +8 -8
- package/docs/04-REFERENCE/architecture/architecture-01-overview.md +427 -427
- package/docs/04-REFERENCE/architecture/architecture-02-client-architecture.md +424 -424
- package/docs/04-REFERENCE/architecture/architecture-03-data-flow.md +690 -690
- package/docs/04-REFERENCE/architecture/architecture-04-service-layer.md +834 -834
- package/docs/04-REFERENCE/architecture/architecture-05-integration-architecture.md +655 -655
- package/docs/04-REFERENCE/architecture/architecture-06-state-management.md +653 -653
- package/docs/04-REFERENCE/architecture/architecture-adding-new-data-sources.md +686 -686
- package/docs/04-REFERENCE/architecture/readme.md +279 -279
- package/docs/04-REFERENCE/platforms/deno/readme.md +117 -117
- package/docs/04-REFERENCE/platforms/nodejs/readme.md +146 -146
- package/docs/04-REFERENCE/platforms/readme.md +135 -135
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-01-introduction.md +398 -398
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-02-quick-start.md +560 -560
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-03-authentication.md +757 -757
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-04-workflows.md +2476 -2476
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-05-connections.md +1167 -1167
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-kv-storage.md +990 -990
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-state-management.md +121 -121
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-api-reference.md +68 -68
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-deployment.md +731 -731
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-08-best-practices.md +1111 -1111
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-09-signature-reference.md +766 -766
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-readme.md +299 -299
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-s3-sftp-configuration-guide.md +1425 -1425
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-api-key-security.md +816 -816
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-connection-security.md +681 -681
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-workflow-task-types.md +708 -708
- package/docs/04-REFERENCE/platforms/versori/readme.md +108 -108
- package/docs/04-REFERENCE/readme.md +148 -148
- package/docs/04-REFERENCE/resolver-signature/examples/advanced-resolvers.ts +482 -482
- package/docs/04-REFERENCE/resolver-signature/examples/async-resolvers.ts +496 -496
- package/docs/04-REFERENCE/resolver-signature/examples/basic-resolvers.ts +343 -343
- package/docs/04-REFERENCE/resolver-signature/examples/resolver-signature-readme.md +188 -188
- package/docs/04-REFERENCE/resolver-signature/examples/testing-resolvers.ts +463 -463
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-01-foundations.md +286 -286
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-02-parameter-reference.md +643 -643
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-03-basic-examples.md +521 -521
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-04-advanced-patterns.md +739 -739
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-05-sdk-resolvers.md +531 -531
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-migration-guide.md +650 -650
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-testing.md +125 -125
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-07-api-reference.md +794 -794
- package/docs/04-REFERENCE/resolver-signature/readme.md +64 -64
- package/docs/04-REFERENCE/resolver-signature/resolver-signature-quick-reference.md +270 -270
- package/docs/04-REFERENCE/resolver-signature/resolver-signature-readme.md +351 -351
- package/docs/04-REFERENCE/schema/fluent-commerce-schema.json +764 -764
- package/docs/04-REFERENCE/schema/readme.md +141 -141
- package/docs/04-REFERENCE/testing/examples/04-reference-testing-readme.md +158 -158
- package/docs/04-REFERENCE/testing/examples/fluent-testing.ts +62 -62
- package/docs/04-REFERENCE/testing/examples/health-check.ts +155 -155
- package/docs/04-REFERENCE/testing/examples/integration-test.ts +119 -119
- package/docs/04-REFERENCE/testing/examples/performance-test.ts +183 -183
- package/docs/04-REFERENCE/testing/examples/s3-testing.ts +127 -127
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-01-foundations.md +267 -267
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-02-s3-testing.md +599 -599
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-03-fluent-testing.md +589 -589
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-04-integration-testing.md +699 -699
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-05-debugging.md +478 -478
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-cicd-integration.md +463 -463
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-preflight-validation.md +131 -131
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-best-practices.md +499 -499
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-coverage-ci.md +165 -165
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-08-api-reference.md +634 -634
- package/docs/04-REFERENCE/testing/readme.md +86 -86
- package/docs/04-REFERENCE/testing/testing-quick-reference.md +667 -667
- package/docs/04-REFERENCE/testing/testing-readme.md +286 -286
- package/docs/04-REFERENCE/troubleshooting/readme.md +144 -144
- package/docs/04-REFERENCE/troubleshooting/troubleshooting-deno-sftp-compatibility.md +392 -392
- package/docs/template-loading-matrix.md +242 -242
- package/package.json +5 -3
|
@@ -1,1444 +1,1444 @@
|
|
|
1
|
-
# Standalone: SFTP XML → Fluent GraphQL
|
|
2
|
-
|
|
3
|
-
**FC Connect SDK Use Case Guide**
|
|
4
|
-
|
|
5
|
-
> **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
|
|
6
|
-
> **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
|
|
7
|
-
|
|
8
|
-
**Context**: Node.js script that reads XML order files from SFTP and creates orders in Fluent Commerce via GraphQL mutations
|
|
9
|
-
|
|
10
|
-
**Complexity**: Medium
|
|
11
|
-
|
|
12
|
-
**Runtime**: Node.js ≥18 / Deno
|
|
13
|
-
|
|
14
|
-
**Estimated Lines**: ~600 lines
|
|
15
|
-
|
|
16
|
-
## What You'll Build
|
|
17
|
-
|
|
18
|
-
- Standalone Node.js/Deno script
|
|
19
|
-
- OAuth2 authentication
|
|
20
|
-
- SFTP connection and file operations
|
|
21
|
-
- XML parsing with validation
|
|
22
|
-
- GraphQL mutation mapping
|
|
23
|
-
- Custom resolvers for transformations
|
|
24
|
-
- File archival after processing
|
|
25
|
-
- Error handling and logging
|
|
26
|
-
|
|
27
|
-
## SDK Methods Used
|
|
28
|
-
|
|
29
|
-
- `createClient({ config: { baseUrl, clientId, clientSecret, retailerId } })` - OAuth2 client
|
|
30
|
-
- `SftpDataSource(config, logger)` - SFTP operations
|
|
31
|
-
- `XMLParserService` - XML parsing
|
|
32
|
-
- `GraphQLMutationMapper(config, logger, { fluentClient, customResolvers })` - Map XML to GraphQL
|
|
33
|
-
- `mapper.mapSafe(xmlData)` - Transform data with error handling (recommended) OR `mapper.map(xmlData)` - Transform data (throws on error) OR `mapper.mapWithNodes(xmlData)` - Transform with custom resolvers
|
|
34
|
-
- `client.graphql({ query, variables })` - Execute mutation
|
|
35
|
-
|
|
36
|
-
---
|
|
37
|
-
|
|
38
|
-
## SFTP Credential Configuration
|
|
39
|
-
|
|
40
|
-
### For Standalone Scripts (Node.js/Deno)
|
|
41
|
-
|
|
42
|
-
**Standalone environments** load SFTP credentials from environment variables or configuration files:
|
|
43
|
-
|
|
44
|
-
```typescript
|
|
45
|
-
import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
|
|
46
|
-
|
|
47
|
-
// Load from environment variables (recommended)
|
|
48
|
-
const sftp = new SftpDataSource({
|
|
49
|
-
type: 'SFTP_XML',
|
|
50
|
-
connectionId: 'sftp-orders',
|
|
51
|
-
name: 'Order SFTP',
|
|
52
|
-
settings: {
|
|
53
|
-
host: process.env.SFTP_HOST || 'sftp.example.com',
|
|
54
|
-
port: parseInt(process.env.SFTP_PORT || '22'),
|
|
55
|
-
username: process.env.SFTP_USERNAME,
|
|
56
|
-
|
|
57
|
-
// Option 1: Password authentication
|
|
58
|
-
password: process.env.SFTP_PASSWORD,
|
|
59
|
-
|
|
60
|
-
// Option 2: SSH private key authentication (more secure)
|
|
61
|
-
// privateKey: fs.readFileSync(process.env.SFTP_KEY_PATH, 'utf8'),
|
|
62
|
-
// passphrase: process.env.SFTP_PASSPHRASE,
|
|
63
|
-
}
|
|
64
|
-
}, logger);
|
|
65
|
-
```
|
|
66
|
-
|
|
67
|
-
**Environment Variable Setup:**
|
|
68
|
-
|
|
69
|
-
```bash
|
|
70
|
-
# .env file
|
|
71
|
-
SFTP_HOST=sftp.example.com
|
|
72
|
-
SFTP_PORT=22
|
|
73
|
-
SFTP_USERNAME=your-username
|
|
74
|
-
SFTP_PASSWORD=your-password
|
|
75
|
-
|
|
76
|
-
# OR use SSH key
|
|
77
|
-
# SFTP_PRIVATE_KEY_PATH=/path/to/private/key
|
|
78
|
-
# SFTP_PASSPHRASE=key-passphrase
|
|
79
|
-
```
|
|
80
|
-
|
|
81
|
-
### For Versori Platform Deployments
|
|
82
|
-
|
|
83
|
-
**Versori platform** uses connection-based credential management. See the comprehensive guide:
|
|
84
|
-
|
|
85
|
-
- [SFTP Credential Access & Security](../../02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md)
|
|
86
|
-
|
|
87
|
-
**Key Differences:**
|
|
88
|
-
|
|
89
|
-
| Aspect | Standalone (Node.js/Deno) | Versori Platform |
|
|
90
|
-
|--------|---------------------------|------------------|
|
|
91
|
-
| **Credential Storage** | Environment variables, files | Versori connections |
|
|
92
|
-
| **Access Method** | Direct configuration | `connectionVariables`, `credentials()`, or `activation.connections` |
|
|
93
|
-
| **Security** | Manual secret management | Platform-managed encryption |
|
|
94
|
-
| **Rotation** | Manual updates | Centralized rotation |
|
|
95
|
-
|
|
96
|
-
**Security Best Practices:**
|
|
97
|
-
|
|
98
|
-
- Never hardcode credentials in source code
|
|
99
|
-
- Use environment variables or secret management systems
|
|
100
|
-
- Store SSH keys securely with appropriate file permissions
|
|
101
|
-
- Rotate credentials regularly
|
|
102
|
-
- Use SSH key authentication over passwords when possible
|
|
103
|
-
|
|
104
|
-
---
|
|
105
|
-
|
|
106
|
-
## Complete Working Script
|
|
107
|
-
|
|
108
|
-
This standalone script demonstrates a complete SFTP-to-Fluent order integration workflow.
|
|
109
|
-
|
|
110
|
-
### Step 1: Install Dependencies
|
|
111
|
-
|
|
112
|
-
```bash
|
|
113
|
-
npm install @fluentcommerce/fc-connect-sdk ssh2-sftp-client dotenv
|
|
114
|
-
npm install --save-dev @types/ssh2-sftp-client @types/node typescript
|
|
115
|
-
```
|
|
116
|
-
|
|
117
|
-
### Step 2: Environment Configuration
|
|
118
|
-
|
|
119
|
-
Create `.env` file:
|
|
120
|
-
|
|
121
|
-
```bash
|
|
122
|
-
# Fluent Commerce OAuth2
|
|
123
|
-
FLUENT_BASE_URL=https://yourinstance.api.fluentcommerce.com
|
|
124
|
-
FLUENT_CLIENT_ID=your-oauth-client-id
|
|
125
|
-
FLUENT_CLIENT_SECRET=your-oauth-client-secret
|
|
126
|
-
FLUENT_RETAILER_ID=your-retailer-id
|
|
127
|
-
|
|
128
|
-
# SFTP Connection
|
|
129
|
-
SFTP_HOST=sftp.example.com
|
|
130
|
-
SFTP_PORT=22
|
|
131
|
-
SFTP_USERNAME=sftp-user
|
|
132
|
-
SFTP_PASSWORD=sftp-password
|
|
133
|
-
|
|
134
|
-
# OR use SSH key
|
|
135
|
-
# SFTP_PRIVATE_KEY_PATH=/path/to/private/key
|
|
136
|
-
# SFTP_PASSPHRASE=key-passphrase
|
|
137
|
-
|
|
138
|
-
# Processing Options
|
|
139
|
-
SFTP_REMOTE_PATH=/orders/incoming
|
|
140
|
-
SFTP_ARCHIVE_PATH=/orders/processed
|
|
141
|
-
SFTP_ERROR_PATH=/orders/errors
|
|
142
|
-
FILE_PATTERN=ORDER_*.xml
|
|
143
|
-
POLLING_INTERVAL_MS=60000
|
|
144
|
-
```
|
|
145
|
-
|
|
146
|
-
### Step 3: Main Script Implementation
|
|
147
|
-
|
|
148
|
-
Create `sftp-order-sync.ts`:
|
|
149
|
-
|
|
150
|
-
```typescript
|
|
151
|
-
import 'dotenv/config';
|
|
152
|
-
|
|
153
|
-
// FC Connect SDK+
|
|
154
|
-
// Install: npm install @fluentcommerce/fc-connect-sdk@latest
|
|
155
|
-
// Docs: https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk
|
|
156
|
-
// GitHub: https://github.com/fluentcommerce/fc-connect-sdk
|
|
157
|
-
|
|
158
|
-
import {
|
|
159
|
-
createClient,
|
|
160
|
-
classifyErrors,
|
|
161
|
-
SftpDataSource,
|
|
162
|
-
XMLParserService,
|
|
163
|
-
GraphQLMutationMapper,
|
|
164
|
-
createConsoleLogger,
|
|
165
|
-
toStructuredLogger
|
|
166
|
-
} from '@fluentcommerce/fc-connect-sdk';
|
|
167
|
-
|
|
168
|
-
import type {
|
|
169
|
-
FluentClient,
|
|
170
|
-
StructuredLogger,
|
|
171
|
-
MappingConfig
|
|
172
|
-
} from '@fluentcommerce/fc-connect-sdk';
|
|
173
|
-
|
|
174
|
-
import { readFileSync } from 'fs';
|
|
175
|
-
import { join } from 'path';
|
|
176
|
-
|
|
177
|
-
// ============================================================================
|
|
178
|
-
// CONFIGURATION
|
|
179
|
-
// ============================================================================
|
|
180
|
-
|
|
181
|
-
const config = {
|
|
182
|
-
fluent: {
|
|
183
|
-
baseUrl: process.env.FLUENT_BASE_URL!,
|
|
184
|
-
clientId: process.env.FLUENT_CLIENT_ID!,
|
|
185
|
-
clientSecret: process.env.FLUENT_CLIENT_SECRET!,
|
|
186
|
-
retailerId: process.env.FLUENT_RETAILER_ID!,
|
|
187
|
-
},
|
|
188
|
-
sftp: {
|
|
189
|
-
host: process.env.SFTP_HOST!,
|
|
190
|
-
port: parseInt(process.env.SFTP_PORT || '22'),
|
|
191
|
-
username: process.env.SFTP_USERNAME!,
|
|
192
|
-
password: process.env.SFTP_PASSWORD,
|
|
193
|
-
privateKey: process.env.SFTP_PRIVATE_KEY_PATH
|
|
194
|
-
? readFileSync(process.env.SFTP_PRIVATE_KEY_PATH, 'utf8')
|
|
195
|
-
: undefined,
|
|
196
|
-
passphrase: process.env.SFTP_PASSPHRASE,
|
|
197
|
-
remotePath: process.env.SFTP_REMOTE_PATH || '/orders/incoming',
|
|
198
|
-
archivePath: process.env.SFTP_ARCHIVE_PATH || '/orders/processed',
|
|
199
|
-
errorPath: process.env.SFTP_ERROR_PATH || '/orders/errors',
|
|
200
|
-
filePattern: process.env.FILE_PATTERN || 'ORDER_*.xml',
|
|
201
|
-
},
|
|
202
|
-
processing: {
|
|
203
|
-
pollingInterval: parseInt(process.env.POLLING_INTERVAL_MS || '60000'),
|
|
204
|
-
batchSize: 100,
|
|
205
|
-
},
|
|
206
|
-
};
|
|
207
|
-
|
|
208
|
-
// ============================================================================
|
|
209
|
-
// GRAPHQL MAPPING CONFIGURATION
|
|
210
|
-
// ============================================================================
|
|
211
|
-
|
|
212
|
-
const orderMappingConfig: MappingConfig = {
|
|
213
|
-
version: '1.0',
|
|
214
|
-
name: 'SFTP XML Order to Fluent',
|
|
215
|
-
mutation: 'createOrder',
|
|
216
|
-
sourceFormat: 'xml',
|
|
217
|
-
operationName: 'CreateOrderFromSFTP',
|
|
218
|
-
arguments: {
|
|
219
|
-
input: {
|
|
220
|
-
_type: 'CreateOrderInput!',
|
|
221
|
-
// Order reference
|
|
222
|
-
ref: {
|
|
223
|
-
source: 'order.@id',
|
|
224
|
-
required: true,
|
|
225
|
-
},
|
|
226
|
-
// Order type
|
|
227
|
-
type: {
|
|
228
|
-
value: 'HD', // Home delivery
|
|
229
|
-
},
|
|
230
|
-
// Retailer reference
|
|
231
|
-
retailer: {
|
|
232
|
-
id: {
|
|
233
|
-
value: config.fluent.retailerId,
|
|
234
|
-
},
|
|
235
|
-
},
|
|
236
|
-
// Customer information
|
|
237
|
-
customer: {
|
|
238
|
-
firstName: {
|
|
239
|
-
source: 'order.customer.first-name',
|
|
240
|
-
transform: 'trim',
|
|
241
|
-
required: true,
|
|
242
|
-
},
|
|
243
|
-
lastName: {
|
|
244
|
-
source: 'order.customer.last-name',
|
|
245
|
-
transform: 'trim',
|
|
246
|
-
required: true,
|
|
247
|
-
},
|
|
248
|
-
email: {
|
|
249
|
-
source: 'order.customer.email',
|
|
250
|
-
transform: 'toLowerCase',
|
|
251
|
-
required: true,
|
|
252
|
-
},
|
|
253
|
-
phone: {
|
|
254
|
-
source: 'order.customer.phone',
|
|
255
|
-
transform: 'trim',
|
|
256
|
-
},
|
|
257
|
-
},
|
|
258
|
-
// Delivery address
|
|
259
|
-
fulfilmentChoice: {
|
|
260
|
-
deliveryType: {
|
|
261
|
-
value: 'STANDARD',
|
|
262
|
-
},
|
|
263
|
-
deliveryAddress: {
|
|
264
|
-
name: {
|
|
265
|
-
source: 'order.shipping.name',
|
|
266
|
-
},
|
|
267
|
-
street: {
|
|
268
|
-
source: 'order.shipping.street',
|
|
269
|
-
required: true,
|
|
270
|
-
},
|
|
271
|
-
city: {
|
|
272
|
-
source: 'order.shipping.city',
|
|
273
|
-
required: true,
|
|
274
|
-
},
|
|
275
|
-
state: {
|
|
276
|
-
source: 'order.shipping.state',
|
|
277
|
-
},
|
|
278
|
-
postcode: {
|
|
279
|
-
source: 'order.shipping.postcode',
|
|
280
|
-
required: true,
|
|
281
|
-
},
|
|
282
|
-
country: {
|
|
283
|
-
source: 'order.shipping.country',
|
|
284
|
-
required: true,
|
|
285
|
-
},
|
|
286
|
-
},
|
|
287
|
-
},
|
|
288
|
-
// Order items array
|
|
289
|
-
items: {
|
|
290
|
-
_array: true,
|
|
291
|
-
_autoWrap: true, // Convert single item to array
|
|
292
|
-
source: 'order.items.item',
|
|
293
|
-
_validation: {
|
|
294
|
-
minItems: 1,
|
|
295
|
-
},
|
|
296
|
-
// Fields for each item
|
|
297
|
-
ref: {
|
|
298
|
-
source: '@id',
|
|
299
|
-
required: true,
|
|
300
|
-
},
|
|
301
|
-
productRef: {
|
|
302
|
-
source: 'sku',
|
|
303
|
-
required: true,
|
|
304
|
-
},
|
|
305
|
-
quantity: {
|
|
306
|
-
source: 'quantity',
|
|
307
|
-
transform: 'parseInt',
|
|
308
|
-
required: true,
|
|
309
|
-
},
|
|
310
|
-
price: {
|
|
311
|
-
source: 'price',
|
|
312
|
-
transform: 'parseFloat',
|
|
313
|
-
required: true,
|
|
314
|
-
},
|
|
315
|
-
totalPrice: {
|
|
316
|
-
source: 'total-price',
|
|
317
|
-
transform: 'parseFloat',
|
|
318
|
-
required: true,
|
|
319
|
-
},
|
|
320
|
-
currency: {
|
|
321
|
-
value: 'USD',
|
|
322
|
-
},
|
|
323
|
-
},
|
|
324
|
-
// Order totals
|
|
325
|
-
totalPrice: {
|
|
326
|
-
source: 'order.totals.subtotal',
|
|
327
|
-
transform: 'parseFloat',
|
|
328
|
-
required: true,
|
|
329
|
-
},
|
|
330
|
-
totalTaxPrice: {
|
|
331
|
-
source: 'order.totals.tax',
|
|
332
|
-
transform: 'parseFloat',
|
|
333
|
-
defaultValue: 0,
|
|
334
|
-
},
|
|
335
|
-
// Custom attributes
|
|
336
|
-
attributes: {
|
|
337
|
-
orderDate: {
|
|
338
|
-
source: 'order.@order-date',
|
|
339
|
-
transform: 'toISO8601',
|
|
340
|
-
},
|
|
341
|
-
externalSystem: {
|
|
342
|
-
value: 'SFTP_XML',
|
|
343
|
-
},
|
|
344
|
-
sourceFile: {
|
|
345
|
-
// This will be set by custom resolver
|
|
346
|
-
},
|
|
347
|
-
},
|
|
348
|
-
},
|
|
349
|
-
},
|
|
350
|
-
returnFields: [
|
|
351
|
-
'id',
|
|
352
|
-
'ref',
|
|
353
|
-
'status',
|
|
354
|
-
'createdOn',
|
|
355
|
-
'totalPrice',
|
|
356
|
-
'customer { firstName lastName email }',
|
|
357
|
-
'items { ref productRef quantity price }',
|
|
358
|
-
],
|
|
359
|
-
customTransforms: {
|
|
360
|
-
// Add custom transform for phone number normalization
|
|
361
|
-
normalizePhone: (value: unknown) => {
|
|
362
|
-
if (!value || typeof value !== 'string') return value;
|
|
363
|
-
// Remove all non-digit characters
|
|
364
|
-
return value.replace(/\D/g, '');
|
|
365
|
-
},
|
|
366
|
-
},
|
|
367
|
-
};
|
|
368
|
-
|
|
369
|
-
// ============================================================================
|
|
370
|
-
// LOGGING SETUP
|
|
371
|
-
// ============================================================================
|
|
372
|
-
|
|
373
|
-
const logger: StructuredLogger = toStructuredLogger(createConsoleLogger(), { logLevel: 'info' });
|
|
374
|
-
|
|
375
|
-
// ============================================================================
|
|
376
|
-
// INITIALIZE SERVICES
|
|
377
|
-
// ============================================================================
|
|
378
|
-
|
|
379
|
-
let fluentClient: FluentClient;
|
|
380
|
-
let sftpSource: SftpDataSource;
|
|
381
|
-
let xmlParser: XMLParserService;
|
|
382
|
-
let mutationMapper: GraphQLMutationMapper;
|
|
383
|
-
|
|
384
|
-
async function initializeServices() {
|
|
385
|
-
logger.info('Initializing services...');
|
|
386
|
-
|
|
387
|
-
// Create Fluent client with OAuth2
|
|
388
|
-
fluentClient = await createClient({ config: config.fluent });
|
|
389
|
-
|
|
390
|
-
// Create SFTP data source
|
|
391
|
-
sftpSource = new SftpDataSource({
|
|
392
|
-
type: 'SFTP_XML',
|
|
393
|
-
connectionId: 'sftp-orders',
|
|
394
|
-
name: 'Order SFTP',
|
|
395
|
-
settings: {
|
|
396
|
-
host: config.sftp.host,
|
|
397
|
-
port: config.sftp.port,
|
|
398
|
-
username: config.sftp.username,
|
|
399
|
-
password: config.sftp.password,
|
|
400
|
-
privateKey: config.sftp.privateKey,
|
|
401
|
-
passphrase: config.sftp.passphrase,
|
|
402
|
-
remotePath: config.sftp.remotePath,
|
|
403
|
-
filePattern: config.sftp.filePattern,
|
|
404
|
-
connectionTimeout: 30000,
|
|
405
|
-
keepAliveInterval: 10000,
|
|
406
|
-
},
|
|
407
|
-
}, logger);
|
|
408
|
-
|
|
409
|
-
// Create XML parser
|
|
410
|
-
xmlParser = new XMLParserService();
|
|
411
|
-
|
|
412
|
-
// Create GraphQL mutation mapper
|
|
413
|
-
mutationMapper = new GraphQLMutationMapper(orderMappingConfig, logger, { fluentClient: fluentClient });
|
|
414
|
-
|
|
415
|
-
logger.info('Services initialized successfully');
|
|
416
|
-
}
|
|
417
|
-
|
|
418
|
-
// ============================================================================
|
|
419
|
-
// FILE PROCESSING
|
|
420
|
-
// ============================================================================
|
|
421
|
-
|
|
422
|
-
interface ProcessingStats {
|
|
423
|
-
filesProcessed: number;
|
|
424
|
-
filesSucceeded: number;
|
|
425
|
-
filesFailed: number;
|
|
426
|
-
ordersCreated: number;
|
|
427
|
-
errors: Array<{ file: string; error: string }>;
|
|
428
|
-
}
|
|
429
|
-
|
|
430
|
-
async function processOrderFiles(): Promise<ProcessingStats> {
|
|
431
|
-
const stats: ProcessingStats = {
|
|
432
|
-
filesProcessed: 0,
|
|
433
|
-
filesSucceeded: 0,
|
|
434
|
-
filesFailed: 0,
|
|
435
|
-
ordersCreated: 0,
|
|
436
|
-
errors: [],
|
|
437
|
-
};
|
|
438
|
-
|
|
439
|
-
try {
|
|
440
|
-
logger.info('Listing files from SFTP', {
|
|
441
|
-
path: config.sftp.remotePath,
|
|
442
|
-
pattern: config.sftp.filePattern,
|
|
443
|
-
});
|
|
444
|
-
|
|
445
|
-
// List files from SFTP
|
|
446
|
-
const files = await sftpSource.listFiles({
|
|
447
|
-
remotePath: config.sftp.remotePath,
|
|
448
|
-
filePattern: config.sftp.filePattern,
|
|
449
|
-
});
|
|
450
|
-
|
|
451
|
-
logger.info(`Found ${files.length} files to process`);
|
|
452
|
-
|
|
453
|
-
// Process each file
|
|
454
|
-
for (const file of files) {
|
|
455
|
-
stats.filesProcessed++;
|
|
456
|
-
|
|
457
|
-
try {
|
|
458
|
-
logger.info(`Processing file: ${file.name}`, {
|
|
459
|
-
size: file.size,
|
|
460
|
-
lastModified: file.lastModified,
|
|
461
|
-
});
|
|
462
|
-
|
|
463
|
-
await processOrderFile(file.name, stats);
|
|
464
|
-
stats.filesSucceeded++;
|
|
465
|
-
} catch (error: any) {
|
|
466
|
-
stats.filesFailed++;
|
|
467
|
-
stats.errors.push({
|
|
468
|
-
file: file.name,
|
|
469
|
-
error: error.message,
|
|
470
|
-
});
|
|
471
|
-
|
|
472
|
-
logger.error(`Failed to process file: ${file.name}`, error);
|
|
473
|
-
|
|
474
|
-
// Move to error folder
|
|
475
|
-
try {
|
|
476
|
-
const errorPath = `${config.sftp.errorPath}/${file.name}`;
|
|
477
|
-
await sftpSource.moveFile(file.path, errorPath, true); // Use file.path for source (full path)
|
|
478
|
-
logger.info(`Moved failed file to: ${errorPath}`);
|
|
479
|
-
} catch (moveError: any) {
|
|
480
|
-
logger.error('Failed to move file to error folder', moveError);
|
|
481
|
-
}
|
|
482
|
-
}
|
|
483
|
-
}
|
|
484
|
-
|
|
485
|
-
logger.info('Processing cycle completed', stats);
|
|
486
|
-
} catch (error: any) {
|
|
487
|
-
logger.error('Error during file listing', error);
|
|
488
|
-
throw error;
|
|
489
|
-
}
|
|
490
|
-
|
|
491
|
-
return stats;
|
|
492
|
-
}
|
|
493
|
-
|
|
494
|
-
async function processOrderFile(fileName: string, stats: ProcessingStats): Promise<void> {
|
|
495
|
-
// Step 1: Download XML file from SFTP
|
|
496
|
-
logger.debug('Downloading file from SFTP', { fileName });
|
|
497
|
-
const xmlContent = await sftpSource.downloadFile(fileName, { encoding: 'utf8' }) as string;
|
|
498
|
-
logger.debug('Downloaded file', {
|
|
499
|
-
fileName,
|
|
500
|
-
size: xmlContent.length,
|
|
501
|
-
});
|
|
502
|
-
|
|
503
|
-
// Step 2: Parse XML
|
|
504
|
-
logger.debug('Parsing XML', { fileName });
|
|
505
|
-
const parsedXml = await xmlParser.parse(xmlContent, {
|
|
506
|
-
includeAttributes: true,
|
|
507
|
-
parseNumbers: true,
|
|
508
|
-
parseBooleans: true,
|
|
509
|
-
normalizeWhitespace: true,
|
|
510
|
-
});
|
|
511
|
-
|
|
512
|
-
logger.debug('XML parsed successfully', {
|
|
513
|
-
fileName,
|
|
514
|
-
rootElements: Object.keys(parsedXml),
|
|
515
|
-
});
|
|
516
|
-
|
|
517
|
-
// Step 3: Map to GraphQL mutation using custom resolvers
|
|
518
|
-
logger.debug('Mapping XML to GraphQL mutation', { fileName });
|
|
519
|
-
|
|
520
|
-
const customResolvers = {
|
|
521
|
-
// Resolver to inject source file name into attributes
|
|
522
|
-
'custom.sourceFile': () => fileName,
|
|
523
|
-
|
|
524
|
-
// Resolver to calculate item total if not provided
|
|
525
|
-
'custom.calculateItemTotal': (value: any, context: any) => {
|
|
526
|
-
const quantity = context.quantity || 1;
|
|
527
|
-
const price = context.price || 0;
|
|
528
|
-
return quantity * price;
|
|
529
|
-
},
|
|
530
|
-
};
|
|
531
|
-
|
|
532
|
-
// Use mapSafe() for error-safe mapping (recommended for production)
|
|
533
|
-
const result = await mutationMapper.mapSafe(parsedXml, {
|
|
534
|
-
beforeMapping: async (data, ctx) => {
|
|
535
|
-
ctx.logger?.debug('Before mapping hook', {
|
|
536
|
-
hasOrder: !!data.order,
|
|
537
|
-
});
|
|
538
|
-
return data;
|
|
539
|
-
},
|
|
540
|
-
afterMapping: async (variables, ctx) => {
|
|
541
|
-
// Inject source file name via custom attribute
|
|
542
|
-
if (variables.input && typeof variables.input === 'object') {
|
|
543
|
-
const input = variables.input as any;
|
|
544
|
-
if (input.attributes) {
|
|
545
|
-
input.attributes.sourceFile = fileName;
|
|
546
|
-
}
|
|
547
|
-
}
|
|
548
|
-
|
|
549
|
-
ctx.logger?.debug('After mapping hook', {
|
|
550
|
-
ref: (variables.input as any)?.ref,
|
|
551
|
-
itemCount: (variables.input as any)?.items?.length,
|
|
552
|
-
});
|
|
553
|
-
|
|
554
|
-
return variables;
|
|
555
|
-
},
|
|
556
|
-
});
|
|
557
|
-
|
|
558
|
-
// Check for mapping errors
|
|
559
|
-
if (!result.success) {
|
|
560
|
-
logger.error('Mapping failed', {
|
|
561
|
-
fileName,
|
|
562
|
-
errors: result.errors,
|
|
563
|
-
});
|
|
564
|
-
return; // Skip this file
|
|
565
|
-
}
|
|
566
|
-
|
|
567
|
-
logger.debug('Mapping completed', {
|
|
568
|
-
fileName,
|
|
569
|
-
orderRef: (result.variables.input as any)?.ref,
|
|
570
|
-
itemCount: (result.variables.input as any)?.items?.length,
|
|
571
|
-
});
|
|
572
|
-
|
|
573
|
-
// Step 4: Execute GraphQL mutation
|
|
574
|
-
logger.info('Creating order in Fluent', {
|
|
575
|
-
fileName,
|
|
576
|
-
orderRef: (result.variables.input as any)?.ref,
|
|
577
|
-
});
|
|
578
|
-
|
|
579
|
-
const apiResult = await fluentClient.graphql({
|
|
580
|
-
query: result.query,
|
|
581
|
-
variables: result.variables,
|
|
582
|
-
});
|
|
583
|
-
|
|
584
|
-
if (apiResult.errors && apiResult.errors.length > 0) {
|
|
585
|
-
// Classify GraphQL errors to determine retry strategy
|
|
586
|
-
const classification = classifyErrors(apiResult.errors);
|
|
587
|
-
|
|
588
|
-
logger.error('GraphQL errors', {
|
|
589
|
-
fileName,
|
|
590
|
-
errorCode: classification.errorCode,
|
|
591
|
-
retryable: classification.retryable,
|
|
592
|
-
action: classification.action,
|
|
593
|
-
message: classification.userMessage,
|
|
594
|
-
errors: apiResult.errors,
|
|
595
|
-
});
|
|
596
|
-
|
|
597
|
-
// Don't retry non-retryable errors (e.g., order already exists)
|
|
598
|
-
if (!classification.retryable) {
|
|
599
|
-
throw new Error(classification.userMessage);
|
|
600
|
-
}
|
|
601
|
-
|
|
602
|
-
// For retryable errors, you could implement retry logic here
|
|
603
|
-
throw new Error(classification.userMessage);
|
|
604
|
-
}
|
|
605
|
-
|
|
606
|
-
const createdOrder = (result.data as any)?.createOrder;
|
|
607
|
-
|
|
608
|
-
logger.info('Order created successfully', {
|
|
609
|
-
fileName,
|
|
610
|
-
orderId: createdOrder?.id,
|
|
611
|
-
orderRef: createdOrder?.ref,
|
|
612
|
-
status: createdOrder?.status,
|
|
613
|
-
});
|
|
614
|
-
|
|
615
|
-
stats.ordersCreated++;
|
|
616
|
-
|
|
617
|
-
// Step 5: Archive processed file
|
|
618
|
-
const archivePath = `${config.sftp.archivePath}/${new Date().toISOString().split('T')[0]}/${fileName}`;
|
|
619
|
-
logger.debug('Archiving file', { fileName, archivePath });
|
|
620
|
-
|
|
621
|
-
// Create archive directory if it doesn't exist
|
|
622
|
-
const archiveDir = archivePath.substring(0, archivePath.lastIndexOf('/'));
|
|
623
|
-
await sftpSource.createDirectory(archiveDir, true);
|
|
624
|
-
|
|
625
|
-
// Move file to archive
|
|
626
|
-
await sftpSource.moveFile(fileName, archivePath, false);
|
|
627
|
-
|
|
628
|
-
logger.info('File archived successfully', {
|
|
629
|
-
fileName,
|
|
630
|
-
archivePath,
|
|
631
|
-
});
|
|
632
|
-
}
|
|
633
|
-
|
|
634
|
-
// ============================================================================
|
|
635
|
-
// MAIN EXECUTION LOOP
|
|
636
|
-
// ============================================================================
|
|
637
|
-
|
|
638
|
-
let isRunning = false;
|
|
639
|
-
let shutdownRequested = false;
|
|
640
|
-
|
|
641
|
-
async function runProcessingCycle() {
|
|
642
|
-
if (isRunning) {
|
|
643
|
-
logger.warn('Processing cycle already running, skipping...');
|
|
644
|
-
return;
|
|
645
|
-
}
|
|
646
|
-
|
|
647
|
-
isRunning = true;
|
|
648
|
-
|
|
649
|
-
try {
|
|
650
|
-
const stats = await processOrderFiles();
|
|
651
|
-
|
|
652
|
-
// Log summary
|
|
653
|
-
logger.info('Processing cycle summary', {
|
|
654
|
-
filesProcessed: stats.filesProcessed,
|
|
655
|
-
filesSucceeded: stats.filesSucceeded,
|
|
656
|
-
filesFailed: stats.filesFailed,
|
|
657
|
-
ordersCreated: stats.ordersCreated,
|
|
658
|
-
errorCount: stats.errors.length,
|
|
659
|
-
});
|
|
660
|
-
|
|
661
|
-
// Log errors if any
|
|
662
|
-
if (stats.errors.length > 0) {
|
|
663
|
-
logger.warn('Processing errors occurred', {
|
|
664
|
-
errors: stats.errors.slice(0, 5), // First 5 errors
|
|
665
|
-
totalErrors: stats.errors.length,
|
|
666
|
-
});
|
|
667
|
-
}
|
|
668
|
-
} catch (error: any) {
|
|
669
|
-
logger.error('Processing cycle failed', error);
|
|
670
|
-
} finally {
|
|
671
|
-
isRunning = false;
|
|
672
|
-
}
|
|
673
|
-
}
|
|
674
|
-
|
|
675
|
-
async function startPolling() {
|
|
676
|
-
logger.info('Starting polling loop', {
|
|
677
|
-
interval: config.processing.pollingInterval,
|
|
678
|
-
intervalSeconds: config.processing.pollingInterval / 1000,
|
|
679
|
-
});
|
|
680
|
-
|
|
681
|
-
// Run initial cycle
|
|
682
|
-
await runProcessingCycle();
|
|
683
|
-
|
|
684
|
-
// Set up polling interval
|
|
685
|
-
const intervalId = setInterval(async () => {
|
|
686
|
-
if (shutdownRequested) {
|
|
687
|
-
clearInterval(intervalId);
|
|
688
|
-
logger.info('Polling stopped due to shutdown request');
|
|
689
|
-
return;
|
|
690
|
-
}
|
|
691
|
-
|
|
692
|
-
await runProcessingCycle();
|
|
693
|
-
}, config.processing.pollingInterval);
|
|
694
|
-
|
|
695
|
-
// Handle graceful shutdown
|
|
696
|
-
const shutdown = async () => {
|
|
697
|
-
if (shutdownRequested) return;
|
|
698
|
-
|
|
699
|
-
shutdownRequested = true;
|
|
700
|
-
logger.info('Shutdown signal received, stopping gracefully...');
|
|
701
|
-
|
|
702
|
-
clearInterval(intervalId);
|
|
703
|
-
|
|
704
|
-
// Wait for current processing to complete
|
|
705
|
-
let waitCount = 0;
|
|
706
|
-
while (isRunning && waitCount < 30) {
|
|
707
|
-
logger.info('Waiting for current processing to complete...');
|
|
708
|
-
await new Promise(resolve => setTimeout(resolve, 1000));
|
|
709
|
-
waitCount++;
|
|
710
|
-
}
|
|
711
|
-
|
|
712
|
-
// Cleanup
|
|
713
|
-
await sftpSource.dispose();
|
|
714
|
-
|
|
715
|
-
logger.info('Shutdown complete');
|
|
716
|
-
process.exit(0);
|
|
717
|
-
};
|
|
718
|
-
|
|
719
|
-
process.on('SIGINT', shutdown);
|
|
720
|
-
process.on('SIGTERM', shutdown);
|
|
721
|
-
}
|
|
722
|
-
|
|
723
|
-
// ============================================================================
|
|
724
|
-
// ENTRY POINT
|
|
725
|
-
// ============================================================================
|
|
726
|
-
|
|
727
|
-
async function main() {
|
|
728
|
-
try {
|
|
729
|
-
logger.info('SFTP XML Order Sync Starting...', {
|
|
730
|
-
sftpHost: config.sftp.host,
|
|
731
|
-
remotePath: config.sftp.remotePath,
|
|
732
|
-
fluentBaseUrl: config.fluent.baseUrl,
|
|
733
|
-
retailerId: config.fluent.retailerId,
|
|
734
|
-
});
|
|
735
|
-
|
|
736
|
-
// Validate configuration
|
|
737
|
-
if (!config.fluent.clientId || !config.fluent.clientSecret) {
|
|
738
|
-
throw new Error('Fluent OAuth2 credentials are required (FLUENT_CLIENT_ID, FLUENT_CLIENT_SECRET)');
|
|
739
|
-
}
|
|
740
|
-
|
|
741
|
-
if (!config.sftp.host || !config.sftp.username) {
|
|
742
|
-
throw new Error('SFTP connection details are required (SFTP_HOST, SFTP_USERNAME)');
|
|
743
|
-
}
|
|
744
|
-
|
|
745
|
-
if (!config.sftp.password && !config.sftp.privateKey) {
|
|
746
|
-
throw new Error('SFTP authentication required (SFTP_PASSWORD or SFTP_PRIVATE_KEY_PATH)');
|
|
747
|
-
}
|
|
748
|
-
|
|
749
|
-
// Initialize services
|
|
750
|
-
await initializeServices();
|
|
751
|
-
|
|
752
|
-
// Validate connections
|
|
753
|
-
logger.info('Validating SFTP connection...');
|
|
754
|
-
const sftpValid = await sftpSource.validateConnection();
|
|
755
|
-
if (!sftpValid) {
|
|
756
|
-
throw new Error('SFTP connection validation failed');
|
|
757
|
-
}
|
|
758
|
-
logger.info('SFTP connection validated');
|
|
759
|
-
|
|
760
|
-
// Start polling loop
|
|
761
|
-
await startPolling();
|
|
762
|
-
} catch (error: any) {
|
|
763
|
-
logger.error('Fatal error during startup', error);
|
|
764
|
-
process.exit(1);
|
|
765
|
-
}
|
|
766
|
-
}
|
|
767
|
-
|
|
768
|
-
// Run the script
|
|
769
|
-
main();
|
|
770
|
-
```
|
|
771
|
-
|
|
772
|
-
---
|
|
773
|
-
|
|
774
|
-
## Key Patterns Explained
|
|
775
|
-
|
|
776
|
-
### Pattern 1: SFTP Connection & File Operations
|
|
777
|
-
|
|
778
|
-
**Connection Management:**
|
|
779
|
-
|
|
780
|
-
```typescript
|
|
781
|
-
const sftpSource = new SftpDataSource({
|
|
782
|
-
type: 'SFTP_XML',
|
|
783
|
-
connectionId: 'sftp-orders',
|
|
784
|
-
name: 'Order SFTP',
|
|
785
|
-
settings: {
|
|
786
|
-
host: config.sftp.host,
|
|
787
|
-
port: config.sftp.port,
|
|
788
|
-
username: config.sftp.username,
|
|
789
|
-
// Authentication: password OR private key
|
|
790
|
-
password: config.sftp.password,
|
|
791
|
-
privateKey: config.sftp.privateKey,
|
|
792
|
-
passphrase: config.sftp.passphrase,
|
|
793
|
-
remotePath: config.sftp.remotePath,
|
|
794
|
-
filePattern: config.sftp.filePattern,
|
|
795
|
-
connectionTimeout: 30000,
|
|
796
|
-
keepAliveInterval: 10000,
|
|
797
|
-
},
|
|
798
|
-
}, logger);
|
|
799
|
-
```
|
|
800
|
-
|
|
801
|
-
**File Listing:**
|
|
802
|
-
|
|
803
|
-
```typescript
|
|
804
|
-
// List files matching pattern
|
|
805
|
-
const files = await sftpSource.listFiles({
|
|
806
|
-
remotePath: '/orders/incoming',
|
|
807
|
-
filePattern: 'ORDER_*.xml',
|
|
808
|
-
});
|
|
809
|
-
|
|
810
|
-
// Filter by last modified time
|
|
811
|
-
const files = await sftpSource.listFiles({
|
|
812
|
-
remotePath: '/orders/incoming',
|
|
813
|
-
filePattern: 'ORDER_*.xml',
|
|
814
|
-
lastProcessedTimestamp: '2024-01-01T00:00:00Z',
|
|
815
|
-
});
|
|
816
|
-
```
|
|
817
|
-
|
|
818
|
-
**File Download:**
|
|
819
|
-
|
|
820
|
-
```typescript
|
|
821
|
-
// Download as string
|
|
822
|
-
const xmlContent = await sftpSource.downloadFile(fileName, {
|
|
823
|
-
encoding: 'utf8'
|
|
824
|
-
}) as string;
|
|
825
|
-
|
|
826
|
-
// Download as buffer
|
|
827
|
-
const buffer = await sftpSource.downloadFile(fileName, {
|
|
828
|
-
asBuffer: true
|
|
829
|
-
}) as Buffer;
|
|
830
|
-
```
|
|
831
|
-
|
|
832
|
-
**File Archival:**
|
|
833
|
-
|
|
834
|
-
```typescript
|
|
835
|
-
// Create archive directory (recursive)
|
|
836
|
-
await sftpSource.createDirectory(archiveDir, true);
|
|
837
|
-
|
|
838
|
-
// Move file to archive
|
|
839
|
-
await sftpSource.moveFile(
|
|
840
|
-
fileName,
|
|
841
|
-
archivePath,
|
|
842
|
-
false // overwrite
|
|
843
|
-
);
|
|
844
|
-
|
|
845
|
-
// Move to error folder
|
|
846
|
-
await sftpSource.moveFile(
|
|
847
|
-
fileName,
|
|
848
|
-
errorPath,
|
|
849
|
-
true // overwrite
|
|
850
|
-
);
|
|
851
|
-
```
|
|
852
|
-
|
|
853
|
-
### Pattern 2: XML Parsing & Validation
|
|
854
|
-
|
|
855
|
-
**Basic XML Parsing:**
|
|
856
|
-
|
|
857
|
-
```typescript
|
|
858
|
-
const xmlParser = new XMLParserService();
|
|
859
|
-
|
|
860
|
-
const parsedXml = await xmlParser.parse(xmlContent, {
|
|
861
|
-
includeAttributes: true, // Parse @id, @type attributes
|
|
862
|
-
parseNumbers: true, // Auto-convert "123" → 123
|
|
863
|
-
parseBooleans: true, // Auto-convert "true" → true
|
|
864
|
-
normalizeWhitespace: true, // Collapse whitespace
|
|
865
|
-
});
|
|
866
|
-
```
|
|
867
|
-
|
|
868
|
-
**XML with Namespaces:**
|
|
869
|
-
|
|
870
|
-
```typescript
|
|
871
|
-
const parsedXml = await xmlParser.parse(xmlContent, {
|
|
872
|
-
includeAttributes: true,
|
|
873
|
-
removeNamespacePrefix: true, // Remove ns: prefix
|
|
874
|
-
});
|
|
875
|
-
```
|
|
876
|
-
|
|
877
|
-
**Array Element Handling:**
|
|
878
|
-
|
|
879
|
-
```typescript
|
|
880
|
-
const parsedXml = await xmlParser.parse(xmlContent, {
|
|
881
|
-
includeAttributes: true,
|
|
882
|
-
arrayElements: ['item', 'product'], // Force arrays
|
|
883
|
-
});
|
|
884
|
-
```
|
|
885
|
-
|
|
886
|
-
**XML Structure Example:**
|
|
887
|
-
|
|
888
|
-
```xml
|
|
889
|
-
<order id="ORD-123" order-date="2024-01-15T10:30:00Z">
|
|
890
|
-
<customer>
|
|
891
|
-
<first-name>John</first-name>
|
|
892
|
-
<last-name>Doe</last-name>
|
|
893
|
-
<email>john@example.com</email>
|
|
894
|
-
<phone>+1-555-0123</phone>
|
|
895
|
-
</customer>
|
|
896
|
-
<shipping>
|
|
897
|
-
<name>John Doe</name>
|
|
898
|
-
<street>123 Main St</street>
|
|
899
|
-
<city>New York</city>
|
|
900
|
-
<state>NY</state>
|
|
901
|
-
<postcode>10001</postcode>
|
|
902
|
-
<country>US</country>
|
|
903
|
-
</shipping>
|
|
904
|
-
<items>
|
|
905
|
-
<item id="1">
|
|
906
|
-
<sku>PROD-001</sku>
|
|
907
|
-
<quantity>2</quantity>
|
|
908
|
-
<price>29.99</price>
|
|
909
|
-
<total-price>59.98</total-price>
|
|
910
|
-
</item>
|
|
911
|
-
<item id="2">
|
|
912
|
-
<sku>PROD-002</sku>
|
|
913
|
-
<quantity>1</quantity>
|
|
914
|
-
<price>49.99</price>
|
|
915
|
-
<total-price>49.99</total-price>
|
|
916
|
-
</item>
|
|
917
|
-
</items>
|
|
918
|
-
<totals>
|
|
919
|
-
<subtotal>109.97</subtotal>
|
|
920
|
-
<tax>10.00</tax>
|
|
921
|
-
<total>119.97</total>
|
|
922
|
-
</totals>
|
|
923
|
-
</order>
|
|
924
|
-
```
|
|
925
|
-
|
|
926
|
-
**Parsed Result:**
|
|
927
|
-
|
|
928
|
-
```javascript
|
|
929
|
-
{
|
|
930
|
-
order: {
|
|
931
|
-
'@id': 'ORD-123',
|
|
932
|
-
'@order-date': '2024-01-15T10:30:00Z',
|
|
933
|
-
customer: {
|
|
934
|
-
'first-name': 'John',
|
|
935
|
-
'last-name': 'Doe',
|
|
936
|
-
email: 'john@example.com',
|
|
937
|
-
phone: '+1-555-0123'
|
|
938
|
-
},
|
|
939
|
-
shipping: { ... },
|
|
940
|
-
items: {
|
|
941
|
-
item: [
|
|
942
|
-
{ '@id': '1', sku: 'PROD-001', quantity: 2, price: 29.99, ... },
|
|
943
|
-
{ '@id': '2', sku: 'PROD-002', quantity: 1, price: 49.99, ... }
|
|
944
|
-
]
|
|
945
|
-
},
|
|
946
|
-
totals: {
|
|
947
|
-
subtotal: 109.97,
|
|
948
|
-
tax: 10,
|
|
949
|
-
total: 119.97
|
|
950
|
-
}
|
|
951
|
-
}
|
|
952
|
-
}
|
|
953
|
-
```
|
|
954
|
-
|
|
955
|
-
### Pattern 3: GraphQL Mutation Mapping
|
|
956
|
-
|
|
957
|
-
**Mapping Configuration:**
|
|
958
|
-
|
|
959
|
-
```typescript
|
|
960
|
-
const mappingConfig: MappingConfig = {
|
|
961
|
-
version: '1.0',
|
|
962
|
-
mutation: 'createOrder',
|
|
963
|
-
sourceFormat: 'xml',
|
|
964
|
-
arguments: {
|
|
965
|
-
input: {
|
|
966
|
-
_type: 'CreateOrderInput!',
|
|
967
|
-
|
|
968
|
-
// Static value
|
|
969
|
-
type: {
|
|
970
|
-
value: 'HD',
|
|
971
|
-
},
|
|
972
|
-
|
|
973
|
-
// Source path
|
|
974
|
-
ref: {
|
|
975
|
-
source: 'order.@id',
|
|
976
|
-
required: true,
|
|
977
|
-
},
|
|
978
|
-
|
|
979
|
-
// Nested object
|
|
980
|
-
customer: {
|
|
981
|
-
firstName: {
|
|
982
|
-
source: 'order.customer.first-name',
|
|
983
|
-
transform: 'trim',
|
|
984
|
-
},
|
|
985
|
-
email: {
|
|
986
|
-
source: 'order.customer.email',
|
|
987
|
-
transform: 'toLowerCase',
|
|
988
|
-
},
|
|
989
|
-
},
|
|
990
|
-
|
|
991
|
-
// Array mapping
|
|
992
|
-
items: {
|
|
993
|
-
_array: true,
|
|
994
|
-
_autoWrap: true, // Single item → [item]
|
|
995
|
-
source: 'order.items.item',
|
|
996
|
-
ref: { source: '@id' },
|
|
997
|
-
productRef: { source: 'sku' },
|
|
998
|
-
quantity: {
|
|
999
|
-
source: 'quantity',
|
|
1000
|
-
transform: 'parseInt',
|
|
1001
|
-
},
|
|
1002
|
-
price: {
|
|
1003
|
-
source: 'price',
|
|
1004
|
-
transform: 'parseFloat',
|
|
1005
|
-
},
|
|
1006
|
-
},
|
|
1007
|
-
},
|
|
1008
|
-
},
|
|
1009
|
-
returnFields: [
|
|
1010
|
-
'id',
|
|
1011
|
-
'ref',
|
|
1012
|
-
'status',
|
|
1013
|
-
'customer { firstName lastName }',
|
|
1014
|
-
],
|
|
1015
|
-
};
|
|
1016
|
-
```
|
|
1017
|
-
|
|
1018
|
-
**Built-in Transforms:**
|
|
1019
|
-
|
|
1020
|
-
- `parseInt`, `parseFloat` - Number conversion
|
|
1021
|
-
- `toString` - String conversion
|
|
1022
|
-
- `toUpperCase`, `toLowerCase`, `trim` - String manipulation
|
|
1023
|
-
- `toBoolean` - Boolean conversion
|
|
1024
|
-
- `toISO8601`, `toDate` - Date formatting
|
|
1025
|
-
|
|
1026
|
-
**Execute Mapping:**
|
|
1027
|
-
|
|
1028
|
-
```typescript
|
|
1029
|
-
const mapper = new GraphQLMutationMapper(
|
|
1030
|
-
mappingConfig,
|
|
1031
|
-
fluentClient,
|
|
1032
|
-
logger
|
|
1033
|
-
);
|
|
1034
|
-
|
|
1035
|
-
// Use mapSafe() for error-safe mapping (recommended)
|
|
1036
|
-
const result = await mapper.mapSafe(parsedXml);
|
|
1037
|
-
if (!result.success) {
|
|
1038
|
-
console.error('Mapping failed:', result.errors);
|
|
1039
|
-
return;
|
|
1040
|
-
}
|
|
1041
|
-
// Result: { success: true, query: '...', variables: { ... } }
|
|
1042
|
-
```
|
|
1043
|
-
|
|
1044
|
-
### Pattern 4: Custom Resolvers for Complex Logic
|
|
1045
|
-
|
|
1046
|
-
**Custom Resolver Example:**
|
|
1047
|
-
|
|
1048
|
-
```typescript
|
|
1049
|
-
const customResolvers = {
|
|
1050
|
-
// Simple resolver
|
|
1051
|
-
'custom.sourceFile': (value: any, context: any) => {
|
|
1052
|
-
return fileName;
|
|
1053
|
-
},
|
|
1054
|
-
|
|
1055
|
-
// Context-aware resolver
|
|
1056
|
-
'custom.calculateTotal': (value: any, context: any) => {
|
|
1057
|
-
const quantity = context.quantity || 1;
|
|
1058
|
-
const price = context.price || 0;
|
|
1059
|
-
return quantity * price;
|
|
1060
|
-
},
|
|
1061
|
-
|
|
1062
|
-
// Async resolver (API call)
|
|
1063
|
-
'custom.lookupCustomer': async (value: any, context: any, config: any, helpers: any) => {
|
|
1064
|
-
const email = helpers.get(context, 'customer.email');
|
|
1065
|
-
const query = `query GetCustomer($email: String!) {
|
|
1066
|
-
customerByEmail(email: $email) {
|
|
1067
|
-
id
|
|
1068
|
-
ref
|
|
1069
|
-
}
|
|
1070
|
-
}`;
|
|
1071
|
-
const result = await fluentClient.graphql({
|
|
1072
|
-
query,
|
|
1073
|
-
variables: { email },
|
|
1074
|
-
});
|
|
1075
|
-
return result.data?.customerByEmail?.id;
|
|
1076
|
-
},
|
|
1077
|
-
|
|
1078
|
-
// Conditional logic
|
|
1079
|
-
'custom.determineShippingMethod': (value: any, context: any) => {
|
|
1080
|
-
const totalPrice = context.totalPrice || 0;
|
|
1081
|
-
if (totalPrice > 100) {
|
|
1082
|
-
return 'EXPRESS';
|
|
1083
|
-
} else if (totalPrice > 50) {
|
|
1084
|
-
return 'STANDARD';
|
|
1085
|
-
} else {
|
|
1086
|
-
return 'ECONOMY';
|
|
1087
|
-
}
|
|
1088
|
-
},
|
|
1089
|
-
};
|
|
1090
|
-
|
|
1091
|
-
// ✅ CORRECT: Use mapWithNodes() when custom resolvers are defined
|
|
1092
|
-
const payload = await mapper.mapWithNodes(parsedXml, customResolvers, {
|
|
1093
|
-
fluentClient: fluentClient as any,
|
|
1094
|
-
config: {},
|
|
1095
|
-
helpers: { fluentClient: fluentClient as any, logger },
|
|
1096
|
-
});
|
|
1097
|
-
|
|
1098
|
-
// ❌ WRONG: map() does NOT accept custom resolvers
|
|
1099
|
-
// const payload = await mapper.map(parsedXml, customResolvers);
|
|
1100
|
-
```
|
|
1101
|
-
|
|
1102
|
-
**Helper Functions Available:**
|
|
1103
|
-
|
|
1104
|
-
```typescript
|
|
1105
|
-
// helpers.get - Extract nested value
|
|
1106
|
-
const email = helpers.get(context, 'customer.email');
|
|
1107
|
-
|
|
1108
|
-
// helpers.logger - Logging
|
|
1109
|
-
helpers.logger.info('Processing order', { ref: orderRef });
|
|
1110
|
-
|
|
1111
|
-
// helpers.ensureArray - Array conversion
|
|
1112
|
-
const items = helpers.ensureArray(context.items);
|
|
1113
|
-
```
|
|
1114
|
-
|
|
1115
|
-
### Pattern 5: File Management (Archive/Error)
|
|
1116
|
-
|
|
1117
|
-
**Archival Strategy:**
|
|
1118
|
-
|
|
1119
|
-
```typescript
|
|
1120
|
-
// Date-based folder structure
|
|
1121
|
-
const today = new Date().toISOString().split('T')[0];
|
|
1122
|
-
const archivePath = `${config.sftp.archivePath}/${today}/${fileName}`;
|
|
1123
|
-
|
|
1124
|
-
// Create directory if needed
|
|
1125
|
-
await sftpSource.createDirectory(
|
|
1126
|
-
`${config.sftp.archivePath}/${today}`,
|
|
1127
|
-
true // recursive
|
|
1128
|
-
);
|
|
1129
|
-
|
|
1130
|
-
// Move file
|
|
1131
|
-
await sftpSource.moveFile(fileName, archivePath, false);
|
|
1132
|
-
```
|
|
1133
|
-
|
|
1134
|
-
**Error Handling:**
|
|
1135
|
-
|
|
1136
|
-
```typescript
|
|
1137
|
-
try {
|
|
1138
|
-
await processOrderFile(fileName, stats);
|
|
1139
|
-
// Success: Archive
|
|
1140
|
-
await sftpSource.moveFile(fileName, archivePath, false);
|
|
1141
|
-
} catch (error: any) {
|
|
1142
|
-
// Failure: Move to error folder
|
|
1143
|
-
const errorPath = `${config.sftp.errorPath}/${fileName}`;
|
|
1144
|
-
await sftpSource.moveFile(fileName, errorPath, true);
|
|
1145
|
-
|
|
1146
|
-
// Create error report
|
|
1147
|
-
const errorReport = {
|
|
1148
|
-
file: fileName,
|
|
1149
|
-
error: error.message,
|
|
1150
|
-
timestamp: new Date().toISOString(),
|
|
1151
|
-
};
|
|
1152
|
-
await sftpSource.uploadFile(null, 2, JSON.stringify(errorReport),
|
|
1153
|
-
`${config.sftp.errorPath}/${fileName}.error.json`
|
|
1154
|
-
);
|
|
1155
|
-
}
|
|
1156
|
-
```
|
|
1157
|
-
|
|
1158
|
-
---
|
|
1159
|
-
|
|
1160
|
-
## Deployment Options
|
|
1161
|
-
|
|
1162
|
-
### Option 1: Local Execution
|
|
1163
|
-
|
|
1164
|
-
```bash
|
|
1165
|
-
# Development
|
|
1166
|
-
npm run dev
|
|
1167
|
-
|
|
1168
|
-
# Production
|
|
1169
|
-
npm run build
|
|
1170
|
-
node dist/sftp-order-sync.js
|
|
1171
|
-
```
|
|
1172
|
-
|
|
1173
|
-
### Option 2: Docker Container
|
|
1174
|
-
|
|
1175
|
-
Create `Dockerfile`:
|
|
1176
|
-
|
|
1177
|
-
```dockerfile
|
|
1178
|
-
FROM node:18-alpine
|
|
1179
|
-
|
|
1180
|
-
WORKDIR /app
|
|
1181
|
-
|
|
1182
|
-
# Install dependencies
|
|
1183
|
-
COPY package*.json ./
|
|
1184
|
-
RUN npm ci --only=production
|
|
1185
|
-
|
|
1186
|
-
# Copy application
|
|
1187
|
-
COPY dist/ ./dist/
|
|
1188
|
-
COPY .env .env
|
|
1189
|
-
|
|
1190
|
-
# Run
|
|
1191
|
-
CMD ["node", "dist/sftp-order-sync.js"]
|
|
1192
|
-
```
|
|
1193
|
-
|
|
1194
|
-
Build and run:
|
|
1195
|
-
|
|
1196
|
-
```bash
|
|
1197
|
-
docker build -t sftp-order-sync .
|
|
1198
|
-
docker run -d --name sftp-order-sync --env-file .env sftp-order-sync
|
|
1199
|
-
```
|
|
1200
|
-
|
|
1201
|
-
### Option 3: Systemd Service
|
|
1202
|
-
|
|
1203
|
-
Create `/etc/systemd/system/sftp-order-sync.service`:
|
|
1204
|
-
|
|
1205
|
-
```ini
|
|
1206
|
-
[Unit]
|
|
1207
|
-
Description=SFTP Order Sync Service
|
|
1208
|
-
After=network.target
|
|
1209
|
-
|
|
1210
|
-
[Service]
|
|
1211
|
-
Type=simple
|
|
1212
|
-
User=nodeapp
|
|
1213
|
-
WorkingDirectory=/opt/sftp-order-sync
|
|
1214
|
-
ExecStart=/usr/bin/node /opt/sftp-order-sync/dist/sftp-order-sync.js
|
|
1215
|
-
Restart=always
|
|
1216
|
-
RestartSec=10
|
|
1217
|
-
StandardOutput=journal
|
|
1218
|
-
StandardError=journal
|
|
1219
|
-
SyslogIdentifier=sftp-order-sync
|
|
1220
|
-
Environment=NODE_ENV=production
|
|
1221
|
-
|
|
1222
|
-
[Install]
|
|
1223
|
-
WantedBy=multi-user.target
|
|
1224
|
-
```
|
|
1225
|
-
|
|
1226
|
-
Enable and start:
|
|
1227
|
-
|
|
1228
|
-
```bash
|
|
1229
|
-
sudo systemctl enable sftp-order-sync
|
|
1230
|
-
sudo systemctl start sftp-order-sync
|
|
1231
|
-
sudo journalctl -u sftp-order-sync -f
|
|
1232
|
-
```
|
|
1233
|
-
|
|
1234
|
-
### Option 4: Cron Job
|
|
1235
|
-
|
|
1236
|
-
Create `/opt/sftp-order-sync/run.sh`:
|
|
1237
|
-
|
|
1238
|
-
```bash
|
|
1239
|
-
#!/bin/bash
|
|
1240
|
-
cd /opt/sftp-order-sync
|
|
1241
|
-
node dist/sftp-order-sync.js >> /var/log/sftp-order-sync.log 2>&1
|
|
1242
|
-
```
|
|
1243
|
-
|
|
1244
|
-
Add to crontab:
|
|
1245
|
-
|
|
1246
|
-
```bash
|
|
1247
|
-
# Run every 5 minutes
|
|
1248
|
-
*/5 * * * * /opt/sftp-order-sync/run.sh
|
|
1249
|
-
```
|
|
1250
|
-
|
|
1251
|
-
---
|
|
1252
|
-
|
|
1253
|
-
## Testing
|
|
1254
|
-
|
|
1255
|
-
### Test with Sample XML
|
|
1256
|
-
|
|
1257
|
-
Create `test-order.xml`:
|
|
1258
|
-
|
|
1259
|
-
```xml
|
|
1260
|
-
<?xml version="1.0" encoding="UTF-8"?>
|
|
1261
|
-
<order id="TEST-001" order-date="2024-01-15T10:30:00Z">
|
|
1262
|
-
<customer>
|
|
1263
|
-
<first-name>John</first-name>
|
|
1264
|
-
<last-name>Doe</last-name>
|
|
1265
|
-
<email>john.doe@example.com</email>
|
|
1266
|
-
<phone>+1-555-0123</phone>
|
|
1267
|
-
</customer>
|
|
1268
|
-
<shipping>
|
|
1269
|
-
<name>John Doe</name>
|
|
1270
|
-
<street>123 Main Street</street>
|
|
1271
|
-
<city>New York</city>
|
|
1272
|
-
<state>NY</state>
|
|
1273
|
-
<postcode>10001</postcode>
|
|
1274
|
-
<country>US</country>
|
|
1275
|
-
</shipping>
|
|
1276
|
-
<items>
|
|
1277
|
-
<item id="1">
|
|
1278
|
-
<sku>PROD-001</sku>
|
|
1279
|
-
<quantity>2</quantity>
|
|
1280
|
-
<price>29.99</price>
|
|
1281
|
-
<total-price>59.98</total-price>
|
|
1282
|
-
</item>
|
|
1283
|
-
<item id="2">
|
|
1284
|
-
<sku>PROD-002</sku>
|
|
1285
|
-
<quantity>1</quantity>
|
|
1286
|
-
<price>49.99</price>
|
|
1287
|
-
<total-price>49.99</total-price>
|
|
1288
|
-
</item>
|
|
1289
|
-
</items>
|
|
1290
|
-
<totals>
|
|
1291
|
-
<subtotal>109.97</subtotal>
|
|
1292
|
-
<tax>10.00</tax>
|
|
1293
|
-
<total>119.97</total>
|
|
1294
|
-
</totals>
|
|
1295
|
-
</order>
|
|
1296
|
-
```
|
|
1297
|
-
|
|
1298
|
-
Upload to SFTP:
|
|
1299
|
-
|
|
1300
|
-
```bash
|
|
1301
|
-
sftp user@sftp.example.com
|
|
1302
|
-
cd /orders/incoming
|
|
1303
|
-
put test-order.xml ORDER_TEST_001.xml
|
|
1304
|
-
```
|
|
1305
|
-
|
|
1306
|
-
Monitor logs:
|
|
1307
|
-
|
|
1308
|
-
```bash
|
|
1309
|
-
# Local
|
|
1310
|
-
npm run dev
|
|
1311
|
-
|
|
1312
|
-
# Docker
|
|
1313
|
-
docker logs -f sftp-order-sync
|
|
1314
|
-
|
|
1315
|
-
# Systemd
|
|
1316
|
-
sudo journalctl -u sftp-order-sync -f
|
|
1317
|
-
```
|
|
1318
|
-
|
|
1319
|
-
---
|
|
1320
|
-
|
|
1321
|
-
## Common Issues
|
|
1322
|
-
|
|
1323
|
-
### Issue 1: SFTP Connection Timeout
|
|
1324
|
-
|
|
1325
|
-
**Symptoms:**
|
|
1326
|
-
|
|
1327
|
-
```
|
|
1328
|
-
Error: SFTP connection timeout after 30000ms
|
|
1329
|
-
```
|
|
1330
|
-
|
|
1331
|
-
**Solution:**
|
|
1332
|
-
|
|
1333
|
-
```typescript
|
|
1334
|
-
// Increase timeout in SFTP config
|
|
1335
|
-
settings: {
|
|
1336
|
-
connectionTimeout: 60000, // 60 seconds
|
|
1337
|
-
keepAliveInterval: 10000, // 10 seconds
|
|
1338
|
-
}
|
|
1339
|
-
```
|
|
1340
|
-
|
|
1341
|
-
### Issue 2: XML Parsing Fails for Attributes
|
|
1342
|
-
|
|
1343
|
-
**Symptoms:**
|
|
1344
|
-
|
|
1345
|
-
```
|
|
1346
|
-
Error: Cannot read property '@id' of undefined
|
|
1347
|
-
```
|
|
1348
|
-
|
|
1349
|
-
**Solution:**
|
|
1350
|
-
|
|
1351
|
-
```typescript
|
|
1352
|
-
// Ensure includeAttributes is enabled
|
|
1353
|
-
const parsedXml = await xmlParser.parse(xmlContent, {
|
|
1354
|
-
includeAttributes: true, // REQUIRED for @attributes
|
|
1355
|
-
});
|
|
1356
|
-
```
|
|
1357
|
-
|
|
1358
|
-
### Issue 3: Array Items Not Mapping
|
|
1359
|
-
|
|
1360
|
-
**Symptoms:**
|
|
1361
|
-
|
|
1362
|
-
```
|
|
1363
|
-
Error: Expected array at path 'order.items.item' but got object
|
|
1364
|
-
```
|
|
1365
|
-
|
|
1366
|
-
**Solution:**
|
|
1367
|
-
|
|
1368
|
-
```typescript
|
|
1369
|
-
// Enable auto-wrap for single items
|
|
1370
|
-
items: {
|
|
1371
|
-
_array: true,
|
|
1372
|
-
_autoWrap: true, // Convert single item to [item]
|
|
1373
|
-
source: 'order.items.item',
|
|
1374
|
-
}
|
|
1375
|
-
```
|
|
1376
|
-
|
|
1377
|
-
### Issue 4: GraphQL Mutation Fails
|
|
1378
|
-
|
|
1379
|
-
**Symptoms:**
|
|
1380
|
-
|
|
1381
|
-
```
|
|
1382
|
-
GraphQL errors: Field 'ref' is required but was null
|
|
1383
|
-
```
|
|
1384
|
-
|
|
1385
|
-
**Solution:**
|
|
1386
|
-
|
|
1387
|
-
```typescript
|
|
1388
|
-
// Check source path and add validation
|
|
1389
|
-
ref: {
|
|
1390
|
-
source: 'order.@id', // Correct attribute path
|
|
1391
|
-
required: true, // Fail fast if missing
|
|
1392
|
-
}
|
|
1393
|
-
|
|
1394
|
-
// Debug parsed XML structure
|
|
1395
|
-
logger.debug('Parsed XML structure', {
|
|
1396
|
-
keys: Object.keys(parsedXml),
|
|
1397
|
-
orderKeys: Object.keys(parsedXml.order || {}),
|
|
1398
|
-
});
|
|
1399
|
-
```
|
|
1400
|
-
|
|
1401
|
-
### Issue 5: Files Not Archiving
|
|
1402
|
-
|
|
1403
|
-
**Symptoms:**
|
|
1404
|
-
|
|
1405
|
-
```
|
|
1406
|
-
Error: Directory does not exist: /orders/processed/2024-01-15
|
|
1407
|
-
```
|
|
1408
|
-
|
|
1409
|
-
**Solution:**
|
|
1410
|
-
|
|
1411
|
-
```typescript
|
|
1412
|
-
// Create directory before moving
|
|
1413
|
-
const archiveDir = archivePath.substring(0, archivePath.lastIndexOf('/'));
|
|
1414
|
-
await sftpSource.createDirectory(archiveDir, true); // recursive
|
|
1415
|
-
|
|
1416
|
-
// Then move file
|
|
1417
|
-
await sftpSource.moveFile(fileName, archivePath, false);
|
|
1418
|
-
```
|
|
1419
|
-
|
|
1420
|
-
---
|
|
1421
|
-
|
|
1422
|
-
## Related Guides
|
|
1423
|
-
|
|
1424
|
-
- **[SFTP Credential Access & Security](../../02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md)** - Comprehensive SFTP credential management guide
|
|
1425
|
-
- **[S3 CSV Inventory Sync](./s3-csv-batch-api.md)** - Similar pattern for S3 CSV files
|
|
1426
|
-
- **[GraphQL Query Export](./graphql-query-export.md)** - GraphQL extraction pattern
|
|
1427
|
-
- **[Field Mapping Pattern](../patterns/field-mapping-universal.md)** - Complete field mapping reference
|
|
1428
|
-
- **[Error Handling Pattern](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md)** - Comprehensive error strategies
|
|
1429
|
-
|
|
1430
|
-
---
|
|
1431
|
-
|
|
1432
|
-
## Summary
|
|
1433
|
-
|
|
1434
|
-
This guide demonstrates a production-ready SFTP XML to Fluent GraphQL integration with:
|
|
1435
|
-
|
|
1436
|
-
- OAuth2 authentication
|
|
1437
|
-
- SFTP connection pooling
|
|
1438
|
-
- XML parsing with attribute support
|
|
1439
|
-
- GraphQL mutation mapping
|
|
1440
|
-
- Custom resolvers for complex logic
|
|
1441
|
-
- File archival and error handling
|
|
1442
|
-
- Graceful shutdown and retry logic
|
|
1443
|
-
|
|
1444
|
-
The complete script is ~600 lines and includes all necessary error handling, logging, and deployment configurations for production use.
|
|
1
|
+
# Standalone: SFTP XML → Fluent GraphQL
|
|
2
|
+
|
|
3
|
+
**FC Connect SDK Use Case Guide**
|
|
4
|
+
|
|
5
|
+
> **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
|
|
6
|
+
> **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
|
|
7
|
+
|
|
8
|
+
**Context**: Node.js script that reads XML order files from SFTP and creates orders in Fluent Commerce via GraphQL mutations
|
|
9
|
+
|
|
10
|
+
**Complexity**: Medium
|
|
11
|
+
|
|
12
|
+
**Runtime**: Node.js ≥18 / Deno
|
|
13
|
+
|
|
14
|
+
**Estimated Lines**: ~600 lines
|
|
15
|
+
|
|
16
|
+
## What You'll Build
|
|
17
|
+
|
|
18
|
+
- Standalone Node.js/Deno script
|
|
19
|
+
- OAuth2 authentication
|
|
20
|
+
- SFTP connection and file operations
|
|
21
|
+
- XML parsing with validation
|
|
22
|
+
- GraphQL mutation mapping
|
|
23
|
+
- Custom resolvers for transformations
|
|
24
|
+
- File archival after processing
|
|
25
|
+
- Error handling and logging
|
|
26
|
+
|
|
27
|
+
## SDK Methods Used
|
|
28
|
+
|
|
29
|
+
- `createClient({ config: { baseUrl, clientId, clientSecret, retailerId } })` - OAuth2 client
|
|
30
|
+
- `SftpDataSource(config, logger)` - SFTP operations
|
|
31
|
+
- `XMLParserService` - XML parsing
|
|
32
|
+
- `GraphQLMutationMapper(config, logger, { fluentClient, customResolvers })` - Map XML to GraphQL
|
|
33
|
+
- `mapper.mapSafe(xmlData)` - Transform data with error handling (recommended) OR `mapper.map(xmlData)` - Transform data (throws on error) OR `mapper.mapWithNodes(xmlData)` - Transform with custom resolvers
|
|
34
|
+
- `client.graphql({ query, variables })` - Execute mutation
|
|
35
|
+
|
|
36
|
+
---
|
|
37
|
+
|
|
38
|
+
## SFTP Credential Configuration
|
|
39
|
+
|
|
40
|
+
### For Standalone Scripts (Node.js/Deno)
|
|
41
|
+
|
|
42
|
+
**Standalone environments** load SFTP credentials from environment variables or configuration files:
|
|
43
|
+
|
|
44
|
+
```typescript
|
|
45
|
+
import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
|
|
46
|
+
|
|
47
|
+
// Load from environment variables (recommended)
|
|
48
|
+
const sftp = new SftpDataSource({
|
|
49
|
+
type: 'SFTP_XML',
|
|
50
|
+
connectionId: 'sftp-orders',
|
|
51
|
+
name: 'Order SFTP',
|
|
52
|
+
settings: {
|
|
53
|
+
host: process.env.SFTP_HOST || 'sftp.example.com',
|
|
54
|
+
port: parseInt(process.env.SFTP_PORT || '22'),
|
|
55
|
+
username: process.env.SFTP_USERNAME,
|
|
56
|
+
|
|
57
|
+
// Option 1: Password authentication
|
|
58
|
+
password: process.env.SFTP_PASSWORD,
|
|
59
|
+
|
|
60
|
+
// Option 2: SSH private key authentication (more secure)
|
|
61
|
+
// privateKey: fs.readFileSync(process.env.SFTP_KEY_PATH, 'utf8'),
|
|
62
|
+
// passphrase: process.env.SFTP_PASSPHRASE,
|
|
63
|
+
}
|
|
64
|
+
}, logger);
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
**Environment Variable Setup:**
|
|
68
|
+
|
|
69
|
+
```bash
|
|
70
|
+
# .env file
|
|
71
|
+
SFTP_HOST=sftp.example.com
|
|
72
|
+
SFTP_PORT=22
|
|
73
|
+
SFTP_USERNAME=your-username
|
|
74
|
+
SFTP_PASSWORD=your-password
|
|
75
|
+
|
|
76
|
+
# OR use SSH key
|
|
77
|
+
# SFTP_PRIVATE_KEY_PATH=/path/to/private/key
|
|
78
|
+
# SFTP_PASSPHRASE=key-passphrase
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
### For Versori Platform Deployments
|
|
82
|
+
|
|
83
|
+
**Versori platform** uses connection-based credential management. See the comprehensive guide:
|
|
84
|
+
|
|
85
|
+
- [SFTP Credential Access & Security](../../02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md)
|
|
86
|
+
|
|
87
|
+
**Key Differences:**
|
|
88
|
+
|
|
89
|
+
| Aspect | Standalone (Node.js/Deno) | Versori Platform |
|
|
90
|
+
|--------|---------------------------|------------------|
|
|
91
|
+
| **Credential Storage** | Environment variables, files | Versori connections |
|
|
92
|
+
| **Access Method** | Direct configuration | `connectionVariables`, `credentials()`, or `activation.connections` |
|
|
93
|
+
| **Security** | Manual secret management | Platform-managed encryption |
|
|
94
|
+
| **Rotation** | Manual updates | Centralized rotation |
|
|
95
|
+
|
|
96
|
+
**Security Best Practices:**
|
|
97
|
+
|
|
98
|
+
- Never hardcode credentials in source code
|
|
99
|
+
- Use environment variables or secret management systems
|
|
100
|
+
- Store SSH keys securely with appropriate file permissions
|
|
101
|
+
- Rotate credentials regularly
|
|
102
|
+
- Use SSH key authentication over passwords when possible
|
|
103
|
+
|
|
104
|
+
---
|
|
105
|
+
|
|
106
|
+
## Complete Working Script
|
|
107
|
+
|
|
108
|
+
This standalone script demonstrates a complete SFTP-to-Fluent order integration workflow.
|
|
109
|
+
|
|
110
|
+
### Step 1: Install Dependencies
|
|
111
|
+
|
|
112
|
+
```bash
|
|
113
|
+
npm install @fluentcommerce/fc-connect-sdk ssh2-sftp-client dotenv
|
|
114
|
+
npm install --save-dev @types/ssh2-sftp-client @types/node typescript
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
### Step 2: Environment Configuration
|
|
118
|
+
|
|
119
|
+
Create `.env` file:
|
|
120
|
+
|
|
121
|
+
```bash
|
|
122
|
+
# Fluent Commerce OAuth2
|
|
123
|
+
FLUENT_BASE_URL=https://yourinstance.api.fluentcommerce.com
|
|
124
|
+
FLUENT_CLIENT_ID=your-oauth-client-id
|
|
125
|
+
FLUENT_CLIENT_SECRET=your-oauth-client-secret
|
|
126
|
+
FLUENT_RETAILER_ID=your-retailer-id
|
|
127
|
+
|
|
128
|
+
# SFTP Connection
|
|
129
|
+
SFTP_HOST=sftp.example.com
|
|
130
|
+
SFTP_PORT=22
|
|
131
|
+
SFTP_USERNAME=sftp-user
|
|
132
|
+
SFTP_PASSWORD=sftp-password
|
|
133
|
+
|
|
134
|
+
# OR use SSH key
|
|
135
|
+
# SFTP_PRIVATE_KEY_PATH=/path/to/private/key
|
|
136
|
+
# SFTP_PASSPHRASE=key-passphrase
|
|
137
|
+
|
|
138
|
+
# Processing Options
|
|
139
|
+
SFTP_REMOTE_PATH=/orders/incoming
|
|
140
|
+
SFTP_ARCHIVE_PATH=/orders/processed
|
|
141
|
+
SFTP_ERROR_PATH=/orders/errors
|
|
142
|
+
FILE_PATTERN=ORDER_*.xml
|
|
143
|
+
POLLING_INTERVAL_MS=60000
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
### Step 3: Main Script Implementation
|
|
147
|
+
|
|
148
|
+
Create `sftp-order-sync.ts`:
|
|
149
|
+
|
|
150
|
+
```typescript
|
|
151
|
+
import 'dotenv/config';
|
|
152
|
+
|
|
153
|
+
// FC Connect SDK+
|
|
154
|
+
// Install: npm install @fluentcommerce/fc-connect-sdk@latest
|
|
155
|
+
// Docs: https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk
|
|
156
|
+
// GitHub: https://github.com/fluentcommerce/fc-connect-sdk
|
|
157
|
+
|
|
158
|
+
import {
|
|
159
|
+
createClient,
|
|
160
|
+
classifyErrors,
|
|
161
|
+
SftpDataSource,
|
|
162
|
+
XMLParserService,
|
|
163
|
+
GraphQLMutationMapper,
|
|
164
|
+
createConsoleLogger,
|
|
165
|
+
toStructuredLogger
|
|
166
|
+
} from '@fluentcommerce/fc-connect-sdk';
|
|
167
|
+
|
|
168
|
+
import type {
|
|
169
|
+
FluentClient,
|
|
170
|
+
StructuredLogger,
|
|
171
|
+
MappingConfig
|
|
172
|
+
} from '@fluentcommerce/fc-connect-sdk';
|
|
173
|
+
|
|
174
|
+
import { readFileSync } from 'fs';
|
|
175
|
+
import { join } from 'path';
|
|
176
|
+
|
|
177
|
+
// ============================================================================
|
|
178
|
+
// CONFIGURATION
|
|
179
|
+
// ============================================================================
|
|
180
|
+
|
|
181
|
+
const config = {
|
|
182
|
+
fluent: {
|
|
183
|
+
baseUrl: process.env.FLUENT_BASE_URL!,
|
|
184
|
+
clientId: process.env.FLUENT_CLIENT_ID!,
|
|
185
|
+
clientSecret: process.env.FLUENT_CLIENT_SECRET!,
|
|
186
|
+
retailerId: process.env.FLUENT_RETAILER_ID!,
|
|
187
|
+
},
|
|
188
|
+
sftp: {
|
|
189
|
+
host: process.env.SFTP_HOST!,
|
|
190
|
+
port: parseInt(process.env.SFTP_PORT || '22'),
|
|
191
|
+
username: process.env.SFTP_USERNAME!,
|
|
192
|
+
password: process.env.SFTP_PASSWORD,
|
|
193
|
+
privateKey: process.env.SFTP_PRIVATE_KEY_PATH
|
|
194
|
+
? readFileSync(process.env.SFTP_PRIVATE_KEY_PATH, 'utf8')
|
|
195
|
+
: undefined,
|
|
196
|
+
passphrase: process.env.SFTP_PASSPHRASE,
|
|
197
|
+
remotePath: process.env.SFTP_REMOTE_PATH || '/orders/incoming',
|
|
198
|
+
archivePath: process.env.SFTP_ARCHIVE_PATH || '/orders/processed',
|
|
199
|
+
errorPath: process.env.SFTP_ERROR_PATH || '/orders/errors',
|
|
200
|
+
filePattern: process.env.FILE_PATTERN || 'ORDER_*.xml',
|
|
201
|
+
},
|
|
202
|
+
processing: {
|
|
203
|
+
pollingInterval: parseInt(process.env.POLLING_INTERVAL_MS || '60000'),
|
|
204
|
+
batchSize: 100,
|
|
205
|
+
},
|
|
206
|
+
};
|
|
207
|
+
|
|
208
|
+
// ============================================================================
|
|
209
|
+
// GRAPHQL MAPPING CONFIGURATION
|
|
210
|
+
// ============================================================================
|
|
211
|
+
|
|
212
|
+
const orderMappingConfig: MappingConfig = {
|
|
213
|
+
version: '1.0',
|
|
214
|
+
name: 'SFTP XML Order to Fluent',
|
|
215
|
+
mutation: 'createOrder',
|
|
216
|
+
sourceFormat: 'xml',
|
|
217
|
+
operationName: 'CreateOrderFromSFTP',
|
|
218
|
+
arguments: {
|
|
219
|
+
input: {
|
|
220
|
+
_type: 'CreateOrderInput!',
|
|
221
|
+
// Order reference
|
|
222
|
+
ref: {
|
|
223
|
+
source: 'order.@id',
|
|
224
|
+
required: true,
|
|
225
|
+
},
|
|
226
|
+
// Order type
|
|
227
|
+
type: {
|
|
228
|
+
value: 'HD', // Home delivery
|
|
229
|
+
},
|
|
230
|
+
// Retailer reference
|
|
231
|
+
retailer: {
|
|
232
|
+
id: {
|
|
233
|
+
value: config.fluent.retailerId,
|
|
234
|
+
},
|
|
235
|
+
},
|
|
236
|
+
// Customer information
|
|
237
|
+
customer: {
|
|
238
|
+
firstName: {
|
|
239
|
+
source: 'order.customer.first-name',
|
|
240
|
+
transform: 'trim',
|
|
241
|
+
required: true,
|
|
242
|
+
},
|
|
243
|
+
lastName: {
|
|
244
|
+
source: 'order.customer.last-name',
|
|
245
|
+
transform: 'trim',
|
|
246
|
+
required: true,
|
|
247
|
+
},
|
|
248
|
+
email: {
|
|
249
|
+
source: 'order.customer.email',
|
|
250
|
+
transform: 'toLowerCase',
|
|
251
|
+
required: true,
|
|
252
|
+
},
|
|
253
|
+
phone: {
|
|
254
|
+
source: 'order.customer.phone',
|
|
255
|
+
transform: 'trim',
|
|
256
|
+
},
|
|
257
|
+
},
|
|
258
|
+
// Delivery address
|
|
259
|
+
fulfilmentChoice: {
|
|
260
|
+
deliveryType: {
|
|
261
|
+
value: 'STANDARD',
|
|
262
|
+
},
|
|
263
|
+
deliveryAddress: {
|
|
264
|
+
name: {
|
|
265
|
+
source: 'order.shipping.name',
|
|
266
|
+
},
|
|
267
|
+
street: {
|
|
268
|
+
source: 'order.shipping.street',
|
|
269
|
+
required: true,
|
|
270
|
+
},
|
|
271
|
+
city: {
|
|
272
|
+
source: 'order.shipping.city',
|
|
273
|
+
required: true,
|
|
274
|
+
},
|
|
275
|
+
state: {
|
|
276
|
+
source: 'order.shipping.state',
|
|
277
|
+
},
|
|
278
|
+
postcode: {
|
|
279
|
+
source: 'order.shipping.postcode',
|
|
280
|
+
required: true,
|
|
281
|
+
},
|
|
282
|
+
country: {
|
|
283
|
+
source: 'order.shipping.country',
|
|
284
|
+
required: true,
|
|
285
|
+
},
|
|
286
|
+
},
|
|
287
|
+
},
|
|
288
|
+
// Order items array
|
|
289
|
+
items: {
|
|
290
|
+
_array: true,
|
|
291
|
+
_autoWrap: true, // Convert single item to array
|
|
292
|
+
source: 'order.items.item',
|
|
293
|
+
_validation: {
|
|
294
|
+
minItems: 1,
|
|
295
|
+
},
|
|
296
|
+
// Fields for each item
|
|
297
|
+
ref: {
|
|
298
|
+
source: '@id',
|
|
299
|
+
required: true,
|
|
300
|
+
},
|
|
301
|
+
productRef: {
|
|
302
|
+
source: 'sku',
|
|
303
|
+
required: true,
|
|
304
|
+
},
|
|
305
|
+
quantity: {
|
|
306
|
+
source: 'quantity',
|
|
307
|
+
transform: 'parseInt',
|
|
308
|
+
required: true,
|
|
309
|
+
},
|
|
310
|
+
price: {
|
|
311
|
+
source: 'price',
|
|
312
|
+
transform: 'parseFloat',
|
|
313
|
+
required: true,
|
|
314
|
+
},
|
|
315
|
+
totalPrice: {
|
|
316
|
+
source: 'total-price',
|
|
317
|
+
transform: 'parseFloat',
|
|
318
|
+
required: true,
|
|
319
|
+
},
|
|
320
|
+
currency: {
|
|
321
|
+
value: 'USD',
|
|
322
|
+
},
|
|
323
|
+
},
|
|
324
|
+
// Order totals
|
|
325
|
+
totalPrice: {
|
|
326
|
+
source: 'order.totals.subtotal',
|
|
327
|
+
transform: 'parseFloat',
|
|
328
|
+
required: true,
|
|
329
|
+
},
|
|
330
|
+
totalTaxPrice: {
|
|
331
|
+
source: 'order.totals.tax',
|
|
332
|
+
transform: 'parseFloat',
|
|
333
|
+
defaultValue: 0,
|
|
334
|
+
},
|
|
335
|
+
// Custom attributes
|
|
336
|
+
attributes: {
|
|
337
|
+
orderDate: {
|
|
338
|
+
source: 'order.@order-date',
|
|
339
|
+
transform: 'toISO8601',
|
|
340
|
+
},
|
|
341
|
+
externalSystem: {
|
|
342
|
+
value: 'SFTP_XML',
|
|
343
|
+
},
|
|
344
|
+
sourceFile: {
|
|
345
|
+
// This will be set by custom resolver
|
|
346
|
+
},
|
|
347
|
+
},
|
|
348
|
+
},
|
|
349
|
+
},
|
|
350
|
+
returnFields: [
|
|
351
|
+
'id',
|
|
352
|
+
'ref',
|
|
353
|
+
'status',
|
|
354
|
+
'createdOn',
|
|
355
|
+
'totalPrice',
|
|
356
|
+
'customer { firstName lastName email }',
|
|
357
|
+
'items { ref productRef quantity price }',
|
|
358
|
+
],
|
|
359
|
+
customTransforms: {
|
|
360
|
+
// Add custom transform for phone number normalization
|
|
361
|
+
normalizePhone: (value: unknown) => {
|
|
362
|
+
if (!value || typeof value !== 'string') return value;
|
|
363
|
+
// Remove all non-digit characters
|
|
364
|
+
return value.replace(/\D/g, '');
|
|
365
|
+
},
|
|
366
|
+
},
|
|
367
|
+
};
|
|
368
|
+
|
|
369
|
+
// ============================================================================
|
|
370
|
+
// LOGGING SETUP
|
|
371
|
+
// ============================================================================
|
|
372
|
+
|
|
373
|
+
const logger: StructuredLogger = toStructuredLogger(createConsoleLogger(), { logLevel: 'info' });
|
|
374
|
+
|
|
375
|
+
// ============================================================================
|
|
376
|
+
// INITIALIZE SERVICES
|
|
377
|
+
// ============================================================================
|
|
378
|
+
|
|
379
|
+
let fluentClient: FluentClient;
|
|
380
|
+
let sftpSource: SftpDataSource;
|
|
381
|
+
let xmlParser: XMLParserService;
|
|
382
|
+
let mutationMapper: GraphQLMutationMapper;
|
|
383
|
+
|
|
384
|
+
async function initializeServices() {
|
|
385
|
+
logger.info('Initializing services...');
|
|
386
|
+
|
|
387
|
+
// Create Fluent client with OAuth2
|
|
388
|
+
fluentClient = await createClient({ config: config.fluent });
|
|
389
|
+
|
|
390
|
+
// Create SFTP data source
|
|
391
|
+
sftpSource = new SftpDataSource({
|
|
392
|
+
type: 'SFTP_XML',
|
|
393
|
+
connectionId: 'sftp-orders',
|
|
394
|
+
name: 'Order SFTP',
|
|
395
|
+
settings: {
|
|
396
|
+
host: config.sftp.host,
|
|
397
|
+
port: config.sftp.port,
|
|
398
|
+
username: config.sftp.username,
|
|
399
|
+
password: config.sftp.password,
|
|
400
|
+
privateKey: config.sftp.privateKey,
|
|
401
|
+
passphrase: config.sftp.passphrase,
|
|
402
|
+
remotePath: config.sftp.remotePath,
|
|
403
|
+
filePattern: config.sftp.filePattern,
|
|
404
|
+
connectionTimeout: 30000,
|
|
405
|
+
keepAliveInterval: 10000,
|
|
406
|
+
},
|
|
407
|
+
}, logger);
|
|
408
|
+
|
|
409
|
+
// Create XML parser
|
|
410
|
+
xmlParser = new XMLParserService();
|
|
411
|
+
|
|
412
|
+
// Create GraphQL mutation mapper
|
|
413
|
+
mutationMapper = new GraphQLMutationMapper(orderMappingConfig, logger, { fluentClient: fluentClient });
|
|
414
|
+
|
|
415
|
+
logger.info('Services initialized successfully');
|
|
416
|
+
}
|
|
417
|
+
|
|
418
|
+
// ============================================================================
|
|
419
|
+
// FILE PROCESSING
|
|
420
|
+
// ============================================================================
|
|
421
|
+
|
|
422
|
+
interface ProcessingStats {
|
|
423
|
+
filesProcessed: number;
|
|
424
|
+
filesSucceeded: number;
|
|
425
|
+
filesFailed: number;
|
|
426
|
+
ordersCreated: number;
|
|
427
|
+
errors: Array<{ file: string; error: string }>;
|
|
428
|
+
}
|
|
429
|
+
|
|
430
|
+
async function processOrderFiles(): Promise<ProcessingStats> {
|
|
431
|
+
const stats: ProcessingStats = {
|
|
432
|
+
filesProcessed: 0,
|
|
433
|
+
filesSucceeded: 0,
|
|
434
|
+
filesFailed: 0,
|
|
435
|
+
ordersCreated: 0,
|
|
436
|
+
errors: [],
|
|
437
|
+
};
|
|
438
|
+
|
|
439
|
+
try {
|
|
440
|
+
logger.info('Listing files from SFTP', {
|
|
441
|
+
path: config.sftp.remotePath,
|
|
442
|
+
pattern: config.sftp.filePattern,
|
|
443
|
+
});
|
|
444
|
+
|
|
445
|
+
// List files from SFTP
|
|
446
|
+
const files = await sftpSource.listFiles({
|
|
447
|
+
remotePath: config.sftp.remotePath,
|
|
448
|
+
filePattern: config.sftp.filePattern,
|
|
449
|
+
});
|
|
450
|
+
|
|
451
|
+
logger.info(`Found ${files.length} files to process`);
|
|
452
|
+
|
|
453
|
+
// Process each file
|
|
454
|
+
for (const file of files) {
|
|
455
|
+
stats.filesProcessed++;
|
|
456
|
+
|
|
457
|
+
try {
|
|
458
|
+
logger.info(`Processing file: ${file.name}`, {
|
|
459
|
+
size: file.size,
|
|
460
|
+
lastModified: file.lastModified,
|
|
461
|
+
});
|
|
462
|
+
|
|
463
|
+
await processOrderFile(file.name, stats);
|
|
464
|
+
stats.filesSucceeded++;
|
|
465
|
+
} catch (error: any) {
|
|
466
|
+
stats.filesFailed++;
|
|
467
|
+
stats.errors.push({
|
|
468
|
+
file: file.name,
|
|
469
|
+
error: error.message,
|
|
470
|
+
});
|
|
471
|
+
|
|
472
|
+
logger.error(`Failed to process file: ${file.name}`, error);
|
|
473
|
+
|
|
474
|
+
// Move to error folder
|
|
475
|
+
try {
|
|
476
|
+
const errorPath = `${config.sftp.errorPath}/${file.name}`;
|
|
477
|
+
await sftpSource.moveFile(file.path, errorPath, true); // Use file.path for source (full path)
|
|
478
|
+
logger.info(`Moved failed file to: ${errorPath}`);
|
|
479
|
+
} catch (moveError: any) {
|
|
480
|
+
logger.error('Failed to move file to error folder', moveError);
|
|
481
|
+
}
|
|
482
|
+
}
|
|
483
|
+
}
|
|
484
|
+
|
|
485
|
+
logger.info('Processing cycle completed', stats);
|
|
486
|
+
} catch (error: any) {
|
|
487
|
+
logger.error('Error during file listing', error);
|
|
488
|
+
throw error;
|
|
489
|
+
}
|
|
490
|
+
|
|
491
|
+
return stats;
|
|
492
|
+
}
|
|
493
|
+
|
|
494
|
+
async function processOrderFile(fileName: string, stats: ProcessingStats): Promise<void> {
|
|
495
|
+
// Step 1: Download XML file from SFTP
|
|
496
|
+
logger.debug('Downloading file from SFTP', { fileName });
|
|
497
|
+
const xmlContent = await sftpSource.downloadFile(fileName, { encoding: 'utf8' }) as string;
|
|
498
|
+
logger.debug('Downloaded file', {
|
|
499
|
+
fileName,
|
|
500
|
+
size: xmlContent.length,
|
|
501
|
+
});
|
|
502
|
+
|
|
503
|
+
// Step 2: Parse XML
|
|
504
|
+
logger.debug('Parsing XML', { fileName });
|
|
505
|
+
const parsedXml = await xmlParser.parse(xmlContent, {
|
|
506
|
+
includeAttributes: true,
|
|
507
|
+
parseNumbers: true,
|
|
508
|
+
parseBooleans: true,
|
|
509
|
+
normalizeWhitespace: true,
|
|
510
|
+
});
|
|
511
|
+
|
|
512
|
+
logger.debug('XML parsed successfully', {
|
|
513
|
+
fileName,
|
|
514
|
+
rootElements: Object.keys(parsedXml),
|
|
515
|
+
});
|
|
516
|
+
|
|
517
|
+
// Step 3: Map to GraphQL mutation using custom resolvers
|
|
518
|
+
logger.debug('Mapping XML to GraphQL mutation', { fileName });
|
|
519
|
+
|
|
520
|
+
const customResolvers = {
|
|
521
|
+
// Resolver to inject source file name into attributes
|
|
522
|
+
'custom.sourceFile': () => fileName,
|
|
523
|
+
|
|
524
|
+
// Resolver to calculate item total if not provided
|
|
525
|
+
'custom.calculateItemTotal': (value: any, context: any) => {
|
|
526
|
+
const quantity = context.quantity || 1;
|
|
527
|
+
const price = context.price || 0;
|
|
528
|
+
return quantity * price;
|
|
529
|
+
},
|
|
530
|
+
};
|
|
531
|
+
|
|
532
|
+
// Use mapSafe() for error-safe mapping (recommended for production)
|
|
533
|
+
const result = await mutationMapper.mapSafe(parsedXml, {
|
|
534
|
+
beforeMapping: async (data, ctx) => {
|
|
535
|
+
ctx.logger?.debug('Before mapping hook', {
|
|
536
|
+
hasOrder: !!data.order,
|
|
537
|
+
});
|
|
538
|
+
return data;
|
|
539
|
+
},
|
|
540
|
+
afterMapping: async (variables, ctx) => {
|
|
541
|
+
// Inject source file name via custom attribute
|
|
542
|
+
if (variables.input && typeof variables.input === 'object') {
|
|
543
|
+
const input = variables.input as any;
|
|
544
|
+
if (input.attributes) {
|
|
545
|
+
input.attributes.sourceFile = fileName;
|
|
546
|
+
}
|
|
547
|
+
}
|
|
548
|
+
|
|
549
|
+
ctx.logger?.debug('After mapping hook', {
|
|
550
|
+
ref: (variables.input as any)?.ref,
|
|
551
|
+
itemCount: (variables.input as any)?.items?.length,
|
|
552
|
+
});
|
|
553
|
+
|
|
554
|
+
return variables;
|
|
555
|
+
},
|
|
556
|
+
});
|
|
557
|
+
|
|
558
|
+
// Check for mapping errors
|
|
559
|
+
if (!result.success) {
|
|
560
|
+
logger.error('Mapping failed', {
|
|
561
|
+
fileName,
|
|
562
|
+
errors: result.errors,
|
|
563
|
+
});
|
|
564
|
+
return; // Skip this file
|
|
565
|
+
}
|
|
566
|
+
|
|
567
|
+
logger.debug('Mapping completed', {
|
|
568
|
+
fileName,
|
|
569
|
+
orderRef: (result.variables.input as any)?.ref,
|
|
570
|
+
itemCount: (result.variables.input as any)?.items?.length,
|
|
571
|
+
});
|
|
572
|
+
|
|
573
|
+
// Step 4: Execute GraphQL mutation
|
|
574
|
+
logger.info('Creating order in Fluent', {
|
|
575
|
+
fileName,
|
|
576
|
+
orderRef: (result.variables.input as any)?.ref,
|
|
577
|
+
});
|
|
578
|
+
|
|
579
|
+
const apiResult = await fluentClient.graphql({
|
|
580
|
+
query: result.query,
|
|
581
|
+
variables: result.variables,
|
|
582
|
+
});
|
|
583
|
+
|
|
584
|
+
if (apiResult.errors && apiResult.errors.length > 0) {
|
|
585
|
+
// Classify GraphQL errors to determine retry strategy
|
|
586
|
+
const classification = classifyErrors(apiResult.errors);
|
|
587
|
+
|
|
588
|
+
logger.error('GraphQL errors', {
|
|
589
|
+
fileName,
|
|
590
|
+
errorCode: classification.errorCode,
|
|
591
|
+
retryable: classification.retryable,
|
|
592
|
+
action: classification.action,
|
|
593
|
+
message: classification.userMessage,
|
|
594
|
+
errors: apiResult.errors,
|
|
595
|
+
});
|
|
596
|
+
|
|
597
|
+
// Don't retry non-retryable errors (e.g., order already exists)
|
|
598
|
+
if (!classification.retryable) {
|
|
599
|
+
throw new Error(classification.userMessage);
|
|
600
|
+
}
|
|
601
|
+
|
|
602
|
+
// For retryable errors, you could implement retry logic here
|
|
603
|
+
throw new Error(classification.userMessage);
|
|
604
|
+
}
|
|
605
|
+
|
|
606
|
+
const createdOrder = (result.data as any)?.createOrder;
|
|
607
|
+
|
|
608
|
+
logger.info('Order created successfully', {
|
|
609
|
+
fileName,
|
|
610
|
+
orderId: createdOrder?.id,
|
|
611
|
+
orderRef: createdOrder?.ref,
|
|
612
|
+
status: createdOrder?.status,
|
|
613
|
+
});
|
|
614
|
+
|
|
615
|
+
stats.ordersCreated++;
|
|
616
|
+
|
|
617
|
+
// Step 5: Archive processed file
|
|
618
|
+
const archivePath = `${config.sftp.archivePath}/${new Date().toISOString().split('T')[0]}/${fileName}`;
|
|
619
|
+
logger.debug('Archiving file', { fileName, archivePath });
|
|
620
|
+
|
|
621
|
+
// Create archive directory if it doesn't exist
|
|
622
|
+
const archiveDir = archivePath.substring(0, archivePath.lastIndexOf('/'));
|
|
623
|
+
await sftpSource.createDirectory(archiveDir, true);
|
|
624
|
+
|
|
625
|
+
// Move file to archive
|
|
626
|
+
await sftpSource.moveFile(fileName, archivePath, false);
|
|
627
|
+
|
|
628
|
+
logger.info('File archived successfully', {
|
|
629
|
+
fileName,
|
|
630
|
+
archivePath,
|
|
631
|
+
});
|
|
632
|
+
}
|
|
633
|
+
|
|
634
|
+
// ============================================================================
|
|
635
|
+
// MAIN EXECUTION LOOP
|
|
636
|
+
// ============================================================================
|
|
637
|
+
|
|
638
|
+
let isRunning = false;
|
|
639
|
+
let shutdownRequested = false;
|
|
640
|
+
|
|
641
|
+
async function runProcessingCycle() {
|
|
642
|
+
if (isRunning) {
|
|
643
|
+
logger.warn('Processing cycle already running, skipping...');
|
|
644
|
+
return;
|
|
645
|
+
}
|
|
646
|
+
|
|
647
|
+
isRunning = true;
|
|
648
|
+
|
|
649
|
+
try {
|
|
650
|
+
const stats = await processOrderFiles();
|
|
651
|
+
|
|
652
|
+
// Log summary
|
|
653
|
+
logger.info('Processing cycle summary', {
|
|
654
|
+
filesProcessed: stats.filesProcessed,
|
|
655
|
+
filesSucceeded: stats.filesSucceeded,
|
|
656
|
+
filesFailed: stats.filesFailed,
|
|
657
|
+
ordersCreated: stats.ordersCreated,
|
|
658
|
+
errorCount: stats.errors.length,
|
|
659
|
+
});
|
|
660
|
+
|
|
661
|
+
// Log errors if any
|
|
662
|
+
if (stats.errors.length > 0) {
|
|
663
|
+
logger.warn('Processing errors occurred', {
|
|
664
|
+
errors: stats.errors.slice(0, 5), // First 5 errors
|
|
665
|
+
totalErrors: stats.errors.length,
|
|
666
|
+
});
|
|
667
|
+
}
|
|
668
|
+
} catch (error: any) {
|
|
669
|
+
logger.error('Processing cycle failed', error);
|
|
670
|
+
} finally {
|
|
671
|
+
isRunning = false;
|
|
672
|
+
}
|
|
673
|
+
}
|
|
674
|
+
|
|
675
|
+
async function startPolling() {
|
|
676
|
+
logger.info('Starting polling loop', {
|
|
677
|
+
interval: config.processing.pollingInterval,
|
|
678
|
+
intervalSeconds: config.processing.pollingInterval / 1000,
|
|
679
|
+
});
|
|
680
|
+
|
|
681
|
+
// Run initial cycle
|
|
682
|
+
await runProcessingCycle();
|
|
683
|
+
|
|
684
|
+
// Set up polling interval
|
|
685
|
+
const intervalId = setInterval(async () => {
|
|
686
|
+
if (shutdownRequested) {
|
|
687
|
+
clearInterval(intervalId);
|
|
688
|
+
logger.info('Polling stopped due to shutdown request');
|
|
689
|
+
return;
|
|
690
|
+
}
|
|
691
|
+
|
|
692
|
+
await runProcessingCycle();
|
|
693
|
+
}, config.processing.pollingInterval);
|
|
694
|
+
|
|
695
|
+
// Handle graceful shutdown
|
|
696
|
+
const shutdown = async () => {
|
|
697
|
+
if (shutdownRequested) return;
|
|
698
|
+
|
|
699
|
+
shutdownRequested = true;
|
|
700
|
+
logger.info('Shutdown signal received, stopping gracefully...');
|
|
701
|
+
|
|
702
|
+
clearInterval(intervalId);
|
|
703
|
+
|
|
704
|
+
// Wait for current processing to complete
|
|
705
|
+
let waitCount = 0;
|
|
706
|
+
while (isRunning && waitCount < 30) {
|
|
707
|
+
logger.info('Waiting for current processing to complete...');
|
|
708
|
+
await new Promise(resolve => setTimeout(resolve, 1000));
|
|
709
|
+
waitCount++;
|
|
710
|
+
}
|
|
711
|
+
|
|
712
|
+
// Cleanup
|
|
713
|
+
await sftpSource.dispose();
|
|
714
|
+
|
|
715
|
+
logger.info('Shutdown complete');
|
|
716
|
+
process.exit(0);
|
|
717
|
+
};
|
|
718
|
+
|
|
719
|
+
process.on('SIGINT', shutdown);
|
|
720
|
+
process.on('SIGTERM', shutdown);
|
|
721
|
+
}
|
|
722
|
+
|
|
723
|
+
// ============================================================================
|
|
724
|
+
// ENTRY POINT
|
|
725
|
+
// ============================================================================
|
|
726
|
+
|
|
727
|
+
async function main() {
|
|
728
|
+
try {
|
|
729
|
+
logger.info('SFTP XML Order Sync Starting...', {
|
|
730
|
+
sftpHost: config.sftp.host,
|
|
731
|
+
remotePath: config.sftp.remotePath,
|
|
732
|
+
fluentBaseUrl: config.fluent.baseUrl,
|
|
733
|
+
retailerId: config.fluent.retailerId,
|
|
734
|
+
});
|
|
735
|
+
|
|
736
|
+
// Validate configuration
|
|
737
|
+
if (!config.fluent.clientId || !config.fluent.clientSecret) {
|
|
738
|
+
throw new Error('Fluent OAuth2 credentials are required (FLUENT_CLIENT_ID, FLUENT_CLIENT_SECRET)');
|
|
739
|
+
}
|
|
740
|
+
|
|
741
|
+
if (!config.sftp.host || !config.sftp.username) {
|
|
742
|
+
throw new Error('SFTP connection details are required (SFTP_HOST, SFTP_USERNAME)');
|
|
743
|
+
}
|
|
744
|
+
|
|
745
|
+
if (!config.sftp.password && !config.sftp.privateKey) {
|
|
746
|
+
throw new Error('SFTP authentication required (SFTP_PASSWORD or SFTP_PRIVATE_KEY_PATH)');
|
|
747
|
+
}
|
|
748
|
+
|
|
749
|
+
// Initialize services
|
|
750
|
+
await initializeServices();
|
|
751
|
+
|
|
752
|
+
// Validate connections
|
|
753
|
+
logger.info('Validating SFTP connection...');
|
|
754
|
+
const sftpValid = await sftpSource.validateConnection();
|
|
755
|
+
if (!sftpValid) {
|
|
756
|
+
throw new Error('SFTP connection validation failed');
|
|
757
|
+
}
|
|
758
|
+
logger.info('SFTP connection validated');
|
|
759
|
+
|
|
760
|
+
// Start polling loop
|
|
761
|
+
await startPolling();
|
|
762
|
+
} catch (error: any) {
|
|
763
|
+
logger.error('Fatal error during startup', error);
|
|
764
|
+
process.exit(1);
|
|
765
|
+
}
|
|
766
|
+
}
|
|
767
|
+
|
|
768
|
+
// Run the script
|
|
769
|
+
main();
|
|
770
|
+
```
|
|
771
|
+
|
|
772
|
+
---
|
|
773
|
+
|
|
774
|
+
## Key Patterns Explained
|
|
775
|
+
|
|
776
|
+
### Pattern 1: SFTP Connection & File Operations
|
|
777
|
+
|
|
778
|
+
**Connection Management:**
|
|
779
|
+
|
|
780
|
+
```typescript
|
|
781
|
+
const sftpSource = new SftpDataSource({
|
|
782
|
+
type: 'SFTP_XML',
|
|
783
|
+
connectionId: 'sftp-orders',
|
|
784
|
+
name: 'Order SFTP',
|
|
785
|
+
settings: {
|
|
786
|
+
host: config.sftp.host,
|
|
787
|
+
port: config.sftp.port,
|
|
788
|
+
username: config.sftp.username,
|
|
789
|
+
// Authentication: password OR private key
|
|
790
|
+
password: config.sftp.password,
|
|
791
|
+
privateKey: config.sftp.privateKey,
|
|
792
|
+
passphrase: config.sftp.passphrase,
|
|
793
|
+
remotePath: config.sftp.remotePath,
|
|
794
|
+
filePattern: config.sftp.filePattern,
|
|
795
|
+
connectionTimeout: 30000,
|
|
796
|
+
keepAliveInterval: 10000,
|
|
797
|
+
},
|
|
798
|
+
}, logger);
|
|
799
|
+
```
|
|
800
|
+
|
|
801
|
+
**File Listing:**
|
|
802
|
+
|
|
803
|
+
```typescript
|
|
804
|
+
// List files matching pattern
|
|
805
|
+
const files = await sftpSource.listFiles({
|
|
806
|
+
remotePath: '/orders/incoming',
|
|
807
|
+
filePattern: 'ORDER_*.xml',
|
|
808
|
+
});
|
|
809
|
+
|
|
810
|
+
// Filter by last modified time
|
|
811
|
+
const files = await sftpSource.listFiles({
|
|
812
|
+
remotePath: '/orders/incoming',
|
|
813
|
+
filePattern: 'ORDER_*.xml',
|
|
814
|
+
lastProcessedTimestamp: '2024-01-01T00:00:00Z',
|
|
815
|
+
});
|
|
816
|
+
```
|
|
817
|
+
|
|
818
|
+
**File Download:**
|
|
819
|
+
|
|
820
|
+
```typescript
|
|
821
|
+
// Download as string
|
|
822
|
+
const xmlContent = await sftpSource.downloadFile(fileName, {
|
|
823
|
+
encoding: 'utf8'
|
|
824
|
+
}) as string;
|
|
825
|
+
|
|
826
|
+
// Download as buffer
|
|
827
|
+
const buffer = await sftpSource.downloadFile(fileName, {
|
|
828
|
+
asBuffer: true
|
|
829
|
+
}) as Buffer;
|
|
830
|
+
```
|
|
831
|
+
|
|
832
|
+
**File Archival:**
|
|
833
|
+
|
|
834
|
+
```typescript
|
|
835
|
+
// Create archive directory (recursive)
|
|
836
|
+
await sftpSource.createDirectory(archiveDir, true);
|
|
837
|
+
|
|
838
|
+
// Move file to archive
|
|
839
|
+
await sftpSource.moveFile(
|
|
840
|
+
fileName,
|
|
841
|
+
archivePath,
|
|
842
|
+
false // overwrite
|
|
843
|
+
);
|
|
844
|
+
|
|
845
|
+
// Move to error folder
|
|
846
|
+
await sftpSource.moveFile(
|
|
847
|
+
fileName,
|
|
848
|
+
errorPath,
|
|
849
|
+
true // overwrite
|
|
850
|
+
);
|
|
851
|
+
```
|
|
852
|
+
|
|
853
|
+
### Pattern 2: XML Parsing & Validation
|
|
854
|
+
|
|
855
|
+
**Basic XML Parsing:**
|
|
856
|
+
|
|
857
|
+
```typescript
|
|
858
|
+
const xmlParser = new XMLParserService();
|
|
859
|
+
|
|
860
|
+
const parsedXml = await xmlParser.parse(xmlContent, {
|
|
861
|
+
includeAttributes: true, // Parse @id, @type attributes
|
|
862
|
+
parseNumbers: true, // Auto-convert "123" → 123
|
|
863
|
+
parseBooleans: true, // Auto-convert "true" → true
|
|
864
|
+
normalizeWhitespace: true, // Collapse whitespace
|
|
865
|
+
});
|
|
866
|
+
```
|
|
867
|
+
|
|
868
|
+
**XML with Namespaces:**
|
|
869
|
+
|
|
870
|
+
```typescript
|
|
871
|
+
const parsedXml = await xmlParser.parse(xmlContent, {
|
|
872
|
+
includeAttributes: true,
|
|
873
|
+
removeNamespacePrefix: true, // Remove ns: prefix
|
|
874
|
+
});
|
|
875
|
+
```
|
|
876
|
+
|
|
877
|
+
**Array Element Handling:**
|
|
878
|
+
|
|
879
|
+
```typescript
|
|
880
|
+
const parsedXml = await xmlParser.parse(xmlContent, {
|
|
881
|
+
includeAttributes: true,
|
|
882
|
+
arrayElements: ['item', 'product'], // Force arrays
|
|
883
|
+
});
|
|
884
|
+
```
|
|
885
|
+
|
|
886
|
+
**XML Structure Example:**
|
|
887
|
+
|
|
888
|
+
```xml
|
|
889
|
+
<order id="ORD-123" order-date="2024-01-15T10:30:00Z">
|
|
890
|
+
<customer>
|
|
891
|
+
<first-name>John</first-name>
|
|
892
|
+
<last-name>Doe</last-name>
|
|
893
|
+
<email>john@example.com</email>
|
|
894
|
+
<phone>+1-555-0123</phone>
|
|
895
|
+
</customer>
|
|
896
|
+
<shipping>
|
|
897
|
+
<name>John Doe</name>
|
|
898
|
+
<street>123 Main St</street>
|
|
899
|
+
<city>New York</city>
|
|
900
|
+
<state>NY</state>
|
|
901
|
+
<postcode>10001</postcode>
|
|
902
|
+
<country>US</country>
|
|
903
|
+
</shipping>
|
|
904
|
+
<items>
|
|
905
|
+
<item id="1">
|
|
906
|
+
<sku>PROD-001</sku>
|
|
907
|
+
<quantity>2</quantity>
|
|
908
|
+
<price>29.99</price>
|
|
909
|
+
<total-price>59.98</total-price>
|
|
910
|
+
</item>
|
|
911
|
+
<item id="2">
|
|
912
|
+
<sku>PROD-002</sku>
|
|
913
|
+
<quantity>1</quantity>
|
|
914
|
+
<price>49.99</price>
|
|
915
|
+
<total-price>49.99</total-price>
|
|
916
|
+
</item>
|
|
917
|
+
</items>
|
|
918
|
+
<totals>
|
|
919
|
+
<subtotal>109.97</subtotal>
|
|
920
|
+
<tax>10.00</tax>
|
|
921
|
+
<total>119.97</total>
|
|
922
|
+
</totals>
|
|
923
|
+
</order>
|
|
924
|
+
```
|
|
925
|
+
|
|
926
|
+
**Parsed Result:**
|
|
927
|
+
|
|
928
|
+
```javascript
|
|
929
|
+
{
|
|
930
|
+
order: {
|
|
931
|
+
'@id': 'ORD-123',
|
|
932
|
+
'@order-date': '2024-01-15T10:30:00Z',
|
|
933
|
+
customer: {
|
|
934
|
+
'first-name': 'John',
|
|
935
|
+
'last-name': 'Doe',
|
|
936
|
+
email: 'john@example.com',
|
|
937
|
+
phone: '+1-555-0123'
|
|
938
|
+
},
|
|
939
|
+
shipping: { ... },
|
|
940
|
+
items: {
|
|
941
|
+
item: [
|
|
942
|
+
{ '@id': '1', sku: 'PROD-001', quantity: 2, price: 29.99, ... },
|
|
943
|
+
{ '@id': '2', sku: 'PROD-002', quantity: 1, price: 49.99, ... }
|
|
944
|
+
]
|
|
945
|
+
},
|
|
946
|
+
totals: {
|
|
947
|
+
subtotal: 109.97,
|
|
948
|
+
tax: 10,
|
|
949
|
+
total: 119.97
|
|
950
|
+
}
|
|
951
|
+
}
|
|
952
|
+
}
|
|
953
|
+
```
|
|
954
|
+
|
|
955
|
+
### Pattern 3: GraphQL Mutation Mapping
|
|
956
|
+
|
|
957
|
+
**Mapping Configuration:**
|
|
958
|
+
|
|
959
|
+
```typescript
|
|
960
|
+
const mappingConfig: MappingConfig = {
|
|
961
|
+
version: '1.0',
|
|
962
|
+
mutation: 'createOrder',
|
|
963
|
+
sourceFormat: 'xml',
|
|
964
|
+
arguments: {
|
|
965
|
+
input: {
|
|
966
|
+
_type: 'CreateOrderInput!',
|
|
967
|
+
|
|
968
|
+
// Static value
|
|
969
|
+
type: {
|
|
970
|
+
value: 'HD',
|
|
971
|
+
},
|
|
972
|
+
|
|
973
|
+
// Source path
|
|
974
|
+
ref: {
|
|
975
|
+
source: 'order.@id',
|
|
976
|
+
required: true,
|
|
977
|
+
},
|
|
978
|
+
|
|
979
|
+
// Nested object
|
|
980
|
+
customer: {
|
|
981
|
+
firstName: {
|
|
982
|
+
source: 'order.customer.first-name',
|
|
983
|
+
transform: 'trim',
|
|
984
|
+
},
|
|
985
|
+
email: {
|
|
986
|
+
source: 'order.customer.email',
|
|
987
|
+
transform: 'toLowerCase',
|
|
988
|
+
},
|
|
989
|
+
},
|
|
990
|
+
|
|
991
|
+
// Array mapping
|
|
992
|
+
items: {
|
|
993
|
+
_array: true,
|
|
994
|
+
_autoWrap: true, // Single item → [item]
|
|
995
|
+
source: 'order.items.item',
|
|
996
|
+
ref: { source: '@id' },
|
|
997
|
+
productRef: { source: 'sku' },
|
|
998
|
+
quantity: {
|
|
999
|
+
source: 'quantity',
|
|
1000
|
+
transform: 'parseInt',
|
|
1001
|
+
},
|
|
1002
|
+
price: {
|
|
1003
|
+
source: 'price',
|
|
1004
|
+
transform: 'parseFloat',
|
|
1005
|
+
},
|
|
1006
|
+
},
|
|
1007
|
+
},
|
|
1008
|
+
},
|
|
1009
|
+
returnFields: [
|
|
1010
|
+
'id',
|
|
1011
|
+
'ref',
|
|
1012
|
+
'status',
|
|
1013
|
+
'customer { firstName lastName }',
|
|
1014
|
+
],
|
|
1015
|
+
};
|
|
1016
|
+
```
|
|
1017
|
+
|
|
1018
|
+
**Built-in Transforms:**
|
|
1019
|
+
|
|
1020
|
+
- `parseInt`, `parseFloat` - Number conversion
|
|
1021
|
+
- `toString` - String conversion
|
|
1022
|
+
- `toUpperCase`, `toLowerCase`, `trim` - String manipulation
|
|
1023
|
+
- `toBoolean` - Boolean conversion
|
|
1024
|
+
- `toISO8601`, `toDate` - Date formatting
|
|
1025
|
+
|
|
1026
|
+
**Execute Mapping:**
|
|
1027
|
+
|
|
1028
|
+
```typescript
|
|
1029
|
+
const mapper = new GraphQLMutationMapper(
|
|
1030
|
+
mappingConfig,
|
|
1031
|
+
fluentClient,
|
|
1032
|
+
logger
|
|
1033
|
+
);
|
|
1034
|
+
|
|
1035
|
+
// Use mapSafe() for error-safe mapping (recommended)
|
|
1036
|
+
const result = await mapper.mapSafe(parsedXml);
|
|
1037
|
+
if (!result.success) {
|
|
1038
|
+
console.error('Mapping failed:', result.errors);
|
|
1039
|
+
return;
|
|
1040
|
+
}
|
|
1041
|
+
// Result: { success: true, query: '...', variables: { ... } }
|
|
1042
|
+
```
|
|
1043
|
+
|
|
1044
|
+
### Pattern 4: Custom Resolvers for Complex Logic
|
|
1045
|
+
|
|
1046
|
+
**Custom Resolver Example:**
|
|
1047
|
+
|
|
1048
|
+
```typescript
|
|
1049
|
+
const customResolvers = {
|
|
1050
|
+
// Simple resolver
|
|
1051
|
+
'custom.sourceFile': (value: any, context: any) => {
|
|
1052
|
+
return fileName;
|
|
1053
|
+
},
|
|
1054
|
+
|
|
1055
|
+
// Context-aware resolver
|
|
1056
|
+
'custom.calculateTotal': (value: any, context: any) => {
|
|
1057
|
+
const quantity = context.quantity || 1;
|
|
1058
|
+
const price = context.price || 0;
|
|
1059
|
+
return quantity * price;
|
|
1060
|
+
},
|
|
1061
|
+
|
|
1062
|
+
// Async resolver (API call)
|
|
1063
|
+
'custom.lookupCustomer': async (value: any, context: any, config: any, helpers: any) => {
|
|
1064
|
+
const email = helpers.get(context, 'customer.email');
|
|
1065
|
+
const query = `query GetCustomer($email: String!) {
|
|
1066
|
+
customerByEmail(email: $email) {
|
|
1067
|
+
id
|
|
1068
|
+
ref
|
|
1069
|
+
}
|
|
1070
|
+
}`;
|
|
1071
|
+
const result = await fluentClient.graphql({
|
|
1072
|
+
query,
|
|
1073
|
+
variables: { email },
|
|
1074
|
+
});
|
|
1075
|
+
return result.data?.customerByEmail?.id;
|
|
1076
|
+
},
|
|
1077
|
+
|
|
1078
|
+
// Conditional logic
|
|
1079
|
+
'custom.determineShippingMethod': (value: any, context: any) => {
|
|
1080
|
+
const totalPrice = context.totalPrice || 0;
|
|
1081
|
+
if (totalPrice > 100) {
|
|
1082
|
+
return 'EXPRESS';
|
|
1083
|
+
} else if (totalPrice > 50) {
|
|
1084
|
+
return 'STANDARD';
|
|
1085
|
+
} else {
|
|
1086
|
+
return 'ECONOMY';
|
|
1087
|
+
}
|
|
1088
|
+
},
|
|
1089
|
+
};
|
|
1090
|
+
|
|
1091
|
+
// ✅ CORRECT: Use mapWithNodes() when custom resolvers are defined
|
|
1092
|
+
const payload = await mapper.mapWithNodes(parsedXml, customResolvers, {
|
|
1093
|
+
fluentClient: fluentClient as any,
|
|
1094
|
+
config: {},
|
|
1095
|
+
helpers: { fluentClient: fluentClient as any, logger },
|
|
1096
|
+
});
|
|
1097
|
+
|
|
1098
|
+
// ❌ WRONG: map() does NOT accept custom resolvers
|
|
1099
|
+
// const payload = await mapper.map(parsedXml, customResolvers);
|
|
1100
|
+
```
|
|
1101
|
+
|
|
1102
|
+
**Helper Functions Available:**
|
|
1103
|
+
|
|
1104
|
+
```typescript
|
|
1105
|
+
// helpers.get - Extract nested value
|
|
1106
|
+
const email = helpers.get(context, 'customer.email');
|
|
1107
|
+
|
|
1108
|
+
// helpers.logger - Logging
|
|
1109
|
+
helpers.logger.info('Processing order', { ref: orderRef });
|
|
1110
|
+
|
|
1111
|
+
// helpers.ensureArray - Array conversion
|
|
1112
|
+
const items = helpers.ensureArray(context.items);
|
|
1113
|
+
```
|
|
1114
|
+
|
|
1115
|
+
### Pattern 5: File Management (Archive/Error)
|
|
1116
|
+
|
|
1117
|
+
**Archival Strategy:**
|
|
1118
|
+
|
|
1119
|
+
```typescript
|
|
1120
|
+
// Date-based folder structure
|
|
1121
|
+
const today = new Date().toISOString().split('T')[0];
|
|
1122
|
+
const archivePath = `${config.sftp.archivePath}/${today}/${fileName}`;
|
|
1123
|
+
|
|
1124
|
+
// Create directory if needed
|
|
1125
|
+
await sftpSource.createDirectory(
|
|
1126
|
+
`${config.sftp.archivePath}/${today}`,
|
|
1127
|
+
true // recursive
|
|
1128
|
+
);
|
|
1129
|
+
|
|
1130
|
+
// Move file
|
|
1131
|
+
await sftpSource.moveFile(fileName, archivePath, false);
|
|
1132
|
+
```
|
|
1133
|
+
|
|
1134
|
+
**Error Handling:**
|
|
1135
|
+
|
|
1136
|
+
```typescript
|
|
1137
|
+
try {
|
|
1138
|
+
await processOrderFile(fileName, stats);
|
|
1139
|
+
// Success: Archive
|
|
1140
|
+
await sftpSource.moveFile(fileName, archivePath, false);
|
|
1141
|
+
} catch (error: any) {
|
|
1142
|
+
// Failure: Move to error folder
|
|
1143
|
+
const errorPath = `${config.sftp.errorPath}/${fileName}`;
|
|
1144
|
+
await sftpSource.moveFile(fileName, errorPath, true);
|
|
1145
|
+
|
|
1146
|
+
// Create error report
|
|
1147
|
+
const errorReport = {
|
|
1148
|
+
file: fileName,
|
|
1149
|
+
error: error.message,
|
|
1150
|
+
timestamp: new Date().toISOString(),
|
|
1151
|
+
};
|
|
1152
|
+
await sftpSource.uploadFile(null, 2, JSON.stringify(errorReport),
|
|
1153
|
+
`${config.sftp.errorPath}/${fileName}.error.json`
|
|
1154
|
+
);
|
|
1155
|
+
}
|
|
1156
|
+
```
|
|
1157
|
+
|
|
1158
|
+
---
|
|
1159
|
+
|
|
1160
|
+
## Deployment Options
|
|
1161
|
+
|
|
1162
|
+
### Option 1: Local Execution
|
|
1163
|
+
|
|
1164
|
+
```bash
|
|
1165
|
+
# Development
|
|
1166
|
+
npm run dev
|
|
1167
|
+
|
|
1168
|
+
# Production
|
|
1169
|
+
npm run build
|
|
1170
|
+
node dist/sftp-order-sync.js
|
|
1171
|
+
```
|
|
1172
|
+
|
|
1173
|
+
### Option 2: Docker Container
|
|
1174
|
+
|
|
1175
|
+
Create `Dockerfile`:
|
|
1176
|
+
|
|
1177
|
+
```dockerfile
|
|
1178
|
+
FROM node:18-alpine
|
|
1179
|
+
|
|
1180
|
+
WORKDIR /app
|
|
1181
|
+
|
|
1182
|
+
# Install dependencies
|
|
1183
|
+
COPY package*.json ./
|
|
1184
|
+
RUN npm ci --only=production
|
|
1185
|
+
|
|
1186
|
+
# Copy application
|
|
1187
|
+
COPY dist/ ./dist/
|
|
1188
|
+
COPY .env .env
|
|
1189
|
+
|
|
1190
|
+
# Run
|
|
1191
|
+
CMD ["node", "dist/sftp-order-sync.js"]
|
|
1192
|
+
```
|
|
1193
|
+
|
|
1194
|
+
Build and run:
|
|
1195
|
+
|
|
1196
|
+
```bash
|
|
1197
|
+
docker build -t sftp-order-sync .
|
|
1198
|
+
docker run -d --name sftp-order-sync --env-file .env sftp-order-sync
|
|
1199
|
+
```
|
|
1200
|
+
|
|
1201
|
+
### Option 3: Systemd Service
|
|
1202
|
+
|
|
1203
|
+
Create `/etc/systemd/system/sftp-order-sync.service`:
|
|
1204
|
+
|
|
1205
|
+
```ini
|
|
1206
|
+
[Unit]
|
|
1207
|
+
Description=SFTP Order Sync Service
|
|
1208
|
+
After=network.target
|
|
1209
|
+
|
|
1210
|
+
[Service]
|
|
1211
|
+
Type=simple
|
|
1212
|
+
User=nodeapp
|
|
1213
|
+
WorkingDirectory=/opt/sftp-order-sync
|
|
1214
|
+
ExecStart=/usr/bin/node /opt/sftp-order-sync/dist/sftp-order-sync.js
|
|
1215
|
+
Restart=always
|
|
1216
|
+
RestartSec=10
|
|
1217
|
+
StandardOutput=journal
|
|
1218
|
+
StandardError=journal
|
|
1219
|
+
SyslogIdentifier=sftp-order-sync
|
|
1220
|
+
Environment=NODE_ENV=production
|
|
1221
|
+
|
|
1222
|
+
[Install]
|
|
1223
|
+
WantedBy=multi-user.target
|
|
1224
|
+
```
|
|
1225
|
+
|
|
1226
|
+
Enable and start:
|
|
1227
|
+
|
|
1228
|
+
```bash
|
|
1229
|
+
sudo systemctl enable sftp-order-sync
|
|
1230
|
+
sudo systemctl start sftp-order-sync
|
|
1231
|
+
sudo journalctl -u sftp-order-sync -f
|
|
1232
|
+
```
|
|
1233
|
+
|
|
1234
|
+
### Option 4: Cron Job
|
|
1235
|
+
|
|
1236
|
+
Create `/opt/sftp-order-sync/run.sh`:
|
|
1237
|
+
|
|
1238
|
+
```bash
|
|
1239
|
+
#!/bin/bash
|
|
1240
|
+
cd /opt/sftp-order-sync
|
|
1241
|
+
node dist/sftp-order-sync.js >> /var/log/sftp-order-sync.log 2>&1
|
|
1242
|
+
```
|
|
1243
|
+
|
|
1244
|
+
Add to crontab:
|
|
1245
|
+
|
|
1246
|
+
```bash
|
|
1247
|
+
# Run every 5 minutes
|
|
1248
|
+
*/5 * * * * /opt/sftp-order-sync/run.sh
|
|
1249
|
+
```
|
|
1250
|
+
|
|
1251
|
+
---
|
|
1252
|
+
|
|
1253
|
+
## Testing
|
|
1254
|
+
|
|
1255
|
+
### Test with Sample XML
|
|
1256
|
+
|
|
1257
|
+
Create `test-order.xml`:
|
|
1258
|
+
|
|
1259
|
+
```xml
|
|
1260
|
+
<?xml version="1.0" encoding="UTF-8"?>
|
|
1261
|
+
<order id="TEST-001" order-date="2024-01-15T10:30:00Z">
|
|
1262
|
+
<customer>
|
|
1263
|
+
<first-name>John</first-name>
|
|
1264
|
+
<last-name>Doe</last-name>
|
|
1265
|
+
<email>john.doe@example.com</email>
|
|
1266
|
+
<phone>+1-555-0123</phone>
|
|
1267
|
+
</customer>
|
|
1268
|
+
<shipping>
|
|
1269
|
+
<name>John Doe</name>
|
|
1270
|
+
<street>123 Main Street</street>
|
|
1271
|
+
<city>New York</city>
|
|
1272
|
+
<state>NY</state>
|
|
1273
|
+
<postcode>10001</postcode>
|
|
1274
|
+
<country>US</country>
|
|
1275
|
+
</shipping>
|
|
1276
|
+
<items>
|
|
1277
|
+
<item id="1">
|
|
1278
|
+
<sku>PROD-001</sku>
|
|
1279
|
+
<quantity>2</quantity>
|
|
1280
|
+
<price>29.99</price>
|
|
1281
|
+
<total-price>59.98</total-price>
|
|
1282
|
+
</item>
|
|
1283
|
+
<item id="2">
|
|
1284
|
+
<sku>PROD-002</sku>
|
|
1285
|
+
<quantity>1</quantity>
|
|
1286
|
+
<price>49.99</price>
|
|
1287
|
+
<total-price>49.99</total-price>
|
|
1288
|
+
</item>
|
|
1289
|
+
</items>
|
|
1290
|
+
<totals>
|
|
1291
|
+
<subtotal>109.97</subtotal>
|
|
1292
|
+
<tax>10.00</tax>
|
|
1293
|
+
<total>119.97</total>
|
|
1294
|
+
</totals>
|
|
1295
|
+
</order>
|
|
1296
|
+
```
|
|
1297
|
+
|
|
1298
|
+
Upload to SFTP:
|
|
1299
|
+
|
|
1300
|
+
```bash
|
|
1301
|
+
sftp user@sftp.example.com
|
|
1302
|
+
cd /orders/incoming
|
|
1303
|
+
put test-order.xml ORDER_TEST_001.xml
|
|
1304
|
+
```
|
|
1305
|
+
|
|
1306
|
+
Monitor logs:
|
|
1307
|
+
|
|
1308
|
+
```bash
|
|
1309
|
+
# Local
|
|
1310
|
+
npm run dev
|
|
1311
|
+
|
|
1312
|
+
# Docker
|
|
1313
|
+
docker logs -f sftp-order-sync
|
|
1314
|
+
|
|
1315
|
+
# Systemd
|
|
1316
|
+
sudo journalctl -u sftp-order-sync -f
|
|
1317
|
+
```
|
|
1318
|
+
|
|
1319
|
+
---
|
|
1320
|
+
|
|
1321
|
+
## Common Issues
|
|
1322
|
+
|
|
1323
|
+
### Issue 1: SFTP Connection Timeout
|
|
1324
|
+
|
|
1325
|
+
**Symptoms:**
|
|
1326
|
+
|
|
1327
|
+
```
|
|
1328
|
+
Error: SFTP connection timeout after 30000ms
|
|
1329
|
+
```
|
|
1330
|
+
|
|
1331
|
+
**Solution:**
|
|
1332
|
+
|
|
1333
|
+
```typescript
|
|
1334
|
+
// Increase timeout in SFTP config
|
|
1335
|
+
settings: {
|
|
1336
|
+
connectionTimeout: 60000, // 60 seconds
|
|
1337
|
+
keepAliveInterval: 10000, // 10 seconds
|
|
1338
|
+
}
|
|
1339
|
+
```
|
|
1340
|
+
|
|
1341
|
+
### Issue 2: XML Parsing Fails for Attributes
|
|
1342
|
+
|
|
1343
|
+
**Symptoms:**
|
|
1344
|
+
|
|
1345
|
+
```
|
|
1346
|
+
Error: Cannot read property '@id' of undefined
|
|
1347
|
+
```
|
|
1348
|
+
|
|
1349
|
+
**Solution:**
|
|
1350
|
+
|
|
1351
|
+
```typescript
|
|
1352
|
+
// Ensure includeAttributes is enabled
|
|
1353
|
+
const parsedXml = await xmlParser.parse(xmlContent, {
|
|
1354
|
+
includeAttributes: true, // REQUIRED for @attributes
|
|
1355
|
+
});
|
|
1356
|
+
```
|
|
1357
|
+
|
|
1358
|
+
### Issue 3: Array Items Not Mapping
|
|
1359
|
+
|
|
1360
|
+
**Symptoms:**
|
|
1361
|
+
|
|
1362
|
+
```
|
|
1363
|
+
Error: Expected array at path 'order.items.item' but got object
|
|
1364
|
+
```
|
|
1365
|
+
|
|
1366
|
+
**Solution:**
|
|
1367
|
+
|
|
1368
|
+
```typescript
|
|
1369
|
+
// Enable auto-wrap for single items
|
|
1370
|
+
items: {
|
|
1371
|
+
_array: true,
|
|
1372
|
+
_autoWrap: true, // Convert single item to [item]
|
|
1373
|
+
source: 'order.items.item',
|
|
1374
|
+
}
|
|
1375
|
+
```
|
|
1376
|
+
|
|
1377
|
+
### Issue 4: GraphQL Mutation Fails
|
|
1378
|
+
|
|
1379
|
+
**Symptoms:**
|
|
1380
|
+
|
|
1381
|
+
```
|
|
1382
|
+
GraphQL errors: Field 'ref' is required but was null
|
|
1383
|
+
```
|
|
1384
|
+
|
|
1385
|
+
**Solution:**
|
|
1386
|
+
|
|
1387
|
+
```typescript
|
|
1388
|
+
// Check source path and add validation
|
|
1389
|
+
ref: {
|
|
1390
|
+
source: 'order.@id', // Correct attribute path
|
|
1391
|
+
required: true, // Fail fast if missing
|
|
1392
|
+
}
|
|
1393
|
+
|
|
1394
|
+
// Debug parsed XML structure
|
|
1395
|
+
logger.debug('Parsed XML structure', {
|
|
1396
|
+
keys: Object.keys(parsedXml),
|
|
1397
|
+
orderKeys: Object.keys(parsedXml.order || {}),
|
|
1398
|
+
});
|
|
1399
|
+
```
|
|
1400
|
+
|
|
1401
|
+
### Issue 5: Files Not Archiving
|
|
1402
|
+
|
|
1403
|
+
**Symptoms:**
|
|
1404
|
+
|
|
1405
|
+
```
|
|
1406
|
+
Error: Directory does not exist: /orders/processed/2024-01-15
|
|
1407
|
+
```
|
|
1408
|
+
|
|
1409
|
+
**Solution:**
|
|
1410
|
+
|
|
1411
|
+
```typescript
|
|
1412
|
+
// Create directory before moving
|
|
1413
|
+
const archiveDir = archivePath.substring(0, archivePath.lastIndexOf('/'));
|
|
1414
|
+
await sftpSource.createDirectory(archiveDir, true); // recursive
|
|
1415
|
+
|
|
1416
|
+
// Then move file
|
|
1417
|
+
await sftpSource.moveFile(fileName, archivePath, false);
|
|
1418
|
+
```
|
|
1419
|
+
|
|
1420
|
+
---
|
|
1421
|
+
|
|
1422
|
+
## Related Guides
|
|
1423
|
+
|
|
1424
|
+
- **[SFTP Credential Access & Security](../../02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md)** - Comprehensive SFTP credential management guide
|
|
1425
|
+
- **[S3 CSV Inventory Sync](./s3-csv-batch-api.md)** - Similar pattern for S3 CSV files
|
|
1426
|
+
- **[GraphQL Query Export](./graphql-query-export.md)** - GraphQL extraction pattern
|
|
1427
|
+
- **[Field Mapping Pattern](../patterns/field-mapping-universal.md)** - Complete field mapping reference
|
|
1428
|
+
- **[Error Handling Pattern](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md)** - Comprehensive error strategies
|
|
1429
|
+
|
|
1430
|
+
---
|
|
1431
|
+
|
|
1432
|
+
## Summary
|
|
1433
|
+
|
|
1434
|
+
This guide demonstrates a production-ready SFTP XML to Fluent GraphQL integration with:
|
|
1435
|
+
|
|
1436
|
+
- OAuth2 authentication
|
|
1437
|
+
- SFTP connection pooling
|
|
1438
|
+
- XML parsing with attribute support
|
|
1439
|
+
- GraphQL mutation mapping
|
|
1440
|
+
- Custom resolvers for complex logic
|
|
1441
|
+
- File archival and error handling
|
|
1442
|
+
- Graceful shutdown and retry logic
|
|
1443
|
+
|
|
1444
|
+
The complete script is ~600 lines and includes all necessary error handling, logging, and deployment configurations for production use.
|