@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +30 -2
- package/README.md +39 -0
- package/dist/cjs/auth/index.d.ts +3 -0
- package/dist/cjs/auth/index.js +13 -0
- package/dist/cjs/auth/profile-loader.d.ts +18 -0
- package/dist/cjs/auth/profile-loader.js +208 -0
- package/dist/cjs/client-factory.d.ts +4 -0
- package/dist/cjs/client-factory.js +10 -0
- package/dist/cjs/clients/fluent-client.js +13 -6
- package/dist/cjs/index.d.ts +3 -1
- package/dist/cjs/index.js +8 -2
- package/dist/cjs/utils/pagination-helpers.js +38 -2
- package/dist/cjs/versori/fluent-versori-client.js +11 -5
- package/dist/esm/auth/index.d.ts +3 -0
- package/dist/esm/auth/index.js +2 -0
- package/dist/esm/auth/profile-loader.d.ts +18 -0
- package/dist/esm/auth/profile-loader.js +169 -0
- package/dist/esm/client-factory.d.ts +4 -0
- package/dist/esm/client-factory.js +9 -0
- package/dist/esm/clients/fluent-client.js +13 -6
- package/dist/esm/index.d.ts +3 -1
- package/dist/esm/index.js +2 -1
- package/dist/esm/utils/pagination-helpers.js +38 -2
- package/dist/esm/versori/fluent-versori-client.js +11 -5
- package/dist/tsconfig.esm.tsbuildinfo +1 -1
- package/dist/tsconfig.tsbuildinfo +1 -1
- package/dist/tsconfig.types.tsbuildinfo +1 -1
- package/dist/types/auth/index.d.ts +3 -0
- package/dist/types/auth/profile-loader.d.ts +18 -0
- package/dist/types/client-factory.d.ts +4 -0
- package/dist/types/index.d.ts +3 -1
- package/docs/00-START-HERE/EXPORT-VALIDATION.md +158 -158
- package/docs/00-START-HERE/cli-analyze-source-structure-guide.md +655 -655
- package/docs/00-START-HERE/cli-documentation-index.md +202 -202
- package/docs/00-START-HERE/cli-quick-reference.md +252 -252
- package/docs/00-START-HERE/decision-tree.md +552 -552
- package/docs/00-START-HERE/getting-started.md +1070 -1070
- package/docs/00-START-HERE/mapper-quick-decision-guide.md +235 -235
- package/docs/00-START-HERE/readme.md +237 -237
- package/docs/00-START-HERE/retailerid-configuration.md +404 -404
- package/docs/00-START-HERE/sdk-philosophy.md +794 -794
- package/docs/00-START-HERE/troubleshooting-quick-reference.md +1086 -1086
- package/docs/01-TEMPLATES/faq.md +686 -686
- package/docs/01-TEMPLATES/patterns/pattern-templates-guide.md +68 -68
- package/docs/01-TEMPLATES/patterns/patterns-csv-schema-validation-and-rejection-report.md +233 -233
- package/docs/01-TEMPLATES/patterns/patterns-custom-resolvers.md +407 -407
- package/docs/01-TEMPLATES/patterns/patterns-error-handling-retry.md +511 -511
- package/docs/01-TEMPLATES/patterns/patterns-field-mapping-universal.md +701 -701
- package/docs/01-TEMPLATES/patterns/patterns-large-file-splitting.md +1430 -1430
- package/docs/01-TEMPLATES/patterns/patterns-master-data-etl.md +2399 -2399
- package/docs/01-TEMPLATES/patterns/patterns-pagination-streaming.md +447 -447
- package/docs/01-TEMPLATES/patterns/patterns-state-duplicate-prevention.md +385 -385
- package/docs/01-TEMPLATES/readme.md +957 -957
- package/docs/01-TEMPLATES/standalone/standalone-asn-inbound-processing.md +1209 -1209
- package/docs/01-TEMPLATES/standalone/standalone-graphql-query-export.md +1140 -1140
- package/docs/01-TEMPLATES/standalone/standalone-graphql-to-parquet-partitioned-s3.md +432 -432
- package/docs/01-TEMPLATES/standalone/standalone-multi-channel-inventory-sync.md +1185 -1185
- package/docs/01-TEMPLATES/standalone/standalone-multi-source-aggregation.md +1462 -1462
- package/docs/01-TEMPLATES/standalone/standalone-s3-csv-batch-api.md +1390 -1390
- package/docs/01-TEMPLATES/standalone/standalone-s3-csv-inventory-to-batch.md +330 -330
- package/docs/01-TEMPLATES/standalone/standalone-scripts-guide.md +87 -87
- package/docs/01-TEMPLATES/standalone/standalone-sftp-xml-graphql.md +1444 -1444
- package/docs/01-TEMPLATES/standalone/standalone-webhook-payload-processing.md +688 -688
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-dropship-order-routing.md +193 -193
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-graphql-parquet-extraction.md +518 -518
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-inter-location-transfers.md +2162 -2162
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-pre-order-allocation.md +2226 -2226
- package/docs/01-TEMPLATES/versori/business-examples/business-scenarios-guide.md +87 -87
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-connection-validation-pattern.md +656 -656
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-dual-workflow-connector.md +835 -835
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-guide.md +108 -108
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-kv-state-management.md +1533 -1533
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-xml-response-patterns.md +1160 -1160
- package/docs/01-TEMPLATES/versori/versori-platform-guide.md +201 -201
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-asn-purchase-order.md +1906 -1906
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-dropship-routing.md +1074 -1074
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-flash-sale-reserve.md +1395 -1395
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-generic-xml-order.md +888 -888
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-payment-gateway-integration.md +2478 -2478
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-rma-returns-comprehensive.md +2240 -2240
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-xml-order-ingestion.md +2029 -2029
- package/docs/01-TEMPLATES/versori/webhooks/webhook-templates-guide.md +140 -140
- package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/inventory-mapping.json +20 -20
- package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/products_2025-01-22.csv +11 -11
- package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/sample-data-guide.md +34 -34
- package/docs/01-TEMPLATES/versori/workflows/_examples/workflow-examples-guide.md +36 -36
- package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-modes-guide.md +1038 -1038
- package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-workflows-guide.md +138 -138
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/graphql-extraction-guide.md +63 -63
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-csv.md +2062 -2062
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-xml.md +2294 -2294
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-s3-csv.md +2461 -2461
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-sftp-xml.md +2529 -2529
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-csv.md +2464 -2464
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-json.md +1959 -1959
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-s3-csv.md +1953 -1953
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-sftp-xml.md +2541 -2541
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-s3-json.md +2384 -2384
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-sftp-xml.md +2445 -2445
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-csv.md +2355 -2355
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-json.md +2042 -2042
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-sftp-xml.md +2726 -2726
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/batch-api-guide.md +206 -206
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-cycle-count-reconciliation.md +2030 -2030
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-multi-channel-inventory-sync.md +1882 -1882
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-csv-inventory-batch.md +2827 -2827
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-json-inventory-batch.md +1952 -1952
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-xml-inventory-batch.md +3289 -3289
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-csv-inventory-batch.md +3064 -3064
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-json-inventory-batch.md +3238 -3238
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-xml-inventory-batch.md +2977 -2977
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/event-api-guide.md +321 -321
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-json-order-cancel-event.md +959 -959
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-xml-order-cancel-event.md +1170 -1170
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-csv-product-event.md +2312 -2312
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-json-product-event.md +2999 -2999
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-parquet-product-event.md +2836 -2836
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-xml-product-event.md +2395 -2395
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-csv-product-event.md +2295 -2295
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-json-product-event.md +2602 -2602
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-parquet-product-event.md +2589 -2589
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-xml-product-event.md +3578 -3578
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/graphql-mutations-guide.md +93 -93
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-json-order-update-graphql.md +1260 -1260
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-xml-order-update-graphql.md +1472 -1472
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-control-graphql.md +2417 -2417
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-location-graphql.md +2811 -2811
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-price-graphql.md +2619 -2619
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-json-location-graphql.md +2807 -2807
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-xml-location-graphql.md +2373 -2373
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-control-graphql.md +2740 -2740
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-location-graphql.md +2760 -2760
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-json-location-graphql.md +1710 -1710
- package/docs/01-TEMPLATES/versori/workflows/ingestion/ingestion-workflows-guide.md +136 -136
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/rubix-webhooks-guide.md +520 -520
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-inline.md +1418 -1418
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-universal-mapper.md +1785 -1785
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-order-attribute-update.md +824 -824
- package/docs/01-TEMPLATES/versori/workflows/workflows-overview-guide.md +646 -646
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-batch-archival.md +724 -724
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-job-tracker.md +627 -627
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-partial-batch-recovery.md +561 -561
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-quick-reference.md +367 -367
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-readme.md +407 -407
- package/docs/02-CORE-GUIDES/advanced-services/readme.md +49 -49
- package/docs/02-CORE-GUIDES/api-reference/api-reference-quick-reference.md +548 -548
- package/docs/02-CORE-GUIDES/api-reference/event-api-input-output-reference.md +702 -1171
- package/docs/02-CORE-GUIDES/api-reference/examples/client-initialization.ts +286 -286
- package/docs/02-CORE-GUIDES/api-reference/graphql-error-classification.md +337 -337
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-01-client-api.md +399 -482
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-03-authentication.md +199 -199
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-04-graphql-mapping.md +925 -925
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-05-services.md +1198 -1198
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-06-data-sources.md +1083 -1083
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-07-parsers.md +1097 -1097
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-pagination.md +513 -513
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-types.md +545 -597
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-error-handling.md +527 -527
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-webhook-validation.md +514 -514
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-extraction.md +557 -557
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-utilities.md +412 -412
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-cli-tools.md +423 -423
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-error-handling.md +716 -716
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-analyze-source-structure.md +518 -518
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-partial-responses.md +212 -212
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-testing.md +300 -300
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-13-resolver-builder.md +322 -322
- package/docs/02-CORE-GUIDES/api-reference/readme.md +279 -279
- package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-quick-reference.md +351 -351
- package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-readme.md +277 -277
- package/docs/02-CORE-GUIDES/auto-pagination/examples/auto-pagination-readme.md +178 -178
- package/docs/02-CORE-GUIDES/auto-pagination/examples/common-patterns.ts +351 -351
- package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-products.ts +384 -384
- package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-virtual-positions.ts +308 -308
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-01-foundations.md +470 -470
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-02-quick-start.md +713 -713
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-03-configuration.md +754 -754
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-04-advanced-patterns.md +732 -732
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-05-sdk-integration.md +847 -847
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-06-troubleshooting.md +359 -359
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-07-api-reference.md +462 -462
- package/docs/02-CORE-GUIDES/auto-pagination/readme.md +54 -54
- package/docs/02-CORE-GUIDES/data-sources/data-sources-file-operations-error-handling.md +1487 -1487
- package/docs/02-CORE-GUIDES/data-sources/data-sources-quick-reference.md +836 -836
- package/docs/02-CORE-GUIDES/data-sources/data-sources-readme.md +276 -276
- package/docs/02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md +553 -553
- package/docs/02-CORE-GUIDES/data-sources/examples/common-patterns.ts +409 -409
- package/docs/02-CORE-GUIDES/data-sources/examples/data-sources-readme.md +178 -178
- package/docs/02-CORE-GUIDES/data-sources/examples/s3-operations.ts +308 -308
- package/docs/02-CORE-GUIDES/data-sources/examples/sftp-operations.ts +371 -371
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-01-foundations.md +735 -735
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-02-s3-operations.md +1302 -1302
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-03-sftp-operations.md +1379 -1379
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-04-file-patterns.md +941 -941
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-05-advanced-topics.md +813 -813
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-06-integration-patterns.md +486 -486
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-07-troubleshooting.md +387 -387
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-08-api-reference.md +417 -417
- package/docs/02-CORE-GUIDES/data-sources/readme.md +77 -77
- package/docs/02-CORE-GUIDES/error-handling-guide.md +936 -936
- package/docs/02-CORE-GUIDES/extraction/examples/02-core-guides-extraction-readme.md +116 -116
- package/docs/02-CORE-GUIDES/extraction/examples/common-patterns.ts +428 -428
- package/docs/02-CORE-GUIDES/extraction/examples/extract-inventory-basic.ts +187 -187
- package/docs/02-CORE-GUIDES/extraction/extraction-quick-reference.md +596 -596
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-01-foundations.md +514 -514
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-02-basic-extraction.md +823 -823
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-03-parquet-processing.md +507 -507
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-04-data-enrichment.md +546 -546
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-05-transformation.md +494 -494
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-export-formats.md +458 -458
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-performance.md +138 -138
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-api-reference.md +148 -148
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-optimization.md +692 -692
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-08-extraction-orchestrator.md +1008 -1008
- package/docs/02-CORE-GUIDES/extraction/readme.md +151 -151
- package/docs/02-CORE-GUIDES/ingestion/examples/_simple-kv-store.ts +40 -40
- package/docs/02-CORE-GUIDES/ingestion/examples/error-recovery.ts +728 -728
- package/docs/02-CORE-GUIDES/ingestion/examples/event-driven.ts +501 -501
- package/docs/02-CORE-GUIDES/ingestion/examples/local-file-ingestion.ts +88 -88
- package/docs/02-CORE-GUIDES/ingestion/examples/parquet-ingestion.ts +117 -117
- package/docs/02-CORE-GUIDES/ingestion/examples/performance-optimized.ts +647 -647
- package/docs/02-CORE-GUIDES/ingestion/examples/s3-csv-ingestion.ts +169 -169
- package/docs/02-CORE-GUIDES/ingestion/examples/sftp-csv-ingestion.ts +134 -134
- package/docs/02-CORE-GUIDES/ingestion/ingestion-quick-reference.md +546 -546
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-01-introduction.md +626 -626
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-02-quick-start.md +658 -658
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-03-data-sources.md +1052 -1052
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-04-field-mapping.md +763 -763
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-05-advanced-parsers.md +676 -676
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-06-batch-api.md +1295 -1295
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-api-reference.md +138 -138
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-state-management.md +1037 -1037
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-08-performance-optimization.md +1349 -1349
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-09-best-practices.md +1893 -1893
- package/docs/02-CORE-GUIDES/ingestion/readme.md +160 -160
- package/docs/02-CORE-GUIDES/logging-guide.md +585 -585
- package/docs/02-CORE-GUIDES/mapping/error-handling-patterns.md +401 -401
- package/docs/02-CORE-GUIDES/mapping/examples/02-core-guides-mapping-readme.md +128 -128
- package/docs/02-CORE-GUIDES/mapping/examples/common-patterns.ts +273 -273
- package/docs/02-CORE-GUIDES/mapping/examples/csv-location-ingestion.json +36 -36
- package/docs/02-CORE-GUIDES/mapping/examples/csv-mapping.ts +242 -242
- package/docs/02-CORE-GUIDES/mapping/examples/graphql-to-parquet-extraction.json +36 -36
- package/docs/02-CORE-GUIDES/mapping/examples/json-mapping.ts +213 -213
- package/docs/02-CORE-GUIDES/mapping/examples/json-product-to-mutation.json +48 -48
- package/docs/02-CORE-GUIDES/mapping/examples/xml-mapping.ts +291 -291
- package/docs/02-CORE-GUIDES/mapping/examples/xml-order-to-mutation.json +45 -45
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-quick-reference.md +463 -463
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-readme.md +227 -227
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-01-introduction.md +222 -222
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-02-quick-start.md +351 -351
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-03-schema-validation.md +569 -569
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-04-mapping-patterns.md +471 -471
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-05-configuration-reference.md +611 -611
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-advanced-xpath.md +148 -148
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-path-syntax.md +464 -464
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-api-reference.md +94 -94
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-array-handling.md +307 -307
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-08-custom-resolvers.md +544 -544
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-09-advanced-patterns.md +427 -427
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-10-hooks-and-variables.md +336 -336
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-11-error-handling.md +488 -488
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-12-arguments-vs-nodes.md +383 -383
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-13-best-practices.md +477 -477
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/readme.md +62 -62
- package/docs/02-CORE-GUIDES/mapping/mapping-format-decision-tree.md +480 -480
- package/docs/02-CORE-GUIDES/mapping/mapping-graphql-alias-batching-guide.md +820 -820
- package/docs/02-CORE-GUIDES/mapping/mapping-javascript-objects.md +2369 -2369
- package/docs/02-CORE-GUIDES/mapping/mapping-mapper-comparison-guide.md +682 -682
- package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-07-api-reference.md +1327 -1327
- package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-08-error-handling.md +1142 -1142
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-04-use-cases.md +891 -891
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-helpers-resolvers.md +1126 -1126
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-sdk-resolvers.md +199 -199
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-07-api-reference.md +1319 -1319
- package/docs/02-CORE-GUIDES/mapping/readme.md +178 -178
- package/docs/02-CORE-GUIDES/mapping/resolver-registration.md +410 -410
- package/docs/02-CORE-GUIDES/mapping/resolvers/examples/common-patterns.ts +226 -226
- package/docs/02-CORE-GUIDES/mapping/resolvers/examples/custom-resolvers.ts +227 -227
- package/docs/02-CORE-GUIDES/mapping/resolvers/examples/sdk-resolvers-usage.ts +203 -203
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-readme.md +274 -274
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-api-reference.md +679 -679
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-cookbook.md +826 -826
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-guide.md +1330 -1330
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-helpers-reference.md +1437 -1437
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-parameters-reference.md +553 -553
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-troubleshooting.md +854 -854
- package/docs/02-CORE-GUIDES/mapping/resolvers/readme.md +75 -75
- package/docs/02-CORE-GUIDES/parsers/examples/02-core-guides-parsers-readme.md +161 -161
- package/docs/02-CORE-GUIDES/parsers/examples/csv-parser-examples.ts +110 -110
- package/docs/02-CORE-GUIDES/parsers/examples/json-parser-examples.ts +33 -33
- package/docs/02-CORE-GUIDES/parsers/examples/parquet-parser-examples.ts +47 -47
- package/docs/02-CORE-GUIDES/parsers/examples/xml-parser-examples.ts +38 -38
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-01-foundations.md +355 -355
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-02-csv-parser.md +772 -772
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-03-json-parser.md +789 -789
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-04-xml-parser.md +857 -857
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-05-parquet-parser.md +603 -603
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-integration-patterns.md +702 -702
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-streaming.md +121 -121
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-api-reference.md +89 -89
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-troubleshooting.md +727 -727
- package/docs/02-CORE-GUIDES/parsers/parsers-quick-reference.md +482 -482
- package/docs/02-CORE-GUIDES/parsers/parsers-readme.md +258 -258
- package/docs/02-CORE-GUIDES/parsers/readme.md +65 -65
- package/docs/02-CORE-GUIDES/readme.md +194 -194
- package/docs/02-CORE-GUIDES/webhook-validation/examples/basic-validation.ts +108 -108
- package/docs/02-CORE-GUIDES/webhook-validation/examples/common-patterns.ts +316 -316
- package/docs/02-CORE-GUIDES/webhook-validation/examples/webhook-validation-readme.md +61 -61
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-01-foundations.md +440 -440
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-02-quick-start.md +525 -525
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-03-versori-integration.md +741 -741
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-04-platform-integration.md +629 -629
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-05-configuration.md +535 -535
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-error-handling.md +611 -611
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-troubleshooting.md +124 -124
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-07-api-reference.md +511 -511
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-08-rubix-webhooks.md +590 -590
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-09-rubix-event-vs-http-call.md +432 -432
- package/docs/02-CORE-GUIDES/webhook-validation/readme.md +239 -239
- package/docs/02-CORE-GUIDES/webhook-validation/webhook-validation-quick-reference.md +392 -392
- package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-quick-reference.md +498 -498
- package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-readme.md +313 -313
- package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/common-patterns.ts +612 -612
- package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/connector-scenarios-readme.md +253 -253
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-01-foundations.md +452 -452
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-02-simple-scenarios.md +681 -681
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-03-intermediate-scenarios.md +637 -637
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-04-advanced-scenarios.md +650 -650
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-05-bidirectional-sync.md +233 -233
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-06-production-patterns.md +442 -442
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-07-reference.md +445 -445
- package/docs/03-PATTERN-GUIDES/connector-scenarios/readme.md +31 -31
- package/docs/03-PATTERN-GUIDES/enterprise-integration-patterns.md +1528 -1528
- package/docs/03-PATTERN-GUIDES/error-handling/comprehensive-error-handling-guide.md +1437 -1437
- package/docs/03-PATTERN-GUIDES/error-handling/error-handling-quick-reference.md +390 -390
- package/docs/03-PATTERN-GUIDES/error-handling/examples/common-patterns.ts +438 -438
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-01-foundations.md +362 -362
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-02-error-types.md +850 -850
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-03-utf8-handling.md +456 -456
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-04-error-scenarios.md +658 -658
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-05-calling-patterns.md +671 -671
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-06-retry-strategies.md +1034 -1034
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-07-monitoring.md +653 -653
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-08-api-reference.md +847 -847
- package/docs/03-PATTERN-GUIDES/error-handling/readme.md +36 -36
- package/docs/03-PATTERN-GUIDES/examples/__tests__/readme.md +40 -40
- package/docs/03-PATTERN-GUIDES/examples/__tests__/resolver-examples.test.js +282 -282
- package/docs/03-PATTERN-GUIDES/examples/test-data/03-pattern-guides-readme.md +110 -110
- package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-inventory.json +123 -123
- package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-order.json +171 -171
- package/docs/03-PATTERN-GUIDES/examples/test-data/readme.md +28 -28
- package/docs/03-PATTERN-GUIDES/extraction/extraction-readme.md +15 -15
- package/docs/03-PATTERN-GUIDES/extraction/readme.md +25 -25
- package/docs/03-PATTERN-GUIDES/file-operations/examples/common-patterns.ts +407 -407
- package/docs/03-PATTERN-GUIDES/file-operations/examples/file-operations-readme.md +142 -142
- package/docs/03-PATTERN-GUIDES/file-operations/file-operations-quick-reference.md +462 -462
- package/docs/03-PATTERN-GUIDES/file-operations/file-operations-readme.md +379 -379
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-01-foundations.md +430 -430
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-02-quick-start.md +484 -484
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-03-s3-operations.md +507 -507
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-04-sftp-operations.md +963 -963
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-05-streaming-performance.md +503 -503
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-archive-patterns.md +386 -386
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-error-handling.md +117 -117
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-api-reference.md +78 -78
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-testing-troubleshooting.md +567 -567
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-08-api-reference.md +1055 -1055
- package/docs/03-PATTERN-GUIDES/file-operations/readme.md +32 -32
- package/docs/03-PATTERN-GUIDES/ingestion/ingestion-readme.md +15 -15
- package/docs/03-PATTERN-GUIDES/ingestion/readme.md +25 -25
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/batch-processing.ts +130 -130
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/common-patterns.ts +360 -360
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/delta-sync.ts +130 -130
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/integration-patterns-readme.md +100 -100
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/real-time-webhook.ts +398 -398
- package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-quick-reference.md +962 -962
- package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-readme.md +134 -134
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-01-real-time-processing.md +991 -991
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-02-batch-processing.md +1547 -1547
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-03-delta-sync.md +1108 -1108
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-04-webhook-patterns.md +1181 -1181
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-05-error-handling.md +1061 -1061
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-advanced-integration-services.md +1547 -1547
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-performance.md +109 -109
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-07-api-reference.md +34 -34
- package/docs/03-PATTERN-GUIDES/integration-patterns/readme.md +30 -30
- package/docs/03-PATTERN-GUIDES/logging-minimal-mode.md +128 -128
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/common-patterns.ts +380 -380
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/multiple-connections-readme.md +139 -139
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/parallel-root-connections.ts +149 -149
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/real-world-scenarios.ts +405 -405
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-01-foundations.md +378 -378
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-02-quick-start.md +566 -566
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-03-targeting-connections.md +659 -659
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-04-parallel-queries.md +656 -656
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-05-best-practices.md +624 -624
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-api-reference.md +824 -824
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-versori.md +119 -119
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-07-api-reference.md +87 -87
- package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-quick-reference.md +353 -353
- package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-readme.md +270 -270
- package/docs/03-PATTERN-GUIDES/multiple-connections/readme.md +30 -30
- package/docs/03-PATTERN-GUIDES/pagination/pagination-readme.md +14 -14
- package/docs/03-PATTERN-GUIDES/pagination/readme.md +24 -24
- package/docs/03-PATTERN-GUIDES/parquet/examples/common-patterns.ts +180 -180
- package/docs/03-PATTERN-GUIDES/parquet/examples/read-parquet.ts +48 -48
- package/docs/03-PATTERN-GUIDES/parquet/examples/write-parquet.ts +65 -65
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-01-introduction.md +393 -393
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-02-quick-start.md +572 -572
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-03-reading-parquet.md +525 -525
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-04-writing-parquet.md +554 -554
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-05-graphql-extraction.md +405 -405
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-performance.md +104 -104
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-s3-integration.md +511 -511
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-api-reference.md +90 -90
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-performance-optimization.md +525 -525
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-08-best-practices.md +712 -712
- package/docs/03-PATTERN-GUIDES/parquet/parquet-quick-reference.md +683 -683
- package/docs/03-PATTERN-GUIDES/parquet/parquet-readme.md +248 -248
- package/docs/03-PATTERN-GUIDES/parquet/readme.md +32 -32
- package/docs/03-PATTERN-GUIDES/parsers/parsers-readme.md +12 -12
- package/docs/03-PATTERN-GUIDES/parsers/readme.md +24 -24
- package/docs/03-PATTERN-GUIDES/readme.md +159 -159
- package/docs/03-PATTERN-GUIDES/webhooks/readme.md +24 -24
- package/docs/03-PATTERN-GUIDES/webhooks/webhooks-readme.md +8 -8
- package/docs/04-REFERENCE/architecture/architecture-01-overview.md +427 -427
- package/docs/04-REFERENCE/architecture/architecture-02-client-architecture.md +424 -424
- package/docs/04-REFERENCE/architecture/architecture-03-data-flow.md +690 -690
- package/docs/04-REFERENCE/architecture/architecture-04-service-layer.md +834 -834
- package/docs/04-REFERENCE/architecture/architecture-05-integration-architecture.md +655 -655
- package/docs/04-REFERENCE/architecture/architecture-06-state-management.md +653 -653
- package/docs/04-REFERENCE/architecture/architecture-adding-new-data-sources.md +686 -686
- package/docs/04-REFERENCE/architecture/readme.md +279 -279
- package/docs/04-REFERENCE/platforms/deno/readme.md +117 -117
- package/docs/04-REFERENCE/platforms/nodejs/readme.md +146 -146
- package/docs/04-REFERENCE/platforms/readme.md +135 -135
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-01-introduction.md +398 -398
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-02-quick-start.md +560 -560
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-03-authentication.md +757 -757
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-04-workflows.md +2476 -2476
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-05-connections.md +1167 -1167
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-kv-storage.md +990 -990
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-state-management.md +121 -121
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-api-reference.md +68 -68
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-deployment.md +731 -731
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-08-best-practices.md +1111 -1111
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-09-signature-reference.md +766 -766
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-readme.md +299 -299
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-s3-sftp-configuration-guide.md +1425 -1425
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-api-key-security.md +816 -816
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-connection-security.md +681 -681
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-workflow-task-types.md +708 -708
- package/docs/04-REFERENCE/platforms/versori/readme.md +108 -108
- package/docs/04-REFERENCE/readme.md +148 -148
- package/docs/04-REFERENCE/resolver-signature/examples/advanced-resolvers.ts +482 -482
- package/docs/04-REFERENCE/resolver-signature/examples/async-resolvers.ts +496 -496
- package/docs/04-REFERENCE/resolver-signature/examples/basic-resolvers.ts +343 -343
- package/docs/04-REFERENCE/resolver-signature/examples/resolver-signature-readme.md +188 -188
- package/docs/04-REFERENCE/resolver-signature/examples/testing-resolvers.ts +463 -463
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-01-foundations.md +286 -286
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-02-parameter-reference.md +643 -643
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-03-basic-examples.md +521 -521
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-04-advanced-patterns.md +739 -739
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-05-sdk-resolvers.md +531 -531
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-migration-guide.md +650 -650
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-testing.md +125 -125
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-07-api-reference.md +794 -794
- package/docs/04-REFERENCE/resolver-signature/readme.md +64 -64
- package/docs/04-REFERENCE/resolver-signature/resolver-signature-quick-reference.md +270 -270
- package/docs/04-REFERENCE/resolver-signature/resolver-signature-readme.md +351 -351
- package/docs/04-REFERENCE/schema/fluent-commerce-schema.json +764 -764
- package/docs/04-REFERENCE/schema/readme.md +141 -141
- package/docs/04-REFERENCE/testing/examples/04-reference-testing-readme.md +158 -158
- package/docs/04-REFERENCE/testing/examples/fluent-testing.ts +62 -62
- package/docs/04-REFERENCE/testing/examples/health-check.ts +155 -155
- package/docs/04-REFERENCE/testing/examples/integration-test.ts +119 -119
- package/docs/04-REFERENCE/testing/examples/performance-test.ts +183 -183
- package/docs/04-REFERENCE/testing/examples/s3-testing.ts +127 -127
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-01-foundations.md +267 -267
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-02-s3-testing.md +599 -599
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-03-fluent-testing.md +589 -589
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-04-integration-testing.md +699 -699
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-05-debugging.md +478 -478
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-cicd-integration.md +463 -463
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-preflight-validation.md +131 -131
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-best-practices.md +499 -499
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-coverage-ci.md +165 -165
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-08-api-reference.md +634 -634
- package/docs/04-REFERENCE/testing/readme.md +86 -86
- package/docs/04-REFERENCE/testing/testing-quick-reference.md +667 -667
- package/docs/04-REFERENCE/testing/testing-readme.md +286 -286
- package/docs/04-REFERENCE/troubleshooting/readme.md +144 -144
- package/docs/04-REFERENCE/troubleshooting/troubleshooting-deno-sftp-compatibility.md +392 -392
- package/docs/template-loading-matrix.md +242 -242
- package/package.json +5 -3
|
@@ -1,2373 +1,2373 @@
|
|
|
1
|
-
---
|
|
2
|
-
template_id: tpl-ingest-s3-xml-to-location-graphql
|
|
3
|
-
canonical_filename: template-ingestion-s3-xml-location-graphql.md
|
|
4
|
-
version: 2.0.0
|
|
5
|
-
sdk_version: ^0.1.39
|
|
6
|
-
runtime: versori
|
|
7
|
-
direction: ingestion
|
|
8
|
-
source: s3-xml
|
|
9
|
-
destination: fluent-graphql
|
|
10
|
-
entity: location
|
|
11
|
-
format: xml
|
|
12
|
-
logging: versori
|
|
13
|
-
status: stable
|
|
14
|
-
compliance: gold-standard
|
|
15
|
-
features:
|
|
16
|
-
- graphql-mutation-mapper
|
|
17
|
-
- memory-management
|
|
18
|
-
- enhanced-logging
|
|
19
|
-
- attribute-transformation
|
|
20
|
-
---
|
|
21
|
-
|
|
22
|
-
Template: Ingestion - S3 XML to Location GraphQL
|
|
23
|
-
|
|
24
|
-
**Template Version:** 2.0.0
|
|
25
|
-
**SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
|
|
26
|
-
**Last Updated:** 2025-01-24
|
|
27
|
-
|
|
28
|
-
**🆕 Version 2.0.0 Enhancements:**
|
|
29
|
-
- ✅ **GraphQL Mutation Mapper** - Direct field mapping to mutation variables
|
|
30
|
-
- ✅ **Memory Management** - Clear large arrays after processing batches
|
|
31
|
-
- ✅ **Enhanced Logging** - Track mutation execution with emoji indicators
|
|
32
|
-
- ✅ **Attribute Transformation** - Handle complex nested data structures
|
|
33
|
-
|
|
34
|
-
---
|
|
35
|
-
|
|
36
|
-
## Implementation Prompt
|
|
37
|
-
|
|
38
|
-
"Create a Versori scheduled workflow that reads location XML files from S3, transforms the data using GraphQLMutationMapper with nested object mapping, and creates/updates Fluent Commerce locations via direct GraphQL mutations with alias batching support.
|
|
39
|
-
|
|
40
|
-
**Requirements:**
|
|
41
|
-
|
|
42
|
-
1. **S3 Source**:
|
|
43
|
-
- List and download XML files with configurable prefix and pattern
|
|
44
|
-
- Archive processed files to `processed/` directory (deduplication via archiving)
|
|
45
|
-
- Move failed files to `errors/` directory
|
|
46
|
-
- Use S3DataSource with retry logic and built-in archival
|
|
47
|
-
- Support configurable bucket, region, credentials
|
|
48
|
-
|
|
49
|
-
2. **XML Parsing**:
|
|
50
|
-
- Use XMLParserService with @ prefix for attribute access
|
|
51
|
-
- Handle both single and multiple `<location>` elements (array normalization)
|
|
52
|
-
- Support nested XML paths (`location.address.street1`, `location.coordinates.@lat`)
|
|
53
|
-
- Parse complex structures (addresses, coordinates, opening schedules)
|
|
54
|
-
|
|
55
|
-
3. **Field Mapping**:
|
|
56
|
-
- Map XML fields to Location GraphQL input type with nested objects:
|
|
57
|
-
- `ref`, `name`, `type` (root fields)
|
|
58
|
-
- `primaryAddress.*` (nested address object with coordinates)
|
|
59
|
-
- `openingSchedule.*` (nested schedule object with 7-day hours)
|
|
60
|
-
- Use SDK resolvers (trim, uppercase, parseFloat, parseInt, boolean)
|
|
61
|
-
- Support custom resolvers for complex transformations
|
|
62
|
-
- Validate required fields
|
|
63
|
-
- Note: Check your GraphQL schema to determine if `retailer.id` field exists and is mandatory/optional
|
|
64
|
-
|
|
65
|
-
4. **GraphQL Mutations** (Direct - NO Batch API):
|
|
66
|
-
- Execute `createLocation` mutation directly for each location
|
|
67
|
-
- **NO BPP (Batch Pre-Processing)** - Not applicable for direct GraphQL mutations
|
|
68
|
-
- **NO Batch API** - Use direct GraphQL mutations with rate limiting instead
|
|
69
|
-
- Use rate limiting (configurable mutations per second)
|
|
70
|
-
- Add delay between mutations to avoid API throttling
|
|
71
|
-
- Retry failed mutations with exponential backoff
|
|
72
|
-
- Track successful vs failed mutations per file
|
|
73
|
-
|
|
74
|
-
5. **Job Tracking & State Management**:
|
|
75
|
-
- **Use JobTracker** - Track job lifecycle (start, complete, fail) in KV store
|
|
76
|
-
- Use StateService + VersoriKVAdapter for duplicate file prevention
|
|
77
|
-
- Track processed files in KV store with metadata
|
|
78
|
-
- Store error state with exponential backoff tracking
|
|
79
|
-
- Support distributed state across workflow runs
|
|
80
|
-
- Provide job status endpoint for monitoring
|
|
81
|
-
|
|
82
|
-
6. **Error Handling**:
|
|
83
|
-
- File-level errors archived to `/errors/` subdirectory
|
|
84
|
-
- Record-level errors tracked but don't stop file processing
|
|
85
|
-
- Mapping errors logged with specific location context
|
|
86
|
-
- Mutation errors retried with exponential backoff
|
|
87
|
-
- Error state tracking with next retry timestamp
|
|
88
|
-
|
|
89
|
-
7. **Advanced Features**:
|
|
90
|
-
- Configurable rate limiting (mutations per second)
|
|
91
|
-
- Empty file detection and archival
|
|
92
|
-
- Timestamp-based error tracking
|
|
93
|
-
- Comprehensive monitoring and logging
|
|
94
|
-
- Manual webhook trigger with job tracking
|
|
95
|
-
- Job status query endpoint
|
|
96
|
-
|
|
97
|
-
**Use SDK Components:**
|
|
98
|
-
|
|
99
|
-
- `createClient()` - Universal client factory for Versori
|
|
100
|
-
- `S3DataSource` - S3 operations with retry logic
|
|
101
|
-
- `XMLParserService` - XML parsing with @ prefix attribute support
|
|
102
|
-
- `GraphQLMutationMapper` - Field transformation with schema validation and nested object support
|
|
103
|
-
- `StateService` + `VersoriKVAdapter` - Duplicate prevention with KV storage
|
|
104
|
-
- Native Versori `log` - Structured logging
|
|
105
|
-
|
|
106
|
-
**Configuration Variables** (from Versori activation):
|
|
107
|
-
|
|
108
|
-
```typescript
|
|
109
|
-
{
|
|
110
|
-
s3: {
|
|
111
|
-
bucketName: string;
|
|
112
|
-
region: string;
|
|
113
|
-
accessKeyId: string;
|
|
114
|
-
secretAccessKey: string;
|
|
115
|
-
prefix: string; // e.g., 'locations/'
|
|
116
|
-
archivePrefix: string; // e.g., 'processed/'
|
|
117
|
-
errorPrefix: string; // e.g., 'errors/'
|
|
118
|
-
filePattern: string; // e.g., '.xml'
|
|
119
|
-
maxFilesToProcess: number;
|
|
120
|
-
enableArchival: boolean;
|
|
121
|
-
},
|
|
122
|
-
fluent: {
|
|
123
|
-
retailerId: string; // Optional: Only if mutation schema requires retailerId in input
|
|
124
|
-
mutationRateLimit: number; // mutations per second (e.g., 5)
|
|
125
|
-
}
|
|
126
|
-
}
|
|
127
|
-
```
|
|
128
|
-
|
|
129
|
-
**Architecture Pattern:**
|
|
130
|
-
|
|
131
|
-
```
|
|
132
|
-
S3 Bucket → List Files → Download XML → Parse → Map → GraphQL Mutation → Archive/Move
|
|
133
|
-
↓ ↓ ↓ ↓ ↓ ↓ ↓
|
|
134
|
-
Configure Filter by S3DataSource XML Universal Direct S3 moveFile()
|
|
135
|
-
Connection Pattern + Retry Parser Mapper createLocation to processed/
|
|
136
|
-
(@) (nested) + Rate Limit or errors/
|
|
137
|
-
(deduplication)
|
|
138
|
-
```
|
|
139
|
-
|
|
140
|
-
**Deliverables:**
|
|
141
|
-
|
|
142
|
-
1. Complete Versori workflow with package.json
|
|
143
|
-
2. Main workflow logic with S3 + XML + GraphQL patterns
|
|
144
|
-
3. Helper functions for rate limiting and retry
|
|
145
|
-
4. XML path resolution examples
|
|
146
|
-
5. Sample XML files with nested structures
|
|
147
|
-
6. Schema validation CLI commands
|
|
148
|
-
7. Testing and deployment instructions
|
|
149
|
-
8. Monitoring and troubleshooting guidance"
|
|
150
|
-
|
|
151
|
-
---
|
|
152
|
-
|
|
153
|
-
# STEP 3: Complete Implementation
|
|
154
|
-
|
|
155
|
-
## Versori Scheduled: S3 XML → Location GraphQL
|
|
156
|
-
|
|
157
|
-
**FC Connect SDK Use Case Guide**
|
|
158
|
-
|
|
159
|
-
> **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
|
|
160
|
-
> **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
|
|
161
|
-
|
|
162
|
-
**Context**: Versori scheduled workflow that reads location XML files from S3 and creates/updates Fluent Commerce locations via GraphQL mutations with XML path resolution and rate limiting
|
|
163
|
-
|
|
164
|
-
**Complexity**: Medium
|
|
165
|
-
|
|
166
|
-
**Runtime**: Versori Platform
|
|
167
|
-
|
|
168
|
-
**Estimated Lines**: ~850 lines (with comprehensive documentation)
|
|
169
|
-
|
|
170
|
-
---
|
|
171
|
-
|
|
172
|
-
## What You'll Build
|
|
173
|
-
|
|
174
|
-
- Scheduled Versori workflow (daily location sync)
|
|
175
|
-
- S3 file listing, download, and archival/move
|
|
176
|
-
- XML parsing with @ prefix for attributes (XPath-style)
|
|
177
|
-
- Array normalization (single element → array conversion)
|
|
178
|
-
- GraphQLMutationMapper-based field transformations with nested objects
|
|
179
|
-
- GraphQL mutations for location upserts with alias batching support
|
|
180
|
-
- Retry logic with exponential backoff
|
|
181
|
-
- StateService duplicate prevention (KV-backed)
|
|
182
|
-
- Error state tracking and file error archival
|
|
183
|
-
- Manual webhook trigger and job status endpoint
|
|
184
|
-
|
|
185
|
-
---
|
|
186
|
-
|
|
187
|
-
## When to Use GraphQL Mutations vs Batch API vs Event API
|
|
188
|
-
|
|
189
|
-
### ✅ Use GraphQL Mutations For:
|
|
190
|
-
|
|
191
|
-
| Entity Type | Use Case | Why GraphQL |
|
|
192
|
-
| -------------- | ---------------------------------------- | ------------------------------------- |
|
|
193
|
-
| **Locations** | Store/warehouse master data (low volume) | Direct control, immediate validation |
|
|
194
|
-
| **Controls** | System configuration, settings | Single operations, complex queries |
|
|
195
|
-
| **Prices** | Price updates (moderate volume) | Immediate feedback, custom logic |
|
|
196
|
-
| **Single Ops** | One-off creates/updates | Testing, debugging, direct API access |
|
|
197
|
-
|
|
198
|
-
### ❌ Use Event API Instead For:
|
|
199
|
-
|
|
200
|
-
| Entity Type | Use Case | Why Event API |
|
|
201
|
-
| ------------------- | -------------------------------------- | -------------------------------------------- |
|
|
202
|
-
| **Products** | Product catalog sync, variant updates | Triggers workflows, validates business rules |
|
|
203
|
-
| **Customers** | Customer registration, profile updates | Needs workflow for downstream systems |
|
|
204
|
-
| **Orders** | Order creation, status updates | Event-driven fulfillment workflows |
|
|
205
|
-
| **Custom Entities** | Any entity needing workflow triggers | Full Rubix workflow support |
|
|
206
|
-
|
|
207
|
-
### 🔄 Use Batch API For:
|
|
208
|
-
|
|
209
|
-
| Entity Type | Use Case | Why Batch API |
|
|
210
|
-
| ------------------ | ---------------------------------- | ----------------------------------------------- |
|
|
211
|
-
| **Inventory ONLY** | Bulk inventory updates, daily sync | Optimized for high-volume, BPP change detection |
|
|
212
|
-
|
|
213
|
-
---
|
|
214
|
-
|
|
215
|
-
## XML File Format
|
|
216
|
-
|
|
217
|
-
### Sample: locations.xml
|
|
218
|
-
|
|
219
|
-
```xml
|
|
220
|
-
<?xml version="1.0" encoding="UTF-8"?>
|
|
221
|
-
<locations>
|
|
222
|
-
<location ref="LOC-001" type="WAREHOUSE">
|
|
223
|
-
<name>Downtown Warehouse</name>
|
|
224
|
-
<address country="USA">
|
|
225
|
-
<street1>123 Main St</street1>
|
|
226
|
-
<street2>Building A</street2>
|
|
227
|
-
<city>New York</city>
|
|
228
|
-
<state>NY</state>
|
|
229
|
-
<postalCode>10001</postalCode>
|
|
230
|
-
</address>
|
|
231
|
-
<coordinates lat="40.7128" lon="-74.0060"/>
|
|
232
|
-
<timeZone>America/New_York</timeZone>
|
|
233
|
-
<openingSchedule>
|
|
234
|
-
<allHours>false</allHours>
|
|
235
|
-
<monStart>800</monStart>
|
|
236
|
-
<monEnd>1800</monEnd>
|
|
237
|
-
<tueStart>800</tueStart>
|
|
238
|
-
<tueEnd>1800</tueEnd>
|
|
239
|
-
<wedStart>800</wedStart>
|
|
240
|
-
<wedEnd>1800</wedEnd>
|
|
241
|
-
<thuStart>800</thuStart>
|
|
242
|
-
<thuEnd>1800</thuEnd>
|
|
243
|
-
<friStart>800</friStart>
|
|
244
|
-
<friEnd>1800</friEnd>
|
|
245
|
-
<satStart>0</satStart>
|
|
246
|
-
<satEnd>0</satEnd>
|
|
247
|
-
<sunStart>0</sunStart>
|
|
248
|
-
<sunEnd>0</sunEnd>
|
|
249
|
-
</openingSchedule>
|
|
250
|
-
</location>
|
|
251
|
-
|
|
252
|
-
<location ref="LOC-002" type="DC">
|
|
253
|
-
<name>Regional DC</name>
|
|
254
|
-
<address country="USA">
|
|
255
|
-
<street1>456 Industrial Pkwy</street1>
|
|
256
|
-
<city>Los Angeles</city>
|
|
257
|
-
<state>CA</state>
|
|
258
|
-
<postalCode>90001</postalCode>
|
|
259
|
-
</address>
|
|
260
|
-
<coordinates lat="34.0522" lon="-118.2437"/>
|
|
261
|
-
<timeZone>America/Los_Angeles</timeZone>
|
|
262
|
-
<openingSchedule>
|
|
263
|
-
<allHours>true</allHours>
|
|
264
|
-
<monStart>0</monStart>
|
|
265
|
-
<monEnd>0</monEnd>
|
|
266
|
-
<tueStart>0</tueStart>
|
|
267
|
-
<tueEnd>0</tueEnd>
|
|
268
|
-
<wedStart>0</wedStart>
|
|
269
|
-
<wedEnd>0</wedEnd>
|
|
270
|
-
<thuStart>0</thuStart>
|
|
271
|
-
<thuEnd>0</thuEnd>
|
|
272
|
-
<friStart>0</friStart>
|
|
273
|
-
<friEnd>0</friEnd>
|
|
274
|
-
<satStart>0</satStart>
|
|
275
|
-
<satEnd>0</satEnd>
|
|
276
|
-
<sunStart>0</sunStart>
|
|
277
|
-
<sunEnd>0</sunEnd>
|
|
278
|
-
</openingSchedule>
|
|
279
|
-
</location>
|
|
280
|
-
</locations>
|
|
281
|
-
```
|
|
282
|
-
|
|
283
|
-
**XML Path Syntax (with @ prefix for attributes):**
|
|
284
|
-
|
|
285
|
-
- `location.@ref` → XML attribute `ref` on `<location>` element
|
|
286
|
-
- `location.name` → Text content of `<name>` element
|
|
287
|
-
- `location.address.street1` → Nested element path
|
|
288
|
-
- `location.address.@country` → XML attribute on nested `<address>` element
|
|
289
|
-
- `location.coordinates.@lat` → XML attribute for latitude
|
|
290
|
-
- `location.openingSchedule.monStart` → Deeply nested element
|
|
291
|
-
|
|
292
|
-
**Note**: The SDK's `XMLParserService` automatically handles XML attributes using `@` prefix notation.
|
|
293
|
-
|
|
294
|
-
---
|
|
295
|
-
|
|
296
|
-
## Versori Workflows Structure
|
|
297
|
-
|
|
298
|
-
**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
|
|
299
|
-
|
|
300
|
-
**Trigger Types:**
|
|
301
|
-
- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
|
|
302
|
-
- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
|
|
303
|
-
- **`workflow()`** → Durable workflows (advanced, rarely used)
|
|
304
|
-
|
|
305
|
-
**Execution Steps (chained to triggers):**
|
|
306
|
-
- **`http()`** → External API calls (chained from schedule/webhook)
|
|
307
|
-
- **`fn()`** → Internal processing (chained from schedule/webhook)
|
|
308
|
-
|
|
309
|
-
### Recommended Project Structure
|
|
310
|
-
|
|
311
|
-
```
|
|
312
|
-
s3-xml-location-graphql/
|
|
313
|
-
├── index.ts # Entry point - exports all workflows
|
|
314
|
-
└── src/
|
|
315
|
-
├── workflows/
|
|
316
|
-
│ ├── scheduled/
|
|
317
|
-
│ │ └── daily-location-sync.ts # Scheduled: Daily location sync
|
|
318
|
-
│ │
|
|
319
|
-
│ └── webhook/
|
|
320
|
-
│ ├── adhoc-location-sync.ts # Webhook: Manual trigger
|
|
321
|
-
│ └── job-status-check.ts # Webhook: Status query
|
|
322
|
-
│
|
|
323
|
-
├── services/
|
|
324
|
-
│ └── location-sync.service.ts # Shared orchestration logic (reusable)
|
|
325
|
-
│
|
|
326
|
-
└── config/
|
|
327
|
-
└── location-mapping.json # GraphQL mapping config
|
|
328
|
-
```
|
|
329
|
-
|
|
330
|
-
---
|
|
331
|
-
|
|
332
|
-
## Complete Versori Workflow
|
|
333
|
-
|
|
334
|
-
### Step 1: Package Configuration
|
|
335
|
-
|
|
336
|
-
**File: package.json**
|
|
337
|
-
|
|
338
|
-
```json
|
|
339
|
-
{
|
|
340
|
-
"name": "versori-s3-xml-location-sync",
|
|
341
|
-
"version": "1.0.0",
|
|
342
|
-
"description": "Versori workflow: S3 XML location sync to Fluent GraphQL",
|
|
343
|
-
"versori": {
|
|
344
|
-
"workflows": "./index.ts"
|
|
345
|
-
},
|
|
346
|
-
"type": "module",
|
|
347
|
-
"scripts": {
|
|
348
|
-
"deploy": "versori deploy",
|
|
349
|
-
"logs": "versori logs"
|
|
350
|
-
},
|
|
351
|
-
"dependencies": {
|
|
352
|
-
"@fluentcommerce/fc-connect-sdk": "^0.1.39",
|
|
353
|
-
"@versori/run": "latest"
|
|
354
|
-
},
|
|
355
|
-
"devDependencies": {
|
|
356
|
-
"typescript": "^5.0.0",
|
|
357
|
-
"@types/node": "^20.0.0"
|
|
358
|
-
}
|
|
359
|
-
}
|
|
360
|
-
```
|
|
361
|
-
|
|
362
|
-
### Step 2: Workflow Entry Point (`index.ts`)
|
|
363
|
-
|
|
364
|
-
**Purpose**: Register all workflows with Versori platform
|
|
365
|
-
|
|
366
|
-
```typescript
|
|
367
|
-
/**
|
|
368
|
-
* Entry Point - Registers all workflows with Versori platform
|
|
369
|
-
*
|
|
370
|
-
* Versori automatically discovers and registers exported workflows
|
|
371
|
-
*
|
|
372
|
-
* File Structure:
|
|
373
|
-
* - src/workflows/scheduled/ → Time-based triggers (cron)
|
|
374
|
-
* - src/workflows/webhook/ → HTTP-based triggers (webhooks)
|
|
375
|
-
*/
|
|
376
|
-
|
|
377
|
-
// Scheduled workflows
|
|
378
|
-
export { dailyLocationSync } from './src/workflows/scheduled/daily-location-sync';
|
|
379
|
-
|
|
380
|
-
// Webhook workflows
|
|
381
|
-
export { adhocLocationSync } from './src/workflows/webhook/adhoc-location-sync';
|
|
382
|
-
export { locationSyncJobStatus } from './src/workflows/webhook/job-status-check';
|
|
383
|
-
```
|
|
384
|
-
|
|
385
|
-
**What Gets Exposed:**
|
|
386
|
-
- ✅ `adhocLocationSync` → `https://{workspace}.versori.run/location-sync-adhoc`
|
|
387
|
-
- ✅ `locationSyncJobStatus` → `https://{workspace}.versori.run/location-sync-job-status`
|
|
388
|
-
- ❌ `dailyLocationSync` → NOT exposed (runs automatically on cron)
|
|
389
|
-
|
|
390
|
-
---
|
|
391
|
-
|
|
392
|
-
### Step 3: Workflow Files
|
|
393
|
-
|
|
394
|
-
#### `src/workflows/scheduled/daily-location-sync.ts`
|
|
395
|
-
|
|
396
|
-
**Purpose**: Automatic daily location sync
|
|
397
|
-
**Trigger**: Cron schedule (`0 2 * * *`)
|
|
398
|
-
**Exposed as Endpoint**: ❌ NO - Runs automatically
|
|
399
|
-
|
|
400
|
-
```typescript
|
|
401
|
-
import { schedule, http } from '@versori/run';
|
|
402
|
-
import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
|
|
403
|
-
import { executeLocationSync } from '../../services/location-sync.service';
|
|
404
|
-
|
|
405
|
-
/**
|
|
406
|
-
* Scheduled Workflow: Daily Location Sync
|
|
407
|
-
*
|
|
408
|
-
* Runs automatically daily at 2 AM UTC
|
|
409
|
-
* NOT exposed as HTTP endpoint - Versori executes on schedule
|
|
410
|
-
*
|
|
411
|
-
* Uses shared service: location-sync.service.ts
|
|
412
|
-
*/
|
|
413
|
-
export const dailyLocationSync = schedule(
|
|
414
|
-
'location-sync-scheduled',
|
|
415
|
-
'0 2 * * *' // Daily at 2 AM UTC
|
|
416
|
-
).then(
|
|
417
|
-
http('run-location-sync', { connection: 'fluent_commerce' }, async ctx => {
|
|
418
|
-
const startTime = Date.now();
|
|
419
|
-
const { log, openKv } = ctx;
|
|
420
|
-
const jobId = `location-sync-${Date.now()}`;
|
|
421
|
-
const tracker = new JobTracker(openKv(':project:'), log);
|
|
422
|
-
|
|
423
|
-
log.info('🚀 [START] Daily location sync initiated', { jobId, trigger: 'schedule' });
|
|
424
|
-
|
|
425
|
-
await tracker.createJob(jobId, { triggeredBy: 'schedule', stage: 'initialization' });
|
|
426
|
-
await tracker.updateJob(jobId, { status: 'processing' });
|
|
427
|
-
|
|
428
|
-
try {
|
|
429
|
-
log.info('⚙️ [PROCESSING] Starting location synchronization workflow', { jobId });
|
|
430
|
-
const result = await executeLocationSync(ctx, jobId, tracker);
|
|
431
|
-
await tracker.markCompleted(jobId, result);
|
|
432
|
-
|
|
433
|
-
const duration = Date.now() - startTime;
|
|
434
|
-
log.info('✅ [SUCCESS] Daily location sync completed', {
|
|
435
|
-
jobId,
|
|
436
|
-
duration: `${duration}ms`,
|
|
437
|
-
processed: result.processed,
|
|
438
|
-
totalRecords: result.totalRecords
|
|
439
|
-
});
|
|
440
|
-
|
|
441
|
-
return { success: true, jobId, duration, ...result };
|
|
442
|
-
} catch (e: any) {
|
|
443
|
-
await tracker.markFailed(jobId, e);
|
|
444
|
-
const duration = Date.now() - startTime;
|
|
445
|
-
|
|
446
|
-
log.error('❌ [FAILED] Daily location sync failed', {
|
|
447
|
-
jobId,
|
|
448
|
-
duration: `${duration}ms`,
|
|
449
|
-
error: e?.message,
|
|
450
|
-
errorType: e?.name
|
|
451
|
-
});
|
|
452
|
-
|
|
453
|
-
return { success: false, jobId, duration, error: e?.message };
|
|
454
|
-
}
|
|
455
|
-
})
|
|
456
|
-
);
|
|
457
|
-
```
|
|
458
|
-
|
|
459
|
-
---
|
|
460
|
-
|
|
461
|
-
#### `src/workflows/webhook/adhoc-location-sync.ts`
|
|
462
|
-
|
|
463
|
-
**Purpose**: Manual location sync trigger (on-demand)
|
|
464
|
-
**Trigger**: HTTP POST
|
|
465
|
-
**Endpoint**: `POST https://{workspace}.versori.run/location-sync-adhoc`
|
|
466
|
-
**Use Cases**: Testing, priority processing, ad-hoc runs
|
|
467
|
-
|
|
468
|
-
```typescript
|
|
469
|
-
import { webhook, http } from '@versori/run';
|
|
470
|
-
import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
|
|
471
|
-
import { executeLocationSync } from '../../services/location-sync.service';
|
|
472
|
-
|
|
473
|
-
/**
|
|
474
|
-
* Webhook: Manual Location Sync Trigger
|
|
475
|
-
*
|
|
476
|
-
* Endpoint: POST https://{workspace}.versori.run/location-sync-adhoc
|
|
477
|
-
* Request body (optional): { filePattern: "urgent_*.xml", maxFiles: 5 }
|
|
478
|
-
*
|
|
479
|
-
* Pattern: webhook().then(http()) - needs Fluent API access
|
|
480
|
-
* Uses shared service: location-sync.service.ts
|
|
481
|
-
*
|
|
482
|
-
* SECURITY: Authentication handled via connection parameter
|
|
483
|
-
* No manual API key validation needed - Versori manages this via connection auth
|
|
484
|
-
*/
|
|
485
|
-
export const adhocLocationSync = webhook('location-sync-adhoc', {
|
|
486
|
-
response: { mode: 'sync' },
|
|
487
|
-
connection: 'location-sync-adhoc', // Versori validates API key
|
|
488
|
-
}).then(
|
|
489
|
-
http('run-location-sync-adhoc', { connection: 'fluent_commerce' }, async ctx => {
|
|
490
|
-
const startTime = Date.now();
|
|
491
|
-
const { log, openKv, data } = ctx;
|
|
492
|
-
const jobId = `location-sync-adhoc-${Date.now()}`;
|
|
493
|
-
const tracker = new JobTracker(openKv(':project:'), log);
|
|
494
|
-
|
|
495
|
-
log.info('🚀 [START] Manual location sync triggered', { jobId, trigger: 'webhook', options: data });
|
|
496
|
-
|
|
497
|
-
await tracker.createJob(jobId, {
|
|
498
|
-
triggeredBy: 'manual',
|
|
499
|
-
stage: 'initialization',
|
|
500
|
-
options: data // Optional: filePattern, maxFiles, etc.
|
|
501
|
-
});
|
|
502
|
-
await tracker.updateJob(jobId, { status: 'processing' });
|
|
503
|
-
|
|
504
|
-
try {
|
|
505
|
-
log.info('⚙️ [PROCESSING] Starting manual location synchronization', { jobId });
|
|
506
|
-
const result = await executeLocationSync(ctx, jobId, tracker);
|
|
507
|
-
await tracker.markCompleted(jobId, result);
|
|
508
|
-
|
|
509
|
-
const duration = Date.now() - startTime;
|
|
510
|
-
log.info('✅ [SUCCESS] Manual location sync completed', {
|
|
511
|
-
jobId,
|
|
512
|
-
duration: `${duration}ms`,
|
|
513
|
-
processed: result.processed,
|
|
514
|
-
totalRecords: result.totalRecords
|
|
515
|
-
});
|
|
516
|
-
|
|
517
|
-
return { success: true, jobId, duration, ...result };
|
|
518
|
-
} catch (e: any) {
|
|
519
|
-
await tracker.markFailed(jobId, e);
|
|
520
|
-
const duration = Date.now() - startTime;
|
|
521
|
-
|
|
522
|
-
log.error('❌ [FAILED] Manual location sync failed', {
|
|
523
|
-
jobId,
|
|
524
|
-
duration: `${duration}ms`,
|
|
525
|
-
error: e?.message,
|
|
526
|
-
errorType: e?.name
|
|
527
|
-
});
|
|
528
|
-
|
|
529
|
-
return { success: false, jobId, duration, error: e?.message };
|
|
530
|
-
}
|
|
531
|
-
})
|
|
532
|
-
);
|
|
533
|
-
```
|
|
534
|
-
|
|
535
|
-
---
|
|
536
|
-
|
|
537
|
-
#### `src/workflows/webhook/job-status-check.ts`
|
|
538
|
-
|
|
539
|
-
**Purpose**: Query job status
|
|
540
|
-
**Trigger**: HTTP POST
|
|
541
|
-
**Endpoint**: `POST https://{workspace}.versori.run/location-sync-job-status`
|
|
542
|
-
**Request body**: `{ "jobId": "location-sync-1234567890" }`
|
|
543
|
-
|
|
544
|
-
```typescript
|
|
545
|
-
import { webhook, fn } from '@versori/run';
|
|
546
|
-
import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
|
|
547
|
-
|
|
548
|
-
/**
|
|
549
|
-
* Webhook: Job Status Check
|
|
550
|
-
*
|
|
551
|
-
* Endpoint: POST https://{workspace}.versori.run/location-sync-job-status
|
|
552
|
-
* Request body: { "jobId": "location-sync-1234567890" }
|
|
553
|
-
*
|
|
554
|
-
* Pattern: webhook().then(fn()) - no external API needed, only KV storage
|
|
555
|
-
* Lightweight: Only queries KV store, no Fluent API calls
|
|
556
|
-
*
|
|
557
|
-
* SECURITY: Authentication handled via connection parameter
|
|
558
|
-
* No manual API key validation needed - Versori manages this via connection auth
|
|
559
|
-
*/
|
|
560
|
-
export const locationSyncJobStatus = webhook('location-sync-job-status', {
|
|
561
|
-
response: { mode: 'sync' },
|
|
562
|
-
connection: 'location-sync-job-status',
|
|
563
|
-
}).then(
|
|
564
|
-
fn('status', async ctx => {
|
|
565
|
-
const { data, log, openKv } = ctx;
|
|
566
|
-
const jobId = data?.jobId as string;
|
|
567
|
-
|
|
568
|
-
if (!jobId) {
|
|
569
|
-
return { success: false, error: 'jobId required' };
|
|
570
|
-
}
|
|
571
|
-
|
|
572
|
-
const tracker = new JobTracker(openKv(':project:'), log);
|
|
573
|
-
const status = await tracker.getJob(jobId);
|
|
574
|
-
|
|
575
|
-
return status
|
|
576
|
-
? { success: true, jobId, ...status }
|
|
577
|
-
: { success: false, error: 'Job not found', jobId };
|
|
578
|
-
})
|
|
579
|
-
);
|
|
580
|
-
```
|
|
581
|
-
|
|
582
|
-
---
|
|
583
|
-
|
|
584
|
-
### Step 4: Main Orchestration Service (`src/services/location-sync.service.ts`)
|
|
585
|
-
|
|
586
|
-
**Note:** This service file should contain the `executeLocationSync` function (renamed from `runLocationXmlWorkflow`). The main workflow logic should be moved here.
|
|
587
|
-
|
|
588
|
-
```typescript
|
|
589
|
-
/**
|
|
590
|
-
* Main Orchestration Service: Location Sync
|
|
591
|
-
*
|
|
592
|
-
* This service contains the core business logic for location synchronization.
|
|
593
|
-
*
|
|
594
|
-
* Features:
|
|
595
|
-
* - S3 file operations with archival-based deduplication (moveFile to processed/)
|
|
596
|
-
* - XML parsing with @ prefix for attributes
|
|
597
|
-
* - Single element → array normalization
|
|
598
|
-
* - GraphQLMutationMapper for nested field transformations
|
|
599
|
-
* - GraphQL mutations with alias batching support
|
|
600
|
-
* - StateService for metadata tracking (secondary to archival)
|
|
601
|
-
* - Error state tracking with exponential backoff
|
|
602
|
-
*
|
|
603
|
-
* Deduplication Strategy:
|
|
604
|
-
* - PRIMARY: S3 archival via moveFile() - Files in processed/ won't be re-listed
|
|
605
|
-
* - SECONDARY: StateService KV tracking - Provides metadata and processing history
|
|
606
|
-
*/
|
|
607
|
-
import { Buffer } from 'node:buffer'; // Required for Deno/Versori runtime
|
|
608
|
-
import {
|
|
609
|
-
createClient,
|
|
610
|
-
S3DataSource,
|
|
611
|
-
XMLParserService,
|
|
612
|
-
GraphQLMutationMapper,
|
|
613
|
-
StateService,
|
|
614
|
-
VersoriKVAdapter,
|
|
615
|
-
JobTracker,
|
|
616
|
-
FluentClient,
|
|
617
|
-
} from '@fluentcommerce/fc-connect-sdk';
|
|
618
|
-
|
|
619
|
-
// ============================================================================
|
|
620
|
-
// Type Definitions
|
|
621
|
-
// ============================================================================
|
|
622
|
-
|
|
623
|
-
interface FileProcessingResult {
|
|
624
|
-
success: boolean;
|
|
625
|
-
locations: any[];
|
|
626
|
-
errors: string[];
|
|
627
|
-
}
|
|
628
|
-
|
|
629
|
-
interface MutationResult {
|
|
630
|
-
successful: number;
|
|
631
|
-
failed: number;
|
|
632
|
-
errors: string[];
|
|
633
|
-
}
|
|
634
|
-
|
|
635
|
-
interface MutationLogEntry {
|
|
636
|
-
timestamp: string;
|
|
637
|
-
fileName: string;
|
|
638
|
-
locationRef: string;
|
|
639
|
-
status: 'success' | 'failure';
|
|
640
|
-
error?: string;
|
|
641
|
-
}
|
|
642
|
-
|
|
643
|
-
// ============================================================================
|
|
644
|
-
// Utility Functions
|
|
645
|
-
// ============================================================================
|
|
646
|
-
|
|
647
|
-
/**
|
|
648
|
-
* Retry utility with exponential backoff
|
|
649
|
-
* (User-defined - not part of SDK public API)
|
|
650
|
-
*/
|
|
651
|
-
async function retryWithBackoff<T>(
|
|
652
|
-
operation: () => Promise<T>,
|
|
653
|
-
maxRetries = 3,
|
|
654
|
-
baseDelayMs = 1000
|
|
655
|
-
): Promise<T> {
|
|
656
|
-
let lastError: any;
|
|
657
|
-
for (let attempt = 0; attempt < maxRetries; attempt++) {
|
|
658
|
-
try {
|
|
659
|
-
return await operation();
|
|
660
|
-
} catch (error) {
|
|
661
|
-
lastError = error;
|
|
662
|
-
if (attempt < maxRetries - 1) {
|
|
663
|
-
const delay = baseDelayMs * Math.pow(2, attempt) + Math.floor(Math.random() * 200);
|
|
664
|
-
await new Promise(resolve => setTimeout(resolve, delay));
|
|
665
|
-
}
|
|
666
|
-
}
|
|
667
|
-
}
|
|
668
|
-
throw lastError;
|
|
669
|
-
}
|
|
670
|
-
|
|
671
|
-
/**
|
|
672
|
-
* Rate limiter for GraphQL mutations
|
|
673
|
-
*/
|
|
674
|
-
async function rateLimitedMutation(operation: () => Promise<any>, delayMs: number): Promise<any> {
|
|
675
|
-
const result = await operation();
|
|
676
|
-
await new Promise(resolve => setTimeout(resolve, delayMs));
|
|
677
|
-
return result;
|
|
678
|
-
}
|
|
679
|
-
|
|
680
|
-
// ============================================================================
|
|
681
|
-
// Service Functions
|
|
682
|
-
// ============================================================================
|
|
683
|
-
|
|
684
|
-
/**
|
|
685
|
-
* Service Function 1: Process File
|
|
686
|
-
*
|
|
687
|
-
* Downloads XML from S3, parses with XMLParserService, normalizes arrays,
|
|
688
|
-
* and maps fields with GraphQLMutationMapper.
|
|
689
|
-
*
|
|
690
|
-
* @param s3 - S3DataSource instance
|
|
691
|
-
* @param parser - XMLParserService instance
|
|
692
|
-
* @param mapper - GraphQLMutationMapper instance
|
|
693
|
-
* @param filePath - Full S3 path to file
|
|
694
|
-
* @param fileName - File name only (for logging)
|
|
695
|
-
* @param log - Logger instance
|
|
696
|
-
* @returns FileProcessingResult with locations array and errors
|
|
697
|
-
*/
|
|
698
|
-
async function processFile(
|
|
699
|
-
s3: S3DataSource,
|
|
700
|
-
parser: XMLParserService,
|
|
701
|
-
mapper: GraphQLMutationMapper,
|
|
702
|
-
filePath: string,
|
|
703
|
-
fileName: string,
|
|
704
|
-
log: any
|
|
705
|
-
): Promise<FileProcessingResult> {
|
|
706
|
-
try {
|
|
707
|
-
log.info('Processing file', { fileName });
|
|
708
|
-
|
|
709
|
-
// Download with retry
|
|
710
|
-
const content = await retryWithBackoff(
|
|
711
|
-
() => s3.downloadFile(filePath, { encoding: 'utf8' }) as Promise<string>
|
|
712
|
-
);
|
|
713
|
-
|
|
714
|
-
// Parse XML
|
|
715
|
-
const xmlData = await parser.parse(content);
|
|
716
|
-
|
|
717
|
-
// Extract location array (handle both single and multiple locations)
|
|
718
|
-
// CRITICAL: XML array normalization - single <location> becomes object, not array
|
|
719
|
-
const locationsData = xmlData.locations?.location;
|
|
720
|
-
const locationsArray = Array.isArray(locationsData) ? locationsData : [locationsData];
|
|
721
|
-
|
|
722
|
-
if (!locationsArray || locationsArray.length === 0) {
|
|
723
|
-
log.warn('Empty file (no locations)', { fileName });
|
|
724
|
-
return {
|
|
725
|
-
success: true,
|
|
726
|
-
locations: [],
|
|
727
|
-
errors: [],
|
|
728
|
-
};
|
|
729
|
-
}
|
|
730
|
-
|
|
731
|
-
// Map each location using GraphQLMutationMapper
|
|
732
|
-
const mappedLocations: Array<{ query: string; variables: any; input: any }> = [];
|
|
733
|
-
const mappingErrors: string[] = [];
|
|
734
|
-
|
|
735
|
-
// ✅ PRODUCTION ENHANCEMENT: Log transformation start
|
|
736
|
-
log.info('Transforming locations to GraphQL mutations', {
|
|
737
|
-
fileName,
|
|
738
|
-
totalLocations: locationsArray.length,
|
|
739
|
-
});
|
|
740
|
-
|
|
741
|
-
for (let i = 0; i < locationsArray.length; i++) {
|
|
742
|
-
const locationNumber = i + 1;
|
|
743
|
-
|
|
744
|
-
// ✅ PRODUCTION ENHANCEMENT: Log progress every 50 locations
|
|
745
|
-
if (locationNumber % 50 === 0) {
|
|
746
|
-
log.info(`📤 Transforming location ${locationNumber}/${locationsArray.length}`, {
|
|
747
|
-
fileName,
|
|
748
|
-
locationNumber,
|
|
749
|
-
totalLocations: locationsArray.length,
|
|
750
|
-
validSoFar: mappedLocations.length,
|
|
751
|
-
errorsSoFar: mappingErrors.length,
|
|
752
|
-
progress: `${((locationNumber / locationsArray.length) * 100).toFixed(1)}%`,
|
|
753
|
-
});
|
|
754
|
-
}
|
|
755
|
-
|
|
756
|
-
// Wrap location in context for mapping
|
|
757
|
-
const record = { location: locationsArray[i] };
|
|
758
|
-
try {
|
|
759
|
-
// GraphQLMutationMapper returns { query, variables } directly
|
|
760
|
-
const mappingResult = await mapper.map(record);
|
|
761
|
-
|
|
762
|
-
mappedLocations.push({
|
|
763
|
-
query: mappingResult.query,
|
|
764
|
-
variables: mappingResult.variables,
|
|
765
|
-
input: mappingResult.variables.input || mappingResult.variables,
|
|
766
|
-
});
|
|
767
|
-
} catch (error: unknown) {
|
|
768
|
-
const errorMsg = error instanceof Error ? error.message : String(error);
|
|
769
|
-
mappingErrors.push(`Location ${locationNumber}: ${errorMsg}`);
|
|
770
|
-
log.warn('Mapping failed for location', { fileName, index: i, error: errorMsg });
|
|
771
|
-
}
|
|
772
|
-
}
|
|
773
|
-
|
|
774
|
-
log.info('File processed', {
|
|
775
|
-
fileName,
|
|
776
|
-
total: locationsArray.length,
|
|
777
|
-
mapped: mappedLocations.length,
|
|
778
|
-
errors: mappingErrors.length,
|
|
779
|
-
successRate: `${((mappedLocations.length / locationsArray.length) * 100).toFixed(1)}%`,
|
|
780
|
-
});
|
|
781
|
-
|
|
782
|
-
return {
|
|
783
|
-
success: true,
|
|
784
|
-
locations: mappedLocations,
|
|
785
|
-
errors: mappingErrors,
|
|
786
|
-
};
|
|
787
|
-
} catch (error: any) {
|
|
788
|
-
// ✅ Enhanced error logging: Extract all error details for visibility
|
|
789
|
-
const errorDetails = {
|
|
790
|
-
message: error?.message || 'Unknown error',
|
|
791
|
-
stack: error?.stack,
|
|
792
|
-
fileName: error?.fileName,
|
|
793
|
-
lineNumber: error?.lineNumber,
|
|
794
|
-
originalError: error?.context?.originalError?.message,
|
|
795
|
-
errorType: error?.name || 'Error',
|
|
796
|
-
};
|
|
797
|
-
log.error('File processing failed', errorDetails, { fileName });
|
|
798
|
-
return {
|
|
799
|
-
success: false,
|
|
800
|
-
locations: [],
|
|
801
|
-
errors: [error.message || 'Unknown error'],
|
|
802
|
-
};
|
|
803
|
-
}
|
|
804
|
-
}
|
|
805
|
-
|
|
806
|
-
/**
|
|
807
|
-
* Service Function 2: Execute Mutations
|
|
808
|
-
*
|
|
809
|
-
* Executes GraphQL createLocation mutations with alias batching support.
|
|
810
|
-
*
|
|
811
|
-
* @param client - FluentClient instance
|
|
812
|
-
* @param mapper - GraphQLMutationMapper instance
|
|
813
|
-
* @param locations - Array of mapped location objects with query and variables
|
|
814
|
-
* @param log - Logger instance
|
|
815
|
-
* @param retailerId - Fluent retailer ID
|
|
816
|
-
* @param batchSize - Number of concurrent requests (default: 1)
|
|
817
|
-
* @param mutationsPerAliasBatch - Optional: Number of mutations per aliased request (default: undefined)
|
|
818
|
-
* @returns MutationResult with success/failure counts
|
|
819
|
-
*/
|
|
820
|
-
async function executeMutations(
|
|
821
|
-
client: FluentClient,
|
|
822
|
-
mapper: GraphQLMutationMapper,
|
|
823
|
-
locations: Array<{ query: string; variables: any; input: any }>,
|
|
824
|
-
log: any,
|
|
825
|
-
retailerId: string,
|
|
826
|
-
batchSize: number = 1, // ✅ Default: 1 (sequential)
|
|
827
|
-
mutationsPerAliasBatch?: number // ✅ NEW: Alias batching parameter (default: undefined = disabled)
|
|
828
|
-
): Promise<MutationResult> {
|
|
829
|
-
// Determine mode: use aliases if mutationsPerAliasBatch is set and > 1
|
|
830
|
-
const useAliases = mutationsPerAliasBatch && mutationsPerAliasBatch > 1;
|
|
831
|
-
|
|
832
|
-
if (useAliases) {
|
|
833
|
-
return await executeMutationsWithAliases(
|
|
834
|
-
client,
|
|
835
|
-
mapper,
|
|
836
|
-
locations,
|
|
837
|
-
log,
|
|
838
|
-
retailerId,
|
|
839
|
-
batchSize,
|
|
840
|
-
mutationsPerAliasBatch!
|
|
841
|
-
);
|
|
842
|
-
} else {
|
|
843
|
-
return await executeMutationsSeparate(
|
|
844
|
-
client,
|
|
845
|
-
locations,
|
|
846
|
-
log,
|
|
847
|
-
batchSize
|
|
848
|
-
);
|
|
849
|
-
}
|
|
850
|
-
}
|
|
851
|
-
|
|
852
|
-
/**
|
|
853
|
-
* Execute mutations using separate concurrent requests (current mode)
|
|
854
|
-
*/
|
|
855
|
-
async function executeMutationsSeparate(
|
|
856
|
-
client: FluentClient,
|
|
857
|
-
locations: Array<{ query: string; variables: any; input: any }>,
|
|
858
|
-
log: any,
|
|
859
|
-
batchSize: number
|
|
860
|
-
): Promise<MutationResult> {
|
|
861
|
-
const result: MutationResult = {
|
|
862
|
-
successful: 0,
|
|
863
|
-
failed: 0,
|
|
864
|
-
errors: [],
|
|
865
|
-
};
|
|
866
|
-
|
|
867
|
-
const safeConc = Math.max(1, Math.floor(batchSize));
|
|
868
|
-
|
|
869
|
-
// Sequential mode
|
|
870
|
-
if (safeConc === 1) {
|
|
871
|
-
for (const location of locations) {
|
|
872
|
-
try {
|
|
873
|
-
await retryWithBackoff(() =>
|
|
874
|
-
client.graphql({
|
|
875
|
-
query: location.query,
|
|
876
|
-
variables: location.variables,
|
|
877
|
-
})
|
|
878
|
-
);
|
|
879
|
-
result.successful++;
|
|
880
|
-
} catch (error: unknown) {
|
|
881
|
-
result.failed++;
|
|
882
|
-
const errorMsg = error instanceof Error ? error.message : String(error);
|
|
883
|
-
result.errors.push(errorMsg);
|
|
884
|
-
}
|
|
885
|
-
}
|
|
886
|
-
return result;
|
|
887
|
-
}
|
|
888
|
-
|
|
889
|
-
// Parallel mode
|
|
890
|
-
for (let i = 0; i < locations.length; i += safeConc) {
|
|
891
|
-
const chunk = locations.slice(i, i + safeConc);
|
|
892
|
-
const results = await Promise.allSettled(
|
|
893
|
-
chunk.map(loc =>
|
|
894
|
-
retryWithBackoff(() =>
|
|
895
|
-
client.graphql({
|
|
896
|
-
query: loc.query,
|
|
897
|
-
variables: loc.variables,
|
|
898
|
-
})
|
|
899
|
-
)
|
|
900
|
-
)
|
|
901
|
-
);
|
|
902
|
-
|
|
903
|
-
results.forEach((settledResult, idx) => {
|
|
904
|
-
if (settledResult.status === 'fulfilled') {
|
|
905
|
-
result.successful++;
|
|
906
|
-
} else {
|
|
907
|
-
result.failed++;
|
|
908
|
-
result.errors.push(
|
|
909
|
-
settledResult.reason instanceof Error ? settledResult.reason.message : String(settledResult.reason)
|
|
910
|
-
);
|
|
911
|
-
}
|
|
912
|
-
});
|
|
913
|
-
}
|
|
914
|
-
|
|
915
|
-
return result;
|
|
916
|
-
}
|
|
917
|
-
|
|
918
|
-
/**
|
|
919
|
-
* ✅ NEW: Execute mutations using GraphQL aliases (batched requests)
|
|
920
|
-
*/
|
|
921
|
-
async function executeMutationsWithAliases(
|
|
922
|
-
client: FluentClient,
|
|
923
|
-
mapper: GraphQLMutationMapper,
|
|
924
|
-
locations: Array<{ query: string; variables: any; input: any }>,
|
|
925
|
-
log: any,
|
|
926
|
-
retailerId: string,
|
|
927
|
-
maxParallel: number,
|
|
928
|
-
mutationsPerAliasBatch: number
|
|
929
|
-
): Promise<MutationResult> {
|
|
930
|
-
const result: MutationResult = { successful: 0, failed: 0, errors: [] };
|
|
931
|
-
|
|
932
|
-
const mutationName = (mapper as any).config.mutation || 'createLocation';
|
|
933
|
-
const aliasBatches: Array<Array<typeof locations[0]>> = [];
|
|
934
|
-
|
|
935
|
-
for (let i = 0; i < locations.length; i += mutationsPerAliasBatch) {
|
|
936
|
-
aliasBatches.push(locations.slice(i, i + mutationsPerAliasBatch));
|
|
937
|
-
}
|
|
938
|
-
|
|
939
|
-
// Process batches with concurrency control
|
|
940
|
-
for (let i = 0; i < aliasBatches.length; i += maxParallel) {
|
|
941
|
-
const concurrentBatches = aliasBatches.slice(i, i + maxParallel);
|
|
942
|
-
|
|
943
|
-
const batchResults = await Promise.allSettled(
|
|
944
|
-
concurrentBatches.map(async (batch) => {
|
|
945
|
-
const { query, variables } = buildAliasedBatch(batch, mutationName, retailerId);
|
|
946
|
-
const response = await retryWithBackoff(() => client.graphql({ query, variables }));
|
|
947
|
-
return parseAliasResponse(response, batch, mutationName);
|
|
948
|
-
})
|
|
949
|
-
);
|
|
950
|
-
|
|
951
|
-
batchResults.forEach((batchResult, idx) => {
|
|
952
|
-
if (batchResult.status === 'fulfilled') {
|
|
953
|
-
const batchRes = batchResult.value;
|
|
954
|
-
result.successful += batchRes.executed;
|
|
955
|
-
result.failed += batchRes.failed;
|
|
956
|
-
result.errors.push(...batchRes.errors);
|
|
957
|
-
} else {
|
|
958
|
-
const batch = concurrentBatches[idx];
|
|
959
|
-
const errorMsg = batchResult.reason instanceof Error ? batchResult.reason.message : String(batchResult.reason);
|
|
960
|
-
batch.forEach(loc => {
|
|
961
|
-
result.failed++;
|
|
962
|
-
result.errors.push(`Batch execution failed: ${errorMsg}`);
|
|
963
|
-
});
|
|
964
|
-
}
|
|
965
|
-
});
|
|
966
|
-
|
|
967
|
-
if (i + maxParallel < aliasBatches.length) {
|
|
968
|
-
await new Promise(resolve => setTimeout(resolve, 500));
|
|
969
|
-
}
|
|
970
|
-
}
|
|
971
|
-
|
|
972
|
-
return result;
|
|
973
|
-
}
|
|
974
|
-
|
|
975
|
-
/**
|
|
976
|
-
* ✅ NEW: Build aliased batch query and variables
|
|
977
|
-
*/
|
|
978
|
-
function buildAliasedBatch(
|
|
979
|
-
batch: Array<{ query: string; variables: any; input: any }>,
|
|
980
|
-
mutationName: string,
|
|
981
|
-
retailerId: string
|
|
982
|
-
): { query: string; variables: Record<string, any> } {
|
|
983
|
-
const batchSize = batch.length;
|
|
984
|
-
const inputTypeName = `${mutationName.charAt(0).toUpperCase() + mutationName.slice(1)}Input`;
|
|
985
|
-
|
|
986
|
-
const variables = Array.from({ length: batchSize }, (_, i) =>
|
|
987
|
-
`$input${i + 1}: ${inputTypeName}!`
|
|
988
|
-
).join(', ');
|
|
989
|
-
|
|
990
|
-
const aliasedMutations = Array.from({ length: batchSize }, (_, i) => {
|
|
991
|
-
const alias = `${mutationName}${i + 1}`;
|
|
992
|
-
return ` ${alias}: ${mutationName}(input: $input${i + 1}) { id ref name }`;
|
|
993
|
-
}).join('\n');
|
|
994
|
-
|
|
995
|
-
const operationName = `Batch${mutationName.charAt(0).toUpperCase() + mutationName.slice(1)}s`;
|
|
996
|
-
const query = `mutation ${operationName}(${variables}) {\n${aliasedMutations}\n}`;
|
|
997
|
-
|
|
998
|
-
const variablesObj: Record<string, any> = {};
|
|
999
|
-
batch.forEach((loc, index) => {
|
|
1000
|
-
const input = loc.variables.input || loc.variables;
|
|
1001
|
-
if (input && !input.retailer) {
|
|
1002
|
-
input.retailer = { id: parseInt(retailerId) };
|
|
1003
|
-
}
|
|
1004
|
-
variablesObj[`input${index + 1}`] = input;
|
|
1005
|
-
});
|
|
1006
|
-
|
|
1007
|
-
return { query, variables: variablesObj };
|
|
1008
|
-
}
|
|
1009
|
-
|
|
1010
|
-
/**
|
|
1011
|
-
* ✅ NEW: Parse aliased GraphQL response
|
|
1012
|
-
*/
|
|
1013
|
-
function parseAliasResponse(
|
|
1014
|
-
response: any,
|
|
1015
|
-
batch: Array<{ query: string; variables: any; input: any }>,
|
|
1016
|
-
mutationName: string
|
|
1017
|
-
): { executed: number; failed: number; errors: string[] } {
|
|
1018
|
-
const result = { executed: 0, failed: 0, errors: [] as string[] };
|
|
1019
|
-
|
|
1020
|
-
const data = response.data || {};
|
|
1021
|
-
const errors = response.errors || [];
|
|
1022
|
-
|
|
1023
|
-
batch.forEach((loc, index) => {
|
|
1024
|
-
const alias = `${mutationName}${index + 1}`;
|
|
1025
|
-
const aliasData = data[alias];
|
|
1026
|
-
const aliasErrors = errors.filter((e: any) =>
|
|
1027
|
-
e.path && Array.isArray(e.path) && e.path.includes(alias)
|
|
1028
|
-
);
|
|
1029
|
-
|
|
1030
|
-
if (aliasData && !aliasErrors.length) {
|
|
1031
|
-
result.executed++;
|
|
1032
|
-
} else {
|
|
1033
|
-
result.failed++;
|
|
1034
|
-
const errorMsg = aliasErrors[0]?.message || 'Mutation failed';
|
|
1035
|
-
result.errors.push(`${loc.input?.ref || 'unknown'}: ${errorMsg}`);
|
|
1036
|
-
}
|
|
1037
|
-
});
|
|
1038
|
-
|
|
1039
|
-
return result;
|
|
1040
|
-
}
|
|
1041
|
-
|
|
1042
|
-
/**
|
|
1043
|
-
* Service Function 3: Write Mutation Log
|
|
1044
|
-
*
|
|
1045
|
-
* Writes mutation results to S3 as a JSON log file.
|
|
1046
|
-
*
|
|
1047
|
-
* @param s3 - S3DataSource instance
|
|
1048
|
-
* @param logEntries - Array of mutation log entries
|
|
1049
|
-
* @param fileName - Original file name (used to generate log path)
|
|
1050
|
-
* @param logPrefix - S3 prefix for logs (e.g., 'logs/')
|
|
1051
|
-
* @param log - Logger instance
|
|
1052
|
-
*/
|
|
1053
|
-
async function writeMutationLog(
|
|
1054
|
-
s3: S3DataSource,
|
|
1055
|
-
logEntries: MutationLogEntry[],
|
|
1056
|
-
fileName: string,
|
|
1057
|
-
logPrefix: string,
|
|
1058
|
-
log: any
|
|
1059
|
-
): Promise<void> {
|
|
1060
|
-
try {
|
|
1061
|
-
const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
|
|
1062
|
-
const logFileName = `${logPrefix}${fileName.replace('.xml', '')}_${timestamp}.json`;
|
|
1063
|
-
|
|
1064
|
-
const logContent = JSON.stringify(
|
|
1065
|
-
{
|
|
1066
|
-
fileName,
|
|
1067
|
-
timestamp: new Date().toISOString(),
|
|
1068
|
-
totalMutations: logEntries.length,
|
|
1069
|
-
successful: logEntries.filter(e => e.status === 'success').length,
|
|
1070
|
-
failed: logEntries.filter(e => e.status === 'failure').length,
|
|
1071
|
-
entries: logEntries,
|
|
1072
|
-
},
|
|
1073
|
-
null,
|
|
1074
|
-
2
|
|
1075
|
-
);
|
|
1076
|
-
|
|
1077
|
-
// Write to S3 (uploadFile accepts string or Buffer)
|
|
1078
|
-
await s3.uploadFile(logFileName, logContent);
|
|
1079
|
-
|
|
1080
|
-
log.info('Mutation log written', { logFileName, entries: logEntries.length });
|
|
1081
|
-
} catch (error: any) {
|
|
1082
|
-
// ✅ Enhanced error logging: Extract all error details for visibility
|
|
1083
|
-
const errorDetails = {
|
|
1084
|
-
message: error?.message || 'Unknown error',
|
|
1085
|
-
stack: error?.stack,
|
|
1086
|
-
errorType: error?.name || 'Error',
|
|
1087
|
-
};
|
|
1088
|
-
log.error('Failed to write mutation log', errorDetails, { fileName });
|
|
1089
|
-
// Don't throw - logging failure shouldn't stop workflow
|
|
1090
|
-
}
|
|
1091
|
-
}
|
|
1092
|
-
|
|
1093
|
-
// ============================================================================
|
|
1094
|
-
// Main Workflow Function
|
|
1095
|
-
// ============================================================================
|
|
1096
|
-
|
|
1097
|
-
/**
|
|
1098
|
-
* Main Orchestration Function: Execute Location Sync
|
|
1099
|
-
*
|
|
1100
|
-
* This function orchestrates the location synchronization workflow.
|
|
1101
|
-
*
|
|
1102
|
-
* Architecture:
|
|
1103
|
-
* 1. List files from S3
|
|
1104
|
-
* 2. For each file:
|
|
1105
|
-
* a. processFile() - Download, parse, map
|
|
1106
|
-
* b. executeMutations() - Send GraphQL mutations with rate limiting
|
|
1107
|
-
* c. writeMutationLog() - Log results to S3
|
|
1108
|
-
* d. Archive file (primary deduplication)
|
|
1109
|
-
* e. Mark processed in KV (metadata tracking)
|
|
1110
|
-
*
|
|
1111
|
-
* @param ctx - Versori context
|
|
1112
|
-
* @param jobId - Job identifier
|
|
1113
|
-
* @param tracker - JobTracker instance
|
|
1114
|
-
* @returns Processing result
|
|
1115
|
-
*/
|
|
1116
|
-
export async function executeLocationSync(ctx: any, jobId: string, tracker: JobTracker) {
|
|
1117
|
-
const { log, activation } = ctx;
|
|
1118
|
-
|
|
1119
|
-
log.info('📋 [INIT] Reading activation variables', { jobId });
|
|
1120
|
-
|
|
1121
|
-
// Read activation variables
|
|
1122
|
-
const s3Bucket = activation?.getVariable('s3BucketName');
|
|
1123
|
-
const s3Region = activation?.getVariable('awsRegion') || 'us-east-1';
|
|
1124
|
-
const s3AccessKeyId = activation?.getVariable('awsAccessKeyId');
|
|
1125
|
-
const s3SecretAccessKey = activation?.getVariable('awsSecretAccessKey');
|
|
1126
|
-
const s3Prefix = activation?.getVariable('s3Prefix') || 'locations/';
|
|
1127
|
-
const archivePrefix = activation?.getVariable('archivePrefix') || 'processed/';
|
|
1128
|
-
const errorPrefix = activation?.getVariable('errorPrefix') || 'errors/';
|
|
1129
|
-
const logPrefix = activation?.getVariable('logPrefix') || 'logs/';
|
|
1130
|
-
const filePattern = (activation?.getVariable('filePattern') || '.xml').toLowerCase();
|
|
1131
|
-
const maxFiles = parseInt(activation?.getVariable('maxFilesToProcess') || '10', 10);
|
|
1132
|
-
const retailerId = activation?.getVariable('retailerId'); // Optional: Only if mutation schema requires it
|
|
1133
|
-
const enableArchival = activation?.getVariable('enableArchival') !== 'false';
|
|
1134
|
-
const enableMutationLogs = activation?.getVariable('enableMutationLogs') !== 'false';
|
|
1135
|
-
const enableFileTracking = activation?.getVariable('enableFileTracking') !== 'false';
|
|
1136
|
-
|
|
1137
|
-
// ✅ Configuration with defaults
|
|
1138
|
-
const mutationBatchSize = parseInt(
|
|
1139
|
-
activation?.getVariable('mutationBatchSize') || '1', // ✅ Default: 1 (sequential)
|
|
1140
|
-
10
|
|
1141
|
-
);
|
|
1142
|
-
|
|
1143
|
-
const mutationsPerAliasBatch = activation?.getVariable('mutationsPerAliasBatch')
|
|
1144
|
-
? parseInt(activation?.getVariable('mutationsPerAliasBatch') || '1', 10)
|
|
1145
|
-
: undefined; // ✅ Default: undefined (disabled, use separate requests)
|
|
1146
|
-
|
|
1147
|
-
// Validate required variables
|
|
1148
|
-
const missingVars: string[] = [];
|
|
1149
|
-
if (!s3Bucket) missingVars.push('s3BucketName');
|
|
1150
|
-
if (!s3AccessKeyId) missingVars.push('awsAccessKeyId');
|
|
1151
|
-
if (!s3SecretAccessKey) missingVars.push('awsSecretAccessKey');
|
|
1152
|
-
// Note: retailerId is optional - only needed if mutation schema requires it
|
|
1153
|
-
|
|
1154
|
-
if (missingVars.length > 0) {
|
|
1155
|
-
const errorMsg = `Missing required variables: ${missingVars.join(', ')}`;
|
|
1156
|
-
log.error('❌ [VALIDATION] Missing required activation variables', {
|
|
1157
|
-
missingVars,
|
|
1158
|
-
recommendation: 'Add missing variables in Versori activation settings'
|
|
1159
|
-
});
|
|
1160
|
-
return { success: false, error: errorMsg, processed: 0 };
|
|
1161
|
-
}
|
|
1162
|
-
|
|
1163
|
-
log.info('✅ [VALIDATION] All required variables present', {
|
|
1164
|
-
s3Bucket,
|
|
1165
|
-
s3Region,
|
|
1166
|
-
s3Prefix,
|
|
1167
|
-
enableFileTracking,
|
|
1168
|
-
mutationBatchSize
|
|
1169
|
-
});
|
|
1170
|
-
|
|
1171
|
-
try {
|
|
1172
|
-
log.info('🔧 [INIT] Initializing Fluent Commerce client', { jobId });
|
|
1173
|
-
|
|
1174
|
-
// Initialize services with validateConnection
|
|
1175
|
-
const client = await createClient(ctx, { validateConnection: true });
|
|
1176
|
-
if (!client) {
|
|
1177
|
-
throw new Error('Failed to create Fluent Commerce client');
|
|
1178
|
-
}
|
|
1179
|
-
|
|
1180
|
-
log.info('✅ [INIT] Fluent Commerce client validated and ready', { jobId });
|
|
1181
|
-
|
|
1182
|
-
// ✅ CORRECT: GraphQL mutations do NOT need client.setRetailerId()
|
|
1183
|
-
// setRetailerId() is only for Job/Event API, NOT GraphQL
|
|
1184
|
-
// Check your GraphQL schema to determine retailerId handling:
|
|
1185
|
-
// - Mandatory retailerId → Must pass it in mutation input
|
|
1186
|
-
// - Optional retailerId → Can pass it if needed
|
|
1187
|
-
// - No retailerId field → Don't pass it
|
|
1188
|
-
// See: fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md
|
|
1189
|
-
|
|
1190
|
-
log.info('🗄️ [INIT] Initializing S3 data source', { s3Bucket, s3Region, s3Prefix });
|
|
1191
|
-
|
|
1192
|
-
const s3 = new S3DataSource(
|
|
1193
|
-
{
|
|
1194
|
-
type: 'S3_XML',
|
|
1195
|
-
connectionId: 's3-location-sync',
|
|
1196
|
-
name: 'Source S3',
|
|
1197
|
-
s3Config: {
|
|
1198
|
-
bucket: s3Bucket,
|
|
1199
|
-
region: s3Region,
|
|
1200
|
-
accessKeyId: s3AccessKeyId,
|
|
1201
|
-
secretAccessKey: s3SecretAccessKey,
|
|
1202
|
-
},
|
|
1203
|
-
},
|
|
1204
|
-
log
|
|
1205
|
-
);
|
|
1206
|
-
|
|
1207
|
-
const parser = new XMLParserService();
|
|
1208
|
-
|
|
1209
|
-
// Initialize state tracking (only if enabled)
|
|
1210
|
-
let stateService: StateService | null = null;
|
|
1211
|
-
if (enableFileTracking) {
|
|
1212
|
-
log.info('🔄 [INIT] Enabling file tracking with StateService', { jobId });
|
|
1213
|
-
const stateKV = new VersoriKVAdapter(ctx);
|
|
1214
|
-
stateService = new StateService(stateKV);
|
|
1215
|
-
} else {
|
|
1216
|
-
log.info('⏭️ [INIT] File tracking disabled - relying on S3 archival only', { jobId });
|
|
1217
|
-
}
|
|
1218
|
-
|
|
1219
|
-
log.info('📝 [INIT] Loading mapping configuration', { jobId });
|
|
1220
|
-
|
|
1221
|
-
// ✅ CRITICAL: Load mapping config from external JSON file
|
|
1222
|
-
// Mapping config uses GraphQLMutationMapper structure (nested objects, not dot notation)
|
|
1223
|
-
// File: src/config/location-mapping.json
|
|
1224
|
-
const mappingConfigJson = await import('../config/location-mapping.json', { assert: { type: 'json' } });
|
|
1225
|
-
const mappingConfig = mappingConfigJson.default;
|
|
1226
|
-
|
|
1227
|
-
// Initialize GraphQLMutationMapper with client for schema introspection
|
|
1228
|
-
const mapper = new GraphQLMutationMapper(mappingConfig, log, { fluentClient: client });
|
|
1229
|
-
|
|
1230
|
-
log.info('📂 [S3] Listing files from S3', { s3Bucket, s3Prefix, filePattern });
|
|
1231
|
-
|
|
1232
|
-
// List files (pattern filtering handled by listFiles)
|
|
1233
|
-
const files = await s3.listFiles({
|
|
1234
|
-
prefix: s3Prefix,
|
|
1235
|
-
pattern: filePattern,
|
|
1236
|
-
maxKeys: 1000
|
|
1237
|
-
});
|
|
1238
|
-
|
|
1239
|
-
// Newest-first ordering
|
|
1240
|
-
const xmlFiles = files
|
|
1241
|
-
.sort((a: any, b: any) => {
|
|
1242
|
-
const aTime = a.lastModified ? new Date(a.lastModified).getTime() : 0;
|
|
1243
|
-
const bTime = b.lastModified ? new Date(b.lastModified).getTime() : 0;
|
|
1244
|
-
return bTime - aTime;
|
|
1245
|
-
})
|
|
1246
|
-
.slice(0, maxFiles);
|
|
1247
|
-
|
|
1248
|
-
log.info('📊 [S3] File discovery complete', {
|
|
1249
|
-
totalFiles: files.length,
|
|
1250
|
-
xmlFiles: xmlFiles.length,
|
|
1251
|
-
maxFiles,
|
|
1252
|
-
selectedFiles: xmlFiles.map(f => f.name)
|
|
1253
|
-
});
|
|
1254
|
-
|
|
1255
|
-
const results = {
|
|
1256
|
-
processed: 0,
|
|
1257
|
-
skipped: 0,
|
|
1258
|
-
failed: 0,
|
|
1259
|
-
totalRecords: 0,
|
|
1260
|
-
errors: [] as string[],
|
|
1261
|
-
};
|
|
1262
|
-
|
|
1263
|
-
log.info('🔄 [PROCESSING] Starting file processing loop', {
|
|
1264
|
-
fileCount: xmlFiles.length,
|
|
1265
|
-
jobId
|
|
1266
|
-
});
|
|
1267
|
-
|
|
1268
|
-
// Per-file processing loop
|
|
1269
|
-
for (const file of xmlFiles) {
|
|
1270
|
-
const filePath = file.path;
|
|
1271
|
-
const fileName = file.name;
|
|
1272
|
-
|
|
1273
|
-
log.info('📄 [FILE] Processing file', { fileName, filePath });
|
|
1274
|
-
|
|
1275
|
-
// Duplicate prevention (secondary check - files in processed/ won't be listed)
|
|
1276
|
-
// Primary deduplication: S3 archival (files moved to processed/ subdirectory)
|
|
1277
|
-
if (enableFileTracking && stateService) {
|
|
1278
|
-
const wasProcessed = await stateService.isFileProcessed(fileName);
|
|
1279
|
-
if (wasProcessed) {
|
|
1280
|
-
log.info('⏭️ [SKIP] File already processed (KV check)', { fileName });
|
|
1281
|
-
results.skipped++;
|
|
1282
|
-
continue;
|
|
1283
|
-
}
|
|
1284
|
-
}
|
|
1285
|
-
|
|
1286
|
-
try {
|
|
1287
|
-
// Step 1: Process file (download, parse, map)
|
|
1288
|
-
log.info('📥 [DOWNLOAD] Downloading and parsing file', { fileName });
|
|
1289
|
-
const processingResult = await processFile(s3, parser, mapper, filePath, fileName, log);
|
|
1290
|
-
|
|
1291
|
-
if (!processingResult.success) {
|
|
1292
|
-
throw new Error(`File processing failed: ${processingResult.errors.join(', ')}`);
|
|
1293
|
-
}
|
|
1294
|
-
|
|
1295
|
-
if (processingResult.locations.length === 0) {
|
|
1296
|
-
log.warn('⚠️ [SKIP] Empty file detected, archiving', { fileName });
|
|
1297
|
-
if (enableArchival) {
|
|
1298
|
-
await s3.moveFile(filePath, `${archivePrefix}${fileName}`);
|
|
1299
|
-
log.info('📦 [ARCHIVE] Empty file archived', { fileName, destination: `${archivePrefix}${fileName}` });
|
|
1300
|
-
}
|
|
1301
|
-
results.skipped++;
|
|
1302
|
-
continue;
|
|
1303
|
-
}
|
|
1304
|
-
|
|
1305
|
-
log.info('✅ [PARSE] File parsed successfully', {
|
|
1306
|
-
fileName,
|
|
1307
|
-
locationCount: processingResult.locations.length,
|
|
1308
|
-
mappingErrors: processingResult.errors.length
|
|
1309
|
-
});
|
|
1310
|
-
|
|
1311
|
-
// Step 2: Execute mutations
|
|
1312
|
-
// Step 2: Execute mutations with alias batching support
|
|
1313
|
-
// ? Enhanced: Extract context for progress logging
|
|
1314
|
-
const sampleLocationRefs = processingResult.locations.slice(0, 5).map((loc: any) => loc.input?.ref || loc.ref || 'unknown');
|
|
1315
|
-
const mutationType = mapper?.mutationName || 'createLocation';
|
|
1316
|
-
|
|
1317
|
-
// ? Enhanced: Start logging with context
|
|
1318
|
-
log.info(`[GraphQLMutations] Sending mutations for file "${fileName}"`, {
|
|
1319
|
-
totalMutations: processingResult.locations.length,
|
|
1320
|
-
mutationType,
|
|
1321
|
-
batchSize: mutationBatchSize,
|
|
1322
|
-
batchMode: mutationBatchSize === 1 ? 'sequential' : `parallel (${mutationBatchSize})`,
|
|
1323
|
-
sampleLocationRefs: sampleLocationRefs.join(', '),
|
|
1324
|
-
aliasBatching: mutationsPerAliasBatch ? `enabled (${mutationsPerAliasBatch} per alias)` : 'disabled'
|
|
1325
|
-
});
|
|
1326
|
-
|
|
1327
|
-
const mutationResult = await executeMutations(
|
|
1328
|
-
client,
|
|
1329
|
-
mapper,
|
|
1330
|
-
processingResult.locations,
|
|
1331
|
-
log,
|
|
1332
|
-
retailerId, // Pass retailerId for mutations that require it in input
|
|
1333
|
-
mutationBatchSize, // Concurrency control (default: 1)
|
|
1334
|
-
mutationsPerAliasBatch // ✅ NEW: Alias batching (default: undefined)
|
|
1335
|
-
);
|
|
1336
|
-
|
|
1337
|
-
// ? Enhanced: Completion logging with summary
|
|
1338
|
-
log.info(`[GraphQLMutations] Mutation submission completed for file "${fileName}"`, {
|
|
1339
|
-
totalMutations: processingResult.locations.length,
|
|
1340
|
-
successful: mutationResult.successful,
|
|
1341
|
-
failed: mutationResult.failed,
|
|
1342
|
-
successRate: processingResult.locations.length > 0 ? `${Math.round((mutationResult.successful / processingResult.locations.length) * 100)}%` : '0%',
|
|
1343
|
-
mutationType
|
|
1344
|
-
});
|
|
1345
|
-
|
|
1346
|
-
// Step 3: Write mutation log (if enabled)
|
|
1347
|
-
if (enableMutationLogs) {
|
|
1348
|
-
const logEntries: MutationLogEntry[] = processingResult.locations.map(loc => {
|
|
1349
|
-
const failed = mutationResult.errors.find(e => e.startsWith(loc.ref));
|
|
1350
|
-
return {
|
|
1351
|
-
timestamp: new Date().toISOString(),
|
|
1352
|
-
fileName,
|
|
1353
|
-
locationRef: loc.ref,
|
|
1354
|
-
status: failed ? 'failure' : 'success',
|
|
1355
|
-
error: failed,
|
|
1356
|
-
};
|
|
1357
|
-
});
|
|
1358
|
-
|
|
1359
|
-
await writeMutationLog(s3, logEntries, fileName, logPrefix, log);
|
|
1360
|
-
}
|
|
1361
|
-
|
|
1362
|
-
// Step 4: Archive file (PRIMARY deduplication - file won't be listed again)
|
|
1363
|
-
if (enableArchival) {
|
|
1364
|
-
log.info('📦 [ARCHIVE] Moving file to processed directory', { fileName });
|
|
1365
|
-
await s3.moveFile(filePath, `${archivePrefix}${fileName}`);
|
|
1366
|
-
log.info('✅ [ARCHIVE] File archived successfully', {
|
|
1367
|
-
fileName,
|
|
1368
|
-
destination: `${archivePrefix}${fileName}`
|
|
1369
|
-
});
|
|
1370
|
-
}
|
|
1371
|
-
|
|
1372
|
-
// Step 5: Mark processed in KV (SECONDARY - provides metadata/history)
|
|
1373
|
-
if (enableFileTracking && stateService) {
|
|
1374
|
-
log.info('💾 [STATE] Recording file processing metadata', { fileName });
|
|
1375
|
-
await stateService.markFileProcessed(fileName, {
|
|
1376
|
-
recordCount: processingResult.locations.length,
|
|
1377
|
-
successful: mutationResult.successful,
|
|
1378
|
-
failed: mutationResult.failed,
|
|
1379
|
-
mappingErrors: processingResult.errors.length,
|
|
1380
|
-
timestamp: new Date().toISOString(),
|
|
1381
|
-
});
|
|
1382
|
-
}
|
|
1383
|
-
|
|
1384
|
-
results.processed++;
|
|
1385
|
-
results.totalRecords += mutationResult.successful;
|
|
1386
|
-
|
|
1387
|
-
log.info('✅ [COMPLETE] File processing complete', {
|
|
1388
|
-
fileName,
|
|
1389
|
-
locations: processingResult.locations.length,
|
|
1390
|
-
successful: mutationResult.successful,
|
|
1391
|
-
failed: mutationResult.failed,
|
|
1392
|
-
mappingErrors: processingResult.errors.length,
|
|
1393
|
-
});
|
|
1394
|
-
|
|
1395
|
-
if (processingResult.errors.length > 0 || mutationResult.errors.length > 0) {
|
|
1396
|
-
results.errors.push(
|
|
1397
|
-
`${fileName}: ${processingResult.errors.length} mapping errors, ${mutationResult.errors.length} mutation errors`
|
|
1398
|
-
);
|
|
1399
|
-
}
|
|
1400
|
-
} catch (error: any) {
|
|
1401
|
-
// ✅ Enhanced error logging: Extract all error details for visibility
|
|
1402
|
-
const errorDetails = {
|
|
1403
|
-
message: error?.message || 'Unknown error',
|
|
1404
|
-
stack: error?.stack,
|
|
1405
|
-
fileName: error?.fileName,
|
|
1406
|
-
lineNumber: error?.lineNumber,
|
|
1407
|
-
originalError: error?.context?.originalError?.message,
|
|
1408
|
-
errorType: error?.name || 'Error',
|
|
1409
|
-
};
|
|
1410
|
-
|
|
1411
|
-
log.error('❌ [ERROR] File processing failed', errorDetails, { fileName });
|
|
1412
|
-
|
|
1413
|
-
// Provide error recommendations based on error type
|
|
1414
|
-
const recommendation = getErrorRecommendation(error);
|
|
1415
|
-
if (recommendation) {
|
|
1416
|
-
log.warn('💡 [RECOMMENDATION]', { fileName, recommendation });
|
|
1417
|
-
}
|
|
1418
|
-
|
|
1419
|
-
results.failed++;
|
|
1420
|
-
results.errors.push(`${fileName}: ${error.message}`);
|
|
1421
|
-
|
|
1422
|
-
// Attempt to move to error directory, ignore failures
|
|
1423
|
-
try {
|
|
1424
|
-
log.info('🗂️ [ERROR] Moving failed file to error directory', { fileName });
|
|
1425
|
-
await s3.moveFile(filePath, `${errorPrefix}${fileName}`);
|
|
1426
|
-
log.info('✅ [ERROR] Failed file moved to error directory', {
|
|
1427
|
-
fileName,
|
|
1428
|
-
destination: `${errorPrefix}${fileName}`
|
|
1429
|
-
});
|
|
1430
|
-
} catch (moveError) {
|
|
1431
|
-
log.warn('⚠️ [ERROR] Failed to move file to error directory', {
|
|
1432
|
-
fileName,
|
|
1433
|
-
moveError: moveError instanceof Error ? moveError.message : String(moveError)
|
|
1434
|
-
});
|
|
1435
|
-
}
|
|
1436
|
-
|
|
1437
|
-
// Track error state with exponential backoff (only if file tracking enabled)
|
|
1438
|
-
if (enableFileTracking && stateService) {
|
|
1439
|
-
try {
|
|
1440
|
-
const stateKV = new VersoriKVAdapter(ctx);
|
|
1441
|
-
const key = ['error-state', fileName];
|
|
1442
|
-
const prev = (await stateKV.get(key))?.value as any;
|
|
1443
|
-
const attempts = (prev?.attemptCount || 0) + 1;
|
|
1444
|
-
const backoffMinutes = Math.min(Math.pow(2, attempts) * 5, 24 * 60);
|
|
1445
|
-
const nextRetryAt = new Date(Date.now() + backoffMinutes * 60000).toISOString();
|
|
1446
|
-
|
|
1447
|
-
await stateKV.set(key, {
|
|
1448
|
-
fileName,
|
|
1449
|
-
attemptCount: attempts,
|
|
1450
|
-
lastError: error?.message || 'unknown',
|
|
1451
|
-
lastAttemptAt: new Date().toISOString(),
|
|
1452
|
-
firstFailedAt: prev?.firstFailedAt || new Date().toISOString(),
|
|
1453
|
-
nextRetryAt,
|
|
1454
|
-
});
|
|
1455
|
-
|
|
1456
|
-
log.info('💾 [ERROR] Error state tracked with exponential backoff', {
|
|
1457
|
-
fileName,
|
|
1458
|
-
attempts,
|
|
1459
|
-
nextRetryAt
|
|
1460
|
-
});
|
|
1461
|
-
} catch (stateError) {
|
|
1462
|
-
log.warn('⚠️ [ERROR] Failed to track error state', {
|
|
1463
|
-
fileName,
|
|
1464
|
-
stateError: stateError instanceof Error ? stateError.message : String(stateError)
|
|
1465
|
-
});
|
|
1466
|
-
}
|
|
1467
|
-
}
|
|
1468
|
-
}
|
|
1469
|
-
}
|
|
1470
|
-
|
|
1471
|
-
log.info('🏁 [COMPLETE] File processing loop finished', {
|
|
1472
|
-
processed: results.processed,
|
|
1473
|
-
skipped: results.skipped,
|
|
1474
|
-
failed: results.failed,
|
|
1475
|
-
totalRecords: results.totalRecords
|
|
1476
|
-
});
|
|
1477
|
-
|
|
1478
|
-
return results;
|
|
1479
|
-
} catch (error: any) {
|
|
1480
|
-
// ✅ Enhanced error logging: Extract all error details for visibility
|
|
1481
|
-
const errorDetails = {
|
|
1482
|
-
message: error?.message || 'Unknown error',
|
|
1483
|
-
stack: error?.stack,
|
|
1484
|
-
errorType: error?.name || 'Error',
|
|
1485
|
-
};
|
|
1486
|
-
|
|
1487
|
-
log.error('❌ [FATAL] Location sync failed', errorDetails);
|
|
1488
|
-
|
|
1489
|
-
// Provide fatal error recommendations
|
|
1490
|
-
const recommendation = getErrorRecommendation(error);
|
|
1491
|
-
if (recommendation) {
|
|
1492
|
-
log.warn('💡 [RECOMMENDATION]', { recommendation });
|
|
1493
|
-
}
|
|
1494
|
-
|
|
1495
|
-
return {
|
|
1496
|
-
success: false,
|
|
1497
|
-
error: error.message,
|
|
1498
|
-
processed: 0,
|
|
1499
|
-
timestamp: new Date().toISOString(),
|
|
1500
|
-
};
|
|
1501
|
-
}
|
|
1502
|
-
}
|
|
1503
|
-
|
|
1504
|
-
/**
|
|
1505
|
-
* Get error recommendation based on error type
|
|
1506
|
-
*/
|
|
1507
|
-
function getErrorRecommendation(error: any): string | null {
|
|
1508
|
-
const message = error?.message?.toLowerCase() || '';
|
|
1509
|
-
|
|
1510
|
-
if (message.includes('s3') || message.includes('access denied')) {
|
|
1511
|
-
return 'Check S3 credentials and bucket permissions. Verify IAM policy includes s3:ListBucket, s3:GetObject, s3:PutObject, s3:DeleteObject.';
|
|
1512
|
-
}
|
|
1513
|
-
|
|
1514
|
-
if (message.includes('xml') || message.includes('parse')) {
|
|
1515
|
-
return 'Verify XML file structure matches expected schema. Check for malformed XML or encoding issues.';
|
|
1516
|
-
}
|
|
1517
|
-
|
|
1518
|
-
if (message.includes('mapping') || message.includes('field')) {
|
|
1519
|
-
return 'Review field mapping configuration. Ensure all required fields are present and source paths are correct.';
|
|
1520
|
-
}
|
|
1521
|
-
|
|
1522
|
-
if (message.includes('graphql') || message.includes('mutation')) {
|
|
1523
|
-
return 'Check GraphQL schema and mutation input. Verify all required fields are provided and types match schema.';
|
|
1524
|
-
}
|
|
1525
|
-
|
|
1526
|
-
if (message.includes('auth') || message.includes('401') || message.includes('403')) {
|
|
1527
|
-
return 'Verify Fluent Commerce credentials. Check OAuth2 client ID/secret and ensure connection is active.';
|
|
1528
|
-
}
|
|
1529
|
-
|
|
1530
|
-
if (message.includes('timeout') || message.includes('econnrefused')) {
|
|
1531
|
-
return 'Check network connectivity. Verify API endpoints are accessible and not rate-limited.';
|
|
1532
|
-
}
|
|
1533
|
-
|
|
1534
|
-
return null;
|
|
1535
|
-
}
|
|
1536
|
-
```
|
|
1537
|
-
|
|
1538
|
-
**Note:** The `executeLocationSync` function should contain the full implementation of `runLocationXmlWorkflow` (renamed to `executeLocationSync`), including all the logic for processing files, executing mutations, and logging. The implementation details are shown in the service function code above.
|
|
1539
|
-
|
|
1540
|
-
---
|
|
1541
|
-
|
|
1542
|
-
### Step 5: TypeScript Configuration
|
|
1543
|
-
|
|
1544
|
-
**File: tsconfig.json**
|
|
1545
|
-
|
|
1546
|
-
```json
|
|
1547
|
-
{
|
|
1548
|
-
"compilerOptions": {
|
|
1549
|
-
"module": "ES2022",
|
|
1550
|
-
"target": "ES2024",
|
|
1551
|
-
"moduleResolution": "node"
|
|
1552
|
-
}
|
|
1553
|
-
}
|
|
1554
|
-
```
|
|
1555
|
-
|
|
1556
|
-
---
|
|
1557
|
-
|
|
1558
|
-
## Code Flow Explanation
|
|
1559
|
-
|
|
1560
|
-
### Initialization Phase
|
|
1561
|
-
|
|
1562
|
-
1. **Read activation variables** - S3 config, Fluent config, rate limiting, logging options
|
|
1563
|
-
2. **Validate required variables** - Fail fast if missing credentials
|
|
1564
|
-
3. **Initialize SDK services** - FluentClient, S3DataSource, XMLParserService, StateService
|
|
1565
|
-
4. **Create mapping configuration** - Define XML → GraphQL field mappings with nested objects
|
|
1566
|
-
5. **Calculate rate limit delay** - Convert mutations/second to delay in milliseconds
|
|
1567
|
-
|
|
1568
|
-
### File Discovery Phase
|
|
1569
|
-
|
|
1570
|
-
1. **List S3 files** - Use `s3.listFiles()` with prefix filter (excludes processed/ subdirectory)
|
|
1571
|
-
2. **Filter by pattern** - Match file extension (e.g., `.xml`)
|
|
1572
|
-
3. **Sort newest-first** - Process most recent files first
|
|
1573
|
-
4. **Apply max files limit** - Prevent overwhelming the workflow
|
|
1574
|
-
5. **Note**: Files in `processed/` subdirectory won't be listed (primary deduplication)
|
|
1575
|
-
|
|
1576
|
-
### Per-File Processing (Service Functions)
|
|
1577
|
-
|
|
1578
|
-
**Step 1: processFile()** - Download, parse, map
|
|
1579
|
-
|
|
1580
|
-
1. **Download file** - Use S3DataSource with retry logic
|
|
1581
|
-
2. **Parse XML** - XMLParserService converts to JavaScript object
|
|
1582
|
-
3. **Normalize array** - Handle single vs multiple `<location>` elements (CRITICAL for XML)
|
|
1583
|
-
4. **Map locations** - Use UniversalMapper with nested field mapping
|
|
1584
|
-
5. **Collect errors** - Track mapping errors without stopping
|
|
1585
|
-
6. **Return result** - FileProcessingResult with locations array and errors
|
|
1586
|
-
|
|
1587
|
-
**Step 2: executeMutations()** - GraphQL mutations with rate limiting
|
|
1588
|
-
|
|
1589
|
-
1. **Loop through locations** - Process each location individually
|
|
1590
|
-
2. **Build mutation input** - Extract nested fields (primaryAddress, openingSchedule)
|
|
1591
|
-
3. **Execute mutation** - Direct GraphQL `createLocation` with retry logic
|
|
1592
|
-
4. **Apply rate limiting** - Add configurable delay between mutations
|
|
1593
|
-
5. **Track results** - Count successful vs failed, collect error messages
|
|
1594
|
-
6. **Return result** - MutationResult with counts and errors
|
|
1595
|
-
|
|
1596
|
-
**Step 3: writeMutationLog()** - S3 log file (optional)
|
|
1597
|
-
|
|
1598
|
-
1. **Create log entries** - Map locations to log entries with status
|
|
1599
|
-
2. **Build JSON log** - Include timestamp, summary, detailed entries
|
|
1600
|
-
3. **Write to S3** - Use Buffer.from() for Deno/Versori runtime compatibility
|
|
1601
|
-
4. **Non-blocking** - Logging failure doesn't stop workflow
|
|
1602
|
-
5. **Timestamped naming** - Unique log file per processed file
|
|
1603
|
-
|
|
1604
|
-
### Cleanup Phase
|
|
1605
|
-
|
|
1606
|
-
1. **Archive file FIRST** - Move to `processed/` or `errors/` (PRIMARY deduplication)
|
|
1607
|
-
2. **Mark processed in KV** - StateService tracks metadata and processing history
|
|
1608
|
-
3. **Error state tracking** - Store error info with exponential backoff timestamp
|
|
1609
|
-
4. **Return results** - Summary of processed, skipped, failed files
|
|
1610
|
-
|
|
1611
|
-
**Note**: The order matters! Archive first (primary deduplication), then KV tracking (metadata/history).
|
|
1612
|
-
|
|
1613
|
-
### Service Function Benefits
|
|
1614
|
-
|
|
1615
|
-
- **Composability**: Each function has single responsibility and can be reused
|
|
1616
|
-
- **Testability**: Service functions can be unit tested independently
|
|
1617
|
-
- **Clarity**: Main workflow shows high-level orchestration
|
|
1618
|
-
- **Error Handling**: Isolated error handling per service function
|
|
1619
|
-
- **Logging**: Detailed logging at each step with structured context
|
|
1620
|
-
|
|
1621
|
-
---
|
|
1622
|
-
|
|
1623
|
-
## S3 Archival Deduplication Pattern
|
|
1624
|
-
|
|
1625
|
-
This template uses **S3 archival** as the primary deduplication mechanism, NOT VersoriFileTracker.
|
|
1626
|
-
|
|
1627
|
-
### How It Works
|
|
1628
|
-
|
|
1629
|
-
**S3 Directory Structure:**
|
|
1630
|
-
|
|
1631
|
-
```
|
|
1632
|
-
s3://my-bucket/
|
|
1633
|
-
├── locations/ ← listFiles() reads from here
|
|
1634
|
-
│ ├── new-file-1.xml
|
|
1635
|
-
│ └── new-file-2.xml
|
|
1636
|
-
├── processed/ ← Successfully processed files
|
|
1637
|
-
│ ├── old-file-1.xml
|
|
1638
|
-
│ └── old-file-2.xml
|
|
1639
|
-
└── errors/ ← Failed files
|
|
1640
|
-
└── bad-file.xml
|
|
1641
|
-
```
|
|
1642
|
-
|
|
1643
|
-
**Deduplication Flow:**
|
|
1644
|
-
|
|
1645
|
-
1. **List files**: `s3.listFiles({ prefix: 'locations/' })` - Only lists `locations/` subdirectory
|
|
1646
|
-
2. **Process file**: Download, parse, transform, send mutations
|
|
1647
|
-
3. **Archive**: `s3.moveFile(filePath, 'processed/new-file-1.xml')` - Moves file out of `locations/`
|
|
1648
|
-
4. **Next run**: File is now in `processed/`, won't be listed again
|
|
1649
|
-
|
|
1650
|
-
**Why This Works:**
|
|
1651
|
-
|
|
1652
|
-
- Files in `processed/` subdirectory are **never listed** when prefix is `locations/`
|
|
1653
|
-
- No need to track file state in KV store for deduplication
|
|
1654
|
-
- S3 is the single source of truth for file status
|
|
1655
|
-
- Simple, reliable, scales to millions of files
|
|
1656
|
-
|
|
1657
|
-
**StateService Role (Secondary):**
|
|
1658
|
-
|
|
1659
|
-
- Provides metadata and processing history
|
|
1660
|
-
- Backup check in case archival fails mid-process
|
|
1661
|
-
- Useful for monitoring and debugging
|
|
1662
|
-
- NOT the primary deduplication mechanism
|
|
1663
|
-
|
|
1664
|
-
**When to Use VersoriFileTracker:**
|
|
1665
|
-
|
|
1666
|
-
- **NEVER for S3 sources** - Use archival pattern instead
|
|
1667
|
-
- **Only for SFTP sources** - Where archival might not be possible
|
|
1668
|
-
- See SFTP templates for VersoriFileTracker usage
|
|
1669
|
-
|
|
1670
|
-
---
|
|
1671
|
-
|
|
1672
|
-
## XML Path Resolution Patterns
|
|
1673
|
-
|
|
1674
|
-
### Pattern 1: XML Attribute Access with @ Prefix
|
|
1675
|
-
|
|
1676
|
-
```typescript
|
|
1677
|
-
const mappingConfig = {
|
|
1678
|
-
fields: {
|
|
1679
|
-
// XML attribute access
|
|
1680
|
-
ref: { source: 'location.@ref' }, // <location ref="LOC-001">
|
|
1681
|
-
type: { source: 'location.@type' }, // <location type="WAREHOUSE">
|
|
1682
|
-
country: { source: 'location.address.@country' }, // <address country="USA">
|
|
1683
|
-
|
|
1684
|
-
// XML element text content
|
|
1685
|
-
name: { source: 'location.name' }, // <name>Downtown</name>
|
|
1686
|
-
city: { source: 'location.address.city' }, // <address><city>NYC</city></address>
|
|
1687
|
-
},
|
|
1688
|
-
};
|
|
1689
|
-
```
|
|
1690
|
-
|
|
1691
|
-
### Pattern 2: Handling Single vs Multiple Elements
|
|
1692
|
-
|
|
1693
|
-
```typescript
|
|
1694
|
-
// XML can have single or multiple <location> elements
|
|
1695
|
-
const locationsData = xmlData.locations?.location;
|
|
1696
|
-
|
|
1697
|
-
// Normalize to array
|
|
1698
|
-
const locations = Array.isArray(locationsData) ? locationsData : [locationsData];
|
|
1699
|
-
|
|
1700
|
-
// Process each location
|
|
1701
|
-
for (const loc of locations) {
|
|
1702
|
-
const record = { location: loc }; // Wrap for mapping
|
|
1703
|
-
const result = await mapper.map(record);
|
|
1704
|
-
}
|
|
1705
|
-
```
|
|
1706
|
-
|
|
1707
|
-
### Pattern 3: Nested Object Mapping
|
|
1708
|
-
|
|
1709
|
-
```typescript
|
|
1710
|
-
const mappingConfig = {
|
|
1711
|
-
fields: {
|
|
1712
|
-
// Root fields
|
|
1713
|
-
ref: { source: 'location.@ref', required: true },
|
|
1714
|
-
name: { source: 'location.name', required: true },
|
|
1715
|
-
type: { source: 'location.@type', required: true },
|
|
1716
|
-
|
|
1717
|
-
// Nested primaryAddress object
|
|
1718
|
-
'primaryAddress.ref': { source: 'location.@ref' },
|
|
1719
|
-
'primaryAddress.street': { source: 'location.address.street1' },
|
|
1720
|
-
'primaryAddress.city': { source: 'location.address.city' },
|
|
1721
|
-
'primaryAddress.latitude': { source: 'location.coordinates.@lat', resolver: 'sdk.parseFloat' },
|
|
1722
|
-
|
|
1723
|
-
// Nested openingSchedule object
|
|
1724
|
-
'openingSchedule.allHours': {
|
|
1725
|
-
source: 'location.openingSchedule.allHours',
|
|
1726
|
-
resolver: 'sdk.boolean',
|
|
1727
|
-
},
|
|
1728
|
-
'openingSchedule.monStart': {
|
|
1729
|
-
source: 'location.openingSchedule.monStart',
|
|
1730
|
-
resolver: 'sdk.parseInt',
|
|
1731
|
-
},
|
|
1732
|
-
|
|
1733
|
-
// Note: retailer.id not shown - standard createLocation does not have this field
|
|
1734
|
-
// If your schema requires it, add: 'retailer.id': { value: parseInt(retailerId) }
|
|
1735
|
-
},
|
|
1736
|
-
};
|
|
1737
|
-
```
|
|
1738
|
-
|
|
1739
|
-
---
|
|
1740
|
-
|
|
1741
|
-
## Sample XML Files
|
|
1742
|
-
|
|
1743
|
-
### Minimal Test File
|
|
1744
|
-
|
|
1745
|
-
**File: test-location.xml**
|
|
1746
|
-
|
|
1747
|
-
```xml
|
|
1748
|
-
<?xml version="1.0" encoding="UTF-8"?>
|
|
1749
|
-
<locations>
|
|
1750
|
-
<location ref="TEST-001" type="WAREHOUSE">
|
|
1751
|
-
<name>Test Warehouse</name>
|
|
1752
|
-
<address country="USA">
|
|
1753
|
-
<street1>123 Test St</street1>
|
|
1754
|
-
<city>TestCity</city>
|
|
1755
|
-
<state>TC</state>
|
|
1756
|
-
<postalCode>12345</postalCode>
|
|
1757
|
-
</address>
|
|
1758
|
-
<coordinates lat="40.7128" lon="-74.0060"/>
|
|
1759
|
-
<timeZone>America/New_York</timeZone>
|
|
1760
|
-
<openingSchedule>
|
|
1761
|
-
<allHours>false</allHours>
|
|
1762
|
-
<monStart>800</monStart>
|
|
1763
|
-
<monEnd>1800</monEnd>
|
|
1764
|
-
<tueStart>800</tueStart>
|
|
1765
|
-
<tueEnd>1800</tueEnd>
|
|
1766
|
-
<wedStart>800</wedStart>
|
|
1767
|
-
<wedEnd>1800</wedEnd>
|
|
1768
|
-
<thuStart>800</thuStart>
|
|
1769
|
-
<thuEnd>1800</thuEnd>
|
|
1770
|
-
<friStart>800</friStart>
|
|
1771
|
-
<friEnd>1800</friEnd>
|
|
1772
|
-
<satStart>0</satStart>
|
|
1773
|
-
<satEnd>0</satEnd>
|
|
1774
|
-
<sunStart>0</sunStart>
|
|
1775
|
-
<sunEnd>0</sunEnd>
|
|
1776
|
-
</openingSchedule>
|
|
1777
|
-
</location>
|
|
1778
|
-
</locations>
|
|
1779
|
-
```
|
|
1780
|
-
|
|
1781
|
-
---
|
|
1782
|
-
|
|
1783
|
-
## Service Functions Deep Dive
|
|
1784
|
-
|
|
1785
|
-
This template demonstrates service function composition for maintainable workflows.
|
|
1786
|
-
|
|
1787
|
-
### Function 1: processFile()
|
|
1788
|
-
|
|
1789
|
-
**Purpose**: Download, parse, normalize, and map XML data
|
|
1790
|
-
|
|
1791
|
-
**Inputs**:
|
|
1792
|
-
- `s3: S3DataSource` - For file download
|
|
1793
|
-
- `parser: XMLParserService` - For XML parsing
|
|
1794
|
-
- `mapper: UniversalMapper` - For field mapping
|
|
1795
|
-
- `filePath: string` - S3 path to file
|
|
1796
|
-
- `fileName: string` - File name for logging
|
|
1797
|
-
- `log: any` - Logger instance
|
|
1798
|
-
|
|
1799
|
-
**Outputs**: `FileProcessingResult`
|
|
1800
|
-
```typescript
|
|
1801
|
-
{
|
|
1802
|
-
success: boolean; // Overall success
|
|
1803
|
-
locations: any[]; // Mapped location objects
|
|
1804
|
-
errors: string[]; // Mapping error messages
|
|
1805
|
-
}
|
|
1806
|
-
```
|
|
1807
|
-
|
|
1808
|
-
**Key Operations**:
|
|
1809
|
-
1. Downloads file with retry logic
|
|
1810
|
-
2. Parses XML using XMLParserService
|
|
1811
|
-
3. **Normalizes arrays** - Handles single vs multiple `<location>` elements
|
|
1812
|
-
4. Maps each location using UniversalMapper
|
|
1813
|
-
5. Collects errors without stopping processing
|
|
1814
|
-
6. Returns all mapped locations + errors
|
|
1815
|
-
|
|
1816
|
-
**Why separate function?**
|
|
1817
|
-
- Testable independently with mock data
|
|
1818
|
-
- Reusable across workflows (scheduled, webhook, adhoc)
|
|
1819
|
-
- Clear error boundaries - file-level errors vs record-level errors
|
|
1820
|
-
- Easy to add XML validation or schema checks
|
|
1821
|
-
|
|
1822
|
-
### Function 2: executeMutations()
|
|
1823
|
-
|
|
1824
|
-
**Purpose**: Execute GraphQL createLocation mutations with rate limiting
|
|
1825
|
-
|
|
1826
|
-
**Inputs**:
|
|
1827
|
-
- `client: FluentClient` - For GraphQL mutations
|
|
1828
|
-
- `locations: any[]` - Mapped location data
|
|
1829
|
-
- `mutationDelayMs: number` - Rate limit delay
|
|
1830
|
-
- `fileName: string` - For logging context
|
|
1831
|
-
- `log: any` - Logger instance
|
|
1832
|
-
|
|
1833
|
-
**Outputs**: `MutationResult`
|
|
1834
|
-
```typescript
|
|
1835
|
-
{
|
|
1836
|
-
successful: number; // Count of successful mutations
|
|
1837
|
-
failed: number; // Count of failed mutations
|
|
1838
|
-
errors: string[]; // Error messages with location refs
|
|
1839
|
-
}
|
|
1840
|
-
```
|
|
1841
|
-
|
|
1842
|
-
**Key Operations**:
|
|
1843
|
-
1. Loops through locations
|
|
1844
|
-
2. Builds GraphQL mutation input
|
|
1845
|
-
3. Executes with retry logic + rate limiting
|
|
1846
|
-
4. Tracks success/failure per location
|
|
1847
|
-
5. Returns summary counts + errors
|
|
1848
|
-
|
|
1849
|
-
**Why separate function?**
|
|
1850
|
-
- Clear separation: mapping vs mutation execution
|
|
1851
|
-
- Rate limiting logic isolated and configurable
|
|
1852
|
-
- Easy to swap mutation types (create vs update)
|
|
1853
|
-
- Testable with mock FluentClient
|
|
1854
|
-
- Can parallelize in future (batch mutations)
|
|
1855
|
-
|
|
1856
|
-
### Function 3: writeMutationLog()
|
|
1857
|
-
|
|
1858
|
-
**Purpose**: Write detailed mutation results to S3 as JSON log
|
|
1859
|
-
|
|
1860
|
-
**Inputs**:
|
|
1861
|
-
- `s3: S3DataSource` - For log upload
|
|
1862
|
-
- `logEntries: MutationLogEntry[]` - Mutation status per location
|
|
1863
|
-
- `fileName: string` - Original file name
|
|
1864
|
-
- `logPrefix: string` - S3 prefix for logs (e.g., `logs/`)
|
|
1865
|
-
- `log: any` - Logger instance
|
|
1866
|
-
|
|
1867
|
-
**Outputs**: `void` (non-blocking - errors logged but not thrown)
|
|
1868
|
-
|
|
1869
|
-
**Key Operations**:
|
|
1870
|
-
1. Creates timestamped log file name
|
|
1871
|
-
2. Builds JSON log with summary + entries
|
|
1872
|
-
3. **Uses Buffer.from()** - Required for Deno/Versori runtime
|
|
1873
|
-
4. Writes to S3 with `uploadFile()`
|
|
1874
|
-
5. Errors don't stop workflow (logging is non-critical)
|
|
1875
|
-
|
|
1876
|
-
**Why separate function?**
|
|
1877
|
-
- Optional feature - can be disabled via config
|
|
1878
|
-
- Non-blocking - logging failure doesn't fail workflow
|
|
1879
|
-
- Structured logging for audit trails
|
|
1880
|
-
- Easy to change log format (JSON, CSV, XML)
|
|
1881
|
-
- Can add log rotation/cleanup logic later
|
|
1882
|
-
|
|
1883
|
-
### Service Function Composition Benefits
|
|
1884
|
-
|
|
1885
|
-
**1. Maintainability**
|
|
1886
|
-
```typescript
|
|
1887
|
-
// Clear workflow orchestration
|
|
1888
|
-
const processingResult = await processFile(...);
|
|
1889
|
-
const mutationResult = await executeMutations(...);
|
|
1890
|
-
await writeMutationLog(...);
|
|
1891
|
-
```
|
|
1892
|
-
|
|
1893
|
-
**2. Testability**
|
|
1894
|
-
```typescript
|
|
1895
|
-
// Unit test processFile() with mock XML
|
|
1896
|
-
const mockS3 = { downloadFile: jest.fn() };
|
|
1897
|
-
const result = await processFile(mockS3, parser, mapper, ...);
|
|
1898
|
-
expect(result.locations).toHaveLength(5);
|
|
1899
|
-
```
|
|
1900
|
-
|
|
1901
|
-
**3. Reusability**
|
|
1902
|
-
```typescript
|
|
1903
|
-
// Use processFile() in different workflows
|
|
1904
|
-
export const webhook = webhook('location-webhook').then(async ctx => {
|
|
1905
|
-
const result = await processFile(s3, parser, mapper, filePath, fileName, log);
|
|
1906
|
-
return { locations: result.locations };
|
|
1907
|
-
});
|
|
1908
|
-
```
|
|
1909
|
-
|
|
1910
|
-
**4. Error Isolation**
|
|
1911
|
-
```typescript
|
|
1912
|
-
// Each function has clear error boundaries
|
|
1913
|
-
try {
|
|
1914
|
-
const processingResult = await processFile(...);
|
|
1915
|
-
// File-level errors caught here
|
|
1916
|
-
} catch (error) {
|
|
1917
|
-
// Handle file processing failure
|
|
1918
|
-
}
|
|
1919
|
-
|
|
1920
|
-
// Mutation errors don't stop file processing
|
|
1921
|
-
const mutationResult = await executeMutations(...);
|
|
1922
|
-
// mutationResult.errors contains per-location failures
|
|
1923
|
-
```
|
|
1924
|
-
|
|
1925
|
-
**5. Progressive Enhancement**
|
|
1926
|
-
```typescript
|
|
1927
|
-
// Easy to add features without touching core logic
|
|
1928
|
-
async function validateLocations(locations: any[]) {
|
|
1929
|
-
// Add validation step
|
|
1930
|
-
}
|
|
1931
|
-
|
|
1932
|
-
const processingResult = await processFile(...);
|
|
1933
|
-
await validateLocations(processingResult.locations); // New step
|
|
1934
|
-
const mutationResult = await executeMutations(...);
|
|
1935
|
-
```
|
|
1936
|
-
|
|
1937
|
-
---
|
|
1938
|
-
|
|
1939
|
-
## Versori Environment Variables
|
|
1940
|
-
|
|
1941
|
-
**Activation Variables:**
|
|
1942
|
-
|
|
1943
|
-
```bash
|
|
1944
|
-
# ============================================================================
|
|
1945
|
-
# Required Variables
|
|
1946
|
-
# ============================================================================
|
|
1947
|
-
s3BucketName=my-location-bucket
|
|
1948
|
-
awsAccessKeyId=AKIAXXXXXXXXXXXX
|
|
1949
|
-
awsSecretAccessKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
|
|
1950
|
-
|
|
1951
|
-
# ============================================================================
|
|
1952
|
-
# S3 Configuration (Optional - with defaults)
|
|
1953
|
-
# ============================================================================
|
|
1954
|
-
awsRegion=us-east-1
|
|
1955
|
-
s3Prefix=locations/
|
|
1956
|
-
archivePrefix=processed/
|
|
1957
|
-
errorPrefix=errors/
|
|
1958
|
-
logPrefix=logs/
|
|
1959
|
-
filePattern=.xml
|
|
1960
|
-
maxFilesToProcess=10
|
|
1961
|
-
|
|
1962
|
-
# ============================================================================
|
|
1963
|
-
# Feature Toggles (Optional - with defaults)
|
|
1964
|
-
# ============================================================================
|
|
1965
|
-
# Enable S3 archival (move files to processed/errors directories)
|
|
1966
|
-
enableArchival=true
|
|
1967
|
-
|
|
1968
|
-
# Enable mutation logs (write detailed mutation results to S3)
|
|
1969
|
-
enableMutationLogs=true
|
|
1970
|
-
|
|
1971
|
-
# Enable file tracking via StateService + KV store
|
|
1972
|
-
# When disabled, relies on S3 archival only for deduplication
|
|
1973
|
-
enableFileTracking=true
|
|
1974
|
-
|
|
1975
|
-
# ============================================================================
|
|
1976
|
-
# Mutation Configuration (Optional - with defaults)
|
|
1977
|
-
# ============================================================================
|
|
1978
|
-
# Mutation batch size (concurrent requests)
|
|
1979
|
-
# - 1 = Sequential (default, safest)
|
|
1980
|
-
# - 5 = Process 5 mutations in parallel
|
|
1981
|
-
# - 10 = Process 10 mutations in parallel
|
|
1982
|
-
mutationBatchSize=1
|
|
1983
|
-
|
|
1984
|
-
# Alias batching (combine multiple mutations into single request)
|
|
1985
|
-
# - undefined = Disabled (default, use separate requests)
|
|
1986
|
-
# - 5 = Combine 5 mutations per aliased request
|
|
1987
|
-
# - 10 = Combine 10 mutations per aliased request
|
|
1988
|
-
mutationsPerAliasBatch=
|
|
1989
|
-
|
|
1990
|
-
# ============================================================================
|
|
1991
|
-
# Fluent Commerce Configuration (Optional)
|
|
1992
|
-
# ============================================================================
|
|
1993
|
-
# Retailer ID - Only if mutation schema requires retailerId in input
|
|
1994
|
-
# Standard createLocation does NOT require this field
|
|
1995
|
-
# Check your GraphQL schema to determine if needed
|
|
1996
|
-
retailerId=my-retailer-id
|
|
1997
|
-
```
|
|
1998
|
-
|
|
1999
|
-
**Notes:**
|
|
2000
|
-
- Webhook security is handled by Versori's native connection authentication. No manual API key configuration needed.
|
|
2001
|
-
- `retailerId` - Standard createLocation does not have this field. Only use if YOUR custom schema requires it.
|
|
2002
|
-
- See `fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md` for details.
|
|
2003
|
-
|
|
2004
|
-
---
|
|
2005
|
-
|
|
2006
|
-
## Schema Validation CLI Commands
|
|
2007
|
-
|
|
2008
|
-
Before deploying, validate your field mappings against the Fluent GraphQL schema:
|
|
2009
|
-
|
|
2010
|
-
### 1. Introspect Schema
|
|
2011
|
-
|
|
2012
|
-
```bash
|
|
2013
|
-
# Generate schema.json from live Fluent API
|
|
2014
|
-
npx fc-connect introspect-schema \
|
|
2015
|
-
--url https://api.fluentcommerce.com/graphql \
|
|
2016
|
-
--client-id your-client-id \
|
|
2017
|
-
--client-secret your-client-secret \
|
|
2018
|
-
--output schema.json
|
|
2019
|
-
```
|
|
2020
|
-
|
|
2021
|
-
### 2. Create Mapping Config File
|
|
2022
|
-
|
|
2023
|
-
**File: location-mapping.json**
|
|
2024
|
-
|
|
2025
|
-
```json
|
|
2026
|
-
{
|
|
2027
|
-
"version": "1.0.0",
|
|
2028
|
-
"mutation": "createLocation",
|
|
2029
|
-
"sourceFormat": "xml",
|
|
2030
|
-
"returnFields": ["id", "ref", "name", "type", "status"],
|
|
2031
|
-
"description": "XML location to Fluent Commerce GraphQL mapping",
|
|
2032
|
-
"fields": {
|
|
2033
|
-
"ref": { "source": "location.@ref", "required": true, "resolver": "sdk.trim" },
|
|
2034
|
-
"name": { "source": "location.name", "required": true, "resolver": "sdk.trim" },
|
|
2035
|
-
"type": { "source": "location.@type", "required": true, "resolver": "sdk.uppercase" },
|
|
2036
|
-
"primaryAddress.ref": { "source": "location.@ref", "required": true },
|
|
2037
|
-
"primaryAddress.street": { "source": "location.address.street1" },
|
|
2038
|
-
"primaryAddress.latitude": {
|
|
2039
|
-
"source": "location.coordinates.@lat",
|
|
2040
|
-
"resolver": "sdk.parseFloat"
|
|
2041
|
-
},
|
|
2042
|
-
"primaryAddress.longitude": {
|
|
2043
|
-
"source": "location.coordinates.@lon",
|
|
2044
|
-
"resolver": "sdk.parseFloat"
|
|
2045
|
-
}
|
|
2046
|
-
}
|
|
2047
|
-
}
|
|
2048
|
-
```
|
|
2049
|
-
|
|
2050
|
-
### 3. Validate Mapping
|
|
2051
|
-
|
|
2052
|
-
```bash
|
|
2053
|
-
# Validate that all target fields exist in schema
|
|
2054
|
-
npx fc-connect validate-schema \
|
|
2055
|
-
--mapping location-mapping.json \
|
|
2056
|
-
--schema schema.json
|
|
2057
|
-
```
|
|
2058
|
-
|
|
2059
|
-
### 4. Analyze Coverage
|
|
2060
|
-
|
|
2061
|
-
```bash
|
|
2062
|
-
# Check which Location fields are mapped vs available
|
|
2063
|
-
npx fc-connect analyze-coverage \
|
|
2064
|
-
--mapping location-mapping.json \
|
|
2065
|
-
--schema schema.json \
|
|
2066
|
-
--type CreateLocationInput
|
|
2067
|
-
```
|
|
2068
|
-
|
|
2069
|
-
**Output:**
|
|
2070
|
-
|
|
2071
|
-
```
|
|
2072
|
-
✅ Mapped: 15/42 fields (35%)
|
|
2073
|
-
❌ Missing required: timezone (String!)
|
|
2074
|
-
⚠️ Optional unmapped: supportPhoneNumber, networkId, attributes
|
|
2075
|
-
```
|
|
2076
|
-
|
|
2077
|
-
---
|
|
2078
|
-
|
|
2079
|
-
## Testing Locally
|
|
2080
|
-
|
|
2081
|
-
### 1. Upload Test XML to S3
|
|
2082
|
-
|
|
2083
|
-
```bash
|
|
2084
|
-
aws s3 cp test-location.xml s3://my-location-bucket/locations/test-location.xml
|
|
2085
|
-
```
|
|
2086
|
-
|
|
2087
|
-
### 2. Deploy to Versori
|
|
2088
|
-
|
|
2089
|
-
```bash
|
|
2090
|
-
npm run deploy
|
|
2091
|
-
```
|
|
2092
|
-
|
|
2093
|
-
### 3. Manual Testing
|
|
2094
|
-
|
|
2095
|
-
```bash
|
|
2096
|
-
# Trigger manual sync (auth handled by Versori connection)
|
|
2097
|
-
curl -X POST https://your-workspace.versori.run/location-xml-adhoc
|
|
2098
|
-
|
|
2099
|
-
# Check job status
|
|
2100
|
-
curl -X POST https://your-workspace.versori.run/location-xml-job-status \
|
|
2101
|
-
-H "Content-Type: application/json" \
|
|
2102
|
-
-d '{"jobId": "location-xml-adhoc-1737525600000"}'
|
|
2103
|
-
```
|
|
2104
|
-
|
|
2105
|
-
### 4. Verify Processing
|
|
2106
|
-
|
|
2107
|
-
- Upload a small XML to S3 (2-3 locations) and trigger `adhoc` webhook
|
|
2108
|
-
- Verify GraphQL mutations are executed for each location
|
|
2109
|
-
- Confirm file moved from `locations/` to `processed/` in S3
|
|
2110
|
-
- Check KV state: errors and last processed metadata
|
|
2111
|
-
- Monitor rate limiting: verify mutations respect configured rate
|
|
2112
|
-
|
|
2113
|
-
---
|
|
2114
|
-
|
|
2115
|
-
## Deployment
|
|
2116
|
-
|
|
2117
|
-
```bash
|
|
2118
|
-
# Deploy to Versori
|
|
2119
|
-
npm run deploy
|
|
2120
|
-
|
|
2121
|
-
# View logs
|
|
2122
|
-
npm run logs
|
|
2123
|
-
|
|
2124
|
-
# Monitor execution
|
|
2125
|
-
versori logs --follow
|
|
2126
|
-
```
|
|
2127
|
-
|
|
2128
|
-
---
|
|
2129
|
-
|
|
2130
|
-
## Monitoring
|
|
2131
|
-
|
|
2132
|
-
### Success Response
|
|
2133
|
-
|
|
2134
|
-
```json
|
|
2135
|
-
{
|
|
2136
|
-
"success": true,
|
|
2137
|
-
"jobId": "location-xml-scheduled-1737525600000",
|
|
2138
|
-
"processed": 3,
|
|
2139
|
-
"skipped": 0,
|
|
2140
|
-
"failed": 0,
|
|
2141
|
-
"totalRecords": 12,
|
|
2142
|
-
"errors": []
|
|
2143
|
-
}
|
|
2144
|
-
```
|
|
2145
|
-
|
|
2146
|
-
### Partial Success Response
|
|
2147
|
-
|
|
2148
|
-
```json
|
|
2149
|
-
{
|
|
2150
|
-
"success": true,
|
|
2151
|
-
"jobId": "location-xml-scheduled-1737525600000",
|
|
2152
|
-
"processed": 3,
|
|
2153
|
-
"skipped": 1,
|
|
2154
|
-
"failed": 0,
|
|
2155
|
-
"totalRecords": 10,
|
|
2156
|
-
"errors": ["locations-003.xml: 2 mapping errors"]
|
|
2157
|
-
}
|
|
2158
|
-
```
|
|
2159
|
-
|
|
2160
|
-
### Error Response
|
|
2161
|
-
|
|
2162
|
-
```json
|
|
2163
|
-
{
|
|
2164
|
-
"success": false,
|
|
2165
|
-
"jobId": "location-xml-scheduled-1737525600000",
|
|
2166
|
-
"processed": 0,
|
|
2167
|
-
"skipped": 0,
|
|
2168
|
-
"failed": 1,
|
|
2169
|
-
"totalRecords": 0,
|
|
2170
|
-
"errors": ["locations-001.xml: Invalid XML structure"]
|
|
2171
|
-
}
|
|
2172
|
-
```
|
|
2173
|
-
|
|
2174
|
-
---
|
|
2175
|
-
|
|
2176
|
-
## Common Pitfalls and Solutions
|
|
2177
|
-
|
|
2178
|
-
### 1. XML Attribute Not Found
|
|
2179
|
-
|
|
2180
|
-
**Symptoms**: Mapping errors like "field not found"
|
|
2181
|
-
|
|
2182
|
-
**Solution**:
|
|
2183
|
-
|
|
2184
|
-
```typescript
|
|
2185
|
-
// ❌ WRONG - Missing @ prefix for attribute
|
|
2186
|
-
ref: {
|
|
2187
|
-
source: 'location.ref';
|
|
2188
|
-
}
|
|
2189
|
-
|
|
2190
|
-
// ✅ CORRECT - Use @ prefix for XML attributes
|
|
2191
|
-
ref: {
|
|
2192
|
-
source: 'location.@ref';
|
|
2193
|
-
}
|
|
2194
|
-
```
|
|
2195
|
-
|
|
2196
|
-
### 2. Single Element Not Array
|
|
2197
|
-
|
|
2198
|
-
**Symptoms**: "Cannot read property forEach of undefined"
|
|
2199
|
-
|
|
2200
|
-
**Solution**:
|
|
2201
|
-
|
|
2202
|
-
```typescript
|
|
2203
|
-
// Always normalize to array
|
|
2204
|
-
const locationsData = xmlData.locations?.location;
|
|
2205
|
-
const locations = Array.isArray(locationsData) ? locationsData : [locationsData];
|
|
2206
|
-
```
|
|
2207
|
-
|
|
2208
|
-
### 3. Rate Limiting Too Aggressive
|
|
2209
|
-
|
|
2210
|
-
**Symptoms**: Slow processing, mutations taking too long
|
|
2211
|
-
|
|
2212
|
-
**Solution**:
|
|
2213
|
-
|
|
2214
|
-
```bash
|
|
2215
|
-
# Increase rate limit (mutations per second)
|
|
2216
|
-
mutationRateLimit=10 # Default is 5
|
|
2217
|
-
```
|
|
2218
|
-
|
|
2219
|
-
### 4. Empty Element vs Missing Element
|
|
2220
|
-
|
|
2221
|
-
**Solution**:
|
|
2222
|
-
|
|
2223
|
-
```typescript
|
|
2224
|
-
// Use required: false and defaultValue for optional fields
|
|
2225
|
-
'primaryAddress.street2': {
|
|
2226
|
-
source: 'location.address.street2',
|
|
2227
|
-
required: false,
|
|
2228
|
-
defaultValue: ''
|
|
2229
|
-
}
|
|
2230
|
-
```
|
|
2231
|
-
|
|
2232
|
-
### 5. S3 Access Denied
|
|
2233
|
-
|
|
2234
|
-
**Symptoms**: S3 operations fail with 403 errors
|
|
2235
|
-
|
|
2236
|
-
**Solution**: Validate IAM permissions
|
|
2237
|
-
|
|
2238
|
-
**Required IAM Permissions:**
|
|
2239
|
-
|
|
2240
|
-
```json
|
|
2241
|
-
{
|
|
2242
|
-
"Version": "2012-10-17",
|
|
2243
|
-
"Statement": [
|
|
2244
|
-
{
|
|
2245
|
-
"Effect": "Allow",
|
|
2246
|
-
"Action": ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
|
|
2247
|
-
"Resource": ["arn:aws:s3:::my-location-bucket", "arn:aws:s3:::my-location-bucket/*"]
|
|
2248
|
-
}
|
|
2249
|
-
]
|
|
2250
|
-
}
|
|
2251
|
-
```
|
|
2252
|
-
|
|
2253
|
-
### 6. GraphQL Schema Mismatch
|
|
2254
|
-
|
|
2255
|
-
**Symptoms**: Mutation errors like "Unknown field", "Invalid input type"
|
|
2256
|
-
|
|
2257
|
-
**Solution**: Use CLI tools to validate mappings
|
|
2258
|
-
|
|
2259
|
-
```bash
|
|
2260
|
-
npx fc-connect validate-schema --mapping location-mapping.json --schema schema.json
|
|
2261
|
-
```
|
|
2262
|
-
|
|
2263
|
-
### 7. Nested Object Mapping Errors
|
|
2264
|
-
|
|
2265
|
-
**Symptoms**: Flat structure instead of nested objects in mutation input
|
|
2266
|
-
|
|
2267
|
-
**Solution**: Use dot notation in field mapping
|
|
2268
|
-
|
|
2269
|
-
```typescript
|
|
2270
|
-
// ✅ CORRECT - Creates nested structure
|
|
2271
|
-
'primaryAddress.city': { source: 'location.address.city' }
|
|
2272
|
-
|
|
2273
|
-
// ❌ WRONG - Creates flat structure
|
|
2274
|
-
primaryAddress_city: { source: 'location.address.city' }
|
|
2275
|
-
```
|
|
2276
|
-
|
|
2277
|
-
### 8. retailerId Configuration Errors
|
|
2278
|
-
|
|
2279
|
-
**Symptoms**: "retailerId is required" errors or confusion about when to use `setRetailerId()`
|
|
2280
|
-
|
|
2281
|
-
**Solution**: Understand the correct pattern
|
|
2282
|
-
|
|
2283
|
-
```typescript
|
|
2284
|
-
// ✅ CORRECT - GraphQL mutations don't need setRetailerId()
|
|
2285
|
-
// Check your GraphQL schema to determine retailerId handling:
|
|
2286
|
-
// - Mandatory retailerId → Must pass it in mutation input
|
|
2287
|
-
// - Optional retailerId → Can pass it if needed
|
|
2288
|
-
// - No retailerId field → Don't pass it
|
|
2289
|
-
// Standard createLocation does not have retailerId field in schema
|
|
2290
|
-
|
|
2291
|
-
// ✅ IF mutation schema requires retailerId (mandatory):
|
|
2292
|
-
const { query, variables } = await mapper.map(location);
|
|
2293
|
-
if (retailerId && variables.input) {
|
|
2294
|
-
variables.input.retailer = { id: parseInt(retailerId) };
|
|
2295
|
-
}
|
|
2296
|
-
await client.graphql({ query, variables });
|
|
2297
|
-
```
|
|
2298
|
-
|
|
2299
|
-
**Reference:** `fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md`
|
|
2300
|
-
|
|
2301
|
-
---
|
|
2302
|
-
|
|
2303
|
-
## Key Takeaways
|
|
2304
|
-
|
|
2305
|
-
### Architecture Patterns
|
|
2306
|
-
|
|
2307
|
-
- **Service Function Composition**: Workflow broken into 3 service functions - `processFile()`, `executeMutations()`, `writeMutationLog()`
|
|
2308
|
-
- **Per-File Processing**: Main workflow orchestrates service functions for each file
|
|
2309
|
-
- **Clear Separation of Concerns**: Parse/map, execute mutations, logging are independent functions
|
|
2310
|
-
|
|
2311
|
-
### SDK Usage
|
|
2312
|
-
|
|
2313
|
-
- **Buffer Import (CRITICAL)**: Always `import { Buffer } from 'node:buffer'` for Deno/Versori runtime
|
|
2314
|
-
- **S3 Archival Deduplication**: Use `s3.moveFile()` to `processed/` subdirectory - PRIMARY deduplication mechanism
|
|
2315
|
-
- **NO VersoriFileTracker for S3**: S3 archival is simpler and more reliable
|
|
2316
|
-
- **StateService Role**: SECONDARY - provides metadata/history, not primary deduplication
|
|
2317
|
-
|
|
2318
|
-
### XML Processing
|
|
2319
|
-
|
|
2320
|
-
- **XML @ Prefix**: Always use `@` prefix for XML attributes (`location.@ref`)
|
|
2321
|
-
- **Array Normalization (CRITICAL)**: Handle single vs multiple elements with `Array.isArray()` check - single `<location>` becomes object, not array
|
|
2322
|
-
- **Nested Mapping**: Use dot notation for nested objects (`primaryAddress.street`, `openingSchedule.monStart`)
|
|
2323
|
-
|
|
2324
|
-
### GraphQL Mutations
|
|
2325
|
-
|
|
2326
|
-
- **NO setRetailerId() Required**: GraphQL mutations do NOT need `client.setRetailerId()` - only Job/Event API needs it
|
|
2327
|
-
- **retailerId in Input**: Check your GraphQL schema to determine retailerId handling:
|
|
2328
|
-
- **Mandatory retailerId** - Field exists and is required (`!`) → Must pass it
|
|
2329
|
-
- **Optional retailerId** - Field exists and is optional → Can pass it if needed
|
|
2330
|
-
- **No retailerId field** - Field doesn't exist → Don't pass it
|
|
2331
|
-
- **Rate Limiting**: Implement configurable delays between mutations to avoid API throttling
|
|
2332
|
-
- **Retry Logic**: Exponential backoff for failed mutations with `retryWithBackoff()`
|
|
2333
|
-
- **Direct Mutations**: Use `client.graphql()` for location upserts (NOT Batch API)
|
|
2334
|
-
- **GraphQL vs Batch API**: Use GraphQL for low-volume master data, Batch API for high-volume inventory
|
|
2335
|
-
|
|
2336
|
-
### Error Handling & Monitoring
|
|
2337
|
-
|
|
2338
|
-
- **Error Recovery**: Exponential backoff for error state tracking with retry timestamps
|
|
2339
|
-
- **Mutation Logging**: Optional S3 JSON logs with detailed per-location mutation status
|
|
2340
|
-
- **Schema Validation**: Use CLI tools before deployment to catch mapping errors
|
|
2341
|
-
- **Archival Order**: Archive FIRST (deduplication), then KV tracking (metadata)
|
|
2342
|
-
|
|
2343
|
-
---
|
|
2344
|
-
|
|
2345
|
-
## Related Documentation
|
|
2346
|
-
|
|
2347
|
-
### Core Guides
|
|
2348
|
-
|
|
2349
|
-
- **GraphQL Mutation Mapping**: `fc-connect-sdk/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/readme.md`
|
|
2350
|
-
- **Universal Mapping**: `fc-connect-sdk/docs/02-CORE-GUIDES/mapping/modules/readme.md`
|
|
2351
|
-
- **XML Parser**: `fc-connect-sdk/docs/02-CORE-GUIDES/parsers/modules/05-xml-parser.md`
|
|
2352
|
-
- **Data Sources**: `fc-connect-sdk/docs/02-CORE-GUIDES/data-sources/readme.md`
|
|
2353
|
-
- **State Management**: `fc-connect-sdk/docs/03-PATTERN-GUIDES/file-operations/state-duplicate-prevention.md`
|
|
2354
|
-
|
|
2355
|
-
### Related Templates
|
|
2356
|
-
|
|
2357
|
-
- **CSV Version**: `template-ingestion-s3-csv-location-graphql.md`
|
|
2358
|
-
- **JSON Version**: `template-ingestion-s3-json-location-graphql.md`
|
|
2359
|
-
- **SFTP XML Version**: `template-ingestion-sftp-xml-location-graphql.md`
|
|
2360
|
-
- **Event API Pattern**: `../event-api/template-ingestion-s3-xml-product-event.md`
|
|
2361
|
-
- **Batch API Pattern**: `../batch-api/template-ingestion-s3-xml-inventory-batch.md`
|
|
2362
|
-
|
|
2363
|
-
### CLI Tools
|
|
2364
|
-
|
|
2365
|
-
- **Schema Introspection**: `fc-connect-sdk/bin/readme.md#introspect-schema`
|
|
2366
|
-
- **Mapping Validation**: `fc-connect-sdk/bin/readme.md#validate-schema`
|
|
2367
|
-
- **Coverage Analysis**: `fc-connect-sdk/bin/readme.md#analyze-coverage`
|
|
2368
|
-
|
|
2369
|
-
### Patterns
|
|
2370
|
-
|
|
2371
|
-
- **Error Handling**: `fc-connect-sdk/docs/01-TEMPLATES/patterns/error-handling-retry.md`
|
|
2372
|
-
- **Rate Limiting**: `fc-connect-sdk/docs/03-PATTERN-GUIDES/integration-patterns/rate-limiting.md`
|
|
2373
|
-
- **XML Patterns**: `fc-connect-sdk/docs/01-TEMPLATES/versori/patterns/xml-response-patterns.md`
|
|
1
|
+
---
|
|
2
|
+
template_id: tpl-ingest-s3-xml-to-location-graphql
|
|
3
|
+
canonical_filename: template-ingestion-s3-xml-location-graphql.md
|
|
4
|
+
version: 2.0.0
|
|
5
|
+
sdk_version: ^0.1.39
|
|
6
|
+
runtime: versori
|
|
7
|
+
direction: ingestion
|
|
8
|
+
source: s3-xml
|
|
9
|
+
destination: fluent-graphql
|
|
10
|
+
entity: location
|
|
11
|
+
format: xml
|
|
12
|
+
logging: versori
|
|
13
|
+
status: stable
|
|
14
|
+
compliance: gold-standard
|
|
15
|
+
features:
|
|
16
|
+
- graphql-mutation-mapper
|
|
17
|
+
- memory-management
|
|
18
|
+
- enhanced-logging
|
|
19
|
+
- attribute-transformation
|
|
20
|
+
---
|
|
21
|
+
|
|
22
|
+
Template: Ingestion - S3 XML to Location GraphQL
|
|
23
|
+
|
|
24
|
+
**Template Version:** 2.0.0
|
|
25
|
+
**SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
|
|
26
|
+
**Last Updated:** 2025-01-24
|
|
27
|
+
|
|
28
|
+
**🆕 Version 2.0.0 Enhancements:**
|
|
29
|
+
- ✅ **GraphQL Mutation Mapper** - Direct field mapping to mutation variables
|
|
30
|
+
- ✅ **Memory Management** - Clear large arrays after processing batches
|
|
31
|
+
- ✅ **Enhanced Logging** - Track mutation execution with emoji indicators
|
|
32
|
+
- ✅ **Attribute Transformation** - Handle complex nested data structures
|
|
33
|
+
|
|
34
|
+
---
|
|
35
|
+
|
|
36
|
+
## Implementation Prompt
|
|
37
|
+
|
|
38
|
+
"Create a Versori scheduled workflow that reads location XML files from S3, transforms the data using GraphQLMutationMapper with nested object mapping, and creates/updates Fluent Commerce locations via direct GraphQL mutations with alias batching support.
|
|
39
|
+
|
|
40
|
+
**Requirements:**
|
|
41
|
+
|
|
42
|
+
1. **S3 Source**:
|
|
43
|
+
- List and download XML files with configurable prefix and pattern
|
|
44
|
+
- Archive processed files to `processed/` directory (deduplication via archiving)
|
|
45
|
+
- Move failed files to `errors/` directory
|
|
46
|
+
- Use S3DataSource with retry logic and built-in archival
|
|
47
|
+
- Support configurable bucket, region, credentials
|
|
48
|
+
|
|
49
|
+
2. **XML Parsing**:
|
|
50
|
+
- Use XMLParserService with @ prefix for attribute access
|
|
51
|
+
- Handle both single and multiple `<location>` elements (array normalization)
|
|
52
|
+
- Support nested XML paths (`location.address.street1`, `location.coordinates.@lat`)
|
|
53
|
+
- Parse complex structures (addresses, coordinates, opening schedules)
|
|
54
|
+
|
|
55
|
+
3. **Field Mapping**:
|
|
56
|
+
- Map XML fields to Location GraphQL input type with nested objects:
|
|
57
|
+
- `ref`, `name`, `type` (root fields)
|
|
58
|
+
- `primaryAddress.*` (nested address object with coordinates)
|
|
59
|
+
- `openingSchedule.*` (nested schedule object with 7-day hours)
|
|
60
|
+
- Use SDK resolvers (trim, uppercase, parseFloat, parseInt, boolean)
|
|
61
|
+
- Support custom resolvers for complex transformations
|
|
62
|
+
- Validate required fields
|
|
63
|
+
- Note: Check your GraphQL schema to determine if `retailer.id` field exists and is mandatory/optional
|
|
64
|
+
|
|
65
|
+
4. **GraphQL Mutations** (Direct - NO Batch API):
|
|
66
|
+
- Execute `createLocation` mutation directly for each location
|
|
67
|
+
- **NO BPP (Batch Pre-Processing)** - Not applicable for direct GraphQL mutations
|
|
68
|
+
- **NO Batch API** - Use direct GraphQL mutations with rate limiting instead
|
|
69
|
+
- Use rate limiting (configurable mutations per second)
|
|
70
|
+
- Add delay between mutations to avoid API throttling
|
|
71
|
+
- Retry failed mutations with exponential backoff
|
|
72
|
+
- Track successful vs failed mutations per file
|
|
73
|
+
|
|
74
|
+
5. **Job Tracking & State Management**:
|
|
75
|
+
- **Use JobTracker** - Track job lifecycle (start, complete, fail) in KV store
|
|
76
|
+
- Use StateService + VersoriKVAdapter for duplicate file prevention
|
|
77
|
+
- Track processed files in KV store with metadata
|
|
78
|
+
- Store error state with exponential backoff tracking
|
|
79
|
+
- Support distributed state across workflow runs
|
|
80
|
+
- Provide job status endpoint for monitoring
|
|
81
|
+
|
|
82
|
+
6. **Error Handling**:
|
|
83
|
+
- File-level errors archived to `/errors/` subdirectory
|
|
84
|
+
- Record-level errors tracked but don't stop file processing
|
|
85
|
+
- Mapping errors logged with specific location context
|
|
86
|
+
- Mutation errors retried with exponential backoff
|
|
87
|
+
- Error state tracking with next retry timestamp
|
|
88
|
+
|
|
89
|
+
7. **Advanced Features**:
|
|
90
|
+
- Configurable rate limiting (mutations per second)
|
|
91
|
+
- Empty file detection and archival
|
|
92
|
+
- Timestamp-based error tracking
|
|
93
|
+
- Comprehensive monitoring and logging
|
|
94
|
+
- Manual webhook trigger with job tracking
|
|
95
|
+
- Job status query endpoint
|
|
96
|
+
|
|
97
|
+
**Use SDK Components:**
|
|
98
|
+
|
|
99
|
+
- `createClient()` - Universal client factory for Versori
|
|
100
|
+
- `S3DataSource` - S3 operations with retry logic
|
|
101
|
+
- `XMLParserService` - XML parsing with @ prefix attribute support
|
|
102
|
+
- `GraphQLMutationMapper` - Field transformation with schema validation and nested object support
|
|
103
|
+
- `StateService` + `VersoriKVAdapter` - Duplicate prevention with KV storage
|
|
104
|
+
- Native Versori `log` - Structured logging
|
|
105
|
+
|
|
106
|
+
**Configuration Variables** (from Versori activation):
|
|
107
|
+
|
|
108
|
+
```typescript
|
|
109
|
+
{
|
|
110
|
+
s3: {
|
|
111
|
+
bucketName: string;
|
|
112
|
+
region: string;
|
|
113
|
+
accessKeyId: string;
|
|
114
|
+
secretAccessKey: string;
|
|
115
|
+
prefix: string; // e.g., 'locations/'
|
|
116
|
+
archivePrefix: string; // e.g., 'processed/'
|
|
117
|
+
errorPrefix: string; // e.g., 'errors/'
|
|
118
|
+
filePattern: string; // e.g., '.xml'
|
|
119
|
+
maxFilesToProcess: number;
|
|
120
|
+
enableArchival: boolean;
|
|
121
|
+
},
|
|
122
|
+
fluent: {
|
|
123
|
+
retailerId: string; // Optional: Only if mutation schema requires retailerId in input
|
|
124
|
+
mutationRateLimit: number; // mutations per second (e.g., 5)
|
|
125
|
+
}
|
|
126
|
+
}
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
**Architecture Pattern:**
|
|
130
|
+
|
|
131
|
+
```
|
|
132
|
+
S3 Bucket → List Files → Download XML → Parse → Map → GraphQL Mutation → Archive/Move
|
|
133
|
+
↓ ↓ ↓ ↓ ↓ ↓ ↓
|
|
134
|
+
Configure Filter by S3DataSource XML Universal Direct S3 moveFile()
|
|
135
|
+
Connection Pattern + Retry Parser Mapper createLocation to processed/
|
|
136
|
+
(@) (nested) + Rate Limit or errors/
|
|
137
|
+
(deduplication)
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
**Deliverables:**
|
|
141
|
+
|
|
142
|
+
1. Complete Versori workflow with package.json
|
|
143
|
+
2. Main workflow logic with S3 + XML + GraphQL patterns
|
|
144
|
+
3. Helper functions for rate limiting and retry
|
|
145
|
+
4. XML path resolution examples
|
|
146
|
+
5. Sample XML files with nested structures
|
|
147
|
+
6. Schema validation CLI commands
|
|
148
|
+
7. Testing and deployment instructions
|
|
149
|
+
8. Monitoring and troubleshooting guidance"
|
|
150
|
+
|
|
151
|
+
---
|
|
152
|
+
|
|
153
|
+
# STEP 3: Complete Implementation
|
|
154
|
+
|
|
155
|
+
## Versori Scheduled: S3 XML → Location GraphQL
|
|
156
|
+
|
|
157
|
+
**FC Connect SDK Use Case Guide**
|
|
158
|
+
|
|
159
|
+
> **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
|
|
160
|
+
> **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
|
|
161
|
+
|
|
162
|
+
**Context**: Versori scheduled workflow that reads location XML files from S3 and creates/updates Fluent Commerce locations via GraphQL mutations with XML path resolution and rate limiting
|
|
163
|
+
|
|
164
|
+
**Complexity**: Medium
|
|
165
|
+
|
|
166
|
+
**Runtime**: Versori Platform
|
|
167
|
+
|
|
168
|
+
**Estimated Lines**: ~850 lines (with comprehensive documentation)
|
|
169
|
+
|
|
170
|
+
---
|
|
171
|
+
|
|
172
|
+
## What You'll Build
|
|
173
|
+
|
|
174
|
+
- Scheduled Versori workflow (daily location sync)
|
|
175
|
+
- S3 file listing, download, and archival/move
|
|
176
|
+
- XML parsing with @ prefix for attributes (XPath-style)
|
|
177
|
+
- Array normalization (single element → array conversion)
|
|
178
|
+
- GraphQLMutationMapper-based field transformations with nested objects
|
|
179
|
+
- GraphQL mutations for location upserts with alias batching support
|
|
180
|
+
- Retry logic with exponential backoff
|
|
181
|
+
- StateService duplicate prevention (KV-backed)
|
|
182
|
+
- Error state tracking and file error archival
|
|
183
|
+
- Manual webhook trigger and job status endpoint
|
|
184
|
+
|
|
185
|
+
---
|
|
186
|
+
|
|
187
|
+
## When to Use GraphQL Mutations vs Batch API vs Event API
|
|
188
|
+
|
|
189
|
+
### ✅ Use GraphQL Mutations For:
|
|
190
|
+
|
|
191
|
+
| Entity Type | Use Case | Why GraphQL |
|
|
192
|
+
| -------------- | ---------------------------------------- | ------------------------------------- |
|
|
193
|
+
| **Locations** | Store/warehouse master data (low volume) | Direct control, immediate validation |
|
|
194
|
+
| **Controls** | System configuration, settings | Single operations, complex queries |
|
|
195
|
+
| **Prices** | Price updates (moderate volume) | Immediate feedback, custom logic |
|
|
196
|
+
| **Single Ops** | One-off creates/updates | Testing, debugging, direct API access |
|
|
197
|
+
|
|
198
|
+
### ❌ Use Event API Instead For:
|
|
199
|
+
|
|
200
|
+
| Entity Type | Use Case | Why Event API |
|
|
201
|
+
| ------------------- | -------------------------------------- | -------------------------------------------- |
|
|
202
|
+
| **Products** | Product catalog sync, variant updates | Triggers workflows, validates business rules |
|
|
203
|
+
| **Customers** | Customer registration, profile updates | Needs workflow for downstream systems |
|
|
204
|
+
| **Orders** | Order creation, status updates | Event-driven fulfillment workflows |
|
|
205
|
+
| **Custom Entities** | Any entity needing workflow triggers | Full Rubix workflow support |
|
|
206
|
+
|
|
207
|
+
### 🔄 Use Batch API For:
|
|
208
|
+
|
|
209
|
+
| Entity Type | Use Case | Why Batch API |
|
|
210
|
+
| ------------------ | ---------------------------------- | ----------------------------------------------- |
|
|
211
|
+
| **Inventory ONLY** | Bulk inventory updates, daily sync | Optimized for high-volume, BPP change detection |
|
|
212
|
+
|
|
213
|
+
---
|
|
214
|
+
|
|
215
|
+
## XML File Format
|
|
216
|
+
|
|
217
|
+
### Sample: locations.xml
|
|
218
|
+
|
|
219
|
+
```xml
|
|
220
|
+
<?xml version="1.0" encoding="UTF-8"?>
|
|
221
|
+
<locations>
|
|
222
|
+
<location ref="LOC-001" type="WAREHOUSE">
|
|
223
|
+
<name>Downtown Warehouse</name>
|
|
224
|
+
<address country="USA">
|
|
225
|
+
<street1>123 Main St</street1>
|
|
226
|
+
<street2>Building A</street2>
|
|
227
|
+
<city>New York</city>
|
|
228
|
+
<state>NY</state>
|
|
229
|
+
<postalCode>10001</postalCode>
|
|
230
|
+
</address>
|
|
231
|
+
<coordinates lat="40.7128" lon="-74.0060"/>
|
|
232
|
+
<timeZone>America/New_York</timeZone>
|
|
233
|
+
<openingSchedule>
|
|
234
|
+
<allHours>false</allHours>
|
|
235
|
+
<monStart>800</monStart>
|
|
236
|
+
<monEnd>1800</monEnd>
|
|
237
|
+
<tueStart>800</tueStart>
|
|
238
|
+
<tueEnd>1800</tueEnd>
|
|
239
|
+
<wedStart>800</wedStart>
|
|
240
|
+
<wedEnd>1800</wedEnd>
|
|
241
|
+
<thuStart>800</thuStart>
|
|
242
|
+
<thuEnd>1800</thuEnd>
|
|
243
|
+
<friStart>800</friStart>
|
|
244
|
+
<friEnd>1800</friEnd>
|
|
245
|
+
<satStart>0</satStart>
|
|
246
|
+
<satEnd>0</satEnd>
|
|
247
|
+
<sunStart>0</sunStart>
|
|
248
|
+
<sunEnd>0</sunEnd>
|
|
249
|
+
</openingSchedule>
|
|
250
|
+
</location>
|
|
251
|
+
|
|
252
|
+
<location ref="LOC-002" type="DC">
|
|
253
|
+
<name>Regional DC</name>
|
|
254
|
+
<address country="USA">
|
|
255
|
+
<street1>456 Industrial Pkwy</street1>
|
|
256
|
+
<city>Los Angeles</city>
|
|
257
|
+
<state>CA</state>
|
|
258
|
+
<postalCode>90001</postalCode>
|
|
259
|
+
</address>
|
|
260
|
+
<coordinates lat="34.0522" lon="-118.2437"/>
|
|
261
|
+
<timeZone>America/Los_Angeles</timeZone>
|
|
262
|
+
<openingSchedule>
|
|
263
|
+
<allHours>true</allHours>
|
|
264
|
+
<monStart>0</monStart>
|
|
265
|
+
<monEnd>0</monEnd>
|
|
266
|
+
<tueStart>0</tueStart>
|
|
267
|
+
<tueEnd>0</tueEnd>
|
|
268
|
+
<wedStart>0</wedStart>
|
|
269
|
+
<wedEnd>0</wedEnd>
|
|
270
|
+
<thuStart>0</thuStart>
|
|
271
|
+
<thuEnd>0</thuEnd>
|
|
272
|
+
<friStart>0</friStart>
|
|
273
|
+
<friEnd>0</friEnd>
|
|
274
|
+
<satStart>0</satStart>
|
|
275
|
+
<satEnd>0</satEnd>
|
|
276
|
+
<sunStart>0</sunStart>
|
|
277
|
+
<sunEnd>0</sunEnd>
|
|
278
|
+
</openingSchedule>
|
|
279
|
+
</location>
|
|
280
|
+
</locations>
|
|
281
|
+
```
|
|
282
|
+
|
|
283
|
+
**XML Path Syntax (with @ prefix for attributes):**
|
|
284
|
+
|
|
285
|
+
- `location.@ref` → XML attribute `ref` on `<location>` element
|
|
286
|
+
- `location.name` → Text content of `<name>` element
|
|
287
|
+
- `location.address.street1` → Nested element path
|
|
288
|
+
- `location.address.@country` → XML attribute on nested `<address>` element
|
|
289
|
+
- `location.coordinates.@lat` → XML attribute for latitude
|
|
290
|
+
- `location.openingSchedule.monStart` → Deeply nested element
|
|
291
|
+
|
|
292
|
+
**Note**: The SDK's `XMLParserService` automatically handles XML attributes using `@` prefix notation.
|
|
293
|
+
|
|
294
|
+
---
|
|
295
|
+
|
|
296
|
+
## Versori Workflows Structure
|
|
297
|
+
|
|
298
|
+
**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
|
|
299
|
+
|
|
300
|
+
**Trigger Types:**
|
|
301
|
+
- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
|
|
302
|
+
- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
|
|
303
|
+
- **`workflow()`** → Durable workflows (advanced, rarely used)
|
|
304
|
+
|
|
305
|
+
**Execution Steps (chained to triggers):**
|
|
306
|
+
- **`http()`** → External API calls (chained from schedule/webhook)
|
|
307
|
+
- **`fn()`** → Internal processing (chained from schedule/webhook)
|
|
308
|
+
|
|
309
|
+
### Recommended Project Structure
|
|
310
|
+
|
|
311
|
+
```
|
|
312
|
+
s3-xml-location-graphql/
|
|
313
|
+
├── index.ts # Entry point - exports all workflows
|
|
314
|
+
└── src/
|
|
315
|
+
├── workflows/
|
|
316
|
+
│ ├── scheduled/
|
|
317
|
+
│ │ └── daily-location-sync.ts # Scheduled: Daily location sync
|
|
318
|
+
│ │
|
|
319
|
+
│ └── webhook/
|
|
320
|
+
│ ├── adhoc-location-sync.ts # Webhook: Manual trigger
|
|
321
|
+
│ └── job-status-check.ts # Webhook: Status query
|
|
322
|
+
│
|
|
323
|
+
├── services/
|
|
324
|
+
│ └── location-sync.service.ts # Shared orchestration logic (reusable)
|
|
325
|
+
│
|
|
326
|
+
└── config/
|
|
327
|
+
└── location-mapping.json # GraphQL mapping config
|
|
328
|
+
```
|
|
329
|
+
|
|
330
|
+
---
|
|
331
|
+
|
|
332
|
+
## Complete Versori Workflow
|
|
333
|
+
|
|
334
|
+
### Step 1: Package Configuration
|
|
335
|
+
|
|
336
|
+
**File: package.json**
|
|
337
|
+
|
|
338
|
+
```json
|
|
339
|
+
{
|
|
340
|
+
"name": "versori-s3-xml-location-sync",
|
|
341
|
+
"version": "1.0.0",
|
|
342
|
+
"description": "Versori workflow: S3 XML location sync to Fluent GraphQL",
|
|
343
|
+
"versori": {
|
|
344
|
+
"workflows": "./index.ts"
|
|
345
|
+
},
|
|
346
|
+
"type": "module",
|
|
347
|
+
"scripts": {
|
|
348
|
+
"deploy": "versori deploy",
|
|
349
|
+
"logs": "versori logs"
|
|
350
|
+
},
|
|
351
|
+
"dependencies": {
|
|
352
|
+
"@fluentcommerce/fc-connect-sdk": "^0.1.39",
|
|
353
|
+
"@versori/run": "latest"
|
|
354
|
+
},
|
|
355
|
+
"devDependencies": {
|
|
356
|
+
"typescript": "^5.0.0",
|
|
357
|
+
"@types/node": "^20.0.0"
|
|
358
|
+
}
|
|
359
|
+
}
|
|
360
|
+
```
|
|
361
|
+
|
|
362
|
+
### Step 2: Workflow Entry Point (`index.ts`)
|
|
363
|
+
|
|
364
|
+
**Purpose**: Register all workflows with Versori platform
|
|
365
|
+
|
|
366
|
+
```typescript
|
|
367
|
+
/**
|
|
368
|
+
* Entry Point - Registers all workflows with Versori platform
|
|
369
|
+
*
|
|
370
|
+
* Versori automatically discovers and registers exported workflows
|
|
371
|
+
*
|
|
372
|
+
* File Structure:
|
|
373
|
+
* - src/workflows/scheduled/ → Time-based triggers (cron)
|
|
374
|
+
* - src/workflows/webhook/ → HTTP-based triggers (webhooks)
|
|
375
|
+
*/
|
|
376
|
+
|
|
377
|
+
// Scheduled workflows
|
|
378
|
+
export { dailyLocationSync } from './src/workflows/scheduled/daily-location-sync';
|
|
379
|
+
|
|
380
|
+
// Webhook workflows
|
|
381
|
+
export { adhocLocationSync } from './src/workflows/webhook/adhoc-location-sync';
|
|
382
|
+
export { locationSyncJobStatus } from './src/workflows/webhook/job-status-check';
|
|
383
|
+
```
|
|
384
|
+
|
|
385
|
+
**What Gets Exposed:**
|
|
386
|
+
- ✅ `adhocLocationSync` → `https://{workspace}.versori.run/location-sync-adhoc`
|
|
387
|
+
- ✅ `locationSyncJobStatus` → `https://{workspace}.versori.run/location-sync-job-status`
|
|
388
|
+
- ❌ `dailyLocationSync` → NOT exposed (runs automatically on cron)
|
|
389
|
+
|
|
390
|
+
---
|
|
391
|
+
|
|
392
|
+
### Step 3: Workflow Files
|
|
393
|
+
|
|
394
|
+
#### `src/workflows/scheduled/daily-location-sync.ts`
|
|
395
|
+
|
|
396
|
+
**Purpose**: Automatic daily location sync
|
|
397
|
+
**Trigger**: Cron schedule (`0 2 * * *`)
|
|
398
|
+
**Exposed as Endpoint**: ❌ NO - Runs automatically
|
|
399
|
+
|
|
400
|
+
```typescript
|
|
401
|
+
import { schedule, http } from '@versori/run';
|
|
402
|
+
import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
|
|
403
|
+
import { executeLocationSync } from '../../services/location-sync.service';
|
|
404
|
+
|
|
405
|
+
/**
|
|
406
|
+
* Scheduled Workflow: Daily Location Sync
|
|
407
|
+
*
|
|
408
|
+
* Runs automatically daily at 2 AM UTC
|
|
409
|
+
* NOT exposed as HTTP endpoint - Versori executes on schedule
|
|
410
|
+
*
|
|
411
|
+
* Uses shared service: location-sync.service.ts
|
|
412
|
+
*/
|
|
413
|
+
export const dailyLocationSync = schedule(
|
|
414
|
+
'location-sync-scheduled',
|
|
415
|
+
'0 2 * * *' // Daily at 2 AM UTC
|
|
416
|
+
).then(
|
|
417
|
+
http('run-location-sync', { connection: 'fluent_commerce' }, async ctx => {
|
|
418
|
+
const startTime = Date.now();
|
|
419
|
+
const { log, openKv } = ctx;
|
|
420
|
+
const jobId = `location-sync-${Date.now()}`;
|
|
421
|
+
const tracker = new JobTracker(openKv(':project:'), log);
|
|
422
|
+
|
|
423
|
+
log.info('🚀 [START] Daily location sync initiated', { jobId, trigger: 'schedule' });
|
|
424
|
+
|
|
425
|
+
await tracker.createJob(jobId, { triggeredBy: 'schedule', stage: 'initialization' });
|
|
426
|
+
await tracker.updateJob(jobId, { status: 'processing' });
|
|
427
|
+
|
|
428
|
+
try {
|
|
429
|
+
log.info('⚙️ [PROCESSING] Starting location synchronization workflow', { jobId });
|
|
430
|
+
const result = await executeLocationSync(ctx, jobId, tracker);
|
|
431
|
+
await tracker.markCompleted(jobId, result);
|
|
432
|
+
|
|
433
|
+
const duration = Date.now() - startTime;
|
|
434
|
+
log.info('✅ [SUCCESS] Daily location sync completed', {
|
|
435
|
+
jobId,
|
|
436
|
+
duration: `${duration}ms`,
|
|
437
|
+
processed: result.processed,
|
|
438
|
+
totalRecords: result.totalRecords
|
|
439
|
+
});
|
|
440
|
+
|
|
441
|
+
return { success: true, jobId, duration, ...result };
|
|
442
|
+
} catch (e: any) {
|
|
443
|
+
await tracker.markFailed(jobId, e);
|
|
444
|
+
const duration = Date.now() - startTime;
|
|
445
|
+
|
|
446
|
+
log.error('❌ [FAILED] Daily location sync failed', {
|
|
447
|
+
jobId,
|
|
448
|
+
duration: `${duration}ms`,
|
|
449
|
+
error: e?.message,
|
|
450
|
+
errorType: e?.name
|
|
451
|
+
});
|
|
452
|
+
|
|
453
|
+
return { success: false, jobId, duration, error: e?.message };
|
|
454
|
+
}
|
|
455
|
+
})
|
|
456
|
+
);
|
|
457
|
+
```
|
|
458
|
+
|
|
459
|
+
---
|
|
460
|
+
|
|
461
|
+
#### `src/workflows/webhook/adhoc-location-sync.ts`
|
|
462
|
+
|
|
463
|
+
**Purpose**: Manual location sync trigger (on-demand)
|
|
464
|
+
**Trigger**: HTTP POST
|
|
465
|
+
**Endpoint**: `POST https://{workspace}.versori.run/location-sync-adhoc`
|
|
466
|
+
**Use Cases**: Testing, priority processing, ad-hoc runs
|
|
467
|
+
|
|
468
|
+
```typescript
|
|
469
|
+
import { webhook, http } from '@versori/run';
|
|
470
|
+
import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
|
|
471
|
+
import { executeLocationSync } from '../../services/location-sync.service';
|
|
472
|
+
|
|
473
|
+
/**
|
|
474
|
+
* Webhook: Manual Location Sync Trigger
|
|
475
|
+
*
|
|
476
|
+
* Endpoint: POST https://{workspace}.versori.run/location-sync-adhoc
|
|
477
|
+
* Request body (optional): { filePattern: "urgent_*.xml", maxFiles: 5 }
|
|
478
|
+
*
|
|
479
|
+
* Pattern: webhook().then(http()) - needs Fluent API access
|
|
480
|
+
* Uses shared service: location-sync.service.ts
|
|
481
|
+
*
|
|
482
|
+
* SECURITY: Authentication handled via connection parameter
|
|
483
|
+
* No manual API key validation needed - Versori manages this via connection auth
|
|
484
|
+
*/
|
|
485
|
+
export const adhocLocationSync = webhook('location-sync-adhoc', {
|
|
486
|
+
response: { mode: 'sync' },
|
|
487
|
+
connection: 'location-sync-adhoc', // Versori validates API key
|
|
488
|
+
}).then(
|
|
489
|
+
http('run-location-sync-adhoc', { connection: 'fluent_commerce' }, async ctx => {
|
|
490
|
+
const startTime = Date.now();
|
|
491
|
+
const { log, openKv, data } = ctx;
|
|
492
|
+
const jobId = `location-sync-adhoc-${Date.now()}`;
|
|
493
|
+
const tracker = new JobTracker(openKv(':project:'), log);
|
|
494
|
+
|
|
495
|
+
log.info('🚀 [START] Manual location sync triggered', { jobId, trigger: 'webhook', options: data });
|
|
496
|
+
|
|
497
|
+
await tracker.createJob(jobId, {
|
|
498
|
+
triggeredBy: 'manual',
|
|
499
|
+
stage: 'initialization',
|
|
500
|
+
options: data // Optional: filePattern, maxFiles, etc.
|
|
501
|
+
});
|
|
502
|
+
await tracker.updateJob(jobId, { status: 'processing' });
|
|
503
|
+
|
|
504
|
+
try {
|
|
505
|
+
log.info('⚙️ [PROCESSING] Starting manual location synchronization', { jobId });
|
|
506
|
+
const result = await executeLocationSync(ctx, jobId, tracker);
|
|
507
|
+
await tracker.markCompleted(jobId, result);
|
|
508
|
+
|
|
509
|
+
const duration = Date.now() - startTime;
|
|
510
|
+
log.info('✅ [SUCCESS] Manual location sync completed', {
|
|
511
|
+
jobId,
|
|
512
|
+
duration: `${duration}ms`,
|
|
513
|
+
processed: result.processed,
|
|
514
|
+
totalRecords: result.totalRecords
|
|
515
|
+
});
|
|
516
|
+
|
|
517
|
+
return { success: true, jobId, duration, ...result };
|
|
518
|
+
} catch (e: any) {
|
|
519
|
+
await tracker.markFailed(jobId, e);
|
|
520
|
+
const duration = Date.now() - startTime;
|
|
521
|
+
|
|
522
|
+
log.error('❌ [FAILED] Manual location sync failed', {
|
|
523
|
+
jobId,
|
|
524
|
+
duration: `${duration}ms`,
|
|
525
|
+
error: e?.message,
|
|
526
|
+
errorType: e?.name
|
|
527
|
+
});
|
|
528
|
+
|
|
529
|
+
return { success: false, jobId, duration, error: e?.message };
|
|
530
|
+
}
|
|
531
|
+
})
|
|
532
|
+
);
|
|
533
|
+
```
|
|
534
|
+
|
|
535
|
+
---
|
|
536
|
+
|
|
537
|
+
#### `src/workflows/webhook/job-status-check.ts`
|
|
538
|
+
|
|
539
|
+
**Purpose**: Query job status
|
|
540
|
+
**Trigger**: HTTP POST
|
|
541
|
+
**Endpoint**: `POST https://{workspace}.versori.run/location-sync-job-status`
|
|
542
|
+
**Request body**: `{ "jobId": "location-sync-1234567890" }`
|
|
543
|
+
|
|
544
|
+
```typescript
|
|
545
|
+
import { webhook, fn } from '@versori/run';
|
|
546
|
+
import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
|
|
547
|
+
|
|
548
|
+
/**
|
|
549
|
+
* Webhook: Job Status Check
|
|
550
|
+
*
|
|
551
|
+
* Endpoint: POST https://{workspace}.versori.run/location-sync-job-status
|
|
552
|
+
* Request body: { "jobId": "location-sync-1234567890" }
|
|
553
|
+
*
|
|
554
|
+
* Pattern: webhook().then(fn()) - no external API needed, only KV storage
|
|
555
|
+
* Lightweight: Only queries KV store, no Fluent API calls
|
|
556
|
+
*
|
|
557
|
+
* SECURITY: Authentication handled via connection parameter
|
|
558
|
+
* No manual API key validation needed - Versori manages this via connection auth
|
|
559
|
+
*/
|
|
560
|
+
export const locationSyncJobStatus = webhook('location-sync-job-status', {
|
|
561
|
+
response: { mode: 'sync' },
|
|
562
|
+
connection: 'location-sync-job-status',
|
|
563
|
+
}).then(
|
|
564
|
+
fn('status', async ctx => {
|
|
565
|
+
const { data, log, openKv } = ctx;
|
|
566
|
+
const jobId = data?.jobId as string;
|
|
567
|
+
|
|
568
|
+
if (!jobId) {
|
|
569
|
+
return { success: false, error: 'jobId required' };
|
|
570
|
+
}
|
|
571
|
+
|
|
572
|
+
const tracker = new JobTracker(openKv(':project:'), log);
|
|
573
|
+
const status = await tracker.getJob(jobId);
|
|
574
|
+
|
|
575
|
+
return status
|
|
576
|
+
? { success: true, jobId, ...status }
|
|
577
|
+
: { success: false, error: 'Job not found', jobId };
|
|
578
|
+
})
|
|
579
|
+
);
|
|
580
|
+
```
|
|
581
|
+
|
|
582
|
+
---
|
|
583
|
+
|
|
584
|
+
### Step 4: Main Orchestration Service (`src/services/location-sync.service.ts`)
|
|
585
|
+
|
|
586
|
+
**Note:** This service file should contain the `executeLocationSync` function (renamed from `runLocationXmlWorkflow`). The main workflow logic should be moved here.
|
|
587
|
+
|
|
588
|
+
```typescript
|
|
589
|
+
/**
|
|
590
|
+
* Main Orchestration Service: Location Sync
|
|
591
|
+
*
|
|
592
|
+
* This service contains the core business logic for location synchronization.
|
|
593
|
+
*
|
|
594
|
+
* Features:
|
|
595
|
+
* - S3 file operations with archival-based deduplication (moveFile to processed/)
|
|
596
|
+
* - XML parsing with @ prefix for attributes
|
|
597
|
+
* - Single element → array normalization
|
|
598
|
+
* - GraphQLMutationMapper for nested field transformations
|
|
599
|
+
* - GraphQL mutations with alias batching support
|
|
600
|
+
* - StateService for metadata tracking (secondary to archival)
|
|
601
|
+
* - Error state tracking with exponential backoff
|
|
602
|
+
*
|
|
603
|
+
* Deduplication Strategy:
|
|
604
|
+
* - PRIMARY: S3 archival via moveFile() - Files in processed/ won't be re-listed
|
|
605
|
+
* - SECONDARY: StateService KV tracking - Provides metadata and processing history
|
|
606
|
+
*/
|
|
607
|
+
import { Buffer } from 'node:buffer'; // Required for Deno/Versori runtime
|
|
608
|
+
import {
|
|
609
|
+
createClient,
|
|
610
|
+
S3DataSource,
|
|
611
|
+
XMLParserService,
|
|
612
|
+
GraphQLMutationMapper,
|
|
613
|
+
StateService,
|
|
614
|
+
VersoriKVAdapter,
|
|
615
|
+
JobTracker,
|
|
616
|
+
FluentClient,
|
|
617
|
+
} from '@fluentcommerce/fc-connect-sdk';
|
|
618
|
+
|
|
619
|
+
// ============================================================================
|
|
620
|
+
// Type Definitions
|
|
621
|
+
// ============================================================================
|
|
622
|
+
|
|
623
|
+
interface FileProcessingResult {
|
|
624
|
+
success: boolean;
|
|
625
|
+
locations: any[];
|
|
626
|
+
errors: string[];
|
|
627
|
+
}
|
|
628
|
+
|
|
629
|
+
interface MutationResult {
|
|
630
|
+
successful: number;
|
|
631
|
+
failed: number;
|
|
632
|
+
errors: string[];
|
|
633
|
+
}
|
|
634
|
+
|
|
635
|
+
interface MutationLogEntry {
|
|
636
|
+
timestamp: string;
|
|
637
|
+
fileName: string;
|
|
638
|
+
locationRef: string;
|
|
639
|
+
status: 'success' | 'failure';
|
|
640
|
+
error?: string;
|
|
641
|
+
}
|
|
642
|
+
|
|
643
|
+
// ============================================================================
|
|
644
|
+
// Utility Functions
|
|
645
|
+
// ============================================================================
|
|
646
|
+
|
|
647
|
+
/**
|
|
648
|
+
* Retry utility with exponential backoff
|
|
649
|
+
* (User-defined - not part of SDK public API)
|
|
650
|
+
*/
|
|
651
|
+
async function retryWithBackoff<T>(
|
|
652
|
+
operation: () => Promise<T>,
|
|
653
|
+
maxRetries = 3,
|
|
654
|
+
baseDelayMs = 1000
|
|
655
|
+
): Promise<T> {
|
|
656
|
+
let lastError: any;
|
|
657
|
+
for (let attempt = 0; attempt < maxRetries; attempt++) {
|
|
658
|
+
try {
|
|
659
|
+
return await operation();
|
|
660
|
+
} catch (error) {
|
|
661
|
+
lastError = error;
|
|
662
|
+
if (attempt < maxRetries - 1) {
|
|
663
|
+
const delay = baseDelayMs * Math.pow(2, attempt) + Math.floor(Math.random() * 200);
|
|
664
|
+
await new Promise(resolve => setTimeout(resolve, delay));
|
|
665
|
+
}
|
|
666
|
+
}
|
|
667
|
+
}
|
|
668
|
+
throw lastError;
|
|
669
|
+
}
|
|
670
|
+
|
|
671
|
+
/**
|
|
672
|
+
* Rate limiter for GraphQL mutations
|
|
673
|
+
*/
|
|
674
|
+
async function rateLimitedMutation(operation: () => Promise<any>, delayMs: number): Promise<any> {
|
|
675
|
+
const result = await operation();
|
|
676
|
+
await new Promise(resolve => setTimeout(resolve, delayMs));
|
|
677
|
+
return result;
|
|
678
|
+
}
|
|
679
|
+
|
|
680
|
+
// ============================================================================
|
|
681
|
+
// Service Functions
|
|
682
|
+
// ============================================================================
|
|
683
|
+
|
|
684
|
+
/**
|
|
685
|
+
* Service Function 1: Process File
|
|
686
|
+
*
|
|
687
|
+
* Downloads XML from S3, parses with XMLParserService, normalizes arrays,
|
|
688
|
+
* and maps fields with GraphQLMutationMapper.
|
|
689
|
+
*
|
|
690
|
+
* @param s3 - S3DataSource instance
|
|
691
|
+
* @param parser - XMLParserService instance
|
|
692
|
+
* @param mapper - GraphQLMutationMapper instance
|
|
693
|
+
* @param filePath - Full S3 path to file
|
|
694
|
+
* @param fileName - File name only (for logging)
|
|
695
|
+
* @param log - Logger instance
|
|
696
|
+
* @returns FileProcessingResult with locations array and errors
|
|
697
|
+
*/
|
|
698
|
+
async function processFile(
|
|
699
|
+
s3: S3DataSource,
|
|
700
|
+
parser: XMLParserService,
|
|
701
|
+
mapper: GraphQLMutationMapper,
|
|
702
|
+
filePath: string,
|
|
703
|
+
fileName: string,
|
|
704
|
+
log: any
|
|
705
|
+
): Promise<FileProcessingResult> {
|
|
706
|
+
try {
|
|
707
|
+
log.info('Processing file', { fileName });
|
|
708
|
+
|
|
709
|
+
// Download with retry
|
|
710
|
+
const content = await retryWithBackoff(
|
|
711
|
+
() => s3.downloadFile(filePath, { encoding: 'utf8' }) as Promise<string>
|
|
712
|
+
);
|
|
713
|
+
|
|
714
|
+
// Parse XML
|
|
715
|
+
const xmlData = await parser.parse(content);
|
|
716
|
+
|
|
717
|
+
// Extract location array (handle both single and multiple locations)
|
|
718
|
+
// CRITICAL: XML array normalization - single <location> becomes object, not array
|
|
719
|
+
const locationsData = xmlData.locations?.location;
|
|
720
|
+
const locationsArray = Array.isArray(locationsData) ? locationsData : [locationsData];
|
|
721
|
+
|
|
722
|
+
if (!locationsArray || locationsArray.length === 0) {
|
|
723
|
+
log.warn('Empty file (no locations)', { fileName });
|
|
724
|
+
return {
|
|
725
|
+
success: true,
|
|
726
|
+
locations: [],
|
|
727
|
+
errors: [],
|
|
728
|
+
};
|
|
729
|
+
}
|
|
730
|
+
|
|
731
|
+
// Map each location using GraphQLMutationMapper
|
|
732
|
+
const mappedLocations: Array<{ query: string; variables: any; input: any }> = [];
|
|
733
|
+
const mappingErrors: string[] = [];
|
|
734
|
+
|
|
735
|
+
// ✅ PRODUCTION ENHANCEMENT: Log transformation start
|
|
736
|
+
log.info('Transforming locations to GraphQL mutations', {
|
|
737
|
+
fileName,
|
|
738
|
+
totalLocations: locationsArray.length,
|
|
739
|
+
});
|
|
740
|
+
|
|
741
|
+
for (let i = 0; i < locationsArray.length; i++) {
|
|
742
|
+
const locationNumber = i + 1;
|
|
743
|
+
|
|
744
|
+
// ✅ PRODUCTION ENHANCEMENT: Log progress every 50 locations
|
|
745
|
+
if (locationNumber % 50 === 0) {
|
|
746
|
+
log.info(`📤 Transforming location ${locationNumber}/${locationsArray.length}`, {
|
|
747
|
+
fileName,
|
|
748
|
+
locationNumber,
|
|
749
|
+
totalLocations: locationsArray.length,
|
|
750
|
+
validSoFar: mappedLocations.length,
|
|
751
|
+
errorsSoFar: mappingErrors.length,
|
|
752
|
+
progress: `${((locationNumber / locationsArray.length) * 100).toFixed(1)}%`,
|
|
753
|
+
});
|
|
754
|
+
}
|
|
755
|
+
|
|
756
|
+
// Wrap location in context for mapping
|
|
757
|
+
const record = { location: locationsArray[i] };
|
|
758
|
+
try {
|
|
759
|
+
// GraphQLMutationMapper returns { query, variables } directly
|
|
760
|
+
const mappingResult = await mapper.map(record);
|
|
761
|
+
|
|
762
|
+
mappedLocations.push({
|
|
763
|
+
query: mappingResult.query,
|
|
764
|
+
variables: mappingResult.variables,
|
|
765
|
+
input: mappingResult.variables.input || mappingResult.variables,
|
|
766
|
+
});
|
|
767
|
+
} catch (error: unknown) {
|
|
768
|
+
const errorMsg = error instanceof Error ? error.message : String(error);
|
|
769
|
+
mappingErrors.push(`Location ${locationNumber}: ${errorMsg}`);
|
|
770
|
+
log.warn('Mapping failed for location', { fileName, index: i, error: errorMsg });
|
|
771
|
+
}
|
|
772
|
+
}
|
|
773
|
+
|
|
774
|
+
log.info('File processed', {
|
|
775
|
+
fileName,
|
|
776
|
+
total: locationsArray.length,
|
|
777
|
+
mapped: mappedLocations.length,
|
|
778
|
+
errors: mappingErrors.length,
|
|
779
|
+
successRate: `${((mappedLocations.length / locationsArray.length) * 100).toFixed(1)}%`,
|
|
780
|
+
});
|
|
781
|
+
|
|
782
|
+
return {
|
|
783
|
+
success: true,
|
|
784
|
+
locations: mappedLocations,
|
|
785
|
+
errors: mappingErrors,
|
|
786
|
+
};
|
|
787
|
+
} catch (error: any) {
|
|
788
|
+
// ✅ Enhanced error logging: Extract all error details for visibility
|
|
789
|
+
const errorDetails = {
|
|
790
|
+
message: error?.message || 'Unknown error',
|
|
791
|
+
stack: error?.stack,
|
|
792
|
+
fileName: error?.fileName,
|
|
793
|
+
lineNumber: error?.lineNumber,
|
|
794
|
+
originalError: error?.context?.originalError?.message,
|
|
795
|
+
errorType: error?.name || 'Error',
|
|
796
|
+
};
|
|
797
|
+
log.error('File processing failed', errorDetails, { fileName });
|
|
798
|
+
return {
|
|
799
|
+
success: false,
|
|
800
|
+
locations: [],
|
|
801
|
+
errors: [error.message || 'Unknown error'],
|
|
802
|
+
};
|
|
803
|
+
}
|
|
804
|
+
}
|
|
805
|
+
|
|
806
|
+
/**
|
|
807
|
+
* Service Function 2: Execute Mutations
|
|
808
|
+
*
|
|
809
|
+
* Executes GraphQL createLocation mutations with alias batching support.
|
|
810
|
+
*
|
|
811
|
+
* @param client - FluentClient instance
|
|
812
|
+
* @param mapper - GraphQLMutationMapper instance
|
|
813
|
+
* @param locations - Array of mapped location objects with query and variables
|
|
814
|
+
* @param log - Logger instance
|
|
815
|
+
* @param retailerId - Fluent retailer ID
|
|
816
|
+
* @param batchSize - Number of concurrent requests (default: 1)
|
|
817
|
+
* @param mutationsPerAliasBatch - Optional: Number of mutations per aliased request (default: undefined)
|
|
818
|
+
* @returns MutationResult with success/failure counts
|
|
819
|
+
*/
|
|
820
|
+
async function executeMutations(
|
|
821
|
+
client: FluentClient,
|
|
822
|
+
mapper: GraphQLMutationMapper,
|
|
823
|
+
locations: Array<{ query: string; variables: any; input: any }>,
|
|
824
|
+
log: any,
|
|
825
|
+
retailerId: string,
|
|
826
|
+
batchSize: number = 1, // ✅ Default: 1 (sequential)
|
|
827
|
+
mutationsPerAliasBatch?: number // ✅ NEW: Alias batching parameter (default: undefined = disabled)
|
|
828
|
+
): Promise<MutationResult> {
|
|
829
|
+
// Determine mode: use aliases if mutationsPerAliasBatch is set and > 1
|
|
830
|
+
const useAliases = mutationsPerAliasBatch && mutationsPerAliasBatch > 1;
|
|
831
|
+
|
|
832
|
+
if (useAliases) {
|
|
833
|
+
return await executeMutationsWithAliases(
|
|
834
|
+
client,
|
|
835
|
+
mapper,
|
|
836
|
+
locations,
|
|
837
|
+
log,
|
|
838
|
+
retailerId,
|
|
839
|
+
batchSize,
|
|
840
|
+
mutationsPerAliasBatch!
|
|
841
|
+
);
|
|
842
|
+
} else {
|
|
843
|
+
return await executeMutationsSeparate(
|
|
844
|
+
client,
|
|
845
|
+
locations,
|
|
846
|
+
log,
|
|
847
|
+
batchSize
|
|
848
|
+
);
|
|
849
|
+
}
|
|
850
|
+
}
|
|
851
|
+
|
|
852
|
+
/**
|
|
853
|
+
* Execute mutations using separate concurrent requests (current mode)
|
|
854
|
+
*/
|
|
855
|
+
async function executeMutationsSeparate(
|
|
856
|
+
client: FluentClient,
|
|
857
|
+
locations: Array<{ query: string; variables: any; input: any }>,
|
|
858
|
+
log: any,
|
|
859
|
+
batchSize: number
|
|
860
|
+
): Promise<MutationResult> {
|
|
861
|
+
const result: MutationResult = {
|
|
862
|
+
successful: 0,
|
|
863
|
+
failed: 0,
|
|
864
|
+
errors: [],
|
|
865
|
+
};
|
|
866
|
+
|
|
867
|
+
const safeConc = Math.max(1, Math.floor(batchSize));
|
|
868
|
+
|
|
869
|
+
// Sequential mode
|
|
870
|
+
if (safeConc === 1) {
|
|
871
|
+
for (const location of locations) {
|
|
872
|
+
try {
|
|
873
|
+
await retryWithBackoff(() =>
|
|
874
|
+
client.graphql({
|
|
875
|
+
query: location.query,
|
|
876
|
+
variables: location.variables,
|
|
877
|
+
})
|
|
878
|
+
);
|
|
879
|
+
result.successful++;
|
|
880
|
+
} catch (error: unknown) {
|
|
881
|
+
result.failed++;
|
|
882
|
+
const errorMsg = error instanceof Error ? error.message : String(error);
|
|
883
|
+
result.errors.push(errorMsg);
|
|
884
|
+
}
|
|
885
|
+
}
|
|
886
|
+
return result;
|
|
887
|
+
}
|
|
888
|
+
|
|
889
|
+
// Parallel mode
|
|
890
|
+
for (let i = 0; i < locations.length; i += safeConc) {
|
|
891
|
+
const chunk = locations.slice(i, i + safeConc);
|
|
892
|
+
const results = await Promise.allSettled(
|
|
893
|
+
chunk.map(loc =>
|
|
894
|
+
retryWithBackoff(() =>
|
|
895
|
+
client.graphql({
|
|
896
|
+
query: loc.query,
|
|
897
|
+
variables: loc.variables,
|
|
898
|
+
})
|
|
899
|
+
)
|
|
900
|
+
)
|
|
901
|
+
);
|
|
902
|
+
|
|
903
|
+
results.forEach((settledResult, idx) => {
|
|
904
|
+
if (settledResult.status === 'fulfilled') {
|
|
905
|
+
result.successful++;
|
|
906
|
+
} else {
|
|
907
|
+
result.failed++;
|
|
908
|
+
result.errors.push(
|
|
909
|
+
settledResult.reason instanceof Error ? settledResult.reason.message : String(settledResult.reason)
|
|
910
|
+
);
|
|
911
|
+
}
|
|
912
|
+
});
|
|
913
|
+
}
|
|
914
|
+
|
|
915
|
+
return result;
|
|
916
|
+
}
|
|
917
|
+
|
|
918
|
+
/**
|
|
919
|
+
* ✅ NEW: Execute mutations using GraphQL aliases (batched requests)
|
|
920
|
+
*/
|
|
921
|
+
async function executeMutationsWithAliases(
|
|
922
|
+
client: FluentClient,
|
|
923
|
+
mapper: GraphQLMutationMapper,
|
|
924
|
+
locations: Array<{ query: string; variables: any; input: any }>,
|
|
925
|
+
log: any,
|
|
926
|
+
retailerId: string,
|
|
927
|
+
maxParallel: number,
|
|
928
|
+
mutationsPerAliasBatch: number
|
|
929
|
+
): Promise<MutationResult> {
|
|
930
|
+
const result: MutationResult = { successful: 0, failed: 0, errors: [] };
|
|
931
|
+
|
|
932
|
+
const mutationName = (mapper as any).config.mutation || 'createLocation';
|
|
933
|
+
const aliasBatches: Array<Array<typeof locations[0]>> = [];
|
|
934
|
+
|
|
935
|
+
for (let i = 0; i < locations.length; i += mutationsPerAliasBatch) {
|
|
936
|
+
aliasBatches.push(locations.slice(i, i + mutationsPerAliasBatch));
|
|
937
|
+
}
|
|
938
|
+
|
|
939
|
+
// Process batches with concurrency control
|
|
940
|
+
for (let i = 0; i < aliasBatches.length; i += maxParallel) {
|
|
941
|
+
const concurrentBatches = aliasBatches.slice(i, i + maxParallel);
|
|
942
|
+
|
|
943
|
+
const batchResults = await Promise.allSettled(
|
|
944
|
+
concurrentBatches.map(async (batch) => {
|
|
945
|
+
const { query, variables } = buildAliasedBatch(batch, mutationName, retailerId);
|
|
946
|
+
const response = await retryWithBackoff(() => client.graphql({ query, variables }));
|
|
947
|
+
return parseAliasResponse(response, batch, mutationName);
|
|
948
|
+
})
|
|
949
|
+
);
|
|
950
|
+
|
|
951
|
+
batchResults.forEach((batchResult, idx) => {
|
|
952
|
+
if (batchResult.status === 'fulfilled') {
|
|
953
|
+
const batchRes = batchResult.value;
|
|
954
|
+
result.successful += batchRes.executed;
|
|
955
|
+
result.failed += batchRes.failed;
|
|
956
|
+
result.errors.push(...batchRes.errors);
|
|
957
|
+
} else {
|
|
958
|
+
const batch = concurrentBatches[idx];
|
|
959
|
+
const errorMsg = batchResult.reason instanceof Error ? batchResult.reason.message : String(batchResult.reason);
|
|
960
|
+
batch.forEach(loc => {
|
|
961
|
+
result.failed++;
|
|
962
|
+
result.errors.push(`Batch execution failed: ${errorMsg}`);
|
|
963
|
+
});
|
|
964
|
+
}
|
|
965
|
+
});
|
|
966
|
+
|
|
967
|
+
if (i + maxParallel < aliasBatches.length) {
|
|
968
|
+
await new Promise(resolve => setTimeout(resolve, 500));
|
|
969
|
+
}
|
|
970
|
+
}
|
|
971
|
+
|
|
972
|
+
return result;
|
|
973
|
+
}
|
|
974
|
+
|
|
975
|
+
/**
|
|
976
|
+
* ✅ NEW: Build aliased batch query and variables
|
|
977
|
+
*/
|
|
978
|
+
function buildAliasedBatch(
|
|
979
|
+
batch: Array<{ query: string; variables: any; input: any }>,
|
|
980
|
+
mutationName: string,
|
|
981
|
+
retailerId: string
|
|
982
|
+
): { query: string; variables: Record<string, any> } {
|
|
983
|
+
const batchSize = batch.length;
|
|
984
|
+
const inputTypeName = `${mutationName.charAt(0).toUpperCase() + mutationName.slice(1)}Input`;
|
|
985
|
+
|
|
986
|
+
const variables = Array.from({ length: batchSize }, (_, i) =>
|
|
987
|
+
`$input${i + 1}: ${inputTypeName}!`
|
|
988
|
+
).join(', ');
|
|
989
|
+
|
|
990
|
+
const aliasedMutations = Array.from({ length: batchSize }, (_, i) => {
|
|
991
|
+
const alias = `${mutationName}${i + 1}`;
|
|
992
|
+
return ` ${alias}: ${mutationName}(input: $input${i + 1}) { id ref name }`;
|
|
993
|
+
}).join('\n');
|
|
994
|
+
|
|
995
|
+
const operationName = `Batch${mutationName.charAt(0).toUpperCase() + mutationName.slice(1)}s`;
|
|
996
|
+
const query = `mutation ${operationName}(${variables}) {\n${aliasedMutations}\n}`;
|
|
997
|
+
|
|
998
|
+
const variablesObj: Record<string, any> = {};
|
|
999
|
+
batch.forEach((loc, index) => {
|
|
1000
|
+
const input = loc.variables.input || loc.variables;
|
|
1001
|
+
if (input && !input.retailer) {
|
|
1002
|
+
input.retailer = { id: parseInt(retailerId) };
|
|
1003
|
+
}
|
|
1004
|
+
variablesObj[`input${index + 1}`] = input;
|
|
1005
|
+
});
|
|
1006
|
+
|
|
1007
|
+
return { query, variables: variablesObj };
|
|
1008
|
+
}
|
|
1009
|
+
|
|
1010
|
+
/**
|
|
1011
|
+
* ✅ NEW: Parse aliased GraphQL response
|
|
1012
|
+
*/
|
|
1013
|
+
function parseAliasResponse(
|
|
1014
|
+
response: any,
|
|
1015
|
+
batch: Array<{ query: string; variables: any; input: any }>,
|
|
1016
|
+
mutationName: string
|
|
1017
|
+
): { executed: number; failed: number; errors: string[] } {
|
|
1018
|
+
const result = { executed: 0, failed: 0, errors: [] as string[] };
|
|
1019
|
+
|
|
1020
|
+
const data = response.data || {};
|
|
1021
|
+
const errors = response.errors || [];
|
|
1022
|
+
|
|
1023
|
+
batch.forEach((loc, index) => {
|
|
1024
|
+
const alias = `${mutationName}${index + 1}`;
|
|
1025
|
+
const aliasData = data[alias];
|
|
1026
|
+
const aliasErrors = errors.filter((e: any) =>
|
|
1027
|
+
e.path && Array.isArray(e.path) && e.path.includes(alias)
|
|
1028
|
+
);
|
|
1029
|
+
|
|
1030
|
+
if (aliasData && !aliasErrors.length) {
|
|
1031
|
+
result.executed++;
|
|
1032
|
+
} else {
|
|
1033
|
+
result.failed++;
|
|
1034
|
+
const errorMsg = aliasErrors[0]?.message || 'Mutation failed';
|
|
1035
|
+
result.errors.push(`${loc.input?.ref || 'unknown'}: ${errorMsg}`);
|
|
1036
|
+
}
|
|
1037
|
+
});
|
|
1038
|
+
|
|
1039
|
+
return result;
|
|
1040
|
+
}
|
|
1041
|
+
|
|
1042
|
+
/**
|
|
1043
|
+
* Service Function 3: Write Mutation Log
|
|
1044
|
+
*
|
|
1045
|
+
* Writes mutation results to S3 as a JSON log file.
|
|
1046
|
+
*
|
|
1047
|
+
* @param s3 - S3DataSource instance
|
|
1048
|
+
* @param logEntries - Array of mutation log entries
|
|
1049
|
+
* @param fileName - Original file name (used to generate log path)
|
|
1050
|
+
* @param logPrefix - S3 prefix for logs (e.g., 'logs/')
|
|
1051
|
+
* @param log - Logger instance
|
|
1052
|
+
*/
|
|
1053
|
+
async function writeMutationLog(
|
|
1054
|
+
s3: S3DataSource,
|
|
1055
|
+
logEntries: MutationLogEntry[],
|
|
1056
|
+
fileName: string,
|
|
1057
|
+
logPrefix: string,
|
|
1058
|
+
log: any
|
|
1059
|
+
): Promise<void> {
|
|
1060
|
+
try {
|
|
1061
|
+
const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
|
|
1062
|
+
const logFileName = `${logPrefix}${fileName.replace('.xml', '')}_${timestamp}.json`;
|
|
1063
|
+
|
|
1064
|
+
const logContent = JSON.stringify(
|
|
1065
|
+
{
|
|
1066
|
+
fileName,
|
|
1067
|
+
timestamp: new Date().toISOString(),
|
|
1068
|
+
totalMutations: logEntries.length,
|
|
1069
|
+
successful: logEntries.filter(e => e.status === 'success').length,
|
|
1070
|
+
failed: logEntries.filter(e => e.status === 'failure').length,
|
|
1071
|
+
entries: logEntries,
|
|
1072
|
+
},
|
|
1073
|
+
null,
|
|
1074
|
+
2
|
|
1075
|
+
);
|
|
1076
|
+
|
|
1077
|
+
// Write to S3 (uploadFile accepts string or Buffer)
|
|
1078
|
+
await s3.uploadFile(logFileName, logContent);
|
|
1079
|
+
|
|
1080
|
+
log.info('Mutation log written', { logFileName, entries: logEntries.length });
|
|
1081
|
+
} catch (error: any) {
|
|
1082
|
+
// ✅ Enhanced error logging: Extract all error details for visibility
|
|
1083
|
+
const errorDetails = {
|
|
1084
|
+
message: error?.message || 'Unknown error',
|
|
1085
|
+
stack: error?.stack,
|
|
1086
|
+
errorType: error?.name || 'Error',
|
|
1087
|
+
};
|
|
1088
|
+
log.error('Failed to write mutation log', errorDetails, { fileName });
|
|
1089
|
+
// Don't throw - logging failure shouldn't stop workflow
|
|
1090
|
+
}
|
|
1091
|
+
}
|
|
1092
|
+
|
|
1093
|
+
// ============================================================================
|
|
1094
|
+
// Main Workflow Function
|
|
1095
|
+
// ============================================================================
|
|
1096
|
+
|
|
1097
|
+
/**
|
|
1098
|
+
* Main Orchestration Function: Execute Location Sync
|
|
1099
|
+
*
|
|
1100
|
+
* This function orchestrates the location synchronization workflow.
|
|
1101
|
+
*
|
|
1102
|
+
* Architecture:
|
|
1103
|
+
* 1. List files from S3
|
|
1104
|
+
* 2. For each file:
|
|
1105
|
+
* a. processFile() - Download, parse, map
|
|
1106
|
+
* b. executeMutations() - Send GraphQL mutations with rate limiting
|
|
1107
|
+
* c. writeMutationLog() - Log results to S3
|
|
1108
|
+
* d. Archive file (primary deduplication)
|
|
1109
|
+
* e. Mark processed in KV (metadata tracking)
|
|
1110
|
+
*
|
|
1111
|
+
* @param ctx - Versori context
|
|
1112
|
+
* @param jobId - Job identifier
|
|
1113
|
+
* @param tracker - JobTracker instance
|
|
1114
|
+
* @returns Processing result
|
|
1115
|
+
*/
|
|
1116
|
+
export async function executeLocationSync(ctx: any, jobId: string, tracker: JobTracker) {
|
|
1117
|
+
const { log, activation } = ctx;
|
|
1118
|
+
|
|
1119
|
+
log.info('📋 [INIT] Reading activation variables', { jobId });
|
|
1120
|
+
|
|
1121
|
+
// Read activation variables
|
|
1122
|
+
const s3Bucket = activation?.getVariable('s3BucketName');
|
|
1123
|
+
const s3Region = activation?.getVariable('awsRegion') || 'us-east-1';
|
|
1124
|
+
const s3AccessKeyId = activation?.getVariable('awsAccessKeyId');
|
|
1125
|
+
const s3SecretAccessKey = activation?.getVariable('awsSecretAccessKey');
|
|
1126
|
+
const s3Prefix = activation?.getVariable('s3Prefix') || 'locations/';
|
|
1127
|
+
const archivePrefix = activation?.getVariable('archivePrefix') || 'processed/';
|
|
1128
|
+
const errorPrefix = activation?.getVariable('errorPrefix') || 'errors/';
|
|
1129
|
+
const logPrefix = activation?.getVariable('logPrefix') || 'logs/';
|
|
1130
|
+
const filePattern = (activation?.getVariable('filePattern') || '.xml').toLowerCase();
|
|
1131
|
+
const maxFiles = parseInt(activation?.getVariable('maxFilesToProcess') || '10', 10);
|
|
1132
|
+
const retailerId = activation?.getVariable('retailerId'); // Optional: Only if mutation schema requires it
|
|
1133
|
+
const enableArchival = activation?.getVariable('enableArchival') !== 'false';
|
|
1134
|
+
const enableMutationLogs = activation?.getVariable('enableMutationLogs') !== 'false';
|
|
1135
|
+
const enableFileTracking = activation?.getVariable('enableFileTracking') !== 'false';
|
|
1136
|
+
|
|
1137
|
+
// ✅ Configuration with defaults
|
|
1138
|
+
const mutationBatchSize = parseInt(
|
|
1139
|
+
activation?.getVariable('mutationBatchSize') || '1', // ✅ Default: 1 (sequential)
|
|
1140
|
+
10
|
|
1141
|
+
);
|
|
1142
|
+
|
|
1143
|
+
const mutationsPerAliasBatch = activation?.getVariable('mutationsPerAliasBatch')
|
|
1144
|
+
? parseInt(activation?.getVariable('mutationsPerAliasBatch') || '1', 10)
|
|
1145
|
+
: undefined; // ✅ Default: undefined (disabled, use separate requests)
|
|
1146
|
+
|
|
1147
|
+
// Validate required variables
|
|
1148
|
+
const missingVars: string[] = [];
|
|
1149
|
+
if (!s3Bucket) missingVars.push('s3BucketName');
|
|
1150
|
+
if (!s3AccessKeyId) missingVars.push('awsAccessKeyId');
|
|
1151
|
+
if (!s3SecretAccessKey) missingVars.push('awsSecretAccessKey');
|
|
1152
|
+
// Note: retailerId is optional - only needed if mutation schema requires it
|
|
1153
|
+
|
|
1154
|
+
if (missingVars.length > 0) {
|
|
1155
|
+
const errorMsg = `Missing required variables: ${missingVars.join(', ')}`;
|
|
1156
|
+
log.error('❌ [VALIDATION] Missing required activation variables', {
|
|
1157
|
+
missingVars,
|
|
1158
|
+
recommendation: 'Add missing variables in Versori activation settings'
|
|
1159
|
+
});
|
|
1160
|
+
return { success: false, error: errorMsg, processed: 0 };
|
|
1161
|
+
}
|
|
1162
|
+
|
|
1163
|
+
log.info('✅ [VALIDATION] All required variables present', {
|
|
1164
|
+
s3Bucket,
|
|
1165
|
+
s3Region,
|
|
1166
|
+
s3Prefix,
|
|
1167
|
+
enableFileTracking,
|
|
1168
|
+
mutationBatchSize
|
|
1169
|
+
});
|
|
1170
|
+
|
|
1171
|
+
try {
|
|
1172
|
+
log.info('🔧 [INIT] Initializing Fluent Commerce client', { jobId });
|
|
1173
|
+
|
|
1174
|
+
// Initialize services with validateConnection
|
|
1175
|
+
const client = await createClient(ctx, { validateConnection: true });
|
|
1176
|
+
if (!client) {
|
|
1177
|
+
throw new Error('Failed to create Fluent Commerce client');
|
|
1178
|
+
}
|
|
1179
|
+
|
|
1180
|
+
log.info('✅ [INIT] Fluent Commerce client validated and ready', { jobId });
|
|
1181
|
+
|
|
1182
|
+
// ✅ CORRECT: GraphQL mutations do NOT need client.setRetailerId()
|
|
1183
|
+
// setRetailerId() is only for Job/Event API, NOT GraphQL
|
|
1184
|
+
// Check your GraphQL schema to determine retailerId handling:
|
|
1185
|
+
// - Mandatory retailerId → Must pass it in mutation input
|
|
1186
|
+
// - Optional retailerId → Can pass it if needed
|
|
1187
|
+
// - No retailerId field → Don't pass it
|
|
1188
|
+
// See: fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md
|
|
1189
|
+
|
|
1190
|
+
log.info('🗄️ [INIT] Initializing S3 data source', { s3Bucket, s3Region, s3Prefix });
|
|
1191
|
+
|
|
1192
|
+
const s3 = new S3DataSource(
|
|
1193
|
+
{
|
|
1194
|
+
type: 'S3_XML',
|
|
1195
|
+
connectionId: 's3-location-sync',
|
|
1196
|
+
name: 'Source S3',
|
|
1197
|
+
s3Config: {
|
|
1198
|
+
bucket: s3Bucket,
|
|
1199
|
+
region: s3Region,
|
|
1200
|
+
accessKeyId: s3AccessKeyId,
|
|
1201
|
+
secretAccessKey: s3SecretAccessKey,
|
|
1202
|
+
},
|
|
1203
|
+
},
|
|
1204
|
+
log
|
|
1205
|
+
);
|
|
1206
|
+
|
|
1207
|
+
const parser = new XMLParserService();
|
|
1208
|
+
|
|
1209
|
+
// Initialize state tracking (only if enabled)
|
|
1210
|
+
let stateService: StateService | null = null;
|
|
1211
|
+
if (enableFileTracking) {
|
|
1212
|
+
log.info('🔄 [INIT] Enabling file tracking with StateService', { jobId });
|
|
1213
|
+
const stateKV = new VersoriKVAdapter(ctx);
|
|
1214
|
+
stateService = new StateService(stateKV);
|
|
1215
|
+
} else {
|
|
1216
|
+
log.info('⏭️ [INIT] File tracking disabled - relying on S3 archival only', { jobId });
|
|
1217
|
+
}
|
|
1218
|
+
|
|
1219
|
+
log.info('📝 [INIT] Loading mapping configuration', { jobId });
|
|
1220
|
+
|
|
1221
|
+
// ✅ CRITICAL: Load mapping config from external JSON file
|
|
1222
|
+
// Mapping config uses GraphQLMutationMapper structure (nested objects, not dot notation)
|
|
1223
|
+
// File: src/config/location-mapping.json
|
|
1224
|
+
const mappingConfigJson = await import('../config/location-mapping.json', { assert: { type: 'json' } });
|
|
1225
|
+
const mappingConfig = mappingConfigJson.default;
|
|
1226
|
+
|
|
1227
|
+
// Initialize GraphQLMutationMapper with client for schema introspection
|
|
1228
|
+
const mapper = new GraphQLMutationMapper(mappingConfig, log, { fluentClient: client });
|
|
1229
|
+
|
|
1230
|
+
log.info('📂 [S3] Listing files from S3', { s3Bucket, s3Prefix, filePattern });
|
|
1231
|
+
|
|
1232
|
+
// List files (pattern filtering handled by listFiles)
|
|
1233
|
+
const files = await s3.listFiles({
|
|
1234
|
+
prefix: s3Prefix,
|
|
1235
|
+
pattern: filePattern,
|
|
1236
|
+
maxKeys: 1000
|
|
1237
|
+
});
|
|
1238
|
+
|
|
1239
|
+
// Newest-first ordering
|
|
1240
|
+
const xmlFiles = files
|
|
1241
|
+
.sort((a: any, b: any) => {
|
|
1242
|
+
const aTime = a.lastModified ? new Date(a.lastModified).getTime() : 0;
|
|
1243
|
+
const bTime = b.lastModified ? new Date(b.lastModified).getTime() : 0;
|
|
1244
|
+
return bTime - aTime;
|
|
1245
|
+
})
|
|
1246
|
+
.slice(0, maxFiles);
|
|
1247
|
+
|
|
1248
|
+
log.info('📊 [S3] File discovery complete', {
|
|
1249
|
+
totalFiles: files.length,
|
|
1250
|
+
xmlFiles: xmlFiles.length,
|
|
1251
|
+
maxFiles,
|
|
1252
|
+
selectedFiles: xmlFiles.map(f => f.name)
|
|
1253
|
+
});
|
|
1254
|
+
|
|
1255
|
+
const results = {
|
|
1256
|
+
processed: 0,
|
|
1257
|
+
skipped: 0,
|
|
1258
|
+
failed: 0,
|
|
1259
|
+
totalRecords: 0,
|
|
1260
|
+
errors: [] as string[],
|
|
1261
|
+
};
|
|
1262
|
+
|
|
1263
|
+
log.info('🔄 [PROCESSING] Starting file processing loop', {
|
|
1264
|
+
fileCount: xmlFiles.length,
|
|
1265
|
+
jobId
|
|
1266
|
+
});
|
|
1267
|
+
|
|
1268
|
+
// Per-file processing loop
|
|
1269
|
+
for (const file of xmlFiles) {
|
|
1270
|
+
const filePath = file.path;
|
|
1271
|
+
const fileName = file.name;
|
|
1272
|
+
|
|
1273
|
+
log.info('📄 [FILE] Processing file', { fileName, filePath });
|
|
1274
|
+
|
|
1275
|
+
// Duplicate prevention (secondary check - files in processed/ won't be listed)
|
|
1276
|
+
// Primary deduplication: S3 archival (files moved to processed/ subdirectory)
|
|
1277
|
+
if (enableFileTracking && stateService) {
|
|
1278
|
+
const wasProcessed = await stateService.isFileProcessed(fileName);
|
|
1279
|
+
if (wasProcessed) {
|
|
1280
|
+
log.info('⏭️ [SKIP] File already processed (KV check)', { fileName });
|
|
1281
|
+
results.skipped++;
|
|
1282
|
+
continue;
|
|
1283
|
+
}
|
|
1284
|
+
}
|
|
1285
|
+
|
|
1286
|
+
try {
|
|
1287
|
+
// Step 1: Process file (download, parse, map)
|
|
1288
|
+
log.info('📥 [DOWNLOAD] Downloading and parsing file', { fileName });
|
|
1289
|
+
const processingResult = await processFile(s3, parser, mapper, filePath, fileName, log);
|
|
1290
|
+
|
|
1291
|
+
if (!processingResult.success) {
|
|
1292
|
+
throw new Error(`File processing failed: ${processingResult.errors.join(', ')}`);
|
|
1293
|
+
}
|
|
1294
|
+
|
|
1295
|
+
if (processingResult.locations.length === 0) {
|
|
1296
|
+
log.warn('⚠️ [SKIP] Empty file detected, archiving', { fileName });
|
|
1297
|
+
if (enableArchival) {
|
|
1298
|
+
await s3.moveFile(filePath, `${archivePrefix}${fileName}`);
|
|
1299
|
+
log.info('📦 [ARCHIVE] Empty file archived', { fileName, destination: `${archivePrefix}${fileName}` });
|
|
1300
|
+
}
|
|
1301
|
+
results.skipped++;
|
|
1302
|
+
continue;
|
|
1303
|
+
}
|
|
1304
|
+
|
|
1305
|
+
log.info('✅ [PARSE] File parsed successfully', {
|
|
1306
|
+
fileName,
|
|
1307
|
+
locationCount: processingResult.locations.length,
|
|
1308
|
+
mappingErrors: processingResult.errors.length
|
|
1309
|
+
});
|
|
1310
|
+
|
|
1311
|
+
// Step 2: Execute mutations
|
|
1312
|
+
// Step 2: Execute mutations with alias batching support
|
|
1313
|
+
// ? Enhanced: Extract context for progress logging
|
|
1314
|
+
const sampleLocationRefs = processingResult.locations.slice(0, 5).map((loc: any) => loc.input?.ref || loc.ref || 'unknown');
|
|
1315
|
+
const mutationType = mapper?.mutationName || 'createLocation';
|
|
1316
|
+
|
|
1317
|
+
// ? Enhanced: Start logging with context
|
|
1318
|
+
log.info(`[GraphQLMutations] Sending mutations for file "${fileName}"`, {
|
|
1319
|
+
totalMutations: processingResult.locations.length,
|
|
1320
|
+
mutationType,
|
|
1321
|
+
batchSize: mutationBatchSize,
|
|
1322
|
+
batchMode: mutationBatchSize === 1 ? 'sequential' : `parallel (${mutationBatchSize})`,
|
|
1323
|
+
sampleLocationRefs: sampleLocationRefs.join(', '),
|
|
1324
|
+
aliasBatching: mutationsPerAliasBatch ? `enabled (${mutationsPerAliasBatch} per alias)` : 'disabled'
|
|
1325
|
+
});
|
|
1326
|
+
|
|
1327
|
+
const mutationResult = await executeMutations(
|
|
1328
|
+
client,
|
|
1329
|
+
mapper,
|
|
1330
|
+
processingResult.locations,
|
|
1331
|
+
log,
|
|
1332
|
+
retailerId, // Pass retailerId for mutations that require it in input
|
|
1333
|
+
mutationBatchSize, // Concurrency control (default: 1)
|
|
1334
|
+
mutationsPerAliasBatch // ✅ NEW: Alias batching (default: undefined)
|
|
1335
|
+
);
|
|
1336
|
+
|
|
1337
|
+
// ? Enhanced: Completion logging with summary
|
|
1338
|
+
log.info(`[GraphQLMutations] Mutation submission completed for file "${fileName}"`, {
|
|
1339
|
+
totalMutations: processingResult.locations.length,
|
|
1340
|
+
successful: mutationResult.successful,
|
|
1341
|
+
failed: mutationResult.failed,
|
|
1342
|
+
successRate: processingResult.locations.length > 0 ? `${Math.round((mutationResult.successful / processingResult.locations.length) * 100)}%` : '0%',
|
|
1343
|
+
mutationType
|
|
1344
|
+
});
|
|
1345
|
+
|
|
1346
|
+
// Step 3: Write mutation log (if enabled)
|
|
1347
|
+
if (enableMutationLogs) {
|
|
1348
|
+
const logEntries: MutationLogEntry[] = processingResult.locations.map(loc => {
|
|
1349
|
+
const failed = mutationResult.errors.find(e => e.startsWith(loc.ref));
|
|
1350
|
+
return {
|
|
1351
|
+
timestamp: new Date().toISOString(),
|
|
1352
|
+
fileName,
|
|
1353
|
+
locationRef: loc.ref,
|
|
1354
|
+
status: failed ? 'failure' : 'success',
|
|
1355
|
+
error: failed,
|
|
1356
|
+
};
|
|
1357
|
+
});
|
|
1358
|
+
|
|
1359
|
+
await writeMutationLog(s3, logEntries, fileName, logPrefix, log);
|
|
1360
|
+
}
|
|
1361
|
+
|
|
1362
|
+
// Step 4: Archive file (PRIMARY deduplication - file won't be listed again)
|
|
1363
|
+
if (enableArchival) {
|
|
1364
|
+
log.info('📦 [ARCHIVE] Moving file to processed directory', { fileName });
|
|
1365
|
+
await s3.moveFile(filePath, `${archivePrefix}${fileName}`);
|
|
1366
|
+
log.info('✅ [ARCHIVE] File archived successfully', {
|
|
1367
|
+
fileName,
|
|
1368
|
+
destination: `${archivePrefix}${fileName}`
|
|
1369
|
+
});
|
|
1370
|
+
}
|
|
1371
|
+
|
|
1372
|
+
// Step 5: Mark processed in KV (SECONDARY - provides metadata/history)
|
|
1373
|
+
if (enableFileTracking && stateService) {
|
|
1374
|
+
log.info('💾 [STATE] Recording file processing metadata', { fileName });
|
|
1375
|
+
await stateService.markFileProcessed(fileName, {
|
|
1376
|
+
recordCount: processingResult.locations.length,
|
|
1377
|
+
successful: mutationResult.successful,
|
|
1378
|
+
failed: mutationResult.failed,
|
|
1379
|
+
mappingErrors: processingResult.errors.length,
|
|
1380
|
+
timestamp: new Date().toISOString(),
|
|
1381
|
+
});
|
|
1382
|
+
}
|
|
1383
|
+
|
|
1384
|
+
results.processed++;
|
|
1385
|
+
results.totalRecords += mutationResult.successful;
|
|
1386
|
+
|
|
1387
|
+
log.info('✅ [COMPLETE] File processing complete', {
|
|
1388
|
+
fileName,
|
|
1389
|
+
locations: processingResult.locations.length,
|
|
1390
|
+
successful: mutationResult.successful,
|
|
1391
|
+
failed: mutationResult.failed,
|
|
1392
|
+
mappingErrors: processingResult.errors.length,
|
|
1393
|
+
});
|
|
1394
|
+
|
|
1395
|
+
if (processingResult.errors.length > 0 || mutationResult.errors.length > 0) {
|
|
1396
|
+
results.errors.push(
|
|
1397
|
+
`${fileName}: ${processingResult.errors.length} mapping errors, ${mutationResult.errors.length} mutation errors`
|
|
1398
|
+
);
|
|
1399
|
+
}
|
|
1400
|
+
} catch (error: any) {
|
|
1401
|
+
// ✅ Enhanced error logging: Extract all error details for visibility
|
|
1402
|
+
const errorDetails = {
|
|
1403
|
+
message: error?.message || 'Unknown error',
|
|
1404
|
+
stack: error?.stack,
|
|
1405
|
+
fileName: error?.fileName,
|
|
1406
|
+
lineNumber: error?.lineNumber,
|
|
1407
|
+
originalError: error?.context?.originalError?.message,
|
|
1408
|
+
errorType: error?.name || 'Error',
|
|
1409
|
+
};
|
|
1410
|
+
|
|
1411
|
+
log.error('❌ [ERROR] File processing failed', errorDetails, { fileName });
|
|
1412
|
+
|
|
1413
|
+
// Provide error recommendations based on error type
|
|
1414
|
+
const recommendation = getErrorRecommendation(error);
|
|
1415
|
+
if (recommendation) {
|
|
1416
|
+
log.warn('💡 [RECOMMENDATION]', { fileName, recommendation });
|
|
1417
|
+
}
|
|
1418
|
+
|
|
1419
|
+
results.failed++;
|
|
1420
|
+
results.errors.push(`${fileName}: ${error.message}`);
|
|
1421
|
+
|
|
1422
|
+
// Attempt to move to error directory, ignore failures
|
|
1423
|
+
try {
|
|
1424
|
+
log.info('🗂️ [ERROR] Moving failed file to error directory', { fileName });
|
|
1425
|
+
await s3.moveFile(filePath, `${errorPrefix}${fileName}`);
|
|
1426
|
+
log.info('✅ [ERROR] Failed file moved to error directory', {
|
|
1427
|
+
fileName,
|
|
1428
|
+
destination: `${errorPrefix}${fileName}`
|
|
1429
|
+
});
|
|
1430
|
+
} catch (moveError) {
|
|
1431
|
+
log.warn('⚠️ [ERROR] Failed to move file to error directory', {
|
|
1432
|
+
fileName,
|
|
1433
|
+
moveError: moveError instanceof Error ? moveError.message : String(moveError)
|
|
1434
|
+
});
|
|
1435
|
+
}
|
|
1436
|
+
|
|
1437
|
+
// Track error state with exponential backoff (only if file tracking enabled)
|
|
1438
|
+
if (enableFileTracking && stateService) {
|
|
1439
|
+
try {
|
|
1440
|
+
const stateKV = new VersoriKVAdapter(ctx);
|
|
1441
|
+
const key = ['error-state', fileName];
|
|
1442
|
+
const prev = (await stateKV.get(key))?.value as any;
|
|
1443
|
+
const attempts = (prev?.attemptCount || 0) + 1;
|
|
1444
|
+
const backoffMinutes = Math.min(Math.pow(2, attempts) * 5, 24 * 60);
|
|
1445
|
+
const nextRetryAt = new Date(Date.now() + backoffMinutes * 60000).toISOString();
|
|
1446
|
+
|
|
1447
|
+
await stateKV.set(key, {
|
|
1448
|
+
fileName,
|
|
1449
|
+
attemptCount: attempts,
|
|
1450
|
+
lastError: error?.message || 'unknown',
|
|
1451
|
+
lastAttemptAt: new Date().toISOString(),
|
|
1452
|
+
firstFailedAt: prev?.firstFailedAt || new Date().toISOString(),
|
|
1453
|
+
nextRetryAt,
|
|
1454
|
+
});
|
|
1455
|
+
|
|
1456
|
+
log.info('💾 [ERROR] Error state tracked with exponential backoff', {
|
|
1457
|
+
fileName,
|
|
1458
|
+
attempts,
|
|
1459
|
+
nextRetryAt
|
|
1460
|
+
});
|
|
1461
|
+
} catch (stateError) {
|
|
1462
|
+
log.warn('⚠️ [ERROR] Failed to track error state', {
|
|
1463
|
+
fileName,
|
|
1464
|
+
stateError: stateError instanceof Error ? stateError.message : String(stateError)
|
|
1465
|
+
});
|
|
1466
|
+
}
|
|
1467
|
+
}
|
|
1468
|
+
}
|
|
1469
|
+
}
|
|
1470
|
+
|
|
1471
|
+
log.info('🏁 [COMPLETE] File processing loop finished', {
|
|
1472
|
+
processed: results.processed,
|
|
1473
|
+
skipped: results.skipped,
|
|
1474
|
+
failed: results.failed,
|
|
1475
|
+
totalRecords: results.totalRecords
|
|
1476
|
+
});
|
|
1477
|
+
|
|
1478
|
+
return results;
|
|
1479
|
+
} catch (error: any) {
|
|
1480
|
+
// ✅ Enhanced error logging: Extract all error details for visibility
|
|
1481
|
+
const errorDetails = {
|
|
1482
|
+
message: error?.message || 'Unknown error',
|
|
1483
|
+
stack: error?.stack,
|
|
1484
|
+
errorType: error?.name || 'Error',
|
|
1485
|
+
};
|
|
1486
|
+
|
|
1487
|
+
log.error('❌ [FATAL] Location sync failed', errorDetails);
|
|
1488
|
+
|
|
1489
|
+
// Provide fatal error recommendations
|
|
1490
|
+
const recommendation = getErrorRecommendation(error);
|
|
1491
|
+
if (recommendation) {
|
|
1492
|
+
log.warn('💡 [RECOMMENDATION]', { recommendation });
|
|
1493
|
+
}
|
|
1494
|
+
|
|
1495
|
+
return {
|
|
1496
|
+
success: false,
|
|
1497
|
+
error: error.message,
|
|
1498
|
+
processed: 0,
|
|
1499
|
+
timestamp: new Date().toISOString(),
|
|
1500
|
+
};
|
|
1501
|
+
}
|
|
1502
|
+
}
|
|
1503
|
+
|
|
1504
|
+
/**
|
|
1505
|
+
* Get error recommendation based on error type
|
|
1506
|
+
*/
|
|
1507
|
+
function getErrorRecommendation(error: any): string | null {
|
|
1508
|
+
const message = error?.message?.toLowerCase() || '';
|
|
1509
|
+
|
|
1510
|
+
if (message.includes('s3') || message.includes('access denied')) {
|
|
1511
|
+
return 'Check S3 credentials and bucket permissions. Verify IAM policy includes s3:ListBucket, s3:GetObject, s3:PutObject, s3:DeleteObject.';
|
|
1512
|
+
}
|
|
1513
|
+
|
|
1514
|
+
if (message.includes('xml') || message.includes('parse')) {
|
|
1515
|
+
return 'Verify XML file structure matches expected schema. Check for malformed XML or encoding issues.';
|
|
1516
|
+
}
|
|
1517
|
+
|
|
1518
|
+
if (message.includes('mapping') || message.includes('field')) {
|
|
1519
|
+
return 'Review field mapping configuration. Ensure all required fields are present and source paths are correct.';
|
|
1520
|
+
}
|
|
1521
|
+
|
|
1522
|
+
if (message.includes('graphql') || message.includes('mutation')) {
|
|
1523
|
+
return 'Check GraphQL schema and mutation input. Verify all required fields are provided and types match schema.';
|
|
1524
|
+
}
|
|
1525
|
+
|
|
1526
|
+
if (message.includes('auth') || message.includes('401') || message.includes('403')) {
|
|
1527
|
+
return 'Verify Fluent Commerce credentials. Check OAuth2 client ID/secret and ensure connection is active.';
|
|
1528
|
+
}
|
|
1529
|
+
|
|
1530
|
+
if (message.includes('timeout') || message.includes('econnrefused')) {
|
|
1531
|
+
return 'Check network connectivity. Verify API endpoints are accessible and not rate-limited.';
|
|
1532
|
+
}
|
|
1533
|
+
|
|
1534
|
+
return null;
|
|
1535
|
+
}
|
|
1536
|
+
```
|
|
1537
|
+
|
|
1538
|
+
**Note:** The `executeLocationSync` function should contain the full implementation of `runLocationXmlWorkflow` (renamed to `executeLocationSync`), including all the logic for processing files, executing mutations, and logging. The implementation details are shown in the service function code above.
|
|
1539
|
+
|
|
1540
|
+
---
|
|
1541
|
+
|
|
1542
|
+
### Step 5: TypeScript Configuration
|
|
1543
|
+
|
|
1544
|
+
**File: tsconfig.json**
|
|
1545
|
+
|
|
1546
|
+
```json
|
|
1547
|
+
{
|
|
1548
|
+
"compilerOptions": {
|
|
1549
|
+
"module": "ES2022",
|
|
1550
|
+
"target": "ES2024",
|
|
1551
|
+
"moduleResolution": "node"
|
|
1552
|
+
}
|
|
1553
|
+
}
|
|
1554
|
+
```
|
|
1555
|
+
|
|
1556
|
+
---
|
|
1557
|
+
|
|
1558
|
+
## Code Flow Explanation
|
|
1559
|
+
|
|
1560
|
+
### Initialization Phase
|
|
1561
|
+
|
|
1562
|
+
1. **Read activation variables** - S3 config, Fluent config, rate limiting, logging options
|
|
1563
|
+
2. **Validate required variables** - Fail fast if missing credentials
|
|
1564
|
+
3. **Initialize SDK services** - FluentClient, S3DataSource, XMLParserService, StateService
|
|
1565
|
+
4. **Create mapping configuration** - Define XML → GraphQL field mappings with nested objects
|
|
1566
|
+
5. **Calculate rate limit delay** - Convert mutations/second to delay in milliseconds
|
|
1567
|
+
|
|
1568
|
+
### File Discovery Phase
|
|
1569
|
+
|
|
1570
|
+
1. **List S3 files** - Use `s3.listFiles()` with prefix filter (excludes processed/ subdirectory)
|
|
1571
|
+
2. **Filter by pattern** - Match file extension (e.g., `.xml`)
|
|
1572
|
+
3. **Sort newest-first** - Process most recent files first
|
|
1573
|
+
4. **Apply max files limit** - Prevent overwhelming the workflow
|
|
1574
|
+
5. **Note**: Files in `processed/` subdirectory won't be listed (primary deduplication)
|
|
1575
|
+
|
|
1576
|
+
### Per-File Processing (Service Functions)
|
|
1577
|
+
|
|
1578
|
+
**Step 1: processFile()** - Download, parse, map
|
|
1579
|
+
|
|
1580
|
+
1. **Download file** - Use S3DataSource with retry logic
|
|
1581
|
+
2. **Parse XML** - XMLParserService converts to JavaScript object
|
|
1582
|
+
3. **Normalize array** - Handle single vs multiple `<location>` elements (CRITICAL for XML)
|
|
1583
|
+
4. **Map locations** - Use UniversalMapper with nested field mapping
|
|
1584
|
+
5. **Collect errors** - Track mapping errors without stopping
|
|
1585
|
+
6. **Return result** - FileProcessingResult with locations array and errors
|
|
1586
|
+
|
|
1587
|
+
**Step 2: executeMutations()** - GraphQL mutations with rate limiting
|
|
1588
|
+
|
|
1589
|
+
1. **Loop through locations** - Process each location individually
|
|
1590
|
+
2. **Build mutation input** - Extract nested fields (primaryAddress, openingSchedule)
|
|
1591
|
+
3. **Execute mutation** - Direct GraphQL `createLocation` with retry logic
|
|
1592
|
+
4. **Apply rate limiting** - Add configurable delay between mutations
|
|
1593
|
+
5. **Track results** - Count successful vs failed, collect error messages
|
|
1594
|
+
6. **Return result** - MutationResult with counts and errors
|
|
1595
|
+
|
|
1596
|
+
**Step 3: writeMutationLog()** - S3 log file (optional)
|
|
1597
|
+
|
|
1598
|
+
1. **Create log entries** - Map locations to log entries with status
|
|
1599
|
+
2. **Build JSON log** - Include timestamp, summary, detailed entries
|
|
1600
|
+
3. **Write to S3** - Use Buffer.from() for Deno/Versori runtime compatibility
|
|
1601
|
+
4. **Non-blocking** - Logging failure doesn't stop workflow
|
|
1602
|
+
5. **Timestamped naming** - Unique log file per processed file
|
|
1603
|
+
|
|
1604
|
+
### Cleanup Phase
|
|
1605
|
+
|
|
1606
|
+
1. **Archive file FIRST** - Move to `processed/` or `errors/` (PRIMARY deduplication)
|
|
1607
|
+
2. **Mark processed in KV** - StateService tracks metadata and processing history
|
|
1608
|
+
3. **Error state tracking** - Store error info with exponential backoff timestamp
|
|
1609
|
+
4. **Return results** - Summary of processed, skipped, failed files
|
|
1610
|
+
|
|
1611
|
+
**Note**: The order matters! Archive first (primary deduplication), then KV tracking (metadata/history).
|
|
1612
|
+
|
|
1613
|
+
### Service Function Benefits
|
|
1614
|
+
|
|
1615
|
+
- **Composability**: Each function has single responsibility and can be reused
|
|
1616
|
+
- **Testability**: Service functions can be unit tested independently
|
|
1617
|
+
- **Clarity**: Main workflow shows high-level orchestration
|
|
1618
|
+
- **Error Handling**: Isolated error handling per service function
|
|
1619
|
+
- **Logging**: Detailed logging at each step with structured context
|
|
1620
|
+
|
|
1621
|
+
---
|
|
1622
|
+
|
|
1623
|
+
## S3 Archival Deduplication Pattern
|
|
1624
|
+
|
|
1625
|
+
This template uses **S3 archival** as the primary deduplication mechanism, NOT VersoriFileTracker.
|
|
1626
|
+
|
|
1627
|
+
### How It Works
|
|
1628
|
+
|
|
1629
|
+
**S3 Directory Structure:**
|
|
1630
|
+
|
|
1631
|
+
```
|
|
1632
|
+
s3://my-bucket/
|
|
1633
|
+
├── locations/ ← listFiles() reads from here
|
|
1634
|
+
│ ├── new-file-1.xml
|
|
1635
|
+
│ └── new-file-2.xml
|
|
1636
|
+
├── processed/ ← Successfully processed files
|
|
1637
|
+
│ ├── old-file-1.xml
|
|
1638
|
+
│ └── old-file-2.xml
|
|
1639
|
+
└── errors/ ← Failed files
|
|
1640
|
+
└── bad-file.xml
|
|
1641
|
+
```
|
|
1642
|
+
|
|
1643
|
+
**Deduplication Flow:**
|
|
1644
|
+
|
|
1645
|
+
1. **List files**: `s3.listFiles({ prefix: 'locations/' })` - Only lists `locations/` subdirectory
|
|
1646
|
+
2. **Process file**: Download, parse, transform, send mutations
|
|
1647
|
+
3. **Archive**: `s3.moveFile(filePath, 'processed/new-file-1.xml')` - Moves file out of `locations/`
|
|
1648
|
+
4. **Next run**: File is now in `processed/`, won't be listed again
|
|
1649
|
+
|
|
1650
|
+
**Why This Works:**
|
|
1651
|
+
|
|
1652
|
+
- Files in `processed/` subdirectory are **never listed** when prefix is `locations/`
|
|
1653
|
+
- No need to track file state in KV store for deduplication
|
|
1654
|
+
- S3 is the single source of truth for file status
|
|
1655
|
+
- Simple, reliable, scales to millions of files
|
|
1656
|
+
|
|
1657
|
+
**StateService Role (Secondary):**
|
|
1658
|
+
|
|
1659
|
+
- Provides metadata and processing history
|
|
1660
|
+
- Backup check in case archival fails mid-process
|
|
1661
|
+
- Useful for monitoring and debugging
|
|
1662
|
+
- NOT the primary deduplication mechanism
|
|
1663
|
+
|
|
1664
|
+
**When to Use VersoriFileTracker:**
|
|
1665
|
+
|
|
1666
|
+
- **NEVER for S3 sources** - Use archival pattern instead
|
|
1667
|
+
- **Only for SFTP sources** - Where archival might not be possible
|
|
1668
|
+
- See SFTP templates for VersoriFileTracker usage
|
|
1669
|
+
|
|
1670
|
+
---
|
|
1671
|
+
|
|
1672
|
+
## XML Path Resolution Patterns
|
|
1673
|
+
|
|
1674
|
+
### Pattern 1: XML Attribute Access with @ Prefix
|
|
1675
|
+
|
|
1676
|
+
```typescript
|
|
1677
|
+
const mappingConfig = {
|
|
1678
|
+
fields: {
|
|
1679
|
+
// XML attribute access
|
|
1680
|
+
ref: { source: 'location.@ref' }, // <location ref="LOC-001">
|
|
1681
|
+
type: { source: 'location.@type' }, // <location type="WAREHOUSE">
|
|
1682
|
+
country: { source: 'location.address.@country' }, // <address country="USA">
|
|
1683
|
+
|
|
1684
|
+
// XML element text content
|
|
1685
|
+
name: { source: 'location.name' }, // <name>Downtown</name>
|
|
1686
|
+
city: { source: 'location.address.city' }, // <address><city>NYC</city></address>
|
|
1687
|
+
},
|
|
1688
|
+
};
|
|
1689
|
+
```
|
|
1690
|
+
|
|
1691
|
+
### Pattern 2: Handling Single vs Multiple Elements
|
|
1692
|
+
|
|
1693
|
+
```typescript
|
|
1694
|
+
// XML can have single or multiple <location> elements
|
|
1695
|
+
const locationsData = xmlData.locations?.location;
|
|
1696
|
+
|
|
1697
|
+
// Normalize to array
|
|
1698
|
+
const locations = Array.isArray(locationsData) ? locationsData : [locationsData];
|
|
1699
|
+
|
|
1700
|
+
// Process each location
|
|
1701
|
+
for (const loc of locations) {
|
|
1702
|
+
const record = { location: loc }; // Wrap for mapping
|
|
1703
|
+
const result = await mapper.map(record);
|
|
1704
|
+
}
|
|
1705
|
+
```
|
|
1706
|
+
|
|
1707
|
+
### Pattern 3: Nested Object Mapping
|
|
1708
|
+
|
|
1709
|
+
```typescript
|
|
1710
|
+
const mappingConfig = {
|
|
1711
|
+
fields: {
|
|
1712
|
+
// Root fields
|
|
1713
|
+
ref: { source: 'location.@ref', required: true },
|
|
1714
|
+
name: { source: 'location.name', required: true },
|
|
1715
|
+
type: { source: 'location.@type', required: true },
|
|
1716
|
+
|
|
1717
|
+
// Nested primaryAddress object
|
|
1718
|
+
'primaryAddress.ref': { source: 'location.@ref' },
|
|
1719
|
+
'primaryAddress.street': { source: 'location.address.street1' },
|
|
1720
|
+
'primaryAddress.city': { source: 'location.address.city' },
|
|
1721
|
+
'primaryAddress.latitude': { source: 'location.coordinates.@lat', resolver: 'sdk.parseFloat' },
|
|
1722
|
+
|
|
1723
|
+
// Nested openingSchedule object
|
|
1724
|
+
'openingSchedule.allHours': {
|
|
1725
|
+
source: 'location.openingSchedule.allHours',
|
|
1726
|
+
resolver: 'sdk.boolean',
|
|
1727
|
+
},
|
|
1728
|
+
'openingSchedule.monStart': {
|
|
1729
|
+
source: 'location.openingSchedule.monStart',
|
|
1730
|
+
resolver: 'sdk.parseInt',
|
|
1731
|
+
},
|
|
1732
|
+
|
|
1733
|
+
// Note: retailer.id not shown - standard createLocation does not have this field
|
|
1734
|
+
// If your schema requires it, add: 'retailer.id': { value: parseInt(retailerId) }
|
|
1735
|
+
},
|
|
1736
|
+
};
|
|
1737
|
+
```
|
|
1738
|
+
|
|
1739
|
+
---
|
|
1740
|
+
|
|
1741
|
+
## Sample XML Files
|
|
1742
|
+
|
|
1743
|
+
### Minimal Test File
|
|
1744
|
+
|
|
1745
|
+
**File: test-location.xml**
|
|
1746
|
+
|
|
1747
|
+
```xml
|
|
1748
|
+
<?xml version="1.0" encoding="UTF-8"?>
|
|
1749
|
+
<locations>
|
|
1750
|
+
<location ref="TEST-001" type="WAREHOUSE">
|
|
1751
|
+
<name>Test Warehouse</name>
|
|
1752
|
+
<address country="USA">
|
|
1753
|
+
<street1>123 Test St</street1>
|
|
1754
|
+
<city>TestCity</city>
|
|
1755
|
+
<state>TC</state>
|
|
1756
|
+
<postalCode>12345</postalCode>
|
|
1757
|
+
</address>
|
|
1758
|
+
<coordinates lat="40.7128" lon="-74.0060"/>
|
|
1759
|
+
<timeZone>America/New_York</timeZone>
|
|
1760
|
+
<openingSchedule>
|
|
1761
|
+
<allHours>false</allHours>
|
|
1762
|
+
<monStart>800</monStart>
|
|
1763
|
+
<monEnd>1800</monEnd>
|
|
1764
|
+
<tueStart>800</tueStart>
|
|
1765
|
+
<tueEnd>1800</tueEnd>
|
|
1766
|
+
<wedStart>800</wedStart>
|
|
1767
|
+
<wedEnd>1800</wedEnd>
|
|
1768
|
+
<thuStart>800</thuStart>
|
|
1769
|
+
<thuEnd>1800</thuEnd>
|
|
1770
|
+
<friStart>800</friStart>
|
|
1771
|
+
<friEnd>1800</friEnd>
|
|
1772
|
+
<satStart>0</satStart>
|
|
1773
|
+
<satEnd>0</satEnd>
|
|
1774
|
+
<sunStart>0</sunStart>
|
|
1775
|
+
<sunEnd>0</sunEnd>
|
|
1776
|
+
</openingSchedule>
|
|
1777
|
+
</location>
|
|
1778
|
+
</locations>
|
|
1779
|
+
```
|
|
1780
|
+
|
|
1781
|
+
---
|
|
1782
|
+
|
|
1783
|
+
## Service Functions Deep Dive
|
|
1784
|
+
|
|
1785
|
+
This template demonstrates service function composition for maintainable workflows.
|
|
1786
|
+
|
|
1787
|
+
### Function 1: processFile()
|
|
1788
|
+
|
|
1789
|
+
**Purpose**: Download, parse, normalize, and map XML data
|
|
1790
|
+
|
|
1791
|
+
**Inputs**:
|
|
1792
|
+
- `s3: S3DataSource` - For file download
|
|
1793
|
+
- `parser: XMLParserService` - For XML parsing
|
|
1794
|
+
- `mapper: UniversalMapper` - For field mapping
|
|
1795
|
+
- `filePath: string` - S3 path to file
|
|
1796
|
+
- `fileName: string` - File name for logging
|
|
1797
|
+
- `log: any` - Logger instance
|
|
1798
|
+
|
|
1799
|
+
**Outputs**: `FileProcessingResult`
|
|
1800
|
+
```typescript
|
|
1801
|
+
{
|
|
1802
|
+
success: boolean; // Overall success
|
|
1803
|
+
locations: any[]; // Mapped location objects
|
|
1804
|
+
errors: string[]; // Mapping error messages
|
|
1805
|
+
}
|
|
1806
|
+
```
|
|
1807
|
+
|
|
1808
|
+
**Key Operations**:
|
|
1809
|
+
1. Downloads file with retry logic
|
|
1810
|
+
2. Parses XML using XMLParserService
|
|
1811
|
+
3. **Normalizes arrays** - Handles single vs multiple `<location>` elements
|
|
1812
|
+
4. Maps each location using UniversalMapper
|
|
1813
|
+
5. Collects errors without stopping processing
|
|
1814
|
+
6. Returns all mapped locations + errors
|
|
1815
|
+
|
|
1816
|
+
**Why separate function?**
|
|
1817
|
+
- Testable independently with mock data
|
|
1818
|
+
- Reusable across workflows (scheduled, webhook, adhoc)
|
|
1819
|
+
- Clear error boundaries - file-level errors vs record-level errors
|
|
1820
|
+
- Easy to add XML validation or schema checks
|
|
1821
|
+
|
|
1822
|
+
### Function 2: executeMutations()
|
|
1823
|
+
|
|
1824
|
+
**Purpose**: Execute GraphQL createLocation mutations with rate limiting
|
|
1825
|
+
|
|
1826
|
+
**Inputs**:
|
|
1827
|
+
- `client: FluentClient` - For GraphQL mutations
|
|
1828
|
+
- `locations: any[]` - Mapped location data
|
|
1829
|
+
- `mutationDelayMs: number` - Rate limit delay
|
|
1830
|
+
- `fileName: string` - For logging context
|
|
1831
|
+
- `log: any` - Logger instance
|
|
1832
|
+
|
|
1833
|
+
**Outputs**: `MutationResult`
|
|
1834
|
+
```typescript
|
|
1835
|
+
{
|
|
1836
|
+
successful: number; // Count of successful mutations
|
|
1837
|
+
failed: number; // Count of failed mutations
|
|
1838
|
+
errors: string[]; // Error messages with location refs
|
|
1839
|
+
}
|
|
1840
|
+
```
|
|
1841
|
+
|
|
1842
|
+
**Key Operations**:
|
|
1843
|
+
1. Loops through locations
|
|
1844
|
+
2. Builds GraphQL mutation input
|
|
1845
|
+
3. Executes with retry logic + rate limiting
|
|
1846
|
+
4. Tracks success/failure per location
|
|
1847
|
+
5. Returns summary counts + errors
|
|
1848
|
+
|
|
1849
|
+
**Why separate function?**
|
|
1850
|
+
- Clear separation: mapping vs mutation execution
|
|
1851
|
+
- Rate limiting logic isolated and configurable
|
|
1852
|
+
- Easy to swap mutation types (create vs update)
|
|
1853
|
+
- Testable with mock FluentClient
|
|
1854
|
+
- Can parallelize in future (batch mutations)
|
|
1855
|
+
|
|
1856
|
+
### Function 3: writeMutationLog()
|
|
1857
|
+
|
|
1858
|
+
**Purpose**: Write detailed mutation results to S3 as JSON log
|
|
1859
|
+
|
|
1860
|
+
**Inputs**:
|
|
1861
|
+
- `s3: S3DataSource` - For log upload
|
|
1862
|
+
- `logEntries: MutationLogEntry[]` - Mutation status per location
|
|
1863
|
+
- `fileName: string` - Original file name
|
|
1864
|
+
- `logPrefix: string` - S3 prefix for logs (e.g., `logs/`)
|
|
1865
|
+
- `log: any` - Logger instance
|
|
1866
|
+
|
|
1867
|
+
**Outputs**: `void` (non-blocking - errors logged but not thrown)
|
|
1868
|
+
|
|
1869
|
+
**Key Operations**:
|
|
1870
|
+
1. Creates timestamped log file name
|
|
1871
|
+
2. Builds JSON log with summary + entries
|
|
1872
|
+
3. **Uses Buffer.from()** - Required for Deno/Versori runtime
|
|
1873
|
+
4. Writes to S3 with `uploadFile()`
|
|
1874
|
+
5. Errors don't stop workflow (logging is non-critical)
|
|
1875
|
+
|
|
1876
|
+
**Why separate function?**
|
|
1877
|
+
- Optional feature - can be disabled via config
|
|
1878
|
+
- Non-blocking - logging failure doesn't fail workflow
|
|
1879
|
+
- Structured logging for audit trails
|
|
1880
|
+
- Easy to change log format (JSON, CSV, XML)
|
|
1881
|
+
- Can add log rotation/cleanup logic later
|
|
1882
|
+
|
|
1883
|
+
### Service Function Composition Benefits
|
|
1884
|
+
|
|
1885
|
+
**1. Maintainability**
|
|
1886
|
+
```typescript
|
|
1887
|
+
// Clear workflow orchestration
|
|
1888
|
+
const processingResult = await processFile(...);
|
|
1889
|
+
const mutationResult = await executeMutations(...);
|
|
1890
|
+
await writeMutationLog(...);
|
|
1891
|
+
```
|
|
1892
|
+
|
|
1893
|
+
**2. Testability**
|
|
1894
|
+
```typescript
|
|
1895
|
+
// Unit test processFile() with mock XML
|
|
1896
|
+
const mockS3 = { downloadFile: jest.fn() };
|
|
1897
|
+
const result = await processFile(mockS3, parser, mapper, ...);
|
|
1898
|
+
expect(result.locations).toHaveLength(5);
|
|
1899
|
+
```
|
|
1900
|
+
|
|
1901
|
+
**3. Reusability**
|
|
1902
|
+
```typescript
|
|
1903
|
+
// Use processFile() in different workflows
|
|
1904
|
+
export const webhook = webhook('location-webhook').then(async ctx => {
|
|
1905
|
+
const result = await processFile(s3, parser, mapper, filePath, fileName, log);
|
|
1906
|
+
return { locations: result.locations };
|
|
1907
|
+
});
|
|
1908
|
+
```
|
|
1909
|
+
|
|
1910
|
+
**4. Error Isolation**
|
|
1911
|
+
```typescript
|
|
1912
|
+
// Each function has clear error boundaries
|
|
1913
|
+
try {
|
|
1914
|
+
const processingResult = await processFile(...);
|
|
1915
|
+
// File-level errors caught here
|
|
1916
|
+
} catch (error) {
|
|
1917
|
+
// Handle file processing failure
|
|
1918
|
+
}
|
|
1919
|
+
|
|
1920
|
+
// Mutation errors don't stop file processing
|
|
1921
|
+
const mutationResult = await executeMutations(...);
|
|
1922
|
+
// mutationResult.errors contains per-location failures
|
|
1923
|
+
```
|
|
1924
|
+
|
|
1925
|
+
**5. Progressive Enhancement**
|
|
1926
|
+
```typescript
|
|
1927
|
+
// Easy to add features without touching core logic
|
|
1928
|
+
async function validateLocations(locations: any[]) {
|
|
1929
|
+
// Add validation step
|
|
1930
|
+
}
|
|
1931
|
+
|
|
1932
|
+
const processingResult = await processFile(...);
|
|
1933
|
+
await validateLocations(processingResult.locations); // New step
|
|
1934
|
+
const mutationResult = await executeMutations(...);
|
|
1935
|
+
```
|
|
1936
|
+
|
|
1937
|
+
---
|
|
1938
|
+
|
|
1939
|
+
## Versori Environment Variables
|
|
1940
|
+
|
|
1941
|
+
**Activation Variables:**
|
|
1942
|
+
|
|
1943
|
+
```bash
|
|
1944
|
+
# ============================================================================
|
|
1945
|
+
# Required Variables
|
|
1946
|
+
# ============================================================================
|
|
1947
|
+
s3BucketName=my-location-bucket
|
|
1948
|
+
awsAccessKeyId=AKIAXXXXXXXXXXXX
|
|
1949
|
+
awsSecretAccessKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
|
|
1950
|
+
|
|
1951
|
+
# ============================================================================
|
|
1952
|
+
# S3 Configuration (Optional - with defaults)
|
|
1953
|
+
# ============================================================================
|
|
1954
|
+
awsRegion=us-east-1
|
|
1955
|
+
s3Prefix=locations/
|
|
1956
|
+
archivePrefix=processed/
|
|
1957
|
+
errorPrefix=errors/
|
|
1958
|
+
logPrefix=logs/
|
|
1959
|
+
filePattern=.xml
|
|
1960
|
+
maxFilesToProcess=10
|
|
1961
|
+
|
|
1962
|
+
# ============================================================================
|
|
1963
|
+
# Feature Toggles (Optional - with defaults)
|
|
1964
|
+
# ============================================================================
|
|
1965
|
+
# Enable S3 archival (move files to processed/errors directories)
|
|
1966
|
+
enableArchival=true
|
|
1967
|
+
|
|
1968
|
+
# Enable mutation logs (write detailed mutation results to S3)
|
|
1969
|
+
enableMutationLogs=true
|
|
1970
|
+
|
|
1971
|
+
# Enable file tracking via StateService + KV store
|
|
1972
|
+
# When disabled, relies on S3 archival only for deduplication
|
|
1973
|
+
enableFileTracking=true
|
|
1974
|
+
|
|
1975
|
+
# ============================================================================
|
|
1976
|
+
# Mutation Configuration (Optional - with defaults)
|
|
1977
|
+
# ============================================================================
|
|
1978
|
+
# Mutation batch size (concurrent requests)
|
|
1979
|
+
# - 1 = Sequential (default, safest)
|
|
1980
|
+
# - 5 = Process 5 mutations in parallel
|
|
1981
|
+
# - 10 = Process 10 mutations in parallel
|
|
1982
|
+
mutationBatchSize=1
|
|
1983
|
+
|
|
1984
|
+
# Alias batching (combine multiple mutations into single request)
|
|
1985
|
+
# - undefined = Disabled (default, use separate requests)
|
|
1986
|
+
# - 5 = Combine 5 mutations per aliased request
|
|
1987
|
+
# - 10 = Combine 10 mutations per aliased request
|
|
1988
|
+
mutationsPerAliasBatch=
|
|
1989
|
+
|
|
1990
|
+
# ============================================================================
|
|
1991
|
+
# Fluent Commerce Configuration (Optional)
|
|
1992
|
+
# ============================================================================
|
|
1993
|
+
# Retailer ID - Only if mutation schema requires retailerId in input
|
|
1994
|
+
# Standard createLocation does NOT require this field
|
|
1995
|
+
# Check your GraphQL schema to determine if needed
|
|
1996
|
+
retailerId=my-retailer-id
|
|
1997
|
+
```
|
|
1998
|
+
|
|
1999
|
+
**Notes:**
|
|
2000
|
+
- Webhook security is handled by Versori's native connection authentication. No manual API key configuration needed.
|
|
2001
|
+
- `retailerId` - Standard createLocation does not have this field. Only use if YOUR custom schema requires it.
|
|
2002
|
+
- See `fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md` for details.
|
|
2003
|
+
|
|
2004
|
+
---
|
|
2005
|
+
|
|
2006
|
+
## Schema Validation CLI Commands
|
|
2007
|
+
|
|
2008
|
+
Before deploying, validate your field mappings against the Fluent GraphQL schema:
|
|
2009
|
+
|
|
2010
|
+
### 1. Introspect Schema
|
|
2011
|
+
|
|
2012
|
+
```bash
|
|
2013
|
+
# Generate schema.json from live Fluent API
|
|
2014
|
+
npx fc-connect introspect-schema \
|
|
2015
|
+
--url https://api.fluentcommerce.com/graphql \
|
|
2016
|
+
--client-id your-client-id \
|
|
2017
|
+
--client-secret your-client-secret \
|
|
2018
|
+
--output schema.json
|
|
2019
|
+
```
|
|
2020
|
+
|
|
2021
|
+
### 2. Create Mapping Config File
|
|
2022
|
+
|
|
2023
|
+
**File: location-mapping.json**
|
|
2024
|
+
|
|
2025
|
+
```json
|
|
2026
|
+
{
|
|
2027
|
+
"version": "1.0.0",
|
|
2028
|
+
"mutation": "createLocation",
|
|
2029
|
+
"sourceFormat": "xml",
|
|
2030
|
+
"returnFields": ["id", "ref", "name", "type", "status"],
|
|
2031
|
+
"description": "XML location to Fluent Commerce GraphQL mapping",
|
|
2032
|
+
"fields": {
|
|
2033
|
+
"ref": { "source": "location.@ref", "required": true, "resolver": "sdk.trim" },
|
|
2034
|
+
"name": { "source": "location.name", "required": true, "resolver": "sdk.trim" },
|
|
2035
|
+
"type": { "source": "location.@type", "required": true, "resolver": "sdk.uppercase" },
|
|
2036
|
+
"primaryAddress.ref": { "source": "location.@ref", "required": true },
|
|
2037
|
+
"primaryAddress.street": { "source": "location.address.street1" },
|
|
2038
|
+
"primaryAddress.latitude": {
|
|
2039
|
+
"source": "location.coordinates.@lat",
|
|
2040
|
+
"resolver": "sdk.parseFloat"
|
|
2041
|
+
},
|
|
2042
|
+
"primaryAddress.longitude": {
|
|
2043
|
+
"source": "location.coordinates.@lon",
|
|
2044
|
+
"resolver": "sdk.parseFloat"
|
|
2045
|
+
}
|
|
2046
|
+
}
|
|
2047
|
+
}
|
|
2048
|
+
```
|
|
2049
|
+
|
|
2050
|
+
### 3. Validate Mapping
|
|
2051
|
+
|
|
2052
|
+
```bash
|
|
2053
|
+
# Validate that all target fields exist in schema
|
|
2054
|
+
npx fc-connect validate-schema \
|
|
2055
|
+
--mapping location-mapping.json \
|
|
2056
|
+
--schema schema.json
|
|
2057
|
+
```
|
|
2058
|
+
|
|
2059
|
+
### 4. Analyze Coverage
|
|
2060
|
+
|
|
2061
|
+
```bash
|
|
2062
|
+
# Check which Location fields are mapped vs available
|
|
2063
|
+
npx fc-connect analyze-coverage \
|
|
2064
|
+
--mapping location-mapping.json \
|
|
2065
|
+
--schema schema.json \
|
|
2066
|
+
--type CreateLocationInput
|
|
2067
|
+
```
|
|
2068
|
+
|
|
2069
|
+
**Output:**
|
|
2070
|
+
|
|
2071
|
+
```
|
|
2072
|
+
✅ Mapped: 15/42 fields (35%)
|
|
2073
|
+
❌ Missing required: timezone (String!)
|
|
2074
|
+
⚠️ Optional unmapped: supportPhoneNumber, networkId, attributes
|
|
2075
|
+
```
|
|
2076
|
+
|
|
2077
|
+
---
|
|
2078
|
+
|
|
2079
|
+
## Testing Locally
|
|
2080
|
+
|
|
2081
|
+
### 1. Upload Test XML to S3
|
|
2082
|
+
|
|
2083
|
+
```bash
|
|
2084
|
+
aws s3 cp test-location.xml s3://my-location-bucket/locations/test-location.xml
|
|
2085
|
+
```
|
|
2086
|
+
|
|
2087
|
+
### 2. Deploy to Versori
|
|
2088
|
+
|
|
2089
|
+
```bash
|
|
2090
|
+
npm run deploy
|
|
2091
|
+
```
|
|
2092
|
+
|
|
2093
|
+
### 3. Manual Testing
|
|
2094
|
+
|
|
2095
|
+
```bash
|
|
2096
|
+
# Trigger manual sync (auth handled by Versori connection)
|
|
2097
|
+
curl -X POST https://your-workspace.versori.run/location-xml-adhoc
|
|
2098
|
+
|
|
2099
|
+
# Check job status
|
|
2100
|
+
curl -X POST https://your-workspace.versori.run/location-xml-job-status \
|
|
2101
|
+
-H "Content-Type: application/json" \
|
|
2102
|
+
-d '{"jobId": "location-xml-adhoc-1737525600000"}'
|
|
2103
|
+
```
|
|
2104
|
+
|
|
2105
|
+
### 4. Verify Processing
|
|
2106
|
+
|
|
2107
|
+
- Upload a small XML to S3 (2-3 locations) and trigger `adhoc` webhook
|
|
2108
|
+
- Verify GraphQL mutations are executed for each location
|
|
2109
|
+
- Confirm file moved from `locations/` to `processed/` in S3
|
|
2110
|
+
- Check KV state: errors and last processed metadata
|
|
2111
|
+
- Monitor rate limiting: verify mutations respect configured rate
|
|
2112
|
+
|
|
2113
|
+
---
|
|
2114
|
+
|
|
2115
|
+
## Deployment
|
|
2116
|
+
|
|
2117
|
+
```bash
|
|
2118
|
+
# Deploy to Versori
|
|
2119
|
+
npm run deploy
|
|
2120
|
+
|
|
2121
|
+
# View logs
|
|
2122
|
+
npm run logs
|
|
2123
|
+
|
|
2124
|
+
# Monitor execution
|
|
2125
|
+
versori logs --follow
|
|
2126
|
+
```
|
|
2127
|
+
|
|
2128
|
+
---
|
|
2129
|
+
|
|
2130
|
+
## Monitoring
|
|
2131
|
+
|
|
2132
|
+
### Success Response
|
|
2133
|
+
|
|
2134
|
+
```json
|
|
2135
|
+
{
|
|
2136
|
+
"success": true,
|
|
2137
|
+
"jobId": "location-xml-scheduled-1737525600000",
|
|
2138
|
+
"processed": 3,
|
|
2139
|
+
"skipped": 0,
|
|
2140
|
+
"failed": 0,
|
|
2141
|
+
"totalRecords": 12,
|
|
2142
|
+
"errors": []
|
|
2143
|
+
}
|
|
2144
|
+
```
|
|
2145
|
+
|
|
2146
|
+
### Partial Success Response
|
|
2147
|
+
|
|
2148
|
+
```json
|
|
2149
|
+
{
|
|
2150
|
+
"success": true,
|
|
2151
|
+
"jobId": "location-xml-scheduled-1737525600000",
|
|
2152
|
+
"processed": 3,
|
|
2153
|
+
"skipped": 1,
|
|
2154
|
+
"failed": 0,
|
|
2155
|
+
"totalRecords": 10,
|
|
2156
|
+
"errors": ["locations-003.xml: 2 mapping errors"]
|
|
2157
|
+
}
|
|
2158
|
+
```
|
|
2159
|
+
|
|
2160
|
+
### Error Response
|
|
2161
|
+
|
|
2162
|
+
```json
|
|
2163
|
+
{
|
|
2164
|
+
"success": false,
|
|
2165
|
+
"jobId": "location-xml-scheduled-1737525600000",
|
|
2166
|
+
"processed": 0,
|
|
2167
|
+
"skipped": 0,
|
|
2168
|
+
"failed": 1,
|
|
2169
|
+
"totalRecords": 0,
|
|
2170
|
+
"errors": ["locations-001.xml: Invalid XML structure"]
|
|
2171
|
+
}
|
|
2172
|
+
```
|
|
2173
|
+
|
|
2174
|
+
---
|
|
2175
|
+
|
|
2176
|
+
## Common Pitfalls and Solutions
|
|
2177
|
+
|
|
2178
|
+
### 1. XML Attribute Not Found
|
|
2179
|
+
|
|
2180
|
+
**Symptoms**: Mapping errors like "field not found"
|
|
2181
|
+
|
|
2182
|
+
**Solution**:
|
|
2183
|
+
|
|
2184
|
+
```typescript
|
|
2185
|
+
// ❌ WRONG - Missing @ prefix for attribute
|
|
2186
|
+
ref: {
|
|
2187
|
+
source: 'location.ref';
|
|
2188
|
+
}
|
|
2189
|
+
|
|
2190
|
+
// ✅ CORRECT - Use @ prefix for XML attributes
|
|
2191
|
+
ref: {
|
|
2192
|
+
source: 'location.@ref';
|
|
2193
|
+
}
|
|
2194
|
+
```
|
|
2195
|
+
|
|
2196
|
+
### 2. Single Element Not Array
|
|
2197
|
+
|
|
2198
|
+
**Symptoms**: "Cannot read property forEach of undefined"
|
|
2199
|
+
|
|
2200
|
+
**Solution**:
|
|
2201
|
+
|
|
2202
|
+
```typescript
|
|
2203
|
+
// Always normalize to array
|
|
2204
|
+
const locationsData = xmlData.locations?.location;
|
|
2205
|
+
const locations = Array.isArray(locationsData) ? locationsData : [locationsData];
|
|
2206
|
+
```
|
|
2207
|
+
|
|
2208
|
+
### 3. Rate Limiting Too Aggressive
|
|
2209
|
+
|
|
2210
|
+
**Symptoms**: Slow processing, mutations taking too long
|
|
2211
|
+
|
|
2212
|
+
**Solution**:
|
|
2213
|
+
|
|
2214
|
+
```bash
|
|
2215
|
+
# Increase rate limit (mutations per second)
|
|
2216
|
+
mutationRateLimit=10 # Default is 5
|
|
2217
|
+
```
|
|
2218
|
+
|
|
2219
|
+
### 4. Empty Element vs Missing Element
|
|
2220
|
+
|
|
2221
|
+
**Solution**:
|
|
2222
|
+
|
|
2223
|
+
```typescript
|
|
2224
|
+
// Use required: false and defaultValue for optional fields
|
|
2225
|
+
'primaryAddress.street2': {
|
|
2226
|
+
source: 'location.address.street2',
|
|
2227
|
+
required: false,
|
|
2228
|
+
defaultValue: ''
|
|
2229
|
+
}
|
|
2230
|
+
```
|
|
2231
|
+
|
|
2232
|
+
### 5. S3 Access Denied
|
|
2233
|
+
|
|
2234
|
+
**Symptoms**: S3 operations fail with 403 errors
|
|
2235
|
+
|
|
2236
|
+
**Solution**: Validate IAM permissions
|
|
2237
|
+
|
|
2238
|
+
**Required IAM Permissions:**
|
|
2239
|
+
|
|
2240
|
+
```json
|
|
2241
|
+
{
|
|
2242
|
+
"Version": "2012-10-17",
|
|
2243
|
+
"Statement": [
|
|
2244
|
+
{
|
|
2245
|
+
"Effect": "Allow",
|
|
2246
|
+
"Action": ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
|
|
2247
|
+
"Resource": ["arn:aws:s3:::my-location-bucket", "arn:aws:s3:::my-location-bucket/*"]
|
|
2248
|
+
}
|
|
2249
|
+
]
|
|
2250
|
+
}
|
|
2251
|
+
```
|
|
2252
|
+
|
|
2253
|
+
### 6. GraphQL Schema Mismatch
|
|
2254
|
+
|
|
2255
|
+
**Symptoms**: Mutation errors like "Unknown field", "Invalid input type"
|
|
2256
|
+
|
|
2257
|
+
**Solution**: Use CLI tools to validate mappings
|
|
2258
|
+
|
|
2259
|
+
```bash
|
|
2260
|
+
npx fc-connect validate-schema --mapping location-mapping.json --schema schema.json
|
|
2261
|
+
```
|
|
2262
|
+
|
|
2263
|
+
### 7. Nested Object Mapping Errors
|
|
2264
|
+
|
|
2265
|
+
**Symptoms**: Flat structure instead of nested objects in mutation input
|
|
2266
|
+
|
|
2267
|
+
**Solution**: Use dot notation in field mapping
|
|
2268
|
+
|
|
2269
|
+
```typescript
|
|
2270
|
+
// ✅ CORRECT - Creates nested structure
|
|
2271
|
+
'primaryAddress.city': { source: 'location.address.city' }
|
|
2272
|
+
|
|
2273
|
+
// ❌ WRONG - Creates flat structure
|
|
2274
|
+
primaryAddress_city: { source: 'location.address.city' }
|
|
2275
|
+
```
|
|
2276
|
+
|
|
2277
|
+
### 8. retailerId Configuration Errors
|
|
2278
|
+
|
|
2279
|
+
**Symptoms**: "retailerId is required" errors or confusion about when to use `setRetailerId()`
|
|
2280
|
+
|
|
2281
|
+
**Solution**: Understand the correct pattern
|
|
2282
|
+
|
|
2283
|
+
```typescript
|
|
2284
|
+
// ✅ CORRECT - GraphQL mutations don't need setRetailerId()
|
|
2285
|
+
// Check your GraphQL schema to determine retailerId handling:
|
|
2286
|
+
// - Mandatory retailerId → Must pass it in mutation input
|
|
2287
|
+
// - Optional retailerId → Can pass it if needed
|
|
2288
|
+
// - No retailerId field → Don't pass it
|
|
2289
|
+
// Standard createLocation does not have retailerId field in schema
|
|
2290
|
+
|
|
2291
|
+
// ✅ IF mutation schema requires retailerId (mandatory):
|
|
2292
|
+
const { query, variables } = await mapper.map(location);
|
|
2293
|
+
if (retailerId && variables.input) {
|
|
2294
|
+
variables.input.retailer = { id: parseInt(retailerId) };
|
|
2295
|
+
}
|
|
2296
|
+
await client.graphql({ query, variables });
|
|
2297
|
+
```
|
|
2298
|
+
|
|
2299
|
+
**Reference:** `fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md`
|
|
2300
|
+
|
|
2301
|
+
---
|
|
2302
|
+
|
|
2303
|
+
## Key Takeaways
|
|
2304
|
+
|
|
2305
|
+
### Architecture Patterns
|
|
2306
|
+
|
|
2307
|
+
- **Service Function Composition**: Workflow broken into 3 service functions - `processFile()`, `executeMutations()`, `writeMutationLog()`
|
|
2308
|
+
- **Per-File Processing**: Main workflow orchestrates service functions for each file
|
|
2309
|
+
- **Clear Separation of Concerns**: Parse/map, execute mutations, logging are independent functions
|
|
2310
|
+
|
|
2311
|
+
### SDK Usage
|
|
2312
|
+
|
|
2313
|
+
- **Buffer Import (CRITICAL)**: Always `import { Buffer } from 'node:buffer'` for Deno/Versori runtime
|
|
2314
|
+
- **S3 Archival Deduplication**: Use `s3.moveFile()` to `processed/` subdirectory - PRIMARY deduplication mechanism
|
|
2315
|
+
- **NO VersoriFileTracker for S3**: S3 archival is simpler and more reliable
|
|
2316
|
+
- **StateService Role**: SECONDARY - provides metadata/history, not primary deduplication
|
|
2317
|
+
|
|
2318
|
+
### XML Processing
|
|
2319
|
+
|
|
2320
|
+
- **XML @ Prefix**: Always use `@` prefix for XML attributes (`location.@ref`)
|
|
2321
|
+
- **Array Normalization (CRITICAL)**: Handle single vs multiple elements with `Array.isArray()` check - single `<location>` becomes object, not array
|
|
2322
|
+
- **Nested Mapping**: Use dot notation for nested objects (`primaryAddress.street`, `openingSchedule.monStart`)
|
|
2323
|
+
|
|
2324
|
+
### GraphQL Mutations
|
|
2325
|
+
|
|
2326
|
+
- **NO setRetailerId() Required**: GraphQL mutations do NOT need `client.setRetailerId()` - only Job/Event API needs it
|
|
2327
|
+
- **retailerId in Input**: Check your GraphQL schema to determine retailerId handling:
|
|
2328
|
+
- **Mandatory retailerId** - Field exists and is required (`!`) → Must pass it
|
|
2329
|
+
- **Optional retailerId** - Field exists and is optional → Can pass it if needed
|
|
2330
|
+
- **No retailerId field** - Field doesn't exist → Don't pass it
|
|
2331
|
+
- **Rate Limiting**: Implement configurable delays between mutations to avoid API throttling
|
|
2332
|
+
- **Retry Logic**: Exponential backoff for failed mutations with `retryWithBackoff()`
|
|
2333
|
+
- **Direct Mutations**: Use `client.graphql()` for location upserts (NOT Batch API)
|
|
2334
|
+
- **GraphQL vs Batch API**: Use GraphQL for low-volume master data, Batch API for high-volume inventory
|
|
2335
|
+
|
|
2336
|
+
### Error Handling & Monitoring
|
|
2337
|
+
|
|
2338
|
+
- **Error Recovery**: Exponential backoff for error state tracking with retry timestamps
|
|
2339
|
+
- **Mutation Logging**: Optional S3 JSON logs with detailed per-location mutation status
|
|
2340
|
+
- **Schema Validation**: Use CLI tools before deployment to catch mapping errors
|
|
2341
|
+
- **Archival Order**: Archive FIRST (deduplication), then KV tracking (metadata)
|
|
2342
|
+
|
|
2343
|
+
---
|
|
2344
|
+
|
|
2345
|
+
## Related Documentation
|
|
2346
|
+
|
|
2347
|
+
### Core Guides
|
|
2348
|
+
|
|
2349
|
+
- **GraphQL Mutation Mapping**: `fc-connect-sdk/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/readme.md`
|
|
2350
|
+
- **Universal Mapping**: `fc-connect-sdk/docs/02-CORE-GUIDES/mapping/modules/readme.md`
|
|
2351
|
+
- **XML Parser**: `fc-connect-sdk/docs/02-CORE-GUIDES/parsers/modules/05-xml-parser.md`
|
|
2352
|
+
- **Data Sources**: `fc-connect-sdk/docs/02-CORE-GUIDES/data-sources/readme.md`
|
|
2353
|
+
- **State Management**: `fc-connect-sdk/docs/03-PATTERN-GUIDES/file-operations/state-duplicate-prevention.md`
|
|
2354
|
+
|
|
2355
|
+
### Related Templates
|
|
2356
|
+
|
|
2357
|
+
- **CSV Version**: `template-ingestion-s3-csv-location-graphql.md`
|
|
2358
|
+
- **JSON Version**: `template-ingestion-s3-json-location-graphql.md`
|
|
2359
|
+
- **SFTP XML Version**: `template-ingestion-sftp-xml-location-graphql.md`
|
|
2360
|
+
- **Event API Pattern**: `../event-api/template-ingestion-s3-xml-product-event.md`
|
|
2361
|
+
- **Batch API Pattern**: `../batch-api/template-ingestion-s3-xml-inventory-batch.md`
|
|
2362
|
+
|
|
2363
|
+
### CLI Tools
|
|
2364
|
+
|
|
2365
|
+
- **Schema Introspection**: `fc-connect-sdk/bin/readme.md#introspect-schema`
|
|
2366
|
+
- **Mapping Validation**: `fc-connect-sdk/bin/readme.md#validate-schema`
|
|
2367
|
+
- **Coverage Analysis**: `fc-connect-sdk/bin/readme.md#analyze-coverage`
|
|
2368
|
+
|
|
2369
|
+
### Patterns
|
|
2370
|
+
|
|
2371
|
+
- **Error Handling**: `fc-connect-sdk/docs/01-TEMPLATES/patterns/error-handling-retry.md`
|
|
2372
|
+
- **Rate Limiting**: `fc-connect-sdk/docs/03-PATTERN-GUIDES/integration-patterns/rate-limiting.md`
|
|
2373
|
+
- **XML Patterns**: `fc-connect-sdk/docs/01-TEMPLATES/versori/patterns/xml-response-patterns.md`
|