@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.56
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +12 -0
- package/README.md +11 -0
- package/dist/cjs/clients/fluent-client.js +13 -6
- package/dist/cjs/utils/pagination-helpers.js +38 -2
- package/dist/cjs/versori/fluent-versori-client.js +11 -5
- package/dist/esm/clients/fluent-client.js +13 -6
- package/dist/esm/utils/pagination-helpers.js +38 -2
- package/dist/esm/versori/fluent-versori-client.js +11 -5
- package/dist/tsconfig.esm.tsbuildinfo +1 -1
- package/dist/tsconfig.tsbuildinfo +1 -1
- package/dist/tsconfig.types.tsbuildinfo +1 -1
- package/docs/00-START-HERE/EXPORT-VALIDATION.md +158 -158
- package/docs/00-START-HERE/cli-analyze-source-structure-guide.md +655 -655
- package/docs/00-START-HERE/cli-documentation-index.md +202 -202
- package/docs/00-START-HERE/cli-quick-reference.md +252 -252
- package/docs/00-START-HERE/decision-tree.md +552 -552
- package/docs/00-START-HERE/getting-started.md +1070 -1070
- package/docs/00-START-HERE/mapper-quick-decision-guide.md +235 -235
- package/docs/00-START-HERE/readme.md +237 -237
- package/docs/00-START-HERE/retailerid-configuration.md +404 -404
- package/docs/00-START-HERE/sdk-philosophy.md +794 -794
- package/docs/00-START-HERE/troubleshooting-quick-reference.md +1086 -1086
- package/docs/01-TEMPLATES/faq.md +686 -686
- package/docs/01-TEMPLATES/patterns/pattern-templates-guide.md +68 -68
- package/docs/01-TEMPLATES/patterns/patterns-csv-schema-validation-and-rejection-report.md +233 -233
- package/docs/01-TEMPLATES/patterns/patterns-custom-resolvers.md +407 -407
- package/docs/01-TEMPLATES/patterns/patterns-error-handling-retry.md +511 -511
- package/docs/01-TEMPLATES/patterns/patterns-field-mapping-universal.md +701 -701
- package/docs/01-TEMPLATES/patterns/patterns-large-file-splitting.md +1430 -1430
- package/docs/01-TEMPLATES/patterns/patterns-master-data-etl.md +2399 -2399
- package/docs/01-TEMPLATES/patterns/patterns-pagination-streaming.md +447 -447
- package/docs/01-TEMPLATES/patterns/patterns-state-duplicate-prevention.md +385 -385
- package/docs/01-TEMPLATES/readme.md +957 -957
- package/docs/01-TEMPLATES/standalone/standalone-asn-inbound-processing.md +1209 -1209
- package/docs/01-TEMPLATES/standalone/standalone-graphql-query-export.md +1140 -1140
- package/docs/01-TEMPLATES/standalone/standalone-graphql-to-parquet-partitioned-s3.md +432 -432
- package/docs/01-TEMPLATES/standalone/standalone-multi-channel-inventory-sync.md +1185 -1185
- package/docs/01-TEMPLATES/standalone/standalone-multi-source-aggregation.md +1462 -1462
- package/docs/01-TEMPLATES/standalone/standalone-s3-csv-batch-api.md +1390 -1390
- package/docs/01-TEMPLATES/standalone/standalone-s3-csv-inventory-to-batch.md +330 -330
- package/docs/01-TEMPLATES/standalone/standalone-scripts-guide.md +87 -87
- package/docs/01-TEMPLATES/standalone/standalone-sftp-xml-graphql.md +1444 -1444
- package/docs/01-TEMPLATES/standalone/standalone-webhook-payload-processing.md +688 -688
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-dropship-order-routing.md +193 -193
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-graphql-parquet-extraction.md +518 -518
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-inter-location-transfers.md +2162 -2162
- package/docs/01-TEMPLATES/versori/business-examples/business-examples-pre-order-allocation.md +2226 -2226
- package/docs/01-TEMPLATES/versori/business-examples/business-scenarios-guide.md +87 -87
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-connection-validation-pattern.md +656 -656
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-dual-workflow-connector.md +835 -835
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-guide.md +108 -108
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-kv-state-management.md +1533 -1533
- package/docs/01-TEMPLATES/versori/patterns/versori-patterns-xml-response-patterns.md +1160 -1160
- package/docs/01-TEMPLATES/versori/versori-platform-guide.md +201 -201
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-asn-purchase-order.md +1906 -1906
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-dropship-routing.md +1074 -1074
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-flash-sale-reserve.md +1395 -1395
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-generic-xml-order.md +888 -888
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-payment-gateway-integration.md +2478 -2478
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-rma-returns-comprehensive.md +2240 -2240
- package/docs/01-TEMPLATES/versori/webhooks/template-webhook-xml-order-ingestion.md +2029 -2029
- package/docs/01-TEMPLATES/versori/webhooks/webhook-templates-guide.md +140 -140
- package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/inventory-mapping.json +20 -20
- package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/products_2025-01-22.csv +11 -11
- package/docs/01-TEMPLATES/versori/workflows/_examples/sample-data/sample-data-guide.md +34 -34
- package/docs/01-TEMPLATES/versori/workflows/_examples/workflow-examples-guide.md +36 -36
- package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-modes-guide.md +1038 -1038
- package/docs/01-TEMPLATES/versori/workflows/extraction/extraction-workflows-guide.md +138 -138
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/graphql-extraction-guide.md +63 -63
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-csv.md +2062 -2062
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-xml.md +2294 -2294
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-s3-csv.md +2461 -2461
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-positions-to-sftp-xml.md +2529 -2529
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-csv.md +2464 -2464
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-inventory-quantities-to-s3-json.md +1959 -1959
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-s3-csv.md +1953 -1953
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-sftp-xml.md +2541 -2541
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-s3-json.md +2384 -2384
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-sftp-xml.md +2445 -2445
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-csv.md +2355 -2355
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-json.md +2042 -2042
- package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-sftp-xml.md +2726 -2726
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/batch-api-guide.md +206 -206
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-cycle-count-reconciliation.md +2030 -2030
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-multi-channel-inventory-sync.md +1882 -1882
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-csv-inventory-batch.md +2827 -2827
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-json-inventory-batch.md +1952 -1952
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-xml-inventory-batch.md +3289 -3289
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-csv-inventory-batch.md +3064 -3064
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-json-inventory-batch.md +3238 -3238
- package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-sftp-xml-inventory-batch.md +2977 -2977
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/event-api-guide.md +321 -321
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-json-order-cancel-event.md +959 -959
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-payload-xml-order-cancel-event.md +1170 -1170
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-csv-product-event.md +2312 -2312
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-json-product-event.md +2999 -2999
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-parquet-product-event.md +2836 -2836
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-xml-product-event.md +2395 -2395
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-csv-product-event.md +2295 -2295
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-json-product-event.md +2602 -2602
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-parquet-product-event.md +2589 -2589
- package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-xml-product-event.md +3578 -3578
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/graphql-mutations-guide.md +93 -93
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-json-order-update-graphql.md +1260 -1260
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-payload-xml-order-update-graphql.md +1472 -1472
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-control-graphql.md +2417 -2417
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-location-graphql.md +2811 -2811
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-price-graphql.md +2619 -2619
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-json-location-graphql.md +2807 -2807
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-xml-location-graphql.md +2373 -2373
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-control-graphql.md +2740 -2740
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-csv-location-graphql.md +2760 -2760
- package/docs/01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-sftp-json-location-graphql.md +1710 -1710
- package/docs/01-TEMPLATES/versori/workflows/ingestion/ingestion-workflows-guide.md +136 -136
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/rubix-webhooks-guide.md +520 -520
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-inline.md +1418 -1418
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-fulfilment-to-sftp-xml-universal-mapper.md +1785 -1785
- package/docs/01-TEMPLATES/versori/workflows/rubix-webhooks/template-webhook-rubix-order-attribute-update.md +824 -824
- package/docs/01-TEMPLATES/versori/workflows/workflows-overview-guide.md +646 -646
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-batch-archival.md +724 -724
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-job-tracker.md +627 -627
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-partial-batch-recovery.md +561 -561
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-quick-reference.md +367 -367
- package/docs/02-CORE-GUIDES/advanced-services/advanced-services-readme.md +407 -407
- package/docs/02-CORE-GUIDES/advanced-services/readme.md +49 -49
- package/docs/02-CORE-GUIDES/api-reference/api-reference-quick-reference.md +548 -548
- package/docs/02-CORE-GUIDES/api-reference/event-api-input-output-reference.md +702 -1171
- package/docs/02-CORE-GUIDES/api-reference/examples/client-initialization.ts +286 -286
- package/docs/02-CORE-GUIDES/api-reference/graphql-error-classification.md +337 -337
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-01-client-api.md +399 -520
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-03-authentication.md +199 -199
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-04-graphql-mapping.md +925 -925
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-05-services.md +1198 -1198
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-06-data-sources.md +1083 -1083
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-07-parsers.md +1097 -1097
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-pagination.md +513 -513
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-08-types.md +545 -597
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-error-handling.md +527 -527
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-09-webhook-validation.md +514 -514
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-extraction.md +557 -557
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-10-utilities.md +412 -412
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-cli-tools.md +423 -423
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-11-error-handling.md +716 -716
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-analyze-source-structure.md +518 -518
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-partial-responses.md +212 -212
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-12-testing.md +300 -300
- package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-13-resolver-builder.md +322 -322
- package/docs/02-CORE-GUIDES/api-reference/readme.md +279 -279
- package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-quick-reference.md +351 -351
- package/docs/02-CORE-GUIDES/auto-pagination/auto-pagination-readme.md +277 -277
- package/docs/02-CORE-GUIDES/auto-pagination/examples/auto-pagination-readme.md +178 -178
- package/docs/02-CORE-GUIDES/auto-pagination/examples/common-patterns.ts +351 -351
- package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-products.ts +384 -384
- package/docs/02-CORE-GUIDES/auto-pagination/examples/paginate-virtual-positions.ts +308 -308
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-01-foundations.md +470 -470
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-02-quick-start.md +713 -713
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-03-configuration.md +754 -754
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-04-advanced-patterns.md +732 -732
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-05-sdk-integration.md +847 -847
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-06-troubleshooting.md +359 -359
- package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-07-api-reference.md +462 -462
- package/docs/02-CORE-GUIDES/auto-pagination/readme.md +54 -54
- package/docs/02-CORE-GUIDES/data-sources/data-sources-file-operations-error-handling.md +1487 -1487
- package/docs/02-CORE-GUIDES/data-sources/data-sources-quick-reference.md +836 -836
- package/docs/02-CORE-GUIDES/data-sources/data-sources-readme.md +276 -276
- package/docs/02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md +553 -553
- package/docs/02-CORE-GUIDES/data-sources/examples/common-patterns.ts +409 -409
- package/docs/02-CORE-GUIDES/data-sources/examples/data-sources-readme.md +178 -178
- package/docs/02-CORE-GUIDES/data-sources/examples/s3-operations.ts +308 -308
- package/docs/02-CORE-GUIDES/data-sources/examples/sftp-operations.ts +371 -371
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-01-foundations.md +735 -735
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-02-s3-operations.md +1302 -1302
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-03-sftp-operations.md +1379 -1379
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-04-file-patterns.md +941 -941
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-05-advanced-topics.md +813 -813
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-06-integration-patterns.md +486 -486
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-07-troubleshooting.md +387 -387
- package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-08-api-reference.md +417 -417
- package/docs/02-CORE-GUIDES/data-sources/readme.md +77 -77
- package/docs/02-CORE-GUIDES/error-handling-guide.md +936 -936
- package/docs/02-CORE-GUIDES/extraction/examples/02-core-guides-extraction-readme.md +116 -116
- package/docs/02-CORE-GUIDES/extraction/examples/common-patterns.ts +428 -428
- package/docs/02-CORE-GUIDES/extraction/examples/extract-inventory-basic.ts +187 -187
- package/docs/02-CORE-GUIDES/extraction/extraction-quick-reference.md +596 -596
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-01-foundations.md +514 -514
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-02-basic-extraction.md +823 -823
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-03-parquet-processing.md +507 -507
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-04-data-enrichment.md +546 -546
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-05-transformation.md +494 -494
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-export-formats.md +458 -458
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-06-performance.md +138 -138
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-api-reference.md +148 -148
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-07-optimization.md +692 -692
- package/docs/02-CORE-GUIDES/extraction/modules/02-core-guides-extraction-08-extraction-orchestrator.md +1008 -1008
- package/docs/02-CORE-GUIDES/extraction/readme.md +151 -151
- package/docs/02-CORE-GUIDES/ingestion/examples/_simple-kv-store.ts +40 -40
- package/docs/02-CORE-GUIDES/ingestion/examples/error-recovery.ts +728 -728
- package/docs/02-CORE-GUIDES/ingestion/examples/event-driven.ts +501 -501
- package/docs/02-CORE-GUIDES/ingestion/examples/local-file-ingestion.ts +88 -88
- package/docs/02-CORE-GUIDES/ingestion/examples/parquet-ingestion.ts +117 -117
- package/docs/02-CORE-GUIDES/ingestion/examples/performance-optimized.ts +647 -647
- package/docs/02-CORE-GUIDES/ingestion/examples/s3-csv-ingestion.ts +169 -169
- package/docs/02-CORE-GUIDES/ingestion/examples/sftp-csv-ingestion.ts +134 -134
- package/docs/02-CORE-GUIDES/ingestion/ingestion-quick-reference.md +546 -546
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-01-introduction.md +626 -626
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-02-quick-start.md +658 -658
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-03-data-sources.md +1052 -1052
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-04-field-mapping.md +763 -763
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-05-advanced-parsers.md +676 -676
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-06-batch-api.md +1295 -1295
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-api-reference.md +138 -138
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-state-management.md +1037 -1037
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-08-performance-optimization.md +1349 -1349
- package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-09-best-practices.md +1893 -1893
- package/docs/02-CORE-GUIDES/ingestion/readme.md +160 -160
- package/docs/02-CORE-GUIDES/logging-guide.md +585 -585
- package/docs/02-CORE-GUIDES/mapping/error-handling-patterns.md +401 -401
- package/docs/02-CORE-GUIDES/mapping/examples/02-core-guides-mapping-readme.md +128 -128
- package/docs/02-CORE-GUIDES/mapping/examples/common-patterns.ts +273 -273
- package/docs/02-CORE-GUIDES/mapping/examples/csv-location-ingestion.json +36 -36
- package/docs/02-CORE-GUIDES/mapping/examples/csv-mapping.ts +242 -242
- package/docs/02-CORE-GUIDES/mapping/examples/graphql-to-parquet-extraction.json +36 -36
- package/docs/02-CORE-GUIDES/mapping/examples/json-mapping.ts +213 -213
- package/docs/02-CORE-GUIDES/mapping/examples/json-product-to-mutation.json +48 -48
- package/docs/02-CORE-GUIDES/mapping/examples/xml-mapping.ts +291 -291
- package/docs/02-CORE-GUIDES/mapping/examples/xml-order-to-mutation.json +45 -45
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-quick-reference.md +463 -463
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/graphql-mutation-mapping-readme.md +227 -227
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-01-introduction.md +222 -222
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-02-quick-start.md +351 -351
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-03-schema-validation.md +569 -569
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-04-mapping-patterns.md +471 -471
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-05-configuration-reference.md +611 -611
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-advanced-xpath.md +148 -148
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-06-path-syntax.md +464 -464
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-api-reference.md +94 -94
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-07-array-handling.md +307 -307
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-08-custom-resolvers.md +544 -544
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-09-advanced-patterns.md +427 -427
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-10-hooks-and-variables.md +336 -336
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-11-error-handling.md +488 -488
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-12-arguments-vs-nodes.md +383 -383
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/modules/graphql-mutation-mapping-13-best-practices.md +477 -477
- package/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/readme.md +62 -62
- package/docs/02-CORE-GUIDES/mapping/mapping-format-decision-tree.md +480 -480
- package/docs/02-CORE-GUIDES/mapping/mapping-graphql-alias-batching-guide.md +820 -820
- package/docs/02-CORE-GUIDES/mapping/mapping-javascript-objects.md +2369 -2369
- package/docs/02-CORE-GUIDES/mapping/mapping-mapper-comparison-guide.md +682 -682
- package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-07-api-reference.md +1327 -1327
- package/docs/02-CORE-GUIDES/mapping/modules/02-core-guides-mapping-08-error-handling.md +1142 -1142
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-04-use-cases.md +891 -891
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-helpers-resolvers.md +1126 -1126
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-06-sdk-resolvers.md +199 -199
- package/docs/02-CORE-GUIDES/mapping/modules/mapping-07-api-reference.md +1319 -1319
- package/docs/02-CORE-GUIDES/mapping/readme.md +178 -178
- package/docs/02-CORE-GUIDES/mapping/resolver-registration.md +410 -410
- package/docs/02-CORE-GUIDES/mapping/resolvers/examples/common-patterns.ts +226 -226
- package/docs/02-CORE-GUIDES/mapping/resolvers/examples/custom-resolvers.ts +227 -227
- package/docs/02-CORE-GUIDES/mapping/resolvers/examples/sdk-resolvers-usage.ts +203 -203
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-readme.md +274 -274
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-api-reference.md +679 -679
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-cookbook.md +826 -826
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-guide.md +1330 -1330
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-helpers-reference.md +1437 -1437
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-parameters-reference.md +553 -553
- package/docs/02-CORE-GUIDES/mapping/resolvers/mapping-resolvers-resolver-troubleshooting.md +854 -854
- package/docs/02-CORE-GUIDES/mapping/resolvers/readme.md +75 -75
- package/docs/02-CORE-GUIDES/parsers/examples/02-core-guides-parsers-readme.md +161 -161
- package/docs/02-CORE-GUIDES/parsers/examples/csv-parser-examples.ts +110 -110
- package/docs/02-CORE-GUIDES/parsers/examples/json-parser-examples.ts +33 -33
- package/docs/02-CORE-GUIDES/parsers/examples/parquet-parser-examples.ts +47 -47
- package/docs/02-CORE-GUIDES/parsers/examples/xml-parser-examples.ts +38 -38
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-01-foundations.md +355 -355
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-02-csv-parser.md +772 -772
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-03-json-parser.md +789 -789
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-04-xml-parser.md +857 -857
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-05-parquet-parser.md +603 -603
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-integration-patterns.md +702 -702
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-06-streaming.md +121 -121
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-api-reference.md +89 -89
- package/docs/02-CORE-GUIDES/parsers/modules/02-core-guides-parsers-07-troubleshooting.md +727 -727
- package/docs/02-CORE-GUIDES/parsers/parsers-quick-reference.md +482 -482
- package/docs/02-CORE-GUIDES/parsers/parsers-readme.md +258 -258
- package/docs/02-CORE-GUIDES/parsers/readme.md +65 -65
- package/docs/02-CORE-GUIDES/readme.md +194 -194
- package/docs/02-CORE-GUIDES/webhook-validation/examples/basic-validation.ts +108 -108
- package/docs/02-CORE-GUIDES/webhook-validation/examples/common-patterns.ts +316 -316
- package/docs/02-CORE-GUIDES/webhook-validation/examples/webhook-validation-readme.md +61 -61
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-01-foundations.md +440 -440
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-02-quick-start.md +525 -525
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-03-versori-integration.md +741 -741
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-04-platform-integration.md +629 -629
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-05-configuration.md +535 -535
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-error-handling.md +611 -611
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-06-troubleshooting.md +124 -124
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-07-api-reference.md +511 -511
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-08-rubix-webhooks.md +590 -590
- package/docs/02-CORE-GUIDES/webhook-validation/modules/webhook-validation-09-rubix-event-vs-http-call.md +432 -432
- package/docs/02-CORE-GUIDES/webhook-validation/readme.md +239 -239
- package/docs/02-CORE-GUIDES/webhook-validation/webhook-validation-quick-reference.md +392 -392
- package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-quick-reference.md +498 -498
- package/docs/03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-readme.md +313 -313
- package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/common-patterns.ts +612 -612
- package/docs/03-PATTERN-GUIDES/connector-scenarios/examples/connector-scenarios-readme.md +253 -253
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-01-foundations.md +452 -452
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-02-simple-scenarios.md +681 -681
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-03-intermediate-scenarios.md +637 -637
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-04-advanced-scenarios.md +650 -650
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-05-bidirectional-sync.md +233 -233
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-06-production-patterns.md +442 -442
- package/docs/03-PATTERN-GUIDES/connector-scenarios/modules/connector-scenarios-07-reference.md +445 -445
- package/docs/03-PATTERN-GUIDES/connector-scenarios/readme.md +31 -31
- package/docs/03-PATTERN-GUIDES/enterprise-integration-patterns.md +1528 -1528
- package/docs/03-PATTERN-GUIDES/error-handling/comprehensive-error-handling-guide.md +1437 -1437
- package/docs/03-PATTERN-GUIDES/error-handling/error-handling-quick-reference.md +390 -390
- package/docs/03-PATTERN-GUIDES/error-handling/examples/common-patterns.ts +438 -438
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-01-foundations.md +362 -362
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-02-error-types.md +850 -850
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-03-utf8-handling.md +456 -456
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-04-error-scenarios.md +658 -658
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-05-calling-patterns.md +671 -671
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-06-retry-strategies.md +1034 -1034
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-07-monitoring.md +653 -653
- package/docs/03-PATTERN-GUIDES/error-handling/modules/error-handling-08-api-reference.md +847 -847
- package/docs/03-PATTERN-GUIDES/error-handling/readme.md +36 -36
- package/docs/03-PATTERN-GUIDES/examples/__tests__/readme.md +40 -40
- package/docs/03-PATTERN-GUIDES/examples/__tests__/resolver-examples.test.js +282 -282
- package/docs/03-PATTERN-GUIDES/examples/test-data/03-pattern-guides-readme.md +110 -110
- package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-inventory.json +123 -123
- package/docs/03-PATTERN-GUIDES/examples/test-data/canonical-order.json +171 -171
- package/docs/03-PATTERN-GUIDES/examples/test-data/readme.md +28 -28
- package/docs/03-PATTERN-GUIDES/extraction/extraction-readme.md +15 -15
- package/docs/03-PATTERN-GUIDES/extraction/readme.md +25 -25
- package/docs/03-PATTERN-GUIDES/file-operations/examples/common-patterns.ts +407 -407
- package/docs/03-PATTERN-GUIDES/file-operations/examples/file-operations-readme.md +142 -142
- package/docs/03-PATTERN-GUIDES/file-operations/file-operations-quick-reference.md +462 -462
- package/docs/03-PATTERN-GUIDES/file-operations/file-operations-readme.md +379 -379
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-01-foundations.md +430 -430
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-02-quick-start.md +484 -484
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-03-s3-operations.md +507 -507
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-04-sftp-operations.md +963 -963
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-05-streaming-performance.md +503 -503
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-archive-patterns.md +386 -386
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-06-error-handling.md +117 -117
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-api-reference.md +78 -78
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-testing-troubleshooting.md +567 -567
- package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-08-api-reference.md +1055 -1055
- package/docs/03-PATTERN-GUIDES/file-operations/readme.md +32 -32
- package/docs/03-PATTERN-GUIDES/ingestion/ingestion-readme.md +15 -15
- package/docs/03-PATTERN-GUIDES/ingestion/readme.md +25 -25
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/batch-processing.ts +130 -130
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/common-patterns.ts +360 -360
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/delta-sync.ts +130 -130
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/integration-patterns-readme.md +100 -100
- package/docs/03-PATTERN-GUIDES/integration-patterns/examples/real-time-webhook.ts +398 -398
- package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-quick-reference.md +962 -962
- package/docs/03-PATTERN-GUIDES/integration-patterns/integration-patterns-readme.md +134 -134
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-01-real-time-processing.md +991 -991
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-02-batch-processing.md +1547 -1547
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-03-delta-sync.md +1108 -1108
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-04-webhook-patterns.md +1181 -1181
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-05-error-handling.md +1061 -1061
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-advanced-integration-services.md +1547 -1547
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-06-performance.md +109 -109
- package/docs/03-PATTERN-GUIDES/integration-patterns/modules/integration-patterns-07-api-reference.md +34 -34
- package/docs/03-PATTERN-GUIDES/integration-patterns/readme.md +30 -30
- package/docs/03-PATTERN-GUIDES/logging-minimal-mode.md +128 -128
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/common-patterns.ts +380 -380
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/multiple-connections-readme.md +139 -139
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/parallel-root-connections.ts +149 -149
- package/docs/03-PATTERN-GUIDES/multiple-connections/examples/real-world-scenarios.ts +405 -405
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-01-foundations.md +378 -378
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-02-quick-start.md +566 -566
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-03-targeting-connections.md +659 -659
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-04-parallel-queries.md +656 -656
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-05-best-practices.md +624 -624
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-api-reference.md +824 -824
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-06-versori.md +119 -119
- package/docs/03-PATTERN-GUIDES/multiple-connections/modules/multiple-connections-07-api-reference.md +87 -87
- package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-quick-reference.md +353 -353
- package/docs/03-PATTERN-GUIDES/multiple-connections/multiple-connections-readme.md +270 -270
- package/docs/03-PATTERN-GUIDES/multiple-connections/readme.md +30 -30
- package/docs/03-PATTERN-GUIDES/pagination/pagination-readme.md +14 -14
- package/docs/03-PATTERN-GUIDES/pagination/readme.md +24 -24
- package/docs/03-PATTERN-GUIDES/parquet/examples/common-patterns.ts +180 -180
- package/docs/03-PATTERN-GUIDES/parquet/examples/read-parquet.ts +48 -48
- package/docs/03-PATTERN-GUIDES/parquet/examples/write-parquet.ts +65 -65
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-01-introduction.md +393 -393
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-02-quick-start.md +572 -572
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-03-reading-parquet.md +525 -525
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-04-writing-parquet.md +554 -554
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-05-graphql-extraction.md +405 -405
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-performance.md +104 -104
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-06-s3-integration.md +511 -511
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-api-reference.md +90 -90
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-07-performance-optimization.md +525 -525
- package/docs/03-PATTERN-GUIDES/parquet/modules/03-pattern-guides-parquet-08-best-practices.md +712 -712
- package/docs/03-PATTERN-GUIDES/parquet/parquet-quick-reference.md +683 -683
- package/docs/03-PATTERN-GUIDES/parquet/parquet-readme.md +248 -248
- package/docs/03-PATTERN-GUIDES/parquet/readme.md +32 -32
- package/docs/03-PATTERN-GUIDES/parsers/parsers-readme.md +12 -12
- package/docs/03-PATTERN-GUIDES/parsers/readme.md +24 -24
- package/docs/03-PATTERN-GUIDES/readme.md +159 -159
- package/docs/03-PATTERN-GUIDES/webhooks/readme.md +24 -24
- package/docs/03-PATTERN-GUIDES/webhooks/webhooks-readme.md +8 -8
- package/docs/04-REFERENCE/architecture/architecture-01-overview.md +427 -427
- package/docs/04-REFERENCE/architecture/architecture-02-client-architecture.md +424 -424
- package/docs/04-REFERENCE/architecture/architecture-03-data-flow.md +690 -690
- package/docs/04-REFERENCE/architecture/architecture-04-service-layer.md +834 -834
- package/docs/04-REFERENCE/architecture/architecture-05-integration-architecture.md +655 -655
- package/docs/04-REFERENCE/architecture/architecture-06-state-management.md +653 -653
- package/docs/04-REFERENCE/architecture/architecture-adding-new-data-sources.md +686 -686
- package/docs/04-REFERENCE/architecture/readme.md +279 -279
- package/docs/04-REFERENCE/platforms/deno/readme.md +117 -117
- package/docs/04-REFERENCE/platforms/nodejs/readme.md +146 -146
- package/docs/04-REFERENCE/platforms/readme.md +135 -135
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-01-introduction.md +398 -398
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-02-quick-start.md +560 -560
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-03-authentication.md +757 -757
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-04-workflows.md +2476 -2476
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-05-connections.md +1167 -1167
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-kv-storage.md +990 -990
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-06-state-management.md +121 -121
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-api-reference.md +68 -68
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-07-deployment.md +731 -731
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-08-best-practices.md +1111 -1111
- package/docs/04-REFERENCE/platforms/versori/modules/platforms-versori-09-signature-reference.md +766 -766
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-readme.md +299 -299
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-s3-sftp-configuration-guide.md +1425 -1425
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-api-key-security.md +816 -816
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-webhook-connection-security.md +681 -681
- package/docs/04-REFERENCE/platforms/versori/platforms-versori-workflow-task-types.md +708 -708
- package/docs/04-REFERENCE/platforms/versori/readme.md +108 -108
- package/docs/04-REFERENCE/readme.md +148 -148
- package/docs/04-REFERENCE/resolver-signature/examples/advanced-resolvers.ts +482 -482
- package/docs/04-REFERENCE/resolver-signature/examples/async-resolvers.ts +496 -496
- package/docs/04-REFERENCE/resolver-signature/examples/basic-resolvers.ts +343 -343
- package/docs/04-REFERENCE/resolver-signature/examples/resolver-signature-readme.md +188 -188
- package/docs/04-REFERENCE/resolver-signature/examples/testing-resolvers.ts +463 -463
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-01-foundations.md +286 -286
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-02-parameter-reference.md +643 -643
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-03-basic-examples.md +521 -521
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-04-advanced-patterns.md +739 -739
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-05-sdk-resolvers.md +531 -531
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-migration-guide.md +650 -650
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-06-testing.md +125 -125
- package/docs/04-REFERENCE/resolver-signature/modules/resolver-signature-07-api-reference.md +794 -794
- package/docs/04-REFERENCE/resolver-signature/readme.md +64 -64
- package/docs/04-REFERENCE/resolver-signature/resolver-signature-quick-reference.md +270 -270
- package/docs/04-REFERENCE/resolver-signature/resolver-signature-readme.md +351 -351
- package/docs/04-REFERENCE/schema/fluent-commerce-schema.json +764 -764
- package/docs/04-REFERENCE/schema/readme.md +141 -141
- package/docs/04-REFERENCE/testing/examples/04-reference-testing-readme.md +158 -158
- package/docs/04-REFERENCE/testing/examples/fluent-testing.ts +62 -62
- package/docs/04-REFERENCE/testing/examples/health-check.ts +155 -155
- package/docs/04-REFERENCE/testing/examples/integration-test.ts +119 -119
- package/docs/04-REFERENCE/testing/examples/performance-test.ts +183 -183
- package/docs/04-REFERENCE/testing/examples/s3-testing.ts +127 -127
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-01-foundations.md +267 -267
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-02-s3-testing.md +599 -599
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-03-fluent-testing.md +589 -589
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-04-integration-testing.md +699 -699
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-05-debugging.md +478 -478
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-cicd-integration.md +463 -463
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-06-preflight-validation.md +131 -131
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-best-practices.md +499 -499
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-07-coverage-ci.md +165 -165
- package/docs/04-REFERENCE/testing/modules/04-reference-testing-08-api-reference.md +634 -634
- package/docs/04-REFERENCE/testing/readme.md +86 -86
- package/docs/04-REFERENCE/testing/testing-quick-reference.md +667 -667
- package/docs/04-REFERENCE/testing/testing-readme.md +286 -286
- package/docs/04-REFERENCE/troubleshooting/readme.md +144 -144
- package/docs/04-REFERENCE/troubleshooting/troubleshooting-deno-sftp-compatibility.md +392 -392
- package/docs/template-loading-matrix.md +242 -242
- package/package.json +5 -3
- package/docs/02-CORE-GUIDES/api-reference/cli-profile-integration.md +0 -377
|
@@ -1,1710 +1,1710 @@
|
|
|
1
|
-
---
|
|
2
|
-
template_id: tpl-ingest-sftp-json-to-location-graphql
|
|
3
|
-
canonical_filename: template-ingestion-sftp-json-location-graphql.md
|
|
4
|
-
version: 2.0.0
|
|
5
|
-
sdk_version: ^0.1.39
|
|
6
|
-
runtime: versori
|
|
7
|
-
direction: ingestion
|
|
8
|
-
source: sftp-json
|
|
9
|
-
destination: fluent-graphql
|
|
10
|
-
entity: location
|
|
11
|
-
format: json
|
|
12
|
-
logging: versori
|
|
13
|
-
status: stable
|
|
14
|
-
compliance: gold-standard
|
|
15
|
-
features:
|
|
16
|
-
- graphql-mutation-mapper
|
|
17
|
-
- memory-management
|
|
18
|
-
- enhanced-logging
|
|
19
|
-
- attribute-transformation
|
|
20
|
-
- dispose-finally
|
|
21
|
-
---
|
|
22
|
-
|
|
23
|
-
# Template: Ingestion - SFTP JSON to Location GraphQL
|
|
24
|
-
|
|
25
|
-
**Deployment Target:** Versori Platform
|
|
26
|
-
**Compliance Status:** ✅ GOLD STANDARD APPROVED
|
|
27
|
-
|
|
28
|
-
---
|
|
29
|
-
|
|
30
|
-
## 📋 Implementation Prompt
|
|
31
|
-
|
|
32
|
-
```
|
|
33
|
-
I need a Versori scheduled ingestion that:
|
|
34
|
-
|
|
35
|
-
1) Discovers JSON files on SFTP with file tracking to skip duplicates
|
|
36
|
-
2) Downloads and parses JSON (locations array) with JSONParserService
|
|
37
|
-
3) Transforms records with GraphQLMutationMapper (NOT UniversalMapper)
|
|
38
|
-
4) Executes GraphQL createLocation mutations with rate limiting
|
|
39
|
-
5) Archives files to processed/ or errors/ and writes error reports
|
|
40
|
-
6) Tracks progress with JobTracker and exposes job status webhook
|
|
41
|
-
7) Uses native Versori log and follows modular service architecture
|
|
42
|
-
|
|
43
|
-
CRITICAL: This template uses GraphQLMutationMapper for GraphQL mutations,
|
|
44
|
-
NOT UniversalMapper (which is for Batch API). Do NOT use setRetailerId()
|
|
45
|
-
for GraphQL mutations - it's only needed for Job/Event API.
|
|
46
|
-
|
|
47
|
-
Use the loaded docs to fill in SDK specifics and best practices.
|
|
48
|
-
```
|
|
49
|
-
|
|
50
|
-
---
|
|
51
|
-
|
|
52
|
-
## 📋 Template Overview
|
|
53
|
-
|
|
54
|
-
This connector runs on the Versori platform with **gold standard compliance**. It reads location data from SFTP JSON files, transforms it with GraphQLMutationMapper, and creates/updates locations via GraphQL mutations.
|
|
55
|
-
|
|
56
|
-
### What This Template Does
|
|
57
|
-
|
|
58
|
-
```
|
|
59
|
-
┌─────────────────────────────────────────────────────────────────┐
|
|
60
|
-
│ INGESTION WORKFLOW (GraphQL Mutations Pattern) │
|
|
61
|
-
└─────────────────────────────────────────────────────────────────┘
|
|
62
|
-
|
|
63
|
-
1. TRIGGER
|
|
64
|
-
├─ Scheduled (Cron): Daily at 2 AM
|
|
65
|
-
├─ Ad hoc (Webhook): Manual trigger
|
|
66
|
-
└─ Status Query (Webhook): Check progress
|
|
67
|
-
|
|
68
|
-
2. DISCOVER FILES (SftpDataSource)
|
|
69
|
-
├─ List files matching pattern
|
|
70
|
-
├─ Check VersoriFileTracker (skip processed)
|
|
71
|
-
└─ Sort by oldest first
|
|
72
|
-
|
|
73
|
-
3. DOWNLOAD & PARSE (LocationFileProcessorService)
|
|
74
|
-
├─ Download from SFTP
|
|
75
|
-
├─ Parse JSON with JSONParserService
|
|
76
|
-
├─ Validate structure with type guards
|
|
77
|
-
└─ Return LocationRecord[]
|
|
78
|
-
|
|
79
|
-
4. TRANSFORM (GraphQLMutationMapper)
|
|
80
|
-
├─ Map to GraphQL CreateLocationInput
|
|
81
|
-
├─ Handle nested objects (primaryAddress, openingSchedule)
|
|
82
|
-
├─ Validate against GraphQL schema
|
|
83
|
-
└─ Generate mutation query
|
|
84
|
-
|
|
85
|
-
5. EXECUTE MUTATIONS (MutationSenderService)
|
|
86
|
-
├─ Execute createLocation mutations
|
|
87
|
-
├─ Rate limiting (configurable concurrency)
|
|
88
|
-
├─ Per-record error handling
|
|
89
|
-
└─ Track success/failure per mutation
|
|
90
|
-
|
|
91
|
-
6. ARCHIVE & LOG
|
|
92
|
-
├─ Move to processed/ or errors/
|
|
93
|
-
├─ Write error reports
|
|
94
|
-
└─ Track state in VersoriFileTracker
|
|
95
|
-
|
|
96
|
-
7. JOB TRACKING
|
|
97
|
-
├─ Update job status at each step
|
|
98
|
-
└─ Enable status queries via webhook
|
|
99
|
-
```
|
|
100
|
-
|
|
101
|
-
### Key Features
|
|
102
|
-
|
|
103
|
-
- ✅ **Gold Standard Compliant** - Modular architecture, proper patterns
|
|
104
|
-
- ✅ **GraphQLMutationMapper** - Correct mapper for GraphQL mutations
|
|
105
|
-
- ✅ **retailerId Configuration** - CRITICAL for GraphQL API calls
|
|
106
|
-
- ✅ **Type-Safe** - Comprehensive TypeScript interfaces
|
|
107
|
-
- ✅ **Modular Services** - 4 services + utils (testable, reusable)
|
|
108
|
-
- ✅ **External Config** - Mapping in JSON file
|
|
109
|
-
- ✅ **Error Handling** - Per-record failures don't block others
|
|
110
|
-
- ✅ **File Tracking** - Prevents duplicate processing
|
|
111
|
-
- ✅ **Job Tracking** - Status queries via webhook
|
|
112
|
-
|
|
113
|
-
---
|
|
114
|
-
|
|
115
|
-
## 📦 SDK Imports
|
|
116
|
-
|
|
117
|
-
```typescript
|
|
118
|
-
import { Buffer } from 'node:buffer'; // Required for Versori/Deno
|
|
119
|
-
|
|
120
|
-
import {
|
|
121
|
-
createClient,
|
|
122
|
-
SftpDataSource,
|
|
123
|
-
JSONParserService,
|
|
124
|
-
GraphQLMutationMapper, // ✅ CRITICAL: Use this for GraphQL (NOT UniversalMapper)
|
|
125
|
-
VersoriFileTracker,
|
|
126
|
-
JobTracker,
|
|
127
|
-
} from '@fluentcommerce/fc-connect-sdk';
|
|
128
|
-
|
|
129
|
-
import type {
|
|
130
|
-
FluentClient,
|
|
131
|
-
FileMetadata,
|
|
132
|
-
Logger,
|
|
133
|
-
} from '@fluentcommerce/fc-connect-sdk';
|
|
134
|
-
|
|
135
|
-
import { schedule, webhook, http, fn } from '@versori/run';
|
|
136
|
-
```
|
|
137
|
-
|
|
138
|
-
**✅ Type-Only Imports:** Separating value and type imports improves tree-shaking and bundle size.
|
|
139
|
-
|
|
140
|
-
---
|
|
141
|
-
|
|
142
|
-
---
|
|
143
|
-
|
|
144
|
-
## SFTP Connection Setup (Recommended)
|
|
145
|
-
|
|
146
|
-
**🔒 BEST PRACTICE:** Store SFTP credentials in a Versori connection object with Basic Auth:
|
|
147
|
-
|
|
148
|
-
### Connection Configuration
|
|
149
|
-
|
|
150
|
-
1. In Versori platform, create a connection named `SFTP`
|
|
151
|
-
2. Set **Authentication Type**: `Basic Auth`
|
|
152
|
-
3. Enter **Username**: Your SFTP username
|
|
153
|
-
4. Enter **Password**: Your SFTP password
|
|
154
|
-
5. The SDK will automatically retrieve and decode credentials
|
|
155
|
-
|
|
156
|
-
### Code Implementation
|
|
157
|
-
|
|
158
|
-
The workflow retrieves credentials programmatically from the Versori connection:
|
|
159
|
-
|
|
160
|
-
```typescript
|
|
161
|
-
import { Buffer } from 'node:buffer';
|
|
162
|
-
|
|
163
|
-
// Retrieve SFTP credentials from connection configuration
|
|
164
|
-
log.info('Retrieving SFTP credentials from connection configuration');
|
|
165
|
-
|
|
166
|
-
let sftpUsername: string;
|
|
167
|
-
let sftpPassword: string;
|
|
168
|
-
|
|
169
|
-
try {
|
|
170
|
-
// Retrieve credentials from the 'SFTP' connection
|
|
171
|
-
const sftpCred = await ctx.credentials().getAccessToken('SFTP');
|
|
172
|
-
|
|
173
|
-
if (!sftpCred?.accessToken) {
|
|
174
|
-
throw new Error('No SFTP credentials found in connection configuration');
|
|
175
|
-
}
|
|
176
|
-
|
|
177
|
-
// Decode base64 accessToken to get "username:password"
|
|
178
|
-
const rawBasicAuth = Buffer.from(sftpCred.accessToken, 'base64').toString('utf-8');
|
|
179
|
-
|
|
180
|
-
// Split on ':' to extract username and password
|
|
181
|
-
const parts = rawBasicAuth.split(':');
|
|
182
|
-
|
|
183
|
-
if (parts.length !== 2) {
|
|
184
|
-
throw new Error('Invalid SFTP credential format - expected username:password');
|
|
185
|
-
}
|
|
186
|
-
|
|
187
|
-
sftpUsername = parts[0];
|
|
188
|
-
sftpPassword = parts[1];
|
|
189
|
-
|
|
190
|
-
log.info('SFTP credentials retrieved successfully', {
|
|
191
|
-
hasUsername: !!sftpUsername,
|
|
192
|
-
hasPassword: !!sftpPassword,
|
|
193
|
-
usernameLength: sftpUsername.length,
|
|
194
|
-
passwordLength: sftpPassword.length,
|
|
195
|
-
});
|
|
196
|
-
} catch (error: any) {
|
|
197
|
-
log.error('Failed to retrieve SFTP credentials', {
|
|
198
|
-
message: error instanceof Error ? error.message : String(error),
|
|
199
|
-
stack: error instanceof Error ? error.stack : undefined,
|
|
200
|
-
errorType: error instanceof Error ? error.constructor.name : 'Error'
|
|
201
|
-
});
|
|
202
|
-
|
|
203
|
-
return {
|
|
204
|
-
success: false,
|
|
205
|
-
error: 'Failed to retrieve SFTP credentials from connection configuration',
|
|
206
|
-
details: error?.message,
|
|
207
|
-
recommendation: 'Please ensure the SFTP connection is configured in the Connections section with Basic Authentication (username and password)',
|
|
208
|
-
};
|
|
209
|
-
}
|
|
210
|
-
|
|
211
|
-
// Use credentials in SftpDataSource
|
|
212
|
-
const sftp = new SftpDataSource({
|
|
213
|
-
type: 'SFTP_JSON',
|
|
214
|
-
connectionId: 'sftp-location-sync',
|
|
215
|
-
name: 'Location Sync SFTP',
|
|
216
|
-
settings: {
|
|
217
|
-
host: activation.getVariable('sftpHost'),
|
|
218
|
-
port: parseInt(activation.getVariable('sftpPort') || '22', 10),
|
|
219
|
-
username: sftpUsername, // From connection
|
|
220
|
-
password: sftpPassword, // From connection
|
|
221
|
-
remotePath: activation.getVariable('sftpRemotePath'),
|
|
222
|
-
filePattern: activation.getVariable('filePattern'),
|
|
223
|
-
encoding: 'utf8',
|
|
224
|
-
},
|
|
225
|
-
}, log);
|
|
226
|
-
```
|
|
227
|
-
|
|
228
|
-
### Why Use Connections Instead of Activation Variables?
|
|
229
|
-
|
|
230
|
-
| Aspect | Activation Variables ❌ | Versori Connections ✅ |
|
|
231
|
-
|--------|------------------------|------------------------|
|
|
232
|
-
| **Security** | Plain text in config UI | Encrypted in Versori vault |
|
|
233
|
-
| **Visibility** | Visible to all users | Controlled access |
|
|
234
|
-
| **Rotation** | Manual update per workflow | Update once, affects all |
|
|
235
|
-
| **Best Practice** | Legacy approach | **Recommended approach** |
|
|
236
|
-
| **Audit Trail** | Limited | Full connection audit logs |
|
|
237
|
-
|
|
238
|
-
**Benefits:**
|
|
239
|
-
- ✅ Credentials stored securely in Versori vault
|
|
240
|
-
- ✅ Connection can be reused across workflows
|
|
241
|
-
- ✅ No sensitive data in activation variables
|
|
242
|
-
- ✅ Easier credential rotation (one place to update)
|
|
243
|
-
- ✅ Better security posture (credentials never exposed in UI)
|
|
244
|
-
|
|
245
|
-
---
|
|
246
|
-
|
|
247
|
-
## ⚙️ Activation Variables
|
|
248
|
-
|
|
249
|
-
**Configuration is driven by activation variables - modify these instead of code:**
|
|
250
|
-
|
|
251
|
-
> **Note:** `sftpUsername` and `sftpPassword` are fetched from the `SFTP` Basic Auth connection (see SFTP Connection Setup above).
|
|
252
|
-
|
|
253
|
-
```bash
|
|
254
|
-
# SFTP Configuration
|
|
255
|
-
SFTP_HOST=sftp.partner.com
|
|
256
|
-
SFTP_PORT=22
|
|
257
|
-
|
|
258
|
-
# SFTP Paths
|
|
259
|
-
SFTP_REMOTE_PATH=/locations/incoming
|
|
260
|
-
SFTP_ARCHIVE_PATH=/locations/processed
|
|
261
|
-
SFTP_ERROR_PATH=/locations/errors
|
|
262
|
-
|
|
263
|
-
# File Processing
|
|
264
|
-
FILE_PATTERN=locations_*.json
|
|
265
|
-
MAX_FILES_PER_RUN=10
|
|
266
|
-
|
|
267
|
-
# GraphQL Configuration
|
|
268
|
-
FLUENT_RETAILER_ID=1 # Optional: Only if mutation schema requires retailerId in input
|
|
269
|
-
MAX_PARALLEL_MUTATIONS=10 # Concurrent mutation limit (1=sequential, 3-10=parallel)
|
|
270
|
-
|
|
271
|
-
# Feature Toggles
|
|
272
|
-
REQUIRE_ABSOLUTE_PATHS=true # "true" for AWS Transfer Family, "false" for standard OpenSSH
|
|
273
|
-
VALIDATE_CONNECTION=true # Validate SFTP connection on startup (default: true)
|
|
274
|
-
ENABLE_FILE_TRACKING=true # Enable/disable file tracking (default: true)
|
|
275
|
-
```
|
|
276
|
-
|
|
277
|
-
**Security:**
|
|
278
|
-
- **SFTP Authentication:** Credentials stored in Versori connection vault (not in activation variables). See [SFTP Connection Setup](#sftp-connection-setup-recommended) section above.
|
|
279
|
-
- **Webhook Authentication:** Enforced by Versori connection configuration. Configure your webhook connection with API key authentication in the Versori Dashboard, then reference it in `webhook({ connection: 'webhook-auth' })`.
|
|
280
|
-
|
|
281
|
-
---
|
|
282
|
-
|
|
283
|
-
## 📄 Mapping Configuration
|
|
284
|
-
|
|
285
|
-
**File:** `src/config/location-mapping.json`
|
|
286
|
-
|
|
287
|
-
**✅ PRODUCTION STANDARD:** External JSON file with GraphQLMutationMapper structure
|
|
288
|
-
|
|
289
|
-
```json
|
|
290
|
-
{
|
|
291
|
-
"mutation": "createLocation",
|
|
292
|
-
"sourceFormat": "json",
|
|
293
|
-
"version": "1.0.0",
|
|
294
|
-
"returnFields": ["id", "ref", "name", "type", "status"],
|
|
295
|
-
"arguments": {
|
|
296
|
-
"input": {
|
|
297
|
-
"ref": {
|
|
298
|
-
"source": "locationRef",
|
|
299
|
-
"required": true,
|
|
300
|
-
"resolver": "trim"
|
|
301
|
-
},
|
|
302
|
-
"name": {
|
|
303
|
-
"source": "locationName",
|
|
304
|
-
"required": true,
|
|
305
|
-
"resolver": "trim"
|
|
306
|
-
},
|
|
307
|
-
"type": {
|
|
308
|
-
"source": "locationType",
|
|
309
|
-
"required": true,
|
|
310
|
-
"resolver": "toUpperCase"
|
|
311
|
-
},
|
|
312
|
-
"status": {
|
|
313
|
-
"value": "ACTIVE",
|
|
314
|
-
"required": true
|
|
315
|
-
},
|
|
316
|
-
"primaryAddress": {
|
|
317
|
-
"ref": {
|
|
318
|
-
"source": "locationRef",
|
|
319
|
-
"required": true,
|
|
320
|
-
"resolver": "trim"
|
|
321
|
-
},
|
|
322
|
-
"street": {
|
|
323
|
-
"source": "address.street1",
|
|
324
|
-
"resolver": "trim"
|
|
325
|
-
},
|
|
326
|
-
"city": {
|
|
327
|
-
"source": "address.city",
|
|
328
|
-
"resolver": "trim"
|
|
329
|
-
},
|
|
330
|
-
"state": {
|
|
331
|
-
"source": "address.state",
|
|
332
|
-
"resolver": "toUpperCase"
|
|
333
|
-
},
|
|
334
|
-
"postcode": {
|
|
335
|
-
"source": "address.postalCode",
|
|
336
|
-
"resolver": "trim"
|
|
337
|
-
},
|
|
338
|
-
"country": {
|
|
339
|
-
"source": "address.country",
|
|
340
|
-
"resolver": "toUpperCase"
|
|
341
|
-
},
|
|
342
|
-
"latitude": {
|
|
343
|
-
"source": "geo.latitude",
|
|
344
|
-
"required": true,
|
|
345
|
-
"resolver": "parseFloat"
|
|
346
|
-
},
|
|
347
|
-
"longitude": {
|
|
348
|
-
"source": "geo.longitude",
|
|
349
|
-
"required": true,
|
|
350
|
-
"resolver": "parseFloat"
|
|
351
|
-
},
|
|
352
|
-
"timeZone": {
|
|
353
|
-
"source": "timeZone",
|
|
354
|
-
"resolver": "trim"
|
|
355
|
-
}
|
|
356
|
-
},
|
|
357
|
-
"openingSchedule": {
|
|
358
|
-
"allHours": {
|
|
359
|
-
"source": "openingSchedule.allHours",
|
|
360
|
-
"required": true,
|
|
361
|
-
"resolver": "toBoolean"
|
|
362
|
-
},
|
|
363
|
-
"monStart": {
|
|
364
|
-
"source": "openingSchedule.monStart",
|
|
365
|
-
"required": true,
|
|
366
|
-
"resolver": "parseInt"
|
|
367
|
-
},
|
|
368
|
-
"monEnd": {
|
|
369
|
-
"source": "openingSchedule.monEnd",
|
|
370
|
-
"required": true,
|
|
371
|
-
"resolver": "parseInt"
|
|
372
|
-
}
|
|
373
|
-
}
|
|
374
|
-
}
|
|
375
|
-
}
|
|
376
|
-
}
|
|
377
|
-
```
|
|
378
|
-
|
|
379
|
-
**Key Differences from UniversalMapper:**
|
|
380
|
-
- ✅ `mutation` property (not `mutationName`)
|
|
381
|
-
- ✅ `arguments.input` structure (matches GraphQL schema)
|
|
382
|
-
- ✅ Nested objects (not dotted paths like `primaryAddress.street`)
|
|
383
|
-
- ✅ Resolver names without `sdk.` prefix (`trim`, `toUpperCase`, `parseInt`, `parseFloat`, `toBoolean`)
|
|
384
|
-
- ✅ Used with `GraphQLMutationMapper.map()` method (not `UniversalMapper.map()`)
|
|
385
|
-
|
|
386
|
-
---
|
|
387
|
-
|
|
388
|
-
## Expected JSON Format
|
|
389
|
-
|
|
390
|
-
```json
|
|
391
|
-
{
|
|
392
|
-
"locations": [
|
|
393
|
-
{
|
|
394
|
-
"locationRef": "LOC-001",
|
|
395
|
-
"locationName": "Main Warehouse",
|
|
396
|
-
"locationType": "WAREHOUSE",
|
|
397
|
-
"address": {
|
|
398
|
-
"street1": "123 Main St",
|
|
399
|
-
"city": "New York",
|
|
400
|
-
"state": "NY",
|
|
401
|
-
"postalCode": "10001",
|
|
402
|
-
"country": "US"
|
|
403
|
-
},
|
|
404
|
-
"geo": {
|
|
405
|
-
"latitude": 40.7128,
|
|
406
|
-
"longitude": -74.0060
|
|
407
|
-
},
|
|
408
|
-
"timeZone": "America/New_York",
|
|
409
|
-
"openingSchedule": {
|
|
410
|
-
"allHours": false,
|
|
411
|
-
"monStart": 480,
|
|
412
|
-
"monEnd": 1020
|
|
413
|
-
}
|
|
414
|
-
}
|
|
415
|
-
]
|
|
416
|
-
}
|
|
417
|
-
```
|
|
418
|
-
|
|
419
|
-
**Alternative:** Root array `[{...}]` is also supported.
|
|
420
|
-
|
|
421
|
-
---
|
|
422
|
-
|
|
423
|
-
## 🔧 Production Code Structure
|
|
424
|
-
|
|
425
|
-
### Modular Architecture (Gold Standard)
|
|
426
|
-
|
|
427
|
-
```
|
|
428
|
-
sftp-json-location-graphql/
|
|
429
|
-
├── package.json
|
|
430
|
-
├── tsconfig.json
|
|
431
|
-
├── index.ts # Entry point
|
|
432
|
-
└── src/
|
|
433
|
-
├── workflows/
|
|
434
|
-
│ └── location-ingestion.ts # 3 workflows
|
|
435
|
-
├── services/
|
|
436
|
-
│ ├── location-file-processor.service.ts # Download, parse, transform
|
|
437
|
-
│ ├── mutation-sender.service.ts # Execute GraphQL mutations
|
|
438
|
-
│ ├── mutation-logger.service.ts # Write error logs
|
|
439
|
-
│ └── location-ingestion.service.ts # Main orchestration
|
|
440
|
-
├── types/
|
|
441
|
-
│ └── location-ingestion.types.ts # TypeScript interfaces
|
|
442
|
-
├── utils/
|
|
443
|
-
│ ├── sftp-path.utils.ts # Path helpers
|
|
444
|
-
│ ├── retry.utils.ts # Retry logic
|
|
445
|
-
│ └── job-id-generator.ts # Job IDs
|
|
446
|
-
└── config/
|
|
447
|
-
└── location-mapping.json # GraphQL mapping config
|
|
448
|
-
```
|
|
449
|
-
|
|
450
|
-
---
|
|
451
|
-
|
|
452
|
-
## Critical Implementation Patterns
|
|
453
|
-
|
|
454
|
-
### ✅ MUST HAVE: retailerId Configuration ⚠️ VERIFICATION REQUIRED
|
|
455
|
-
|
|
456
|
-
```typescript
|
|
457
|
-
// In main orchestration service (location-ingestion.service.ts)
|
|
458
|
-
|
|
459
|
-
export async function executeLocationIngestion(ctx: VersoriContext, params: LocationIngestionParams) {
|
|
460
|
-
const { log, activation } = ctx;
|
|
461
|
-
|
|
462
|
-
// Step 1: Create client
|
|
463
|
-
const client = await createClient(ctx);
|
|
464
|
-
if (!client) {
|
|
465
|
-
throw new Error('Failed to create Fluent Commerce client');
|
|
466
|
-
}
|
|
467
|
-
|
|
468
|
-
// ✅ CORRECT: GraphQL mutations don't need setRetailerId()
|
|
469
|
-
// Check your GraphQL schema to determine retailerId handling:
|
|
470
|
-
// - Mandatory retailerId → Must pass it in mutation input
|
|
471
|
-
// - Optional retailerId → Can pass it if needed
|
|
472
|
-
// - No retailerId field → Don't pass it
|
|
473
|
-
// See: fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md
|
|
474
|
-
|
|
475
|
-
// Step 2: Initialize GraphQLMutationMapper with client
|
|
476
|
-
import mappingConfig from '../config/location-mapping.json' with { type: 'json' };
|
|
477
|
-
const mapper = new GraphQLMutationMapper(mappingConfig, log, { fluentClient: client });
|
|
478
|
-
|
|
479
|
-
// Step 3: Process files and execute mutations
|
|
480
|
-
// ... rest of workflow
|
|
481
|
-
|
|
482
|
-
// ✅ Configuration with defaults
|
|
483
|
-
const mutationBatchSize = parseInt(
|
|
484
|
-
activation?.getVariable('mutationBatchSize') || '1', // ✅ Default: 1 (sequential)
|
|
485
|
-
10
|
|
486
|
-
);
|
|
487
|
-
|
|
488
|
-
const mutationsPerAliasBatch = activation?.getVariable('mutationsPerAliasBatch')
|
|
489
|
-
? parseInt(activation?.getVariable('mutationsPerAliasBatch') || '1', 10)
|
|
490
|
-
: undefined; // ✅ Default: undefined (disabled, use separate requests)
|
|
491
|
-
|
|
492
|
-
// ? Enhanced: Extract context for progress logging
|
|
493
|
-
const sampleLocationRefs = locations.slice(0, 5).map((loc: any) => loc.ref || loc.input?.ref || 'unknown');
|
|
494
|
-
const mutationType = mapper?.mutationName || 'createLocation';
|
|
495
|
-
|
|
496
|
-
// ? Enhanced: Start logging with context
|
|
497
|
-
log.info(`[GraphQLMutations] Sending mutations for locations`, {
|
|
498
|
-
totalMutations: locations.length,
|
|
499
|
-
mutationType,
|
|
500
|
-
batchSize: mutationBatchSize,
|
|
501
|
-
batchMode: mutationBatchSize === 1 ? 'sequential' : `parallel (${mutationBatchSize})`,
|
|
502
|
-
sampleLocationRefs: sampleLocationRefs.join(', '),
|
|
503
|
-
aliasBatching: mutationsPerAliasBatch ? `enabled (${mutationsPerAliasBatch} per alias)` : 'disabled'
|
|
504
|
-
});
|
|
505
|
-
|
|
506
|
-
// Use in executeMutations:
|
|
507
|
-
const mutationResults = await mutationSender.executeMutations(
|
|
508
|
-
locations,
|
|
509
|
-
mutationBatchSize, // Concurrency control (default: 1)
|
|
510
|
-
mutationsPerAliasBatch // ✅ NEW: Alias batching (default: undefined)
|
|
511
|
-
);
|
|
512
|
-
|
|
513
|
-
// ? Enhanced: Completion logging with summary
|
|
514
|
-
log.info(`[GraphQLMutations] Mutation submission completed`, {
|
|
515
|
-
totalMutations: locations.length,
|
|
516
|
-
mutationsExecuted: mutationResults.mutationsExecuted,
|
|
517
|
-
mutationsFailed: mutationResults.mutationsFailed,
|
|
518
|
-
successRate: locations.length > 0 ? `${Math.round((mutationResults.mutationsExecuted / locations.length) * 100)}%` : '0%',
|
|
519
|
-
mutationType
|
|
520
|
-
});
|
|
521
|
-
}
|
|
522
|
-
```
|
|
523
|
-
|
|
524
|
-
**✅ VERIFIED Pattern:**
|
|
525
|
-
- GraphQL mutations do NOT need `client.setRetailerId()`
|
|
526
|
-
- `setRetailerId()` is only for Job API and Event API
|
|
527
|
-
- Pass retailerId in mutation input variables if schema requires it
|
|
528
|
-
|
|
529
|
-
**Example (For mutations that require retailerId in schema):**
|
|
530
|
-
```typescript
|
|
531
|
-
// Only needed if mutation schema explicitly requires retailerId
|
|
532
|
-
const fluentRetailerId = activation.getVariable('fluentRetailerId');
|
|
533
|
-
const { query, variables } = await mapper.map(location);
|
|
534
|
-
|
|
535
|
-
// Add to input if mutation schema requires it
|
|
536
|
-
if (fluentRetailerId && variables.input) {
|
|
537
|
-
variables.input.retailer = { id: parseInt(fluentRetailerId) };
|
|
538
|
-
}
|
|
539
|
-
|
|
540
|
-
await client.graphql({ query, variables });
|
|
541
|
-
```
|
|
542
|
-
|
|
543
|
-
**Note:** Check your GraphQL schema (via introspection) to determine retailerId handling:
|
|
544
|
-
1. Does the mutation have `retailerId` or `retailer.id` field?
|
|
545
|
-
2. Is it mandatory (`!`) or optional?
|
|
546
|
-
3. If field doesn't exist → don't pass it
|
|
547
|
-
|
|
548
|
-
### ✅ MUST HAVE: Double-Finally SFTP Disposal
|
|
549
|
-
|
|
550
|
-
**What:** Defensive resource cleanup with nested finally blocks (matches Event API gold standard)
|
|
551
|
-
|
|
552
|
-
**Gold Standard Pattern:**
|
|
553
|
-
|
|
554
|
-
```typescript
|
|
555
|
-
// ✅ CRITICAL: Declare SFTP outside try block for safe disposal
|
|
556
|
-
let sftp: SftpDataSource | undefined;
|
|
557
|
-
|
|
558
|
-
try {
|
|
559
|
-
// 📊 Execution boundary: Start workflow
|
|
560
|
-
const startTime = Date.now();
|
|
561
|
-
log.info('🚀 Starting location ingestion workflow', {
|
|
562
|
-
stage: 'initialization',
|
|
563
|
-
jobId,
|
|
564
|
-
triggeredBy: params.triggeredBy
|
|
565
|
-
});
|
|
566
|
-
|
|
567
|
-
// Initialize SFTP
|
|
568
|
-
sftp = new SftpDataSource(...);
|
|
569
|
-
|
|
570
|
-
// ✅ Validate connection (if enabled)
|
|
571
|
-
const validateConnection = activation?.getVariable('validateConnection') === 'true';
|
|
572
|
-
if (validateConnection) {
|
|
573
|
-
log.info('🔌 Validating SFTP connection', { stage: 'connection_validation' });
|
|
574
|
-
await sftp.validateConnection();
|
|
575
|
-
log.info('✅ SFTP connection validated successfully');
|
|
576
|
-
}
|
|
577
|
-
|
|
578
|
-
try {
|
|
579
|
-
// Use SFTP
|
|
580
|
-
await sftp.listFiles(...);
|
|
581
|
-
await sftp.downloadFile(...);
|
|
582
|
-
// ... rest of workflow
|
|
583
|
-
|
|
584
|
-
// 📊 Execution boundary: End workflow
|
|
585
|
-
const duration = Date.now() - startTime;
|
|
586
|
-
log.info('✅ Location ingestion workflow completed', {
|
|
587
|
-
stage: 'completion',
|
|
588
|
-
jobId,
|
|
589
|
-
duration,
|
|
590
|
-
filesProcessed,
|
|
591
|
-
mutationsExecuted
|
|
592
|
-
});
|
|
593
|
-
} finally {
|
|
594
|
-
// Inner finally - normal cleanup
|
|
595
|
-
if (sftp) {
|
|
596
|
-
await sftp.dispose();
|
|
597
|
-
log.info('🔌 SFTP connection disposed');
|
|
598
|
-
}
|
|
599
|
-
}
|
|
600
|
-
} catch (error: unknown) {
|
|
601
|
-
// ✅ Enhanced error logging: Extract all error details + recommendations
|
|
602
|
-
const errorMessage = error instanceof Error ? error.message : String(error);
|
|
603
|
-
const errorDetails = {
|
|
604
|
-
message: errorMessage,
|
|
605
|
-
stack: error instanceof Error ? error.stack : undefined,
|
|
606
|
-
errorType: error instanceof Error ? error.constructor.name : 'Error',
|
|
607
|
-
stage: 'workflow_execution',
|
|
608
|
-
jobId
|
|
609
|
-
};
|
|
610
|
-
|
|
611
|
-
// ✅ Add error recommendations
|
|
612
|
-
const recommendations: string[] = [];
|
|
613
|
-
if (errorMessage.includes('ECONNREFUSED')) {
|
|
614
|
-
recommendations.push('Check SFTP host and port configuration');
|
|
615
|
-
recommendations.push('Verify network connectivity to SFTP server');
|
|
616
|
-
} else if (errorMessage.includes('Authentication failed')) {
|
|
617
|
-
recommendations.push('Verify SFTP username and password');
|
|
618
|
-
recommendations.push('Check if account is locked or expired');
|
|
619
|
-
} else if (errorMessage.includes('No such file')) {
|
|
620
|
-
recommendations.push('Verify incoming path exists on SFTP server');
|
|
621
|
-
recommendations.push('Check file pattern matches existing files');
|
|
622
|
-
}
|
|
623
|
-
|
|
624
|
-
log.error('❌ Workflow failed', { ...errorDetails, recommendations });
|
|
625
|
-
|
|
626
|
-
// Handle error (mark job failed, etc.)
|
|
627
|
-
} finally {
|
|
628
|
-
// Outer finally - defensive cleanup
|
|
629
|
-
// ⚠️ CRITICAL: Ensures disposal even if error occurs after SFTP creation
|
|
630
|
-
// but before inner try block (e.g., during connection validation)
|
|
631
|
-
if (sftp) {
|
|
632
|
-
try {
|
|
633
|
-
await sftp.dispose();
|
|
634
|
-
log.info('🔌 SFTP connection disposed (outer finally)');
|
|
635
|
-
} catch (disposeError: unknown) {
|
|
636
|
-
const disposeErrorMessage = disposeError instanceof Error
|
|
637
|
-
? disposeError.message
|
|
638
|
-
: String(disposeError);
|
|
639
|
-
log.warn('⚠️ Error disposing SFTP in outer finally', { error: disposeErrorMessage });
|
|
640
|
-
}
|
|
641
|
-
}
|
|
642
|
-
}
|
|
643
|
-
```
|
|
644
|
-
|
|
645
|
-
**Why This is Critical:**
|
|
646
|
-
- Ensures disposal even if error occurs after SFTP creation but before inner try block
|
|
647
|
-
- Defensive cleanup catches edge cases
|
|
648
|
-
- Prevents resource leaks in error scenarios
|
|
649
|
-
- Matches Event API gold standard pattern
|
|
650
|
-
|
|
651
|
-
---
|
|
652
|
-
|
|
653
|
-
## Processing Modes
|
|
654
|
-
|
|
655
|
-
**⚠️ IMPORTANT:** Choose ONE processing mode per connector. These are alternative patterns, not features to use together.
|
|
656
|
-
|
|
657
|
-
The SDK supports two processing modes for handling multiple files. **Select the mode that best fits your use case** and configure your connector accordingly:
|
|
658
|
-
|
|
659
|
-
### Mode 1: Per-File Processing (Recommended Default) ✅ IMPLEMENTED
|
|
660
|
-
|
|
661
|
-
**When to use:**
|
|
662
|
-
- Multiple large files that shouldn't be in memory together
|
|
663
|
-
- Need file-level consistency (file 3 fails → files 1-2 already archived)
|
|
664
|
-
- Clear error isolation per file
|
|
665
|
-
|
|
666
|
-
**How it works:**
|
|
667
|
-
1. Process file 1 completely (download → parse → transform → execute mutations → archive)
|
|
668
|
-
2. Move to file 2
|
|
669
|
-
3. Continue sequentially
|
|
670
|
-
|
|
671
|
-
**Benefits:**
|
|
672
|
-
- ✅ Low memory usage (one file at a time)
|
|
673
|
-
- ✅ Clear error isolation (failed file doesn't block others)
|
|
674
|
-
- ✅ Incremental progress (files archived as completed)
|
|
675
|
-
|
|
676
|
-
**Example implementation (see code section below)**
|
|
677
|
-
|
|
678
|
-
### Mode 2: Chunked Per-File Processing ⚠️ OPTIONAL - Choose if needed
|
|
679
|
-
|
|
680
|
-
**When to use:**
|
|
681
|
-
- Many small files that can be processed in parallel
|
|
682
|
-
- Need better throughput for many files
|
|
683
|
-
- Files are independent (no cross-file dependencies)
|
|
684
|
-
|
|
685
|
-
**How it works:**
|
|
686
|
-
1. Group files into chunks (e.g., 5 files per chunk)
|
|
687
|
-
2. Process chunk 1 files in parallel
|
|
688
|
-
3. Wait for chunk completion, then move to chunk 2
|
|
689
|
-
|
|
690
|
-
**Benefits:**
|
|
691
|
-
- ✅ Better throughput (parallel processing)
|
|
692
|
-
- ✅ Still maintains file-level isolation
|
|
693
|
-
- ✅ Configurable chunk size
|
|
694
|
-
|
|
695
|
-
**Note:** GraphQL mutations are typically one-off operations, so per-file processing is usually sufficient. Chunked mode is optional for high-volume scenarios.
|
|
696
|
-
|
|
697
|
-
---
|
|
698
|
-
|
|
699
|
-
## Key Service Patterns
|
|
700
|
-
|
|
701
|
-
### Service 1: Location File Processor (Complete Implementation)
|
|
702
|
-
|
|
703
|
-
**File:** `src/services/location-file-processor.service.ts`
|
|
704
|
-
|
|
705
|
-
```typescript
|
|
706
|
-
/**
|
|
707
|
-
* Location File Processor Service
|
|
708
|
-
*
|
|
709
|
-
* Downloads JSON files from SFTP, parses content, and transforms with GraphQLMutationMapper.
|
|
710
|
-
* Reusable service for location file processing workflows.
|
|
711
|
-
*/
|
|
712
|
-
|
|
713
|
-
import {
|
|
714
|
-
SftpDataSource,
|
|
715
|
-
JSONParserService,
|
|
716
|
-
GraphQLMutationMapper,
|
|
717
|
-
} from '@fluentcommerce/fc-connect-sdk';
|
|
718
|
-
import type {
|
|
719
|
-
ProcessFileResult,
|
|
720
|
-
LocationRecord,
|
|
721
|
-
} from '../types/location-ingestion.types';
|
|
722
|
-
import { extractFileName } from '../utils/sftp-path.utils';
|
|
723
|
-
|
|
724
|
-
/**
|
|
725
|
-
* Location file processing service
|
|
726
|
-
* Coordinates SFTP download, JSON parsing (via SDK), and GraphQL mutation generation
|
|
727
|
-
*/
|
|
728
|
-
export class LocationFileProcessorService {
|
|
729
|
-
constructor(
|
|
730
|
-
private sftp: SftpDataSource,
|
|
731
|
-
private jsonParser: JSONParserService,
|
|
732
|
-
private mapper: GraphQLMutationMapper
|
|
733
|
-
) {} // ✅ No logger param - workflow handles logging
|
|
734
|
-
|
|
735
|
-
/**
|
|
736
|
-
* Download JSON file from SFTP, parse, and generate GraphQL mutations
|
|
737
|
-
*/
|
|
738
|
-
async downloadParseAndTransform(remoteFilePath: string): Promise<ProcessFileResult> {
|
|
739
|
-
// ✅ Extract filename for logging
|
|
740
|
-
const fileName = extractFileName(remoteFilePath);
|
|
741
|
-
|
|
742
|
-
try {
|
|
743
|
-
// Download JSON content
|
|
744
|
-
const content = (await this.sftp.downloadFile(remoteFilePath, {
|
|
745
|
-
encoding: 'utf8',
|
|
746
|
-
})) as string;
|
|
747
|
-
|
|
748
|
-
// Parse JSON with type safety
|
|
749
|
-
let parsed: unknown;
|
|
750
|
-
try {
|
|
751
|
-
parsed = await this.jsonParser.parse(content);
|
|
752
|
-
} catch (parseError: unknown) {
|
|
753
|
-
const errorMessage = parseError instanceof Error ? parseError.message : String(parseError);
|
|
754
|
-
return {
|
|
755
|
-
success: false,
|
|
756
|
-
locations: [],
|
|
757
|
-
error: `JSON parse error: ${errorMessage}`,
|
|
758
|
-
fileName,
|
|
759
|
-
};
|
|
760
|
-
}
|
|
761
|
-
|
|
762
|
-
// Normalize locations array (handle both { locations: [...] } and root array formats)
|
|
763
|
-
const jsonData = parsed as Record<string, unknown>;
|
|
764
|
-
const locations = jsonData.locations
|
|
765
|
-
? (Array.isArray(jsonData.locations) ? jsonData.locations : [jsonData.locations])
|
|
766
|
-
: (Array.isArray(jsonData) ? jsonData : []);
|
|
767
|
-
|
|
768
|
-
if (locations.length === 0) {
|
|
769
|
-
return {
|
|
770
|
-
success: false,
|
|
771
|
-
locations: [],
|
|
772
|
-
error: 'No locations found in JSON',
|
|
773
|
-
fileName,
|
|
774
|
-
};
|
|
775
|
-
}
|
|
776
|
-
|
|
777
|
-
// Validate with type guard
|
|
778
|
-
const validLocations = locations.filter(isLocationRecord);
|
|
779
|
-
|
|
780
|
-
if (validLocations.length === 0) {
|
|
781
|
-
return {
|
|
782
|
-
success: false,
|
|
783
|
-
locations: [],
|
|
784
|
-
error: 'No valid locations found after validation',
|
|
785
|
-
fileName,
|
|
786
|
-
};
|
|
787
|
-
}
|
|
788
|
-
|
|
789
|
-
// Transform with GraphQLMutationMapper
|
|
790
|
-
const mapped = await Promise.all(
|
|
791
|
-
validLocations.map(loc => this.mapper.map(loc))
|
|
792
|
-
);
|
|
793
|
-
|
|
794
|
-
return {
|
|
795
|
-
success: true,
|
|
796
|
-
locations: mapped.map(m => m.variables.input),
|
|
797
|
-
fileName,
|
|
798
|
-
};
|
|
799
|
-
} catch (error: unknown) {
|
|
800
|
-
const errorMessage = error instanceof Error ? error.message : String(error);
|
|
801
|
-
return {
|
|
802
|
-
success: false,
|
|
803
|
-
locations: [],
|
|
804
|
-
error: errorMessage,
|
|
805
|
-
fileName,
|
|
806
|
-
};
|
|
807
|
-
}
|
|
808
|
-
}
|
|
809
|
-
}
|
|
810
|
-
|
|
811
|
-
/**
|
|
812
|
-
* Type guard: Check if parsed value is a valid LocationRecord
|
|
813
|
-
*/
|
|
814
|
-
function isLocationRecord(obj: unknown): obj is LocationRecord {
|
|
815
|
-
if (typeof obj !== 'object' || obj === null) return false;
|
|
816
|
-
const loc = obj as Record<string, unknown>;
|
|
817
|
-
|
|
818
|
-
return (
|
|
819
|
-
typeof loc.locationRef === 'string' &&
|
|
820
|
-
typeof loc.locationName === 'string' &&
|
|
821
|
-
typeof loc.locationType === 'string' &&
|
|
822
|
-
typeof loc.address === 'object' &&
|
|
823
|
-
loc.address !== null &&
|
|
824
|
-
typeof loc.geo === 'object' &&
|
|
825
|
-
loc.geo !== null
|
|
826
|
-
);
|
|
827
|
-
}
|
|
828
|
-
```
|
|
829
|
-
|
|
830
|
-
### Service 2: Mutation Sender (Complete Implementation)
|
|
831
|
-
|
|
832
|
-
**File:** `src/services/mutation-sender.service.ts`
|
|
833
|
-
|
|
834
|
-
```typescript
|
|
835
|
-
/**
|
|
836
|
-
* Mutation Sender Service
|
|
837
|
-
*
|
|
838
|
-
* Executes GraphQL mutations with per-record error handling.
|
|
839
|
-
* Continues processing on individual failures (GraphQL mutation pattern).
|
|
840
|
-
*/
|
|
841
|
-
|
|
842
|
-
import { GraphQLMutationMapper } from '@fluentcommerce/fc-connect-sdk';
|
|
843
|
-
import type { FluentClient } from '@fluentcommerce/fc-connect-sdk';
|
|
844
|
-
import type { MutationResult, MappedLocation } from '../types/location-ingestion.types';
|
|
845
|
-
|
|
846
|
-
/**
|
|
847
|
-
* Service for executing GraphQL mutations
|
|
848
|
-
*/
|
|
849
|
-
export class MutationSenderService {
|
|
850
|
-
constructor(
|
|
851
|
-
private client: FluentClient,
|
|
852
|
-
private mapper: GraphQLMutationMapper
|
|
853
|
-
) {}
|
|
854
|
-
|
|
855
|
-
/**
|
|
856
|
-
* Execute GraphQL mutations with configurable concurrency and alias batching
|
|
857
|
-
*
|
|
858
|
-
* **Performance Characteristics:**
|
|
859
|
-
* - `maxParallel: 1` → Sequential processing (safe default, ~1 mutation/sec)
|
|
860
|
-
* - `maxParallel: 3-5` → Balanced throughput (~3-5 mutations/sec)
|
|
861
|
-
* - `maxParallel: 10` → High-volume processing (~10 mutations/sec, 100+ locations)
|
|
862
|
-
*
|
|
863
|
-
* **Implementation Strategy:**
|
|
864
|
-
* - Sequential (1): Optimized loop (no Promise.allSettled overhead)
|
|
865
|
-
* - Parallel (>1): Chunked processing with bounded concurrency
|
|
866
|
-
* - Alias batching: Groups mutations into aliased GraphQL requests (reduces network overhead)
|
|
867
|
-
* - Both modes: Per-record error tracking (failures don't block others)
|
|
868
|
-
*
|
|
869
|
-
* **Alias Batching:**
|
|
870
|
-
* - When `mutationsPerAliasBatch > 1`, groups mutations into aliased requests
|
|
871
|
-
* - Example: 5 mutations → 1 HTTP request with aliases (createLocation1, createLocation2, ...)
|
|
872
|
-
* - Reduces network overhead by ~80% for high-volume scenarios
|
|
873
|
-
*
|
|
874
|
-
* @param locations - Array of mapped locations to create/update
|
|
875
|
-
* @param maxParallel - Number of concurrent mutations or alias batches (default: 1, min: 1)
|
|
876
|
-
* @param mutationsPerAliasBatch - Optional: Number of mutations per aliased request (default: undefined = disabled)
|
|
877
|
-
* @returns MutationResult with counts (mutationsExecuted/mutationsFailed) and error details
|
|
878
|
-
*/
|
|
879
|
-
async executeMutations(
|
|
880
|
-
locations: MappedLocation[],
|
|
881
|
-
maxParallel: number = 1, // ✅ Default: 1 (sequential)
|
|
882
|
-
mutationsPerAliasBatch?: number // ✅ NEW: Alias batching parameter (default: undefined = disabled)
|
|
883
|
-
): Promise<MutationResult> {
|
|
884
|
-
// Determine mode: use aliases if mutationsPerAliasBatch is set and > 1
|
|
885
|
-
const useAliases = mutationsPerAliasBatch && mutationsPerAliasBatch > 1;
|
|
886
|
-
|
|
887
|
-
if (useAliases) {
|
|
888
|
-
return await this.executeMutationsWithAliases(
|
|
889
|
-
locations,
|
|
890
|
-
maxParallel,
|
|
891
|
-
mutationsPerAliasBatch!
|
|
892
|
-
);
|
|
893
|
-
} else {
|
|
894
|
-
return await this.executeMutationsSeparate(locations, maxParallel);
|
|
895
|
-
}
|
|
896
|
-
}
|
|
897
|
-
|
|
898
|
-
/**
|
|
899
|
-
* Execute mutations using separate concurrent requests (current mode)
|
|
900
|
-
*/
|
|
901
|
-
private async executeMutationsSeparate(
|
|
902
|
-
locations: MappedLocation[],
|
|
903
|
-
maxParallel: number
|
|
904
|
-
): Promise<MutationResult> {
|
|
905
|
-
// Validate concurrency (guard against invalid values)
|
|
906
|
-
const safeConc = Math.max(1, Math.floor(maxParallel));
|
|
907
|
-
|
|
908
|
-
// Result accumulators
|
|
909
|
-
let mutationsExecuted = 0;
|
|
910
|
-
let mutationsFailed = 0;
|
|
911
|
-
const errors: Array<{ locationRef: string; error: string }> = [];
|
|
912
|
-
|
|
913
|
-
// ============================================================================
|
|
914
|
-
// SEQUENTIAL MODE (maxParallel === 1)
|
|
915
|
-
// ============================================================================
|
|
916
|
-
if (safeConc === 1) {
|
|
917
|
-
for (const location of locations) {
|
|
918
|
-
try {
|
|
919
|
-
const { query, variables } = await this.mapper.map(location);
|
|
920
|
-
await this.client.graphql({ query, variables });
|
|
921
|
-
mutationsExecuted++;
|
|
922
|
-
} catch (err: unknown) {
|
|
923
|
-
mutationsFailed++;
|
|
924
|
-
const errorMsg = err instanceof Error ? err.message : String(err);
|
|
925
|
-
errors.push({
|
|
926
|
-
locationRef: location.ref || 'unknown',
|
|
927
|
-
error: errorMsg,
|
|
928
|
-
});
|
|
929
|
-
// Continue processing (failure doesn't block other locations)
|
|
930
|
-
}
|
|
931
|
-
}
|
|
932
|
-
return { mutationsExecuted, mutationsFailed, errors };
|
|
933
|
-
}
|
|
934
|
-
|
|
935
|
-
// ============================================================================
|
|
936
|
-
// PARALLEL MODE (maxParallel > 1)
|
|
937
|
-
// ============================================================================
|
|
938
|
-
// Chunked processing with bounded concurrency
|
|
939
|
-
for (let i = 0; i < locations.length; i += safeConc) {
|
|
940
|
-
const chunk = locations.slice(i, i + safeConc);
|
|
941
|
-
|
|
942
|
-
// Fire all mutations in chunk concurrently
|
|
943
|
-
const results = await Promise.allSettled(
|
|
944
|
-
chunk.map(async (location) => {
|
|
945
|
-
const { query, variables } = await this.mapper.map(location);
|
|
946
|
-
return await this.client.graphql({ query, variables });
|
|
947
|
-
})
|
|
948
|
-
);
|
|
949
|
-
|
|
950
|
-
// Aggregate chunk results
|
|
951
|
-
results.forEach((result, idx) => {
|
|
952
|
-
if (result.status === 'fulfilled') {
|
|
953
|
-
mutationsExecuted++;
|
|
954
|
-
} else {
|
|
955
|
-
mutationsFailed++;
|
|
956
|
-
const error = result.reason;
|
|
957
|
-
errors.push({
|
|
958
|
-
locationRef: chunk[idx]?.ref || 'unknown',
|
|
959
|
-
error: error?.message || String(error) || 'Unknown error',
|
|
960
|
-
});
|
|
961
|
-
}
|
|
962
|
-
});
|
|
963
|
-
|
|
964
|
-
// Small delay between batches to avoid rate limiting
|
|
965
|
-
if (i + safeConc < locations.length) {
|
|
966
|
-
await new Promise(resolve => setTimeout(resolve, 100));
|
|
967
|
-
}
|
|
968
|
-
}
|
|
969
|
-
|
|
970
|
-
return { mutationsExecuted, mutationsFailed, errors };
|
|
971
|
-
}
|
|
972
|
-
|
|
973
|
-
/**
|
|
974
|
-
* ✅ NEW: Execute mutations using GraphQL aliases (batched requests)
|
|
975
|
-
*
|
|
976
|
-
* Groups mutations into alias batches and executes them with concurrency control.
|
|
977
|
-
* Each aliased request contains multiple mutations identified by alias names.
|
|
978
|
-
*
|
|
979
|
-
* @param locations - Array of mapped locations
|
|
980
|
-
* @param maxParallel - Number of concurrent alias batch requests
|
|
981
|
-
* @param mutationsPerAliasBatch - Number of mutations per aliased request
|
|
982
|
-
* @returns Mutation execution results
|
|
983
|
-
*/
|
|
984
|
-
private async executeMutationsWithAliases(
|
|
985
|
-
locations: MappedLocation[],
|
|
986
|
-
maxParallel: number,
|
|
987
|
-
mutationsPerAliasBatch: number
|
|
988
|
-
): Promise<MutationResult> {
|
|
989
|
-
const results: MutationResult = { mutationsExecuted: 0, mutationsFailed: 0, errors: [] };
|
|
990
|
-
|
|
991
|
-
// Extract mutation name from mapper config
|
|
992
|
-
const mutationName = (this.mapper as any).config.mutation || 'createLocation';
|
|
993
|
-
|
|
994
|
-
// Group locations into alias batches
|
|
995
|
-
const aliasBatches: Array<MappedLocation[]> = [];
|
|
996
|
-
for (let i = 0; i < locations.length; i += mutationsPerAliasBatch) {
|
|
997
|
-
aliasBatches.push(locations.slice(i, i + mutationsPerAliasBatch));
|
|
998
|
-
}
|
|
999
|
-
|
|
1000
|
-
// Process batches with concurrency control
|
|
1001
|
-
for (let i = 0; i < aliasBatches.length; i += maxParallel) {
|
|
1002
|
-
const concurrentBatches = aliasBatches.slice(i, i + maxParallel);
|
|
1003
|
-
|
|
1004
|
-
const batchResults = await Promise.allSettled(
|
|
1005
|
-
concurrentBatches.map(async (batch) => {
|
|
1006
|
-
// Build aliased query and variables
|
|
1007
|
-
const { query, variables } = await this.buildAliasedBatch(batch, mutationName);
|
|
1008
|
-
|
|
1009
|
-
// Execute aliased mutation
|
|
1010
|
-
const response = await this.client.graphql({ query, variables });
|
|
1011
|
-
|
|
1012
|
-
// Parse results and errors
|
|
1013
|
-
return this.parseAliasResponse(response, batch, mutationName);
|
|
1014
|
-
})
|
|
1015
|
-
);
|
|
1016
|
-
|
|
1017
|
-
// Aggregate results
|
|
1018
|
-
batchResults.forEach((result, idx) => {
|
|
1019
|
-
if (result.status === 'fulfilled') {
|
|
1020
|
-
const batchResult = result.value;
|
|
1021
|
-
results.mutationsExecuted += batchResult.executed;
|
|
1022
|
-
results.mutationsFailed += batchResult.failed;
|
|
1023
|
-
results.errors.push(...batchResult.errors);
|
|
1024
|
-
} else {
|
|
1025
|
-
// Entire batch failed
|
|
1026
|
-
const batch = concurrentBatches[idx];
|
|
1027
|
-
const errorMsg = result.reason instanceof Error ? result.reason.message : String(result.reason);
|
|
1028
|
-
batch.forEach(loc => {
|
|
1029
|
-
results.mutationsFailed++;
|
|
1030
|
-
results.errors.push({
|
|
1031
|
-
locationRef: loc.ref || 'unknown',
|
|
1032
|
-
error: `Batch execution failed: ${errorMsg}`
|
|
1033
|
-
});
|
|
1034
|
-
});
|
|
1035
|
-
}
|
|
1036
|
-
});
|
|
1037
|
-
|
|
1038
|
-
// Add small delay between concurrent batches to respect rate limits
|
|
1039
|
-
if (i + maxParallel < aliasBatches.length) {
|
|
1040
|
-
await new Promise(resolve => setTimeout(resolve, 500));
|
|
1041
|
-
}
|
|
1042
|
-
}
|
|
1043
|
-
|
|
1044
|
-
return results;
|
|
1045
|
-
}
|
|
1046
|
-
|
|
1047
|
-
/**
|
|
1048
|
-
* ✅ NEW: Build aliased batch query and variables
|
|
1049
|
-
*
|
|
1050
|
-
* @param batch - Array of location objects
|
|
1051
|
-
* @param mutationName - GraphQL mutation name (e.g., 'createLocation')
|
|
1052
|
-
* @returns GraphQL query and variables for aliased batch
|
|
1053
|
-
*/
|
|
1054
|
-
private async buildAliasedBatch(
|
|
1055
|
-
batch: MappedLocation[],
|
|
1056
|
-
mutationName: string
|
|
1057
|
-
): Promise<{ query: string; variables: Record<string, any> }> {
|
|
1058
|
-
const batchSize = batch.length;
|
|
1059
|
-
const inputTypeName = `${mutationName.charAt(0).toUpperCase() + mutationName.slice(1)}Input`;
|
|
1060
|
-
|
|
1061
|
-
// Build aliased query (simple naming: mutationName + index)
|
|
1062
|
-
const variables = Array.from({ length: batchSize }, (_, i) =>
|
|
1063
|
-
`$input${i + 1}: ${inputTypeName}!`
|
|
1064
|
-
).join(', ');
|
|
1065
|
-
|
|
1066
|
-
const aliasedMutations = Array.from({ length: batchSize }, (_, i) => {
|
|
1067
|
-
const alias = `${mutationName}${i + 1}`; // Simple: createLocation1, createLocation2, etc.
|
|
1068
|
-
return ` ${alias}: ${mutationName}(input: $input${i + 1}) { id ref name }`;
|
|
1069
|
-
}).join('\n');
|
|
1070
|
-
|
|
1071
|
-
const operationName = `Batch${mutationName.charAt(0).toUpperCase() + mutationName.slice(1)}s`;
|
|
1072
|
-
const query = `mutation ${operationName}(${variables}) {\n${aliasedMutations}\n}`;
|
|
1073
|
-
|
|
1074
|
-
// Build variables from mapped locations (await all mappings)
|
|
1075
|
-
const mappedItems = await Promise.all(
|
|
1076
|
-
batch.map(loc => this.mapper.map(loc))
|
|
1077
|
-
);
|
|
1078
|
-
|
|
1079
|
-
const variablesObj: Record<string, unknown> = {};
|
|
1080
|
-
mappedItems.forEach((mapped, index) => {
|
|
1081
|
-
const input = mapped.variables.input || mapped.variables;
|
|
1082
|
-
variablesObj[`input${index + 1}`] = input;
|
|
1083
|
-
});
|
|
1084
|
-
|
|
1085
|
-
return { query, variables: variablesObj };
|
|
1086
|
-
}
|
|
1087
|
-
|
|
1088
|
-
/**
|
|
1089
|
-
* ✅ NEW: Parse aliased GraphQL response and extract individual mutation results
|
|
1090
|
-
*
|
|
1091
|
-
* @param response - GraphQL response from aliased mutation
|
|
1092
|
-
* @param batch - Original batch of location objects
|
|
1093
|
-
* @param mutationName - GraphQL mutation name (e.g., 'createLocation')
|
|
1094
|
-
* @returns Parsed results with executed/failed counts and errors
|
|
1095
|
-
*/
|
|
1096
|
-
private parseAliasResponse(
|
|
1097
|
-
response: { data?: Record<string, unknown>; errors?: Array<{ path?: string[]; message: string }> },
|
|
1098
|
-
batch: MappedLocation[],
|
|
1099
|
-
mutationName: string
|
|
1100
|
-
): { executed: number; failed: number; errors: Array<{ locationRef: string; error: string }> } {
|
|
1101
|
-
const result = { executed: 0, failed: 0, errors: [] as Array<{ locationRef: string; error: string }> };
|
|
1102
|
-
|
|
1103
|
-
const data = response.data || {};
|
|
1104
|
-
const errors = response.errors || [];
|
|
1105
|
-
|
|
1106
|
-
// Process each alias result
|
|
1107
|
-
batch.forEach((loc, index) => {
|
|
1108
|
-
const alias = `${mutationName}${index + 1}`; // Simple: createLocation1, createLocation2, etc.
|
|
1109
|
-
const aliasData = data[alias];
|
|
1110
|
-
const aliasErrors = errors.filter((e) =>
|
|
1111
|
-
e.path && Array.isArray(e.path) && e.path.includes(alias)
|
|
1112
|
-
);
|
|
1113
|
-
|
|
1114
|
-
if (aliasData && !aliasErrors.length) {
|
|
1115
|
-
result.executed++;
|
|
1116
|
-
} else {
|
|
1117
|
-
result.failed++;
|
|
1118
|
-
const errorMsg = aliasErrors[0]?.message || 'Mutation failed';
|
|
1119
|
-
const locationRef = loc.ref || 'unknown';
|
|
1120
|
-
result.errors.push({
|
|
1121
|
-
locationRef,
|
|
1122
|
-
error: errorMsg,
|
|
1123
|
-
});
|
|
1124
|
-
}
|
|
1125
|
-
});
|
|
1126
|
-
|
|
1127
|
-
return result;
|
|
1128
|
-
}
|
|
1129
|
-
}
|
|
1130
|
-
```
|
|
1131
|
-
|
|
1132
|
-
### Service 3: Mutation Logger (Complete Implementation)
|
|
1133
|
-
|
|
1134
|
-
**File:** `src/services/mutation-logger.service.ts`
|
|
1135
|
-
|
|
1136
|
-
```typescript
|
|
1137
|
-
/**
|
|
1138
|
-
* Mutation Logger Service
|
|
1139
|
-
*
|
|
1140
|
-
* Writes mutation processing logs to SFTP for audit purposes.
|
|
1141
|
-
* Log writing failures don't stop the workflow (non-critical).
|
|
1142
|
-
*/
|
|
1143
|
-
|
|
1144
|
-
import { Buffer } from 'node:buffer'; // Required for Versori/Deno
|
|
1145
|
-
import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
|
|
1146
|
-
import type { MutationResult } from '../types/location-ingestion.types';
|
|
1147
|
-
|
|
1148
|
-
/**
|
|
1149
|
-
* Service for writing mutation logs to SFTP
|
|
1150
|
-
*/
|
|
1151
|
-
export class MutationLoggerService {
|
|
1152
|
-
constructor(private sftp: SftpDataSource) {}
|
|
1153
|
-
|
|
1154
|
-
/**
|
|
1155
|
-
* Write mutation processing log to SFTP
|
|
1156
|
-
*
|
|
1157
|
-
* Creates JSON log file with mutation results.
|
|
1158
|
-
* Only write logs when there are failures (optional).
|
|
1159
|
-
*
|
|
1160
|
-
* Returns log file path on success, or throws on critical errors.
|
|
1161
|
-
*/
|
|
1162
|
-
async writeMutationLog(
|
|
1163
|
-
fileName: string,
|
|
1164
|
-
result: MutationResult,
|
|
1165
|
-
remotePath: string,
|
|
1166
|
-
requireAbsolutePaths: boolean
|
|
1167
|
-
): Promise<string> {
|
|
1168
|
-
const timestamp = new Date().toISOString();
|
|
1169
|
-
const logContent = JSON.stringify(
|
|
1170
|
-
{
|
|
1171
|
-
fileName,
|
|
1172
|
-
timestamp,
|
|
1173
|
-
mutationsExecuted: result.mutationsExecuted,
|
|
1174
|
-
mutationsFailed: result.mutationsFailed,
|
|
1175
|
-
errors: result.errors,
|
|
1176
|
-
},
|
|
1177
|
-
null,
|
|
1178
|
-
2
|
|
1179
|
-
);
|
|
1180
|
-
|
|
1181
|
-
const logFileName = `${fileName.replace(/\.json$/i, '')}-mutations-${timestamp.replace(/[:.]/g, '-')}.json`;
|
|
1182
|
-
|
|
1183
|
-
const logPath =
|
|
1184
|
-
requireAbsolutePaths && !remotePath.startsWith('/')
|
|
1185
|
-
? `/${remotePath}/${logFileName}`.replace(/\/+/g, '/')
|
|
1186
|
-
: `${remotePath}/${logFileName}`.replace(/\/+/g, '/');
|
|
1187
|
-
|
|
1188
|
-
await this.sftp.uploadFile(logPath, logContent);
|
|
1189
|
-
|
|
1190
|
-
return logPath; // Return path for workflow logging
|
|
1191
|
-
}
|
|
1192
|
-
}
|
|
1193
|
-
```
|
|
1194
|
-
|
|
1195
|
-
---
|
|
1196
|
-
|
|
1197
|
-
## Type Definitions
|
|
1198
|
-
|
|
1199
|
-
**File:** `src/types/location-ingestion.types.ts`
|
|
1200
|
-
|
|
1201
|
-
```typescript
|
|
1202
|
-
export interface LocationRecord {
|
|
1203
|
-
locationRef: string;
|
|
1204
|
-
locationName: string;
|
|
1205
|
-
locationType: string;
|
|
1206
|
-
address: {
|
|
1207
|
-
street1: string;
|
|
1208
|
-
city: string;
|
|
1209
|
-
state: string;
|
|
1210
|
-
postalCode: string;
|
|
1211
|
-
country: string;
|
|
1212
|
-
};
|
|
1213
|
-
geo: {
|
|
1214
|
-
latitude: number;
|
|
1215
|
-
longitude: number;
|
|
1216
|
-
};
|
|
1217
|
-
timeZone: string;
|
|
1218
|
-
openingSchedule: {
|
|
1219
|
-
allHours: boolean;
|
|
1220
|
-
monStart: number;
|
|
1221
|
-
monEnd: number;
|
|
1222
|
-
// ... other schedule fields
|
|
1223
|
-
};
|
|
1224
|
-
}
|
|
1225
|
-
|
|
1226
|
-
export interface MappedLocation {
|
|
1227
|
-
ref: string;
|
|
1228
|
-
name: string;
|
|
1229
|
-
type: string;
|
|
1230
|
-
status: string;
|
|
1231
|
-
primaryAddress: {
|
|
1232
|
-
ref: string;
|
|
1233
|
-
street?: string;
|
|
1234
|
-
city?: string;
|
|
1235
|
-
latitude: number;
|
|
1236
|
-
longitude: number;
|
|
1237
|
-
};
|
|
1238
|
-
openingSchedule: {
|
|
1239
|
-
allHours: boolean;
|
|
1240
|
-
monStart: number;
|
|
1241
|
-
monEnd: number;
|
|
1242
|
-
};
|
|
1243
|
-
}
|
|
1244
|
-
|
|
1245
|
-
export interface MutationResult {
|
|
1246
|
-
mutationsExecuted: number;
|
|
1247
|
-
mutationsFailed: number;
|
|
1248
|
-
errors: Array<{ locationRef: string; error: string }>;
|
|
1249
|
-
}
|
|
1250
|
-
|
|
1251
|
-
export interface ProcessFileResult {
|
|
1252
|
-
success: boolean;
|
|
1253
|
-
locations: Array<{ query: string; variables: Record<string, unknown> }>;
|
|
1254
|
-
error?: string;
|
|
1255
|
-
fileName: string;
|
|
1256
|
-
}
|
|
1257
|
-
|
|
1258
|
-
// Type guard for validation
|
|
1259
|
-
export function isLocationRecord(obj: unknown): obj is LocationRecord {
|
|
1260
|
-
if (typeof obj !== 'object' || obj === null) return false;
|
|
1261
|
-
const loc = obj as Record<string, unknown>;
|
|
1262
|
-
|
|
1263
|
-
return (
|
|
1264
|
-
typeof loc.locationRef === 'string' &&
|
|
1265
|
-
typeof loc.locationName === 'string' &&
|
|
1266
|
-
typeof loc.locationType === 'string' &&
|
|
1267
|
-
typeof loc.address === 'object' &&
|
|
1268
|
-
typeof loc.geo === 'object'
|
|
1269
|
-
);
|
|
1270
|
-
}
|
|
1271
|
-
```
|
|
1272
|
-
|
|
1273
|
-
---
|
|
1274
|
-
|
|
1275
|
-
---
|
|
1276
|
-
|
|
1277
|
-
## Versori Workflows Structure
|
|
1278
|
-
|
|
1279
|
-
**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
|
|
1280
|
-
|
|
1281
|
-
**Trigger Types:**
|
|
1282
|
-
- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
|
|
1283
|
-
- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
|
|
1284
|
-
- **`workflow()`** → Durable workflows (advanced, rarely used)
|
|
1285
|
-
|
|
1286
|
-
**Execution Steps (chained to triggers):**
|
|
1287
|
-
- **`http()`** → External API calls (chained from schedule/webhook)
|
|
1288
|
-
- **`fn()`** → Internal processing (chained from schedule/webhook)
|
|
1289
|
-
|
|
1290
|
-
### Recommended Project Structure
|
|
1291
|
-
|
|
1292
|
-
```
|
|
1293
|
-
sftp-json-location-graphql/
|
|
1294
|
-
├── index.ts # Entry point - exports all workflows
|
|
1295
|
-
└── src/
|
|
1296
|
-
├── workflows/
|
|
1297
|
-
│ ├── scheduled/
|
|
1298
|
-
│ │ └── daily-location-sync.ts # Scheduled: Daily location sync
|
|
1299
|
-
│ │
|
|
1300
|
-
│ └── webhook/
|
|
1301
|
-
│ ├── adhoc-location-sync.ts # Webhook: Manual trigger
|
|
1302
|
-
│ └── job-status-check.ts # Webhook: Status query
|
|
1303
|
-
│
|
|
1304
|
-
├── services/
|
|
1305
|
-
│ └── location-sync.service.ts # Shared orchestration logic (reusable)
|
|
1306
|
-
│
|
|
1307
|
-
└── config/
|
|
1308
|
-
└── location-mapping.json # GraphQL mapping config
|
|
1309
|
-
```
|
|
1310
|
-
|
|
1311
|
-
---
|
|
1312
|
-
|
|
1313
|
-
## Workflow Files
|
|
1314
|
-
|
|
1315
|
-
### 1. Scheduled Workflows (`src/workflows/scheduled/`)
|
|
1316
|
-
|
|
1317
|
-
All time-based triggers that run automatically on cron schedules.
|
|
1318
|
-
|
|
1319
|
-
#### `src/workflows/scheduled/daily-location-sync.ts`
|
|
1320
|
-
|
|
1321
|
-
**Purpose**: Automatic daily location sync
|
|
1322
|
-
**Trigger**: Cron schedule (`0 2 * * *`)
|
|
1323
|
-
**Exposed as Endpoint**: ❌ NO - Runs automatically
|
|
1324
|
-
|
|
1325
|
-
```typescript
|
|
1326
|
-
import { schedule, http } from '@versori/run';
|
|
1327
|
-
import { executeLocationIngestion } from '../../services/location-sync.service';
|
|
1328
|
-
import { generateJobId } from '../../utils/job-id-generator';
|
|
1329
|
-
|
|
1330
|
-
/**
|
|
1331
|
-
* Scheduled Workflow: Daily Location Sync
|
|
1332
|
-
*
|
|
1333
|
-
* Runs automatically daily at 2 AM UTC
|
|
1334
|
-
* NOT exposed as HTTP endpoint - Versori executes on schedule
|
|
1335
|
-
*
|
|
1336
|
-
* Uses shared service: location-sync.service.ts
|
|
1337
|
-
*/
|
|
1338
|
-
export const dailyLocationSync = schedule(
|
|
1339
|
-
'location-sync-scheduled',
|
|
1340
|
-
'0 2 * * *' // Daily at 2 AM UTC
|
|
1341
|
-
).then(
|
|
1342
|
-
http('run-location-sync', { connection: 'fluent_commerce' }, async ctx => {
|
|
1343
|
-
const jobId = generateJobId('SCHEDULED', 'LOC');
|
|
1344
|
-
return await executeLocationIngestion(ctx, { jobId, triggeredBy: 'schedule' });
|
|
1345
|
-
})
|
|
1346
|
-
);
|
|
1347
|
-
```
|
|
1348
|
-
|
|
1349
|
-
---
|
|
1350
|
-
|
|
1351
|
-
### 2. Webhook Workflows (`src/workflows/webhook/`)
|
|
1352
|
-
|
|
1353
|
-
All HTTP-based triggers that create webhook endpoints.
|
|
1354
|
-
|
|
1355
|
-
#### `src/workflows/webhook/adhoc-location-sync.ts`
|
|
1356
|
-
|
|
1357
|
-
**Purpose**: Manual location sync trigger (on-demand)
|
|
1358
|
-
**Trigger**: HTTP POST
|
|
1359
|
-
**Endpoint**: `POST https://{workspace}.versori.run/location-sync-adhoc`
|
|
1360
|
-
**Use Cases**: Testing, priority processing, ad-hoc runs
|
|
1361
|
-
|
|
1362
|
-
```typescript
|
|
1363
|
-
import { webhook, http } from '@versori/run';
|
|
1364
|
-
import { executeLocationIngestion } from '../../services/location-sync.service';
|
|
1365
|
-
import { generateJobId } from '../../utils/job-id-generator';
|
|
1366
|
-
|
|
1367
|
-
/**
|
|
1368
|
-
* Webhook: Manual Location Sync Trigger
|
|
1369
|
-
*
|
|
1370
|
-
* Endpoint: POST https://{workspace}.versori.run/location-sync-adhoc
|
|
1371
|
-
* Request body (optional): { filePattern: "urgent_*.json", maxFiles: 5, forceReprocess: true }
|
|
1372
|
-
*
|
|
1373
|
-
* Pattern: webhook().then(http()) - needs Fluent API access
|
|
1374
|
-
* Uses shared service: location-sync.service.ts
|
|
1375
|
-
*
|
|
1376
|
-
* SECURITY: Authentication handled via connection parameter
|
|
1377
|
-
* No manual API key validation needed - Versori manages this via connection auth
|
|
1378
|
-
*/
|
|
1379
|
-
export const adhocLocationSync = webhook('location-sync-adhoc', {
|
|
1380
|
-
response: { mode: 'sync' },
|
|
1381
|
-
connection: 'location-sync-adhoc', // Versori validates API key
|
|
1382
|
-
}).then(
|
|
1383
|
-
http('run-location-sync-adhoc', { connection: 'fluent_commerce' }, async ctx => {
|
|
1384
|
-
const jobId = generateJobId('ADHOC', 'LOC');
|
|
1385
|
-
const { filePattern, maxFiles, forceReprocess } = ctx.data;
|
|
1386
|
-
return await executeLocationIngestion(ctx, {
|
|
1387
|
-
jobId,
|
|
1388
|
-
triggeredBy: 'manual',
|
|
1389
|
-
filePattern,
|
|
1390
|
-
maxFiles,
|
|
1391
|
-
forceReprocess,
|
|
1392
|
-
});
|
|
1393
|
-
})
|
|
1394
|
-
);
|
|
1395
|
-
```
|
|
1396
|
-
|
|
1397
|
-
---
|
|
1398
|
-
|
|
1399
|
-
#### `src/workflows/webhook/job-status-check.ts`
|
|
1400
|
-
|
|
1401
|
-
**Purpose**: Query job status and progress
|
|
1402
|
-
**Trigger**: HTTP POST
|
|
1403
|
-
**Endpoint**: `POST https://{workspace}.versori.run/location-sync-job-status`
|
|
1404
|
-
**Request Body**: `{ "jobId": "SCHEDULED_LOC_20250124_183045_abc123" }`
|
|
1405
|
-
|
|
1406
|
-
```typescript
|
|
1407
|
-
import { webhook, fn } from '@versori/run';
|
|
1408
|
-
import { getJobStatus } from '../../services/location-sync.service';
|
|
1409
|
-
|
|
1410
|
-
/**
|
|
1411
|
-
* Webhook: Job Status Check
|
|
1412
|
-
*
|
|
1413
|
-
* Endpoint: POST https://{workspace}.versori.run/location-sync-job-status
|
|
1414
|
-
* Request body: { "jobId": "SCHEDULED_LOC_20250124_183045_abc123" }
|
|
1415
|
-
*
|
|
1416
|
-
* Pattern: webhook().then(fn()) - no external API needed, only KV storage
|
|
1417
|
-
* Lightweight: Only queries KV store, no Fluent API calls
|
|
1418
|
-
*
|
|
1419
|
-
* SECURITY: Authentication handled via connection parameter
|
|
1420
|
-
* No manual API key validation needed - Versori manages this via connection auth
|
|
1421
|
-
*/
|
|
1422
|
-
export const locationSyncJobStatus = webhook('location-sync-job-status', {
|
|
1423
|
-
response: { mode: 'sync' },
|
|
1424
|
-
connection: 'location-sync-job-status',
|
|
1425
|
-
}).then(
|
|
1426
|
-
fn('query-job-status', async ctx => {
|
|
1427
|
-
const { jobId } = ctx.data;
|
|
1428
|
-
const status = await getJobStatus(ctx.openKv(':project:'), jobId, ctx.log);
|
|
1429
|
-
return { success: !!status, jobId, ...status };
|
|
1430
|
-
})
|
|
1431
|
-
);
|
|
1432
|
-
```
|
|
1433
|
-
|
|
1434
|
-
---
|
|
1435
|
-
|
|
1436
|
-
### 3. Entry Point (`index.ts`)
|
|
1437
|
-
|
|
1438
|
-
**Purpose**: Register all workflows with Versori platform
|
|
1439
|
-
|
|
1440
|
-
```typescript
|
|
1441
|
-
/**
|
|
1442
|
-
* Entry Point - Registers all workflows with Versori platform
|
|
1443
|
-
*
|
|
1444
|
-
* Versori automatically discovers and registers exported workflows
|
|
1445
|
-
*
|
|
1446
|
-
* File Structure:
|
|
1447
|
-
* - src/workflows/scheduled/ → Time-based triggers (cron)
|
|
1448
|
-
* - src/workflows/webhook/ → HTTP-based triggers (webhooks)
|
|
1449
|
-
*/
|
|
1450
|
-
|
|
1451
|
-
import { MemoryInterpreter } from '@versori/run';
|
|
1452
|
-
|
|
1453
|
-
// Import scheduled workflows
|
|
1454
|
-
import { dailyLocationSync } from './src/workflows/scheduled/daily-location-sync';
|
|
1455
|
-
|
|
1456
|
-
// Import webhook workflows
|
|
1457
|
-
import { adhocLocationSync } from './src/workflows/webhook/adhoc-location-sync';
|
|
1458
|
-
import { locationSyncJobStatus } from './src/workflows/webhook/job-status-check';
|
|
1459
|
-
|
|
1460
|
-
// Register all workflows
|
|
1461
|
-
export {
|
|
1462
|
-
// Scheduled (time-based triggers)
|
|
1463
|
-
dailyLocationSync,
|
|
1464
|
-
|
|
1465
|
-
// Webhooks (HTTP-based triggers)
|
|
1466
|
-
adhocLocationSync,
|
|
1467
|
-
locationSyncJobStatus,
|
|
1468
|
-
};
|
|
1469
|
-
|
|
1470
|
-
// ✅ Memory interpreter for local development
|
|
1471
|
-
export const interpreter = new MemoryInterpreter({
|
|
1472
|
-
workflows: [dailyLocationSync, adhocLocationSync, locationSyncJobStatus],
|
|
1473
|
-
});
|
|
1474
|
-
```
|
|
1475
|
-
|
|
1476
|
-
**What Gets Exposed:**
|
|
1477
|
-
- ✅ `adhocLocationSync` → `https://{workspace}.versori.run/location-sync-adhoc`
|
|
1478
|
-
- ✅ `locationSyncJobStatus` → `https://{workspace}.versori.run/location-sync-job-status`
|
|
1479
|
-
- ❌ `dailyLocationSync` → NOT exposed (runs automatically on cron)
|
|
1480
|
-
|
|
1481
|
-
---
|
|
1482
|
-
|
|
1483
|
-
## Package Configuration
|
|
1484
|
-
|
|
1485
|
-
**File:** `package.json`
|
|
1486
|
-
|
|
1487
|
-
```json
|
|
1488
|
-
{
|
|
1489
|
-
"name": "sftp-json-location-graphql",
|
|
1490
|
-
"type": "module",
|
|
1491
|
-
"main": "index.ts",
|
|
1492
|
-
"dependencies": {
|
|
1493
|
-
"@fluentcommerce/fc-connect-sdk": "^0.1.39",
|
|
1494
|
-
"@versori/run": "latest"
|
|
1495
|
-
},
|
|
1496
|
-
"devDependencies": {
|
|
1497
|
-
"@types/node": "^20.0.0",
|
|
1498
|
-
"typescript": "^5.0.0"
|
|
1499
|
-
}
|
|
1500
|
-
}
|
|
1501
|
-
```
|
|
1502
|
-
|
|
1503
|
-
**File:** `tsconfig.json`
|
|
1504
|
-
|
|
1505
|
-
```json
|
|
1506
|
-
{
|
|
1507
|
-
"compilerOptions": {
|
|
1508
|
-
"module": "ES2022",
|
|
1509
|
-
"target": "ES2024",
|
|
1510
|
-
"moduleResolution": "node"
|
|
1511
|
-
}
|
|
1512
|
-
}
|
|
1513
|
-
```
|
|
1514
|
-
|
|
1515
|
-
---
|
|
1516
|
-
|
|
1517
|
-
## Deployment
|
|
1518
|
-
|
|
1519
|
-
```bash
|
|
1520
|
-
# 1. Install dependencies
|
|
1521
|
-
npm install
|
|
1522
|
-
|
|
1523
|
-
# 2. Deploy to Versori
|
|
1524
|
-
npm run deploy
|
|
1525
|
-
|
|
1526
|
-
# 3. Configure activation variables in Versori Dashboard
|
|
1527
|
-
# CRITICAL: Set fluentRetailerId
|
|
1528
|
-
|
|
1529
|
-
# 4. Test with sample file
|
|
1530
|
-
curl -X POST https://your-workspace.versori.run/location-ingestion-adhoc \
|
|
1531
|
-
-H "Content-Type: application/json" \
|
|
1532
|
-
-d '{"forceReprocess": false}'
|
|
1533
|
-
```
|
|
1534
|
-
|
|
1535
|
-
---
|
|
1536
|
-
|
|
1537
|
-
## Testing
|
|
1538
|
-
|
|
1539
|
-
### Test Scheduled Run
|
|
1540
|
-
Upload test JSON to SFTP and wait for scheduled execution (2 AM daily).
|
|
1541
|
-
|
|
1542
|
-
### Test Ad hoc Trigger
|
|
1543
|
-
```bash
|
|
1544
|
-
curl -X POST https://workspace.versori.run/location-ingestion-adhoc \
|
|
1545
|
-
-d '{"filePattern": "test_*.json"}'
|
|
1546
|
-
```
|
|
1547
|
-
|
|
1548
|
-
### Test Job Status
|
|
1549
|
-
```bash
|
|
1550
|
-
curl -X POST https://workspace.versori.run/location-ingestion-job-status \
|
|
1551
|
-
-d '{"jobId": "ADHOC_LOC_20251101_183045_abc123"}'
|
|
1552
|
-
```
|
|
1553
|
-
|
|
1554
|
-
---
|
|
1555
|
-
|
|
1556
|
-
## Monitoring
|
|
1557
|
-
|
|
1558
|
-
### Success Response
|
|
1559
|
-
|
|
1560
|
-
```json
|
|
1561
|
-
{
|
|
1562
|
-
"success": true,
|
|
1563
|
-
"filesProcessed": 1,
|
|
1564
|
-
"filesSkipped": 0,
|
|
1565
|
-
"filesFailed": 0,
|
|
1566
|
-
"totalRecords": 50,
|
|
1567
|
-
"mutationsExecuted": 50,
|
|
1568
|
-
"mutationsFailed": 0,
|
|
1569
|
-
"results": [
|
|
1570
|
-
{
|
|
1571
|
-
"file": "locations_2025-01-22.json",
|
|
1572
|
-
"success": true,
|
|
1573
|
-
"recordsProcessed": 50,
|
|
1574
|
-
"mutationsExecuted": 50,
|
|
1575
|
-
"mutationsFailed": 0
|
|
1576
|
-
}
|
|
1577
|
-
],
|
|
1578
|
-
"duration": 12345
|
|
1579
|
-
}
|
|
1580
|
-
```
|
|
1581
|
-
|
|
1582
|
-
### Partial Success Response
|
|
1583
|
-
|
|
1584
|
-
```json
|
|
1585
|
-
{
|
|
1586
|
-
"success": true,
|
|
1587
|
-
"filesProcessed": 1,
|
|
1588
|
-
"filesSkipped": 0,
|
|
1589
|
-
"filesFailed": 0,
|
|
1590
|
-
"totalRecords": 50,
|
|
1591
|
-
"mutationsExecuted": 45,
|
|
1592
|
-
"mutationsFailed": 5,
|
|
1593
|
-
"results": [
|
|
1594
|
-
{
|
|
1595
|
-
"file": "locations_2025-01-22.json",
|
|
1596
|
-
"success": true,
|
|
1597
|
-
"recordsProcessed": 50,
|
|
1598
|
-
"mutationsExecuted": 45,
|
|
1599
|
-
"mutationsFailed": 5,
|
|
1600
|
-
"errors": ["LOC-001: Invalid location ref", "LOC-002: Missing required field"]
|
|
1601
|
-
}
|
|
1602
|
-
],
|
|
1603
|
-
"duration": 12345
|
|
1604
|
-
}
|
|
1605
|
-
```
|
|
1606
|
-
|
|
1607
|
-
### Error Response
|
|
1608
|
-
|
|
1609
|
-
```json
|
|
1610
|
-
{
|
|
1611
|
-
"success": false,
|
|
1612
|
-
"filesProcessed": 0,
|
|
1613
|
-
"filesFailed": 1,
|
|
1614
|
-
"totalRecords": 0,
|
|
1615
|
-
"mutationsExecuted": 0,
|
|
1616
|
-
"mutationsFailed": 0,
|
|
1617
|
-
"results": [
|
|
1618
|
-
{
|
|
1619
|
-
"file": "locations_2025-01-22.json",
|
|
1620
|
-
"success": false,
|
|
1621
|
-
"error": "JSON parse error: Invalid structure"
|
|
1622
|
-
}
|
|
1623
|
-
],
|
|
1624
|
-
"duration": 876
|
|
1625
|
-
}
|
|
1626
|
-
```
|
|
1627
|
-
|
|
1628
|
-
### Monitoring Metrics
|
|
1629
|
-
|
|
1630
|
-
Monitor these metrics via Versori logs filtered by `jobId` or `stage`:
|
|
1631
|
-
|
|
1632
|
-
- **Files Processed** - Total files successfully processed
|
|
1633
|
-
- **Mutations Executed** - Total GraphQL mutations executed successfully
|
|
1634
|
-
- **Mutations Failed** - Mutations that failed (check error logs)
|
|
1635
|
-
- **Processing Duration** - Time taken for complete workflow
|
|
1636
|
-
- **Rate Limiting** - Watch for 429 errors indicating GraphQL throttling
|
|
1637
|
-
|
|
1638
|
-
Use the status webhook for dashboards and automated monitoring.
|
|
1639
|
-
|
|
1640
|
-
---
|
|
1641
|
-
|
|
1642
|
-
Before deployment, verify:
|
|
1643
|
-
|
|
1644
|
-
- [ ] ✅ `GraphQLMutationMapper` imported (NOT `UniversalMapper`)
|
|
1645
|
-
- [ ] ✅ NO `client.setRetailerId()` call (not needed for GraphQL mutations)
|
|
1646
|
-
- [ ] ✅ `fluentRetailerId` activation variable configured (only if mutation schema requires it)
|
|
1647
|
-
- [ ] ✅ Mapping config in external JSON file
|
|
1648
|
-
- [ ] ✅ Mapping config uses correct structure (`mutation` property, `arguments.input`, resolver names without `sdk.` prefix)
|
|
1649
|
-
- [ ] ✅ Uses `mapper.map()` method (returns `{ query, variables }`)
|
|
1650
|
-
- [ ] ✅ Type definitions include `LocationRecord`, `MappedLocation`
|
|
1651
|
-
- [ ] ✅ Services follow modular pattern (no logger params in business logic)
|
|
1652
|
-
- [ ] ✅ Three workflows implemented (scheduled, ad hoc, status)
|
|
1653
|
-
- [ ] ✅ Buffer imported from `node:buffer`
|
|
1654
|
-
- [ ] ✅ **Double-finally SFTP disposal** pattern implemented (inner + outer finally)
|
|
1655
|
-
- [ ] ✅ Complete service implementations (processor, sender, logger)
|
|
1656
|
-
- [ ] ✅ Error handling with per-record tracking
|
|
1657
|
-
- [ ] ✅ Processing mode documented (per-file vs chunked)
|
|
1658
|
-
|
|
1659
|
-
---
|
|
1660
|
-
|
|
1661
|
-
## Troubleshooting
|
|
1662
|
-
|
|
1663
|
-
### GraphQL Mutations Failing
|
|
1664
|
-
**Problem:** Mutations return auth errors
|
|
1665
|
-
**Solution:**
|
|
1666
|
-
1. Verify OAuth2 credentials are correct in Versori connection
|
|
1667
|
-
2. Check if mutation schema requires retailerId in input variables
|
|
1668
|
-
3. Do NOT use `client.setRetailerId()` - it's only for Job/Event API
|
|
1669
|
-
|
|
1670
|
-
### Wrong Mapper Error
|
|
1671
|
-
**Problem:** "UniversalMapper not suitable for GraphQL"
|
|
1672
|
-
**Solution:** Replace `UniversalMapper` with `GraphQLMutationMapper`
|
|
1673
|
-
|
|
1674
|
-
### Mapping Config Errors
|
|
1675
|
-
**Problem:** "Invalid mapping structure"
|
|
1676
|
-
**Solution:** Ensure config has `mutation` property (not `mutationName`) and `arguments.input` structure. Resolver names should be without `sdk.` prefix (`trim`, `toUpperCase`, `parseInt`, `parseFloat`, `toBoolean`)
|
|
1677
|
-
|
|
1678
|
-
### Wrong Method Name
|
|
1679
|
-
**Problem:** "generateMutation is not a function"
|
|
1680
|
-
**Solution:** Use `mapper.map()` method, not `generateMutation()`. Returns `{ query, variables }`
|
|
1681
|
-
|
|
1682
|
-
---
|
|
1683
|
-
|
|
1684
|
-
## Key Takeaways
|
|
1685
|
-
|
|
1686
|
-
- ✅ **Use GraphQLMutationMapper** for GraphQL mutations (NOT UniversalMapper)
|
|
1687
|
-
- ⚠️ **Set retailerId** on client before mutations (VERIFICATION REQUIRED - may need mutation input instead)
|
|
1688
|
-
- ✅ **Double-finally SFTP disposal** for resource safety (matches Event API gold standard)
|
|
1689
|
-
- ✅ **External JSON config** for production (not inline)
|
|
1690
|
-
- ✅ **Modular services** for testability and reusability
|
|
1691
|
-
- ✅ **Complete error handling** with per-record tracking
|
|
1692
|
-
- ✅ **Processing modes** documented (per-file recommended, chunked optional)
|
|
1693
|
-
- ✅ **Type-safe** with comprehensive interfaces
|
|
1694
|
-
- ✅ **Native Versori logging** (LoggingService removed - use native log)
|
|
1695
|
-
- ✅ **GraphQL alias batching** support for high-volume scenarios (mutationsPerAliasBatch parameter)
|
|
1696
|
-
|
|
1697
|
-
---
|
|
1698
|
-
|
|
1699
|
-
## References
|
|
1700
|
-
|
|
1701
|
-
- **Gold Standard Guide:** `internal/GOLD-STANDARD-APPLICATION-GUIDE.md`
|
|
1702
|
-
- **GraphQL Mutation Mapping:** `fc-connect-sdk/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/`
|
|
1703
|
-
- **GraphQL Alias Batching:** `fc-connect-sdk/docs/02-CORE-GUIDES/mapping/graphql-alias-batching-guide.md`
|
|
1704
|
-
- **retailerId Configuration:** `fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md`
|
|
1705
|
-
- **Event API Template:** `template-ingestion-sftp-xml-product-event.md` (architectural reference)
|
|
1706
|
-
|
|
1707
|
-
---
|
|
1708
|
-
|
|
1709
|
-
**Status:** ✅ Gold Standard Compliant
|
|
1710
|
-
**Compliance:** All patterns verified against SDK and Fluent GraphQL schema
|
|
1
|
+
---
|
|
2
|
+
template_id: tpl-ingest-sftp-json-to-location-graphql
|
|
3
|
+
canonical_filename: template-ingestion-sftp-json-location-graphql.md
|
|
4
|
+
version: 2.0.0
|
|
5
|
+
sdk_version: ^0.1.39
|
|
6
|
+
runtime: versori
|
|
7
|
+
direction: ingestion
|
|
8
|
+
source: sftp-json
|
|
9
|
+
destination: fluent-graphql
|
|
10
|
+
entity: location
|
|
11
|
+
format: json
|
|
12
|
+
logging: versori
|
|
13
|
+
status: stable
|
|
14
|
+
compliance: gold-standard
|
|
15
|
+
features:
|
|
16
|
+
- graphql-mutation-mapper
|
|
17
|
+
- memory-management
|
|
18
|
+
- enhanced-logging
|
|
19
|
+
- attribute-transformation
|
|
20
|
+
- dispose-finally
|
|
21
|
+
---
|
|
22
|
+
|
|
23
|
+
# Template: Ingestion - SFTP JSON to Location GraphQL
|
|
24
|
+
|
|
25
|
+
**Deployment Target:** Versori Platform
|
|
26
|
+
**Compliance Status:** ✅ GOLD STANDARD APPROVED
|
|
27
|
+
|
|
28
|
+
---
|
|
29
|
+
|
|
30
|
+
## 📋 Implementation Prompt
|
|
31
|
+
|
|
32
|
+
```
|
|
33
|
+
I need a Versori scheduled ingestion that:
|
|
34
|
+
|
|
35
|
+
1) Discovers JSON files on SFTP with file tracking to skip duplicates
|
|
36
|
+
2) Downloads and parses JSON (locations array) with JSONParserService
|
|
37
|
+
3) Transforms records with GraphQLMutationMapper (NOT UniversalMapper)
|
|
38
|
+
4) Executes GraphQL createLocation mutations with rate limiting
|
|
39
|
+
5) Archives files to processed/ or errors/ and writes error reports
|
|
40
|
+
6) Tracks progress with JobTracker and exposes job status webhook
|
|
41
|
+
7) Uses native Versori log and follows modular service architecture
|
|
42
|
+
|
|
43
|
+
CRITICAL: This template uses GraphQLMutationMapper for GraphQL mutations,
|
|
44
|
+
NOT UniversalMapper (which is for Batch API). Do NOT use setRetailerId()
|
|
45
|
+
for GraphQL mutations - it's only needed for Job/Event API.
|
|
46
|
+
|
|
47
|
+
Use the loaded docs to fill in SDK specifics and best practices.
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
---
|
|
51
|
+
|
|
52
|
+
## 📋 Template Overview
|
|
53
|
+
|
|
54
|
+
This connector runs on the Versori platform with **gold standard compliance**. It reads location data from SFTP JSON files, transforms it with GraphQLMutationMapper, and creates/updates locations via GraphQL mutations.
|
|
55
|
+
|
|
56
|
+
### What This Template Does
|
|
57
|
+
|
|
58
|
+
```
|
|
59
|
+
┌─────────────────────────────────────────────────────────────────┐
|
|
60
|
+
│ INGESTION WORKFLOW (GraphQL Mutations Pattern) │
|
|
61
|
+
└─────────────────────────────────────────────────────────────────┘
|
|
62
|
+
|
|
63
|
+
1. TRIGGER
|
|
64
|
+
├─ Scheduled (Cron): Daily at 2 AM
|
|
65
|
+
├─ Ad hoc (Webhook): Manual trigger
|
|
66
|
+
└─ Status Query (Webhook): Check progress
|
|
67
|
+
|
|
68
|
+
2. DISCOVER FILES (SftpDataSource)
|
|
69
|
+
├─ List files matching pattern
|
|
70
|
+
├─ Check VersoriFileTracker (skip processed)
|
|
71
|
+
└─ Sort by oldest first
|
|
72
|
+
|
|
73
|
+
3. DOWNLOAD & PARSE (LocationFileProcessorService)
|
|
74
|
+
├─ Download from SFTP
|
|
75
|
+
├─ Parse JSON with JSONParserService
|
|
76
|
+
├─ Validate structure with type guards
|
|
77
|
+
└─ Return LocationRecord[]
|
|
78
|
+
|
|
79
|
+
4. TRANSFORM (GraphQLMutationMapper)
|
|
80
|
+
├─ Map to GraphQL CreateLocationInput
|
|
81
|
+
├─ Handle nested objects (primaryAddress, openingSchedule)
|
|
82
|
+
├─ Validate against GraphQL schema
|
|
83
|
+
└─ Generate mutation query
|
|
84
|
+
|
|
85
|
+
5. EXECUTE MUTATIONS (MutationSenderService)
|
|
86
|
+
├─ Execute createLocation mutations
|
|
87
|
+
├─ Rate limiting (configurable concurrency)
|
|
88
|
+
├─ Per-record error handling
|
|
89
|
+
└─ Track success/failure per mutation
|
|
90
|
+
|
|
91
|
+
6. ARCHIVE & LOG
|
|
92
|
+
├─ Move to processed/ or errors/
|
|
93
|
+
├─ Write error reports
|
|
94
|
+
└─ Track state in VersoriFileTracker
|
|
95
|
+
|
|
96
|
+
7. JOB TRACKING
|
|
97
|
+
├─ Update job status at each step
|
|
98
|
+
└─ Enable status queries via webhook
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
### Key Features
|
|
102
|
+
|
|
103
|
+
- ✅ **Gold Standard Compliant** - Modular architecture, proper patterns
|
|
104
|
+
- ✅ **GraphQLMutationMapper** - Correct mapper for GraphQL mutations
|
|
105
|
+
- ✅ **retailerId Configuration** - CRITICAL for GraphQL API calls
|
|
106
|
+
- ✅ **Type-Safe** - Comprehensive TypeScript interfaces
|
|
107
|
+
- ✅ **Modular Services** - 4 services + utils (testable, reusable)
|
|
108
|
+
- ✅ **External Config** - Mapping in JSON file
|
|
109
|
+
- ✅ **Error Handling** - Per-record failures don't block others
|
|
110
|
+
- ✅ **File Tracking** - Prevents duplicate processing
|
|
111
|
+
- ✅ **Job Tracking** - Status queries via webhook
|
|
112
|
+
|
|
113
|
+
---
|
|
114
|
+
|
|
115
|
+
## 📦 SDK Imports
|
|
116
|
+
|
|
117
|
+
```typescript
|
|
118
|
+
import { Buffer } from 'node:buffer'; // Required for Versori/Deno
|
|
119
|
+
|
|
120
|
+
import {
|
|
121
|
+
createClient,
|
|
122
|
+
SftpDataSource,
|
|
123
|
+
JSONParserService,
|
|
124
|
+
GraphQLMutationMapper, // ✅ CRITICAL: Use this for GraphQL (NOT UniversalMapper)
|
|
125
|
+
VersoriFileTracker,
|
|
126
|
+
JobTracker,
|
|
127
|
+
} from '@fluentcommerce/fc-connect-sdk';
|
|
128
|
+
|
|
129
|
+
import type {
|
|
130
|
+
FluentClient,
|
|
131
|
+
FileMetadata,
|
|
132
|
+
Logger,
|
|
133
|
+
} from '@fluentcommerce/fc-connect-sdk';
|
|
134
|
+
|
|
135
|
+
import { schedule, webhook, http, fn } from '@versori/run';
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
**✅ Type-Only Imports:** Separating value and type imports improves tree-shaking and bundle size.
|
|
139
|
+
|
|
140
|
+
---
|
|
141
|
+
|
|
142
|
+
---
|
|
143
|
+
|
|
144
|
+
## SFTP Connection Setup (Recommended)
|
|
145
|
+
|
|
146
|
+
**🔒 BEST PRACTICE:** Store SFTP credentials in a Versori connection object with Basic Auth:
|
|
147
|
+
|
|
148
|
+
### Connection Configuration
|
|
149
|
+
|
|
150
|
+
1. In Versori platform, create a connection named `SFTP`
|
|
151
|
+
2. Set **Authentication Type**: `Basic Auth`
|
|
152
|
+
3. Enter **Username**: Your SFTP username
|
|
153
|
+
4. Enter **Password**: Your SFTP password
|
|
154
|
+
5. The SDK will automatically retrieve and decode credentials
|
|
155
|
+
|
|
156
|
+
### Code Implementation
|
|
157
|
+
|
|
158
|
+
The workflow retrieves credentials programmatically from the Versori connection:
|
|
159
|
+
|
|
160
|
+
```typescript
|
|
161
|
+
import { Buffer } from 'node:buffer';
|
|
162
|
+
|
|
163
|
+
// Retrieve SFTP credentials from connection configuration
|
|
164
|
+
log.info('Retrieving SFTP credentials from connection configuration');
|
|
165
|
+
|
|
166
|
+
let sftpUsername: string;
|
|
167
|
+
let sftpPassword: string;
|
|
168
|
+
|
|
169
|
+
try {
|
|
170
|
+
// Retrieve credentials from the 'SFTP' connection
|
|
171
|
+
const sftpCred = await ctx.credentials().getAccessToken('SFTP');
|
|
172
|
+
|
|
173
|
+
if (!sftpCred?.accessToken) {
|
|
174
|
+
throw new Error('No SFTP credentials found in connection configuration');
|
|
175
|
+
}
|
|
176
|
+
|
|
177
|
+
// Decode base64 accessToken to get "username:password"
|
|
178
|
+
const rawBasicAuth = Buffer.from(sftpCred.accessToken, 'base64').toString('utf-8');
|
|
179
|
+
|
|
180
|
+
// Split on ':' to extract username and password
|
|
181
|
+
const parts = rawBasicAuth.split(':');
|
|
182
|
+
|
|
183
|
+
if (parts.length !== 2) {
|
|
184
|
+
throw new Error('Invalid SFTP credential format - expected username:password');
|
|
185
|
+
}
|
|
186
|
+
|
|
187
|
+
sftpUsername = parts[0];
|
|
188
|
+
sftpPassword = parts[1];
|
|
189
|
+
|
|
190
|
+
log.info('SFTP credentials retrieved successfully', {
|
|
191
|
+
hasUsername: !!sftpUsername,
|
|
192
|
+
hasPassword: !!sftpPassword,
|
|
193
|
+
usernameLength: sftpUsername.length,
|
|
194
|
+
passwordLength: sftpPassword.length,
|
|
195
|
+
});
|
|
196
|
+
} catch (error: any) {
|
|
197
|
+
log.error('Failed to retrieve SFTP credentials', {
|
|
198
|
+
message: error instanceof Error ? error.message : String(error),
|
|
199
|
+
stack: error instanceof Error ? error.stack : undefined,
|
|
200
|
+
errorType: error instanceof Error ? error.constructor.name : 'Error'
|
|
201
|
+
});
|
|
202
|
+
|
|
203
|
+
return {
|
|
204
|
+
success: false,
|
|
205
|
+
error: 'Failed to retrieve SFTP credentials from connection configuration',
|
|
206
|
+
details: error?.message,
|
|
207
|
+
recommendation: 'Please ensure the SFTP connection is configured in the Connections section with Basic Authentication (username and password)',
|
|
208
|
+
};
|
|
209
|
+
}
|
|
210
|
+
|
|
211
|
+
// Use credentials in SftpDataSource
|
|
212
|
+
const sftp = new SftpDataSource({
|
|
213
|
+
type: 'SFTP_JSON',
|
|
214
|
+
connectionId: 'sftp-location-sync',
|
|
215
|
+
name: 'Location Sync SFTP',
|
|
216
|
+
settings: {
|
|
217
|
+
host: activation.getVariable('sftpHost'),
|
|
218
|
+
port: parseInt(activation.getVariable('sftpPort') || '22', 10),
|
|
219
|
+
username: sftpUsername, // From connection
|
|
220
|
+
password: sftpPassword, // From connection
|
|
221
|
+
remotePath: activation.getVariable('sftpRemotePath'),
|
|
222
|
+
filePattern: activation.getVariable('filePattern'),
|
|
223
|
+
encoding: 'utf8',
|
|
224
|
+
},
|
|
225
|
+
}, log);
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
### Why Use Connections Instead of Activation Variables?
|
|
229
|
+
|
|
230
|
+
| Aspect | Activation Variables ❌ | Versori Connections ✅ |
|
|
231
|
+
|--------|------------------------|------------------------|
|
|
232
|
+
| **Security** | Plain text in config UI | Encrypted in Versori vault |
|
|
233
|
+
| **Visibility** | Visible to all users | Controlled access |
|
|
234
|
+
| **Rotation** | Manual update per workflow | Update once, affects all |
|
|
235
|
+
| **Best Practice** | Legacy approach | **Recommended approach** |
|
|
236
|
+
| **Audit Trail** | Limited | Full connection audit logs |
|
|
237
|
+
|
|
238
|
+
**Benefits:**
|
|
239
|
+
- ✅ Credentials stored securely in Versori vault
|
|
240
|
+
- ✅ Connection can be reused across workflows
|
|
241
|
+
- ✅ No sensitive data in activation variables
|
|
242
|
+
- ✅ Easier credential rotation (one place to update)
|
|
243
|
+
- ✅ Better security posture (credentials never exposed in UI)
|
|
244
|
+
|
|
245
|
+
---
|
|
246
|
+
|
|
247
|
+
## ⚙️ Activation Variables
|
|
248
|
+
|
|
249
|
+
**Configuration is driven by activation variables - modify these instead of code:**
|
|
250
|
+
|
|
251
|
+
> **Note:** `sftpUsername` and `sftpPassword` are fetched from the `SFTP` Basic Auth connection (see SFTP Connection Setup above).
|
|
252
|
+
|
|
253
|
+
```bash
|
|
254
|
+
# SFTP Configuration
|
|
255
|
+
SFTP_HOST=sftp.partner.com
|
|
256
|
+
SFTP_PORT=22
|
|
257
|
+
|
|
258
|
+
# SFTP Paths
|
|
259
|
+
SFTP_REMOTE_PATH=/locations/incoming
|
|
260
|
+
SFTP_ARCHIVE_PATH=/locations/processed
|
|
261
|
+
SFTP_ERROR_PATH=/locations/errors
|
|
262
|
+
|
|
263
|
+
# File Processing
|
|
264
|
+
FILE_PATTERN=locations_*.json
|
|
265
|
+
MAX_FILES_PER_RUN=10
|
|
266
|
+
|
|
267
|
+
# GraphQL Configuration
|
|
268
|
+
FLUENT_RETAILER_ID=1 # Optional: Only if mutation schema requires retailerId in input
|
|
269
|
+
MAX_PARALLEL_MUTATIONS=10 # Concurrent mutation limit (1=sequential, 3-10=parallel)
|
|
270
|
+
|
|
271
|
+
# Feature Toggles
|
|
272
|
+
REQUIRE_ABSOLUTE_PATHS=true # "true" for AWS Transfer Family, "false" for standard OpenSSH
|
|
273
|
+
VALIDATE_CONNECTION=true # Validate SFTP connection on startup (default: true)
|
|
274
|
+
ENABLE_FILE_TRACKING=true # Enable/disable file tracking (default: true)
|
|
275
|
+
```
|
|
276
|
+
|
|
277
|
+
**Security:**
|
|
278
|
+
- **SFTP Authentication:** Credentials stored in Versori connection vault (not in activation variables). See [SFTP Connection Setup](#sftp-connection-setup-recommended) section above.
|
|
279
|
+
- **Webhook Authentication:** Enforced by Versori connection configuration. Configure your webhook connection with API key authentication in the Versori Dashboard, then reference it in `webhook({ connection: 'webhook-auth' })`.
|
|
280
|
+
|
|
281
|
+
---
|
|
282
|
+
|
|
283
|
+
## 📄 Mapping Configuration
|
|
284
|
+
|
|
285
|
+
**File:** `src/config/location-mapping.json`
|
|
286
|
+
|
|
287
|
+
**✅ PRODUCTION STANDARD:** External JSON file with GraphQLMutationMapper structure
|
|
288
|
+
|
|
289
|
+
```json
|
|
290
|
+
{
|
|
291
|
+
"mutation": "createLocation",
|
|
292
|
+
"sourceFormat": "json",
|
|
293
|
+
"version": "1.0.0",
|
|
294
|
+
"returnFields": ["id", "ref", "name", "type", "status"],
|
|
295
|
+
"arguments": {
|
|
296
|
+
"input": {
|
|
297
|
+
"ref": {
|
|
298
|
+
"source": "locationRef",
|
|
299
|
+
"required": true,
|
|
300
|
+
"resolver": "trim"
|
|
301
|
+
},
|
|
302
|
+
"name": {
|
|
303
|
+
"source": "locationName",
|
|
304
|
+
"required": true,
|
|
305
|
+
"resolver": "trim"
|
|
306
|
+
},
|
|
307
|
+
"type": {
|
|
308
|
+
"source": "locationType",
|
|
309
|
+
"required": true,
|
|
310
|
+
"resolver": "toUpperCase"
|
|
311
|
+
},
|
|
312
|
+
"status": {
|
|
313
|
+
"value": "ACTIVE",
|
|
314
|
+
"required": true
|
|
315
|
+
},
|
|
316
|
+
"primaryAddress": {
|
|
317
|
+
"ref": {
|
|
318
|
+
"source": "locationRef",
|
|
319
|
+
"required": true,
|
|
320
|
+
"resolver": "trim"
|
|
321
|
+
},
|
|
322
|
+
"street": {
|
|
323
|
+
"source": "address.street1",
|
|
324
|
+
"resolver": "trim"
|
|
325
|
+
},
|
|
326
|
+
"city": {
|
|
327
|
+
"source": "address.city",
|
|
328
|
+
"resolver": "trim"
|
|
329
|
+
},
|
|
330
|
+
"state": {
|
|
331
|
+
"source": "address.state",
|
|
332
|
+
"resolver": "toUpperCase"
|
|
333
|
+
},
|
|
334
|
+
"postcode": {
|
|
335
|
+
"source": "address.postalCode",
|
|
336
|
+
"resolver": "trim"
|
|
337
|
+
},
|
|
338
|
+
"country": {
|
|
339
|
+
"source": "address.country",
|
|
340
|
+
"resolver": "toUpperCase"
|
|
341
|
+
},
|
|
342
|
+
"latitude": {
|
|
343
|
+
"source": "geo.latitude",
|
|
344
|
+
"required": true,
|
|
345
|
+
"resolver": "parseFloat"
|
|
346
|
+
},
|
|
347
|
+
"longitude": {
|
|
348
|
+
"source": "geo.longitude",
|
|
349
|
+
"required": true,
|
|
350
|
+
"resolver": "parseFloat"
|
|
351
|
+
},
|
|
352
|
+
"timeZone": {
|
|
353
|
+
"source": "timeZone",
|
|
354
|
+
"resolver": "trim"
|
|
355
|
+
}
|
|
356
|
+
},
|
|
357
|
+
"openingSchedule": {
|
|
358
|
+
"allHours": {
|
|
359
|
+
"source": "openingSchedule.allHours",
|
|
360
|
+
"required": true,
|
|
361
|
+
"resolver": "toBoolean"
|
|
362
|
+
},
|
|
363
|
+
"monStart": {
|
|
364
|
+
"source": "openingSchedule.monStart",
|
|
365
|
+
"required": true,
|
|
366
|
+
"resolver": "parseInt"
|
|
367
|
+
},
|
|
368
|
+
"monEnd": {
|
|
369
|
+
"source": "openingSchedule.monEnd",
|
|
370
|
+
"required": true,
|
|
371
|
+
"resolver": "parseInt"
|
|
372
|
+
}
|
|
373
|
+
}
|
|
374
|
+
}
|
|
375
|
+
}
|
|
376
|
+
}
|
|
377
|
+
```
|
|
378
|
+
|
|
379
|
+
**Key Differences from UniversalMapper:**
|
|
380
|
+
- ✅ `mutation` property (not `mutationName`)
|
|
381
|
+
- ✅ `arguments.input` structure (matches GraphQL schema)
|
|
382
|
+
- ✅ Nested objects (not dotted paths like `primaryAddress.street`)
|
|
383
|
+
- ✅ Resolver names without `sdk.` prefix (`trim`, `toUpperCase`, `parseInt`, `parseFloat`, `toBoolean`)
|
|
384
|
+
- ✅ Used with `GraphQLMutationMapper.map()` method (not `UniversalMapper.map()`)
|
|
385
|
+
|
|
386
|
+
---
|
|
387
|
+
|
|
388
|
+
## Expected JSON Format
|
|
389
|
+
|
|
390
|
+
```json
|
|
391
|
+
{
|
|
392
|
+
"locations": [
|
|
393
|
+
{
|
|
394
|
+
"locationRef": "LOC-001",
|
|
395
|
+
"locationName": "Main Warehouse",
|
|
396
|
+
"locationType": "WAREHOUSE",
|
|
397
|
+
"address": {
|
|
398
|
+
"street1": "123 Main St",
|
|
399
|
+
"city": "New York",
|
|
400
|
+
"state": "NY",
|
|
401
|
+
"postalCode": "10001",
|
|
402
|
+
"country": "US"
|
|
403
|
+
},
|
|
404
|
+
"geo": {
|
|
405
|
+
"latitude": 40.7128,
|
|
406
|
+
"longitude": -74.0060
|
|
407
|
+
},
|
|
408
|
+
"timeZone": "America/New_York",
|
|
409
|
+
"openingSchedule": {
|
|
410
|
+
"allHours": false,
|
|
411
|
+
"monStart": 480,
|
|
412
|
+
"monEnd": 1020
|
|
413
|
+
}
|
|
414
|
+
}
|
|
415
|
+
]
|
|
416
|
+
}
|
|
417
|
+
```
|
|
418
|
+
|
|
419
|
+
**Alternative:** Root array `[{...}]` is also supported.
|
|
420
|
+
|
|
421
|
+
---
|
|
422
|
+
|
|
423
|
+
## 🔧 Production Code Structure
|
|
424
|
+
|
|
425
|
+
### Modular Architecture (Gold Standard)
|
|
426
|
+
|
|
427
|
+
```
|
|
428
|
+
sftp-json-location-graphql/
|
|
429
|
+
├── package.json
|
|
430
|
+
├── tsconfig.json
|
|
431
|
+
├── index.ts # Entry point
|
|
432
|
+
└── src/
|
|
433
|
+
├── workflows/
|
|
434
|
+
│ └── location-ingestion.ts # 3 workflows
|
|
435
|
+
├── services/
|
|
436
|
+
│ ├── location-file-processor.service.ts # Download, parse, transform
|
|
437
|
+
│ ├── mutation-sender.service.ts # Execute GraphQL mutations
|
|
438
|
+
│ ├── mutation-logger.service.ts # Write error logs
|
|
439
|
+
│ └── location-ingestion.service.ts # Main orchestration
|
|
440
|
+
├── types/
|
|
441
|
+
│ └── location-ingestion.types.ts # TypeScript interfaces
|
|
442
|
+
├── utils/
|
|
443
|
+
│ ├── sftp-path.utils.ts # Path helpers
|
|
444
|
+
│ ├── retry.utils.ts # Retry logic
|
|
445
|
+
│ └── job-id-generator.ts # Job IDs
|
|
446
|
+
└── config/
|
|
447
|
+
└── location-mapping.json # GraphQL mapping config
|
|
448
|
+
```
|
|
449
|
+
|
|
450
|
+
---
|
|
451
|
+
|
|
452
|
+
## Critical Implementation Patterns
|
|
453
|
+
|
|
454
|
+
### ✅ MUST HAVE: retailerId Configuration ⚠️ VERIFICATION REQUIRED
|
|
455
|
+
|
|
456
|
+
```typescript
|
|
457
|
+
// In main orchestration service (location-ingestion.service.ts)
|
|
458
|
+
|
|
459
|
+
export async function executeLocationIngestion(ctx: VersoriContext, params: LocationIngestionParams) {
|
|
460
|
+
const { log, activation } = ctx;
|
|
461
|
+
|
|
462
|
+
// Step 1: Create client
|
|
463
|
+
const client = await createClient(ctx);
|
|
464
|
+
if (!client) {
|
|
465
|
+
throw new Error('Failed to create Fluent Commerce client');
|
|
466
|
+
}
|
|
467
|
+
|
|
468
|
+
// ✅ CORRECT: GraphQL mutations don't need setRetailerId()
|
|
469
|
+
// Check your GraphQL schema to determine retailerId handling:
|
|
470
|
+
// - Mandatory retailerId → Must pass it in mutation input
|
|
471
|
+
// - Optional retailerId → Can pass it if needed
|
|
472
|
+
// - No retailerId field → Don't pass it
|
|
473
|
+
// See: fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md
|
|
474
|
+
|
|
475
|
+
// Step 2: Initialize GraphQLMutationMapper with client
|
|
476
|
+
import mappingConfig from '../config/location-mapping.json' with { type: 'json' };
|
|
477
|
+
const mapper = new GraphQLMutationMapper(mappingConfig, log, { fluentClient: client });
|
|
478
|
+
|
|
479
|
+
// Step 3: Process files and execute mutations
|
|
480
|
+
// ... rest of workflow
|
|
481
|
+
|
|
482
|
+
// ✅ Configuration with defaults
|
|
483
|
+
const mutationBatchSize = parseInt(
|
|
484
|
+
activation?.getVariable('mutationBatchSize') || '1', // ✅ Default: 1 (sequential)
|
|
485
|
+
10
|
|
486
|
+
);
|
|
487
|
+
|
|
488
|
+
const mutationsPerAliasBatch = activation?.getVariable('mutationsPerAliasBatch')
|
|
489
|
+
? parseInt(activation?.getVariable('mutationsPerAliasBatch') || '1', 10)
|
|
490
|
+
: undefined; // ✅ Default: undefined (disabled, use separate requests)
|
|
491
|
+
|
|
492
|
+
// ? Enhanced: Extract context for progress logging
|
|
493
|
+
const sampleLocationRefs = locations.slice(0, 5).map((loc: any) => loc.ref || loc.input?.ref || 'unknown');
|
|
494
|
+
const mutationType = mapper?.mutationName || 'createLocation';
|
|
495
|
+
|
|
496
|
+
// ? Enhanced: Start logging with context
|
|
497
|
+
log.info(`[GraphQLMutations] Sending mutations for locations`, {
|
|
498
|
+
totalMutations: locations.length,
|
|
499
|
+
mutationType,
|
|
500
|
+
batchSize: mutationBatchSize,
|
|
501
|
+
batchMode: mutationBatchSize === 1 ? 'sequential' : `parallel (${mutationBatchSize})`,
|
|
502
|
+
sampleLocationRefs: sampleLocationRefs.join(', '),
|
|
503
|
+
aliasBatching: mutationsPerAliasBatch ? `enabled (${mutationsPerAliasBatch} per alias)` : 'disabled'
|
|
504
|
+
});
|
|
505
|
+
|
|
506
|
+
// Use in executeMutations:
|
|
507
|
+
const mutationResults = await mutationSender.executeMutations(
|
|
508
|
+
locations,
|
|
509
|
+
mutationBatchSize, // Concurrency control (default: 1)
|
|
510
|
+
mutationsPerAliasBatch // ✅ NEW: Alias batching (default: undefined)
|
|
511
|
+
);
|
|
512
|
+
|
|
513
|
+
// ? Enhanced: Completion logging with summary
|
|
514
|
+
log.info(`[GraphQLMutations] Mutation submission completed`, {
|
|
515
|
+
totalMutations: locations.length,
|
|
516
|
+
mutationsExecuted: mutationResults.mutationsExecuted,
|
|
517
|
+
mutationsFailed: mutationResults.mutationsFailed,
|
|
518
|
+
successRate: locations.length > 0 ? `${Math.round((mutationResults.mutationsExecuted / locations.length) * 100)}%` : '0%',
|
|
519
|
+
mutationType
|
|
520
|
+
});
|
|
521
|
+
}
|
|
522
|
+
```
|
|
523
|
+
|
|
524
|
+
**✅ VERIFIED Pattern:**
|
|
525
|
+
- GraphQL mutations do NOT need `client.setRetailerId()`
|
|
526
|
+
- `setRetailerId()` is only for Job API and Event API
|
|
527
|
+
- Pass retailerId in mutation input variables if schema requires it
|
|
528
|
+
|
|
529
|
+
**Example (For mutations that require retailerId in schema):**
|
|
530
|
+
```typescript
|
|
531
|
+
// Only needed if mutation schema explicitly requires retailerId
|
|
532
|
+
const fluentRetailerId = activation.getVariable('fluentRetailerId');
|
|
533
|
+
const { query, variables } = await mapper.map(location);
|
|
534
|
+
|
|
535
|
+
// Add to input if mutation schema requires it
|
|
536
|
+
if (fluentRetailerId && variables.input) {
|
|
537
|
+
variables.input.retailer = { id: parseInt(fluentRetailerId) };
|
|
538
|
+
}
|
|
539
|
+
|
|
540
|
+
await client.graphql({ query, variables });
|
|
541
|
+
```
|
|
542
|
+
|
|
543
|
+
**Note:** Check your GraphQL schema (via introspection) to determine retailerId handling:
|
|
544
|
+
1. Does the mutation have `retailerId` or `retailer.id` field?
|
|
545
|
+
2. Is it mandatory (`!`) or optional?
|
|
546
|
+
3. If field doesn't exist → don't pass it
|
|
547
|
+
|
|
548
|
+
### ✅ MUST HAVE: Double-Finally SFTP Disposal
|
|
549
|
+
|
|
550
|
+
**What:** Defensive resource cleanup with nested finally blocks (matches Event API gold standard)
|
|
551
|
+
|
|
552
|
+
**Gold Standard Pattern:**
|
|
553
|
+
|
|
554
|
+
```typescript
|
|
555
|
+
// ✅ CRITICAL: Declare SFTP outside try block for safe disposal
|
|
556
|
+
let sftp: SftpDataSource | undefined;
|
|
557
|
+
|
|
558
|
+
try {
|
|
559
|
+
// 📊 Execution boundary: Start workflow
|
|
560
|
+
const startTime = Date.now();
|
|
561
|
+
log.info('🚀 Starting location ingestion workflow', {
|
|
562
|
+
stage: 'initialization',
|
|
563
|
+
jobId,
|
|
564
|
+
triggeredBy: params.triggeredBy
|
|
565
|
+
});
|
|
566
|
+
|
|
567
|
+
// Initialize SFTP
|
|
568
|
+
sftp = new SftpDataSource(...);
|
|
569
|
+
|
|
570
|
+
// ✅ Validate connection (if enabled)
|
|
571
|
+
const validateConnection = activation?.getVariable('validateConnection') === 'true';
|
|
572
|
+
if (validateConnection) {
|
|
573
|
+
log.info('🔌 Validating SFTP connection', { stage: 'connection_validation' });
|
|
574
|
+
await sftp.validateConnection();
|
|
575
|
+
log.info('✅ SFTP connection validated successfully');
|
|
576
|
+
}
|
|
577
|
+
|
|
578
|
+
try {
|
|
579
|
+
// Use SFTP
|
|
580
|
+
await sftp.listFiles(...);
|
|
581
|
+
await sftp.downloadFile(...);
|
|
582
|
+
// ... rest of workflow
|
|
583
|
+
|
|
584
|
+
// 📊 Execution boundary: End workflow
|
|
585
|
+
const duration = Date.now() - startTime;
|
|
586
|
+
log.info('✅ Location ingestion workflow completed', {
|
|
587
|
+
stage: 'completion',
|
|
588
|
+
jobId,
|
|
589
|
+
duration,
|
|
590
|
+
filesProcessed,
|
|
591
|
+
mutationsExecuted
|
|
592
|
+
});
|
|
593
|
+
} finally {
|
|
594
|
+
// Inner finally - normal cleanup
|
|
595
|
+
if (sftp) {
|
|
596
|
+
await sftp.dispose();
|
|
597
|
+
log.info('🔌 SFTP connection disposed');
|
|
598
|
+
}
|
|
599
|
+
}
|
|
600
|
+
} catch (error: unknown) {
|
|
601
|
+
// ✅ Enhanced error logging: Extract all error details + recommendations
|
|
602
|
+
const errorMessage = error instanceof Error ? error.message : String(error);
|
|
603
|
+
const errorDetails = {
|
|
604
|
+
message: errorMessage,
|
|
605
|
+
stack: error instanceof Error ? error.stack : undefined,
|
|
606
|
+
errorType: error instanceof Error ? error.constructor.name : 'Error',
|
|
607
|
+
stage: 'workflow_execution',
|
|
608
|
+
jobId
|
|
609
|
+
};
|
|
610
|
+
|
|
611
|
+
// ✅ Add error recommendations
|
|
612
|
+
const recommendations: string[] = [];
|
|
613
|
+
if (errorMessage.includes('ECONNREFUSED')) {
|
|
614
|
+
recommendations.push('Check SFTP host and port configuration');
|
|
615
|
+
recommendations.push('Verify network connectivity to SFTP server');
|
|
616
|
+
} else if (errorMessage.includes('Authentication failed')) {
|
|
617
|
+
recommendations.push('Verify SFTP username and password');
|
|
618
|
+
recommendations.push('Check if account is locked or expired');
|
|
619
|
+
} else if (errorMessage.includes('No such file')) {
|
|
620
|
+
recommendations.push('Verify incoming path exists on SFTP server');
|
|
621
|
+
recommendations.push('Check file pattern matches existing files');
|
|
622
|
+
}
|
|
623
|
+
|
|
624
|
+
log.error('❌ Workflow failed', { ...errorDetails, recommendations });
|
|
625
|
+
|
|
626
|
+
// Handle error (mark job failed, etc.)
|
|
627
|
+
} finally {
|
|
628
|
+
// Outer finally - defensive cleanup
|
|
629
|
+
// ⚠️ CRITICAL: Ensures disposal even if error occurs after SFTP creation
|
|
630
|
+
// but before inner try block (e.g., during connection validation)
|
|
631
|
+
if (sftp) {
|
|
632
|
+
try {
|
|
633
|
+
await sftp.dispose();
|
|
634
|
+
log.info('🔌 SFTP connection disposed (outer finally)');
|
|
635
|
+
} catch (disposeError: unknown) {
|
|
636
|
+
const disposeErrorMessage = disposeError instanceof Error
|
|
637
|
+
? disposeError.message
|
|
638
|
+
: String(disposeError);
|
|
639
|
+
log.warn('⚠️ Error disposing SFTP in outer finally', { error: disposeErrorMessage });
|
|
640
|
+
}
|
|
641
|
+
}
|
|
642
|
+
}
|
|
643
|
+
```
|
|
644
|
+
|
|
645
|
+
**Why This is Critical:**
|
|
646
|
+
- Ensures disposal even if error occurs after SFTP creation but before inner try block
|
|
647
|
+
- Defensive cleanup catches edge cases
|
|
648
|
+
- Prevents resource leaks in error scenarios
|
|
649
|
+
- Matches Event API gold standard pattern
|
|
650
|
+
|
|
651
|
+
---
|
|
652
|
+
|
|
653
|
+
## Processing Modes
|
|
654
|
+
|
|
655
|
+
**⚠️ IMPORTANT:** Choose ONE processing mode per connector. These are alternative patterns, not features to use together.
|
|
656
|
+
|
|
657
|
+
The SDK supports two processing modes for handling multiple files. **Select the mode that best fits your use case** and configure your connector accordingly:
|
|
658
|
+
|
|
659
|
+
### Mode 1: Per-File Processing (Recommended Default) ✅ IMPLEMENTED
|
|
660
|
+
|
|
661
|
+
**When to use:**
|
|
662
|
+
- Multiple large files that shouldn't be in memory together
|
|
663
|
+
- Need file-level consistency (file 3 fails → files 1-2 already archived)
|
|
664
|
+
- Clear error isolation per file
|
|
665
|
+
|
|
666
|
+
**How it works:**
|
|
667
|
+
1. Process file 1 completely (download → parse → transform → execute mutations → archive)
|
|
668
|
+
2. Move to file 2
|
|
669
|
+
3. Continue sequentially
|
|
670
|
+
|
|
671
|
+
**Benefits:**
|
|
672
|
+
- ✅ Low memory usage (one file at a time)
|
|
673
|
+
- ✅ Clear error isolation (failed file doesn't block others)
|
|
674
|
+
- ✅ Incremental progress (files archived as completed)
|
|
675
|
+
|
|
676
|
+
**Example implementation (see code section below)**
|
|
677
|
+
|
|
678
|
+
### Mode 2: Chunked Per-File Processing ⚠️ OPTIONAL - Choose if needed
|
|
679
|
+
|
|
680
|
+
**When to use:**
|
|
681
|
+
- Many small files that can be processed in parallel
|
|
682
|
+
- Need better throughput for many files
|
|
683
|
+
- Files are independent (no cross-file dependencies)
|
|
684
|
+
|
|
685
|
+
**How it works:**
|
|
686
|
+
1. Group files into chunks (e.g., 5 files per chunk)
|
|
687
|
+
2. Process chunk 1 files in parallel
|
|
688
|
+
3. Wait for chunk completion, then move to chunk 2
|
|
689
|
+
|
|
690
|
+
**Benefits:**
|
|
691
|
+
- ✅ Better throughput (parallel processing)
|
|
692
|
+
- ✅ Still maintains file-level isolation
|
|
693
|
+
- ✅ Configurable chunk size
|
|
694
|
+
|
|
695
|
+
**Note:** GraphQL mutations are typically one-off operations, so per-file processing is usually sufficient. Chunked mode is optional for high-volume scenarios.
|
|
696
|
+
|
|
697
|
+
---
|
|
698
|
+
|
|
699
|
+
## Key Service Patterns
|
|
700
|
+
|
|
701
|
+
### Service 1: Location File Processor (Complete Implementation)
|
|
702
|
+
|
|
703
|
+
**File:** `src/services/location-file-processor.service.ts`
|
|
704
|
+
|
|
705
|
+
```typescript
|
|
706
|
+
/**
|
|
707
|
+
* Location File Processor Service
|
|
708
|
+
*
|
|
709
|
+
* Downloads JSON files from SFTP, parses content, and transforms with GraphQLMutationMapper.
|
|
710
|
+
* Reusable service for location file processing workflows.
|
|
711
|
+
*/
|
|
712
|
+
|
|
713
|
+
import {
|
|
714
|
+
SftpDataSource,
|
|
715
|
+
JSONParserService,
|
|
716
|
+
GraphQLMutationMapper,
|
|
717
|
+
} from '@fluentcommerce/fc-connect-sdk';
|
|
718
|
+
import type {
|
|
719
|
+
ProcessFileResult,
|
|
720
|
+
LocationRecord,
|
|
721
|
+
} from '../types/location-ingestion.types';
|
|
722
|
+
import { extractFileName } from '../utils/sftp-path.utils';
|
|
723
|
+
|
|
724
|
+
/**
|
|
725
|
+
* Location file processing service
|
|
726
|
+
* Coordinates SFTP download, JSON parsing (via SDK), and GraphQL mutation generation
|
|
727
|
+
*/
|
|
728
|
+
export class LocationFileProcessorService {
|
|
729
|
+
constructor(
|
|
730
|
+
private sftp: SftpDataSource,
|
|
731
|
+
private jsonParser: JSONParserService,
|
|
732
|
+
private mapper: GraphQLMutationMapper
|
|
733
|
+
) {} // ✅ No logger param - workflow handles logging
|
|
734
|
+
|
|
735
|
+
/**
|
|
736
|
+
* Download JSON file from SFTP, parse, and generate GraphQL mutations
|
|
737
|
+
*/
|
|
738
|
+
async downloadParseAndTransform(remoteFilePath: string): Promise<ProcessFileResult> {
|
|
739
|
+
// ✅ Extract filename for logging
|
|
740
|
+
const fileName = extractFileName(remoteFilePath);
|
|
741
|
+
|
|
742
|
+
try {
|
|
743
|
+
// Download JSON content
|
|
744
|
+
const content = (await this.sftp.downloadFile(remoteFilePath, {
|
|
745
|
+
encoding: 'utf8',
|
|
746
|
+
})) as string;
|
|
747
|
+
|
|
748
|
+
// Parse JSON with type safety
|
|
749
|
+
let parsed: unknown;
|
|
750
|
+
try {
|
|
751
|
+
parsed = await this.jsonParser.parse(content);
|
|
752
|
+
} catch (parseError: unknown) {
|
|
753
|
+
const errorMessage = parseError instanceof Error ? parseError.message : String(parseError);
|
|
754
|
+
return {
|
|
755
|
+
success: false,
|
|
756
|
+
locations: [],
|
|
757
|
+
error: `JSON parse error: ${errorMessage}`,
|
|
758
|
+
fileName,
|
|
759
|
+
};
|
|
760
|
+
}
|
|
761
|
+
|
|
762
|
+
// Normalize locations array (handle both { locations: [...] } and root array formats)
|
|
763
|
+
const jsonData = parsed as Record<string, unknown>;
|
|
764
|
+
const locations = jsonData.locations
|
|
765
|
+
? (Array.isArray(jsonData.locations) ? jsonData.locations : [jsonData.locations])
|
|
766
|
+
: (Array.isArray(jsonData) ? jsonData : []);
|
|
767
|
+
|
|
768
|
+
if (locations.length === 0) {
|
|
769
|
+
return {
|
|
770
|
+
success: false,
|
|
771
|
+
locations: [],
|
|
772
|
+
error: 'No locations found in JSON',
|
|
773
|
+
fileName,
|
|
774
|
+
};
|
|
775
|
+
}
|
|
776
|
+
|
|
777
|
+
// Validate with type guard
|
|
778
|
+
const validLocations = locations.filter(isLocationRecord);
|
|
779
|
+
|
|
780
|
+
if (validLocations.length === 0) {
|
|
781
|
+
return {
|
|
782
|
+
success: false,
|
|
783
|
+
locations: [],
|
|
784
|
+
error: 'No valid locations found after validation',
|
|
785
|
+
fileName,
|
|
786
|
+
};
|
|
787
|
+
}
|
|
788
|
+
|
|
789
|
+
// Transform with GraphQLMutationMapper
|
|
790
|
+
const mapped = await Promise.all(
|
|
791
|
+
validLocations.map(loc => this.mapper.map(loc))
|
|
792
|
+
);
|
|
793
|
+
|
|
794
|
+
return {
|
|
795
|
+
success: true,
|
|
796
|
+
locations: mapped.map(m => m.variables.input),
|
|
797
|
+
fileName,
|
|
798
|
+
};
|
|
799
|
+
} catch (error: unknown) {
|
|
800
|
+
const errorMessage = error instanceof Error ? error.message : String(error);
|
|
801
|
+
return {
|
|
802
|
+
success: false,
|
|
803
|
+
locations: [],
|
|
804
|
+
error: errorMessage,
|
|
805
|
+
fileName,
|
|
806
|
+
};
|
|
807
|
+
}
|
|
808
|
+
}
|
|
809
|
+
}
|
|
810
|
+
|
|
811
|
+
/**
|
|
812
|
+
* Type guard: Check if parsed value is a valid LocationRecord
|
|
813
|
+
*/
|
|
814
|
+
function isLocationRecord(obj: unknown): obj is LocationRecord {
|
|
815
|
+
if (typeof obj !== 'object' || obj === null) return false;
|
|
816
|
+
const loc = obj as Record<string, unknown>;
|
|
817
|
+
|
|
818
|
+
return (
|
|
819
|
+
typeof loc.locationRef === 'string' &&
|
|
820
|
+
typeof loc.locationName === 'string' &&
|
|
821
|
+
typeof loc.locationType === 'string' &&
|
|
822
|
+
typeof loc.address === 'object' &&
|
|
823
|
+
loc.address !== null &&
|
|
824
|
+
typeof loc.geo === 'object' &&
|
|
825
|
+
loc.geo !== null
|
|
826
|
+
);
|
|
827
|
+
}
|
|
828
|
+
```
|
|
829
|
+
|
|
830
|
+
### Service 2: Mutation Sender (Complete Implementation)
|
|
831
|
+
|
|
832
|
+
**File:** `src/services/mutation-sender.service.ts`
|
|
833
|
+
|
|
834
|
+
```typescript
|
|
835
|
+
/**
|
|
836
|
+
* Mutation Sender Service
|
|
837
|
+
*
|
|
838
|
+
* Executes GraphQL mutations with per-record error handling.
|
|
839
|
+
* Continues processing on individual failures (GraphQL mutation pattern).
|
|
840
|
+
*/
|
|
841
|
+
|
|
842
|
+
import { GraphQLMutationMapper } from '@fluentcommerce/fc-connect-sdk';
|
|
843
|
+
import type { FluentClient } from '@fluentcommerce/fc-connect-sdk';
|
|
844
|
+
import type { MutationResult, MappedLocation } from '../types/location-ingestion.types';
|
|
845
|
+
|
|
846
|
+
/**
|
|
847
|
+
* Service for executing GraphQL mutations
|
|
848
|
+
*/
|
|
849
|
+
export class MutationSenderService {
|
|
850
|
+
constructor(
|
|
851
|
+
private client: FluentClient,
|
|
852
|
+
private mapper: GraphQLMutationMapper
|
|
853
|
+
) {}
|
|
854
|
+
|
|
855
|
+
/**
|
|
856
|
+
* Execute GraphQL mutations with configurable concurrency and alias batching
|
|
857
|
+
*
|
|
858
|
+
* **Performance Characteristics:**
|
|
859
|
+
* - `maxParallel: 1` → Sequential processing (safe default, ~1 mutation/sec)
|
|
860
|
+
* - `maxParallel: 3-5` → Balanced throughput (~3-5 mutations/sec)
|
|
861
|
+
* - `maxParallel: 10` → High-volume processing (~10 mutations/sec, 100+ locations)
|
|
862
|
+
*
|
|
863
|
+
* **Implementation Strategy:**
|
|
864
|
+
* - Sequential (1): Optimized loop (no Promise.allSettled overhead)
|
|
865
|
+
* - Parallel (>1): Chunked processing with bounded concurrency
|
|
866
|
+
* - Alias batching: Groups mutations into aliased GraphQL requests (reduces network overhead)
|
|
867
|
+
* - Both modes: Per-record error tracking (failures don't block others)
|
|
868
|
+
*
|
|
869
|
+
* **Alias Batching:**
|
|
870
|
+
* - When `mutationsPerAliasBatch > 1`, groups mutations into aliased requests
|
|
871
|
+
* - Example: 5 mutations → 1 HTTP request with aliases (createLocation1, createLocation2, ...)
|
|
872
|
+
* - Reduces network overhead by ~80% for high-volume scenarios
|
|
873
|
+
*
|
|
874
|
+
* @param locations - Array of mapped locations to create/update
|
|
875
|
+
* @param maxParallel - Number of concurrent mutations or alias batches (default: 1, min: 1)
|
|
876
|
+
* @param mutationsPerAliasBatch - Optional: Number of mutations per aliased request (default: undefined = disabled)
|
|
877
|
+
* @returns MutationResult with counts (mutationsExecuted/mutationsFailed) and error details
|
|
878
|
+
*/
|
|
879
|
+
async executeMutations(
|
|
880
|
+
locations: MappedLocation[],
|
|
881
|
+
maxParallel: number = 1, // ✅ Default: 1 (sequential)
|
|
882
|
+
mutationsPerAliasBatch?: number // ✅ NEW: Alias batching parameter (default: undefined = disabled)
|
|
883
|
+
): Promise<MutationResult> {
|
|
884
|
+
// Determine mode: use aliases if mutationsPerAliasBatch is set and > 1
|
|
885
|
+
const useAliases = mutationsPerAliasBatch && mutationsPerAliasBatch > 1;
|
|
886
|
+
|
|
887
|
+
if (useAliases) {
|
|
888
|
+
return await this.executeMutationsWithAliases(
|
|
889
|
+
locations,
|
|
890
|
+
maxParallel,
|
|
891
|
+
mutationsPerAliasBatch!
|
|
892
|
+
);
|
|
893
|
+
} else {
|
|
894
|
+
return await this.executeMutationsSeparate(locations, maxParallel);
|
|
895
|
+
}
|
|
896
|
+
}
|
|
897
|
+
|
|
898
|
+
/**
|
|
899
|
+
* Execute mutations using separate concurrent requests (current mode)
|
|
900
|
+
*/
|
|
901
|
+
private async executeMutationsSeparate(
|
|
902
|
+
locations: MappedLocation[],
|
|
903
|
+
maxParallel: number
|
|
904
|
+
): Promise<MutationResult> {
|
|
905
|
+
// Validate concurrency (guard against invalid values)
|
|
906
|
+
const safeConc = Math.max(1, Math.floor(maxParallel));
|
|
907
|
+
|
|
908
|
+
// Result accumulators
|
|
909
|
+
let mutationsExecuted = 0;
|
|
910
|
+
let mutationsFailed = 0;
|
|
911
|
+
const errors: Array<{ locationRef: string; error: string }> = [];
|
|
912
|
+
|
|
913
|
+
// ============================================================================
|
|
914
|
+
// SEQUENTIAL MODE (maxParallel === 1)
|
|
915
|
+
// ============================================================================
|
|
916
|
+
if (safeConc === 1) {
|
|
917
|
+
for (const location of locations) {
|
|
918
|
+
try {
|
|
919
|
+
const { query, variables } = await this.mapper.map(location);
|
|
920
|
+
await this.client.graphql({ query, variables });
|
|
921
|
+
mutationsExecuted++;
|
|
922
|
+
} catch (err: unknown) {
|
|
923
|
+
mutationsFailed++;
|
|
924
|
+
const errorMsg = err instanceof Error ? err.message : String(err);
|
|
925
|
+
errors.push({
|
|
926
|
+
locationRef: location.ref || 'unknown',
|
|
927
|
+
error: errorMsg,
|
|
928
|
+
});
|
|
929
|
+
// Continue processing (failure doesn't block other locations)
|
|
930
|
+
}
|
|
931
|
+
}
|
|
932
|
+
return { mutationsExecuted, mutationsFailed, errors };
|
|
933
|
+
}
|
|
934
|
+
|
|
935
|
+
// ============================================================================
|
|
936
|
+
// PARALLEL MODE (maxParallel > 1)
|
|
937
|
+
// ============================================================================
|
|
938
|
+
// Chunked processing with bounded concurrency
|
|
939
|
+
for (let i = 0; i < locations.length; i += safeConc) {
|
|
940
|
+
const chunk = locations.slice(i, i + safeConc);
|
|
941
|
+
|
|
942
|
+
// Fire all mutations in chunk concurrently
|
|
943
|
+
const results = await Promise.allSettled(
|
|
944
|
+
chunk.map(async (location) => {
|
|
945
|
+
const { query, variables } = await this.mapper.map(location);
|
|
946
|
+
return await this.client.graphql({ query, variables });
|
|
947
|
+
})
|
|
948
|
+
);
|
|
949
|
+
|
|
950
|
+
// Aggregate chunk results
|
|
951
|
+
results.forEach((result, idx) => {
|
|
952
|
+
if (result.status === 'fulfilled') {
|
|
953
|
+
mutationsExecuted++;
|
|
954
|
+
} else {
|
|
955
|
+
mutationsFailed++;
|
|
956
|
+
const error = result.reason;
|
|
957
|
+
errors.push({
|
|
958
|
+
locationRef: chunk[idx]?.ref || 'unknown',
|
|
959
|
+
error: error?.message || String(error) || 'Unknown error',
|
|
960
|
+
});
|
|
961
|
+
}
|
|
962
|
+
});
|
|
963
|
+
|
|
964
|
+
// Small delay between batches to avoid rate limiting
|
|
965
|
+
if (i + safeConc < locations.length) {
|
|
966
|
+
await new Promise(resolve => setTimeout(resolve, 100));
|
|
967
|
+
}
|
|
968
|
+
}
|
|
969
|
+
|
|
970
|
+
return { mutationsExecuted, mutationsFailed, errors };
|
|
971
|
+
}
|
|
972
|
+
|
|
973
|
+
/**
|
|
974
|
+
* ✅ NEW: Execute mutations using GraphQL aliases (batched requests)
|
|
975
|
+
*
|
|
976
|
+
* Groups mutations into alias batches and executes them with concurrency control.
|
|
977
|
+
* Each aliased request contains multiple mutations identified by alias names.
|
|
978
|
+
*
|
|
979
|
+
* @param locations - Array of mapped locations
|
|
980
|
+
* @param maxParallel - Number of concurrent alias batch requests
|
|
981
|
+
* @param mutationsPerAliasBatch - Number of mutations per aliased request
|
|
982
|
+
* @returns Mutation execution results
|
|
983
|
+
*/
|
|
984
|
+
private async executeMutationsWithAliases(
|
|
985
|
+
locations: MappedLocation[],
|
|
986
|
+
maxParallel: number,
|
|
987
|
+
mutationsPerAliasBatch: number
|
|
988
|
+
): Promise<MutationResult> {
|
|
989
|
+
const results: MutationResult = { mutationsExecuted: 0, mutationsFailed: 0, errors: [] };
|
|
990
|
+
|
|
991
|
+
// Extract mutation name from mapper config
|
|
992
|
+
const mutationName = (this.mapper as any).config.mutation || 'createLocation';
|
|
993
|
+
|
|
994
|
+
// Group locations into alias batches
|
|
995
|
+
const aliasBatches: Array<MappedLocation[]> = [];
|
|
996
|
+
for (let i = 0; i < locations.length; i += mutationsPerAliasBatch) {
|
|
997
|
+
aliasBatches.push(locations.slice(i, i + mutationsPerAliasBatch));
|
|
998
|
+
}
|
|
999
|
+
|
|
1000
|
+
// Process batches with concurrency control
|
|
1001
|
+
for (let i = 0; i < aliasBatches.length; i += maxParallel) {
|
|
1002
|
+
const concurrentBatches = aliasBatches.slice(i, i + maxParallel);
|
|
1003
|
+
|
|
1004
|
+
const batchResults = await Promise.allSettled(
|
|
1005
|
+
concurrentBatches.map(async (batch) => {
|
|
1006
|
+
// Build aliased query and variables
|
|
1007
|
+
const { query, variables } = await this.buildAliasedBatch(batch, mutationName);
|
|
1008
|
+
|
|
1009
|
+
// Execute aliased mutation
|
|
1010
|
+
const response = await this.client.graphql({ query, variables });
|
|
1011
|
+
|
|
1012
|
+
// Parse results and errors
|
|
1013
|
+
return this.parseAliasResponse(response, batch, mutationName);
|
|
1014
|
+
})
|
|
1015
|
+
);
|
|
1016
|
+
|
|
1017
|
+
// Aggregate results
|
|
1018
|
+
batchResults.forEach((result, idx) => {
|
|
1019
|
+
if (result.status === 'fulfilled') {
|
|
1020
|
+
const batchResult = result.value;
|
|
1021
|
+
results.mutationsExecuted += batchResult.executed;
|
|
1022
|
+
results.mutationsFailed += batchResult.failed;
|
|
1023
|
+
results.errors.push(...batchResult.errors);
|
|
1024
|
+
} else {
|
|
1025
|
+
// Entire batch failed
|
|
1026
|
+
const batch = concurrentBatches[idx];
|
|
1027
|
+
const errorMsg = result.reason instanceof Error ? result.reason.message : String(result.reason);
|
|
1028
|
+
batch.forEach(loc => {
|
|
1029
|
+
results.mutationsFailed++;
|
|
1030
|
+
results.errors.push({
|
|
1031
|
+
locationRef: loc.ref || 'unknown',
|
|
1032
|
+
error: `Batch execution failed: ${errorMsg}`
|
|
1033
|
+
});
|
|
1034
|
+
});
|
|
1035
|
+
}
|
|
1036
|
+
});
|
|
1037
|
+
|
|
1038
|
+
// Add small delay between concurrent batches to respect rate limits
|
|
1039
|
+
if (i + maxParallel < aliasBatches.length) {
|
|
1040
|
+
await new Promise(resolve => setTimeout(resolve, 500));
|
|
1041
|
+
}
|
|
1042
|
+
}
|
|
1043
|
+
|
|
1044
|
+
return results;
|
|
1045
|
+
}
|
|
1046
|
+
|
|
1047
|
+
/**
|
|
1048
|
+
* ✅ NEW: Build aliased batch query and variables
|
|
1049
|
+
*
|
|
1050
|
+
* @param batch - Array of location objects
|
|
1051
|
+
* @param mutationName - GraphQL mutation name (e.g., 'createLocation')
|
|
1052
|
+
* @returns GraphQL query and variables for aliased batch
|
|
1053
|
+
*/
|
|
1054
|
+
private async buildAliasedBatch(
|
|
1055
|
+
batch: MappedLocation[],
|
|
1056
|
+
mutationName: string
|
|
1057
|
+
): Promise<{ query: string; variables: Record<string, any> }> {
|
|
1058
|
+
const batchSize = batch.length;
|
|
1059
|
+
const inputTypeName = `${mutationName.charAt(0).toUpperCase() + mutationName.slice(1)}Input`;
|
|
1060
|
+
|
|
1061
|
+
// Build aliased query (simple naming: mutationName + index)
|
|
1062
|
+
const variables = Array.from({ length: batchSize }, (_, i) =>
|
|
1063
|
+
`$input${i + 1}: ${inputTypeName}!`
|
|
1064
|
+
).join(', ');
|
|
1065
|
+
|
|
1066
|
+
const aliasedMutations = Array.from({ length: batchSize }, (_, i) => {
|
|
1067
|
+
const alias = `${mutationName}${i + 1}`; // Simple: createLocation1, createLocation2, etc.
|
|
1068
|
+
return ` ${alias}: ${mutationName}(input: $input${i + 1}) { id ref name }`;
|
|
1069
|
+
}).join('\n');
|
|
1070
|
+
|
|
1071
|
+
const operationName = `Batch${mutationName.charAt(0).toUpperCase() + mutationName.slice(1)}s`;
|
|
1072
|
+
const query = `mutation ${operationName}(${variables}) {\n${aliasedMutations}\n}`;
|
|
1073
|
+
|
|
1074
|
+
// Build variables from mapped locations (await all mappings)
|
|
1075
|
+
const mappedItems = await Promise.all(
|
|
1076
|
+
batch.map(loc => this.mapper.map(loc))
|
|
1077
|
+
);
|
|
1078
|
+
|
|
1079
|
+
const variablesObj: Record<string, unknown> = {};
|
|
1080
|
+
mappedItems.forEach((mapped, index) => {
|
|
1081
|
+
const input = mapped.variables.input || mapped.variables;
|
|
1082
|
+
variablesObj[`input${index + 1}`] = input;
|
|
1083
|
+
});
|
|
1084
|
+
|
|
1085
|
+
return { query, variables: variablesObj };
|
|
1086
|
+
}
|
|
1087
|
+
|
|
1088
|
+
/**
|
|
1089
|
+
* ✅ NEW: Parse aliased GraphQL response and extract individual mutation results
|
|
1090
|
+
*
|
|
1091
|
+
* @param response - GraphQL response from aliased mutation
|
|
1092
|
+
* @param batch - Original batch of location objects
|
|
1093
|
+
* @param mutationName - GraphQL mutation name (e.g., 'createLocation')
|
|
1094
|
+
* @returns Parsed results with executed/failed counts and errors
|
|
1095
|
+
*/
|
|
1096
|
+
private parseAliasResponse(
|
|
1097
|
+
response: { data?: Record<string, unknown>; errors?: Array<{ path?: string[]; message: string }> },
|
|
1098
|
+
batch: MappedLocation[],
|
|
1099
|
+
mutationName: string
|
|
1100
|
+
): { executed: number; failed: number; errors: Array<{ locationRef: string; error: string }> } {
|
|
1101
|
+
const result = { executed: 0, failed: 0, errors: [] as Array<{ locationRef: string; error: string }> };
|
|
1102
|
+
|
|
1103
|
+
const data = response.data || {};
|
|
1104
|
+
const errors = response.errors || [];
|
|
1105
|
+
|
|
1106
|
+
// Process each alias result
|
|
1107
|
+
batch.forEach((loc, index) => {
|
|
1108
|
+
const alias = `${mutationName}${index + 1}`; // Simple: createLocation1, createLocation2, etc.
|
|
1109
|
+
const aliasData = data[alias];
|
|
1110
|
+
const aliasErrors = errors.filter((e) =>
|
|
1111
|
+
e.path && Array.isArray(e.path) && e.path.includes(alias)
|
|
1112
|
+
);
|
|
1113
|
+
|
|
1114
|
+
if (aliasData && !aliasErrors.length) {
|
|
1115
|
+
result.executed++;
|
|
1116
|
+
} else {
|
|
1117
|
+
result.failed++;
|
|
1118
|
+
const errorMsg = aliasErrors[0]?.message || 'Mutation failed';
|
|
1119
|
+
const locationRef = loc.ref || 'unknown';
|
|
1120
|
+
result.errors.push({
|
|
1121
|
+
locationRef,
|
|
1122
|
+
error: errorMsg,
|
|
1123
|
+
});
|
|
1124
|
+
}
|
|
1125
|
+
});
|
|
1126
|
+
|
|
1127
|
+
return result;
|
|
1128
|
+
}
|
|
1129
|
+
}
|
|
1130
|
+
```
|
|
1131
|
+
|
|
1132
|
+
### Service 3: Mutation Logger (Complete Implementation)
|
|
1133
|
+
|
|
1134
|
+
**File:** `src/services/mutation-logger.service.ts`
|
|
1135
|
+
|
|
1136
|
+
```typescript
|
|
1137
|
+
/**
|
|
1138
|
+
* Mutation Logger Service
|
|
1139
|
+
*
|
|
1140
|
+
* Writes mutation processing logs to SFTP for audit purposes.
|
|
1141
|
+
* Log writing failures don't stop the workflow (non-critical).
|
|
1142
|
+
*/
|
|
1143
|
+
|
|
1144
|
+
import { Buffer } from 'node:buffer'; // Required for Versori/Deno
|
|
1145
|
+
import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
|
|
1146
|
+
import type { MutationResult } from '../types/location-ingestion.types';
|
|
1147
|
+
|
|
1148
|
+
/**
|
|
1149
|
+
* Service for writing mutation logs to SFTP
|
|
1150
|
+
*/
|
|
1151
|
+
export class MutationLoggerService {
|
|
1152
|
+
constructor(private sftp: SftpDataSource) {}
|
|
1153
|
+
|
|
1154
|
+
/**
|
|
1155
|
+
* Write mutation processing log to SFTP
|
|
1156
|
+
*
|
|
1157
|
+
* Creates JSON log file with mutation results.
|
|
1158
|
+
* Only write logs when there are failures (optional).
|
|
1159
|
+
*
|
|
1160
|
+
* Returns log file path on success, or throws on critical errors.
|
|
1161
|
+
*/
|
|
1162
|
+
async writeMutationLog(
|
|
1163
|
+
fileName: string,
|
|
1164
|
+
result: MutationResult,
|
|
1165
|
+
remotePath: string,
|
|
1166
|
+
requireAbsolutePaths: boolean
|
|
1167
|
+
): Promise<string> {
|
|
1168
|
+
const timestamp = new Date().toISOString();
|
|
1169
|
+
const logContent = JSON.stringify(
|
|
1170
|
+
{
|
|
1171
|
+
fileName,
|
|
1172
|
+
timestamp,
|
|
1173
|
+
mutationsExecuted: result.mutationsExecuted,
|
|
1174
|
+
mutationsFailed: result.mutationsFailed,
|
|
1175
|
+
errors: result.errors,
|
|
1176
|
+
},
|
|
1177
|
+
null,
|
|
1178
|
+
2
|
|
1179
|
+
);
|
|
1180
|
+
|
|
1181
|
+
const logFileName = `${fileName.replace(/\.json$/i, '')}-mutations-${timestamp.replace(/[:.]/g, '-')}.json`;
|
|
1182
|
+
|
|
1183
|
+
const logPath =
|
|
1184
|
+
requireAbsolutePaths && !remotePath.startsWith('/')
|
|
1185
|
+
? `/${remotePath}/${logFileName}`.replace(/\/+/g, '/')
|
|
1186
|
+
: `${remotePath}/${logFileName}`.replace(/\/+/g, '/');
|
|
1187
|
+
|
|
1188
|
+
await this.sftp.uploadFile(logPath, logContent);
|
|
1189
|
+
|
|
1190
|
+
return logPath; // Return path for workflow logging
|
|
1191
|
+
}
|
|
1192
|
+
}
|
|
1193
|
+
```
|
|
1194
|
+
|
|
1195
|
+
---
|
|
1196
|
+
|
|
1197
|
+
## Type Definitions
|
|
1198
|
+
|
|
1199
|
+
**File:** `src/types/location-ingestion.types.ts`
|
|
1200
|
+
|
|
1201
|
+
```typescript
|
|
1202
|
+
export interface LocationRecord {
|
|
1203
|
+
locationRef: string;
|
|
1204
|
+
locationName: string;
|
|
1205
|
+
locationType: string;
|
|
1206
|
+
address: {
|
|
1207
|
+
street1: string;
|
|
1208
|
+
city: string;
|
|
1209
|
+
state: string;
|
|
1210
|
+
postalCode: string;
|
|
1211
|
+
country: string;
|
|
1212
|
+
};
|
|
1213
|
+
geo: {
|
|
1214
|
+
latitude: number;
|
|
1215
|
+
longitude: number;
|
|
1216
|
+
};
|
|
1217
|
+
timeZone: string;
|
|
1218
|
+
openingSchedule: {
|
|
1219
|
+
allHours: boolean;
|
|
1220
|
+
monStart: number;
|
|
1221
|
+
monEnd: number;
|
|
1222
|
+
// ... other schedule fields
|
|
1223
|
+
};
|
|
1224
|
+
}
|
|
1225
|
+
|
|
1226
|
+
export interface MappedLocation {
|
|
1227
|
+
ref: string;
|
|
1228
|
+
name: string;
|
|
1229
|
+
type: string;
|
|
1230
|
+
status: string;
|
|
1231
|
+
primaryAddress: {
|
|
1232
|
+
ref: string;
|
|
1233
|
+
street?: string;
|
|
1234
|
+
city?: string;
|
|
1235
|
+
latitude: number;
|
|
1236
|
+
longitude: number;
|
|
1237
|
+
};
|
|
1238
|
+
openingSchedule: {
|
|
1239
|
+
allHours: boolean;
|
|
1240
|
+
monStart: number;
|
|
1241
|
+
monEnd: number;
|
|
1242
|
+
};
|
|
1243
|
+
}
|
|
1244
|
+
|
|
1245
|
+
export interface MutationResult {
|
|
1246
|
+
mutationsExecuted: number;
|
|
1247
|
+
mutationsFailed: number;
|
|
1248
|
+
errors: Array<{ locationRef: string; error: string }>;
|
|
1249
|
+
}
|
|
1250
|
+
|
|
1251
|
+
export interface ProcessFileResult {
|
|
1252
|
+
success: boolean;
|
|
1253
|
+
locations: Array<{ query: string; variables: Record<string, unknown> }>;
|
|
1254
|
+
error?: string;
|
|
1255
|
+
fileName: string;
|
|
1256
|
+
}
|
|
1257
|
+
|
|
1258
|
+
// Type guard for validation
|
|
1259
|
+
export function isLocationRecord(obj: unknown): obj is LocationRecord {
|
|
1260
|
+
if (typeof obj !== 'object' || obj === null) return false;
|
|
1261
|
+
const loc = obj as Record<string, unknown>;
|
|
1262
|
+
|
|
1263
|
+
return (
|
|
1264
|
+
typeof loc.locationRef === 'string' &&
|
|
1265
|
+
typeof loc.locationName === 'string' &&
|
|
1266
|
+
typeof loc.locationType === 'string' &&
|
|
1267
|
+
typeof loc.address === 'object' &&
|
|
1268
|
+
typeof loc.geo === 'object'
|
|
1269
|
+
);
|
|
1270
|
+
}
|
|
1271
|
+
```
|
|
1272
|
+
|
|
1273
|
+
---
|
|
1274
|
+
|
|
1275
|
+
---
|
|
1276
|
+
|
|
1277
|
+
## Versori Workflows Structure
|
|
1278
|
+
|
|
1279
|
+
**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
|
|
1280
|
+
|
|
1281
|
+
**Trigger Types:**
|
|
1282
|
+
- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
|
|
1283
|
+
- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
|
|
1284
|
+
- **`workflow()`** → Durable workflows (advanced, rarely used)
|
|
1285
|
+
|
|
1286
|
+
**Execution Steps (chained to triggers):**
|
|
1287
|
+
- **`http()`** → External API calls (chained from schedule/webhook)
|
|
1288
|
+
- **`fn()`** → Internal processing (chained from schedule/webhook)
|
|
1289
|
+
|
|
1290
|
+
### Recommended Project Structure
|
|
1291
|
+
|
|
1292
|
+
```
|
|
1293
|
+
sftp-json-location-graphql/
|
|
1294
|
+
├── index.ts # Entry point - exports all workflows
|
|
1295
|
+
└── src/
|
|
1296
|
+
├── workflows/
|
|
1297
|
+
│ ├── scheduled/
|
|
1298
|
+
│ │ └── daily-location-sync.ts # Scheduled: Daily location sync
|
|
1299
|
+
│ │
|
|
1300
|
+
│ └── webhook/
|
|
1301
|
+
│ ├── adhoc-location-sync.ts # Webhook: Manual trigger
|
|
1302
|
+
│ └── job-status-check.ts # Webhook: Status query
|
|
1303
|
+
│
|
|
1304
|
+
├── services/
|
|
1305
|
+
│ └── location-sync.service.ts # Shared orchestration logic (reusable)
|
|
1306
|
+
│
|
|
1307
|
+
└── config/
|
|
1308
|
+
└── location-mapping.json # GraphQL mapping config
|
|
1309
|
+
```
|
|
1310
|
+
|
|
1311
|
+
---
|
|
1312
|
+
|
|
1313
|
+
## Workflow Files
|
|
1314
|
+
|
|
1315
|
+
### 1. Scheduled Workflows (`src/workflows/scheduled/`)
|
|
1316
|
+
|
|
1317
|
+
All time-based triggers that run automatically on cron schedules.
|
|
1318
|
+
|
|
1319
|
+
#### `src/workflows/scheduled/daily-location-sync.ts`
|
|
1320
|
+
|
|
1321
|
+
**Purpose**: Automatic daily location sync
|
|
1322
|
+
**Trigger**: Cron schedule (`0 2 * * *`)
|
|
1323
|
+
**Exposed as Endpoint**: ❌ NO - Runs automatically
|
|
1324
|
+
|
|
1325
|
+
```typescript
|
|
1326
|
+
import { schedule, http } from '@versori/run';
|
|
1327
|
+
import { executeLocationIngestion } from '../../services/location-sync.service';
|
|
1328
|
+
import { generateJobId } from '../../utils/job-id-generator';
|
|
1329
|
+
|
|
1330
|
+
/**
|
|
1331
|
+
* Scheduled Workflow: Daily Location Sync
|
|
1332
|
+
*
|
|
1333
|
+
* Runs automatically daily at 2 AM UTC
|
|
1334
|
+
* NOT exposed as HTTP endpoint - Versori executes on schedule
|
|
1335
|
+
*
|
|
1336
|
+
* Uses shared service: location-sync.service.ts
|
|
1337
|
+
*/
|
|
1338
|
+
export const dailyLocationSync = schedule(
|
|
1339
|
+
'location-sync-scheduled',
|
|
1340
|
+
'0 2 * * *' // Daily at 2 AM UTC
|
|
1341
|
+
).then(
|
|
1342
|
+
http('run-location-sync', { connection: 'fluent_commerce' }, async ctx => {
|
|
1343
|
+
const jobId = generateJobId('SCHEDULED', 'LOC');
|
|
1344
|
+
return await executeLocationIngestion(ctx, { jobId, triggeredBy: 'schedule' });
|
|
1345
|
+
})
|
|
1346
|
+
);
|
|
1347
|
+
```
|
|
1348
|
+
|
|
1349
|
+
---
|
|
1350
|
+
|
|
1351
|
+
### 2. Webhook Workflows (`src/workflows/webhook/`)
|
|
1352
|
+
|
|
1353
|
+
All HTTP-based triggers that create webhook endpoints.
|
|
1354
|
+
|
|
1355
|
+
#### `src/workflows/webhook/adhoc-location-sync.ts`
|
|
1356
|
+
|
|
1357
|
+
**Purpose**: Manual location sync trigger (on-demand)
|
|
1358
|
+
**Trigger**: HTTP POST
|
|
1359
|
+
**Endpoint**: `POST https://{workspace}.versori.run/location-sync-adhoc`
|
|
1360
|
+
**Use Cases**: Testing, priority processing, ad-hoc runs
|
|
1361
|
+
|
|
1362
|
+
```typescript
|
|
1363
|
+
import { webhook, http } from '@versori/run';
|
|
1364
|
+
import { executeLocationIngestion } from '../../services/location-sync.service';
|
|
1365
|
+
import { generateJobId } from '../../utils/job-id-generator';
|
|
1366
|
+
|
|
1367
|
+
/**
|
|
1368
|
+
* Webhook: Manual Location Sync Trigger
|
|
1369
|
+
*
|
|
1370
|
+
* Endpoint: POST https://{workspace}.versori.run/location-sync-adhoc
|
|
1371
|
+
* Request body (optional): { filePattern: "urgent_*.json", maxFiles: 5, forceReprocess: true }
|
|
1372
|
+
*
|
|
1373
|
+
* Pattern: webhook().then(http()) - needs Fluent API access
|
|
1374
|
+
* Uses shared service: location-sync.service.ts
|
|
1375
|
+
*
|
|
1376
|
+
* SECURITY: Authentication handled via connection parameter
|
|
1377
|
+
* No manual API key validation needed - Versori manages this via connection auth
|
|
1378
|
+
*/
|
|
1379
|
+
export const adhocLocationSync = webhook('location-sync-adhoc', {
|
|
1380
|
+
response: { mode: 'sync' },
|
|
1381
|
+
connection: 'location-sync-adhoc', // Versori validates API key
|
|
1382
|
+
}).then(
|
|
1383
|
+
http('run-location-sync-adhoc', { connection: 'fluent_commerce' }, async ctx => {
|
|
1384
|
+
const jobId = generateJobId('ADHOC', 'LOC');
|
|
1385
|
+
const { filePattern, maxFiles, forceReprocess } = ctx.data;
|
|
1386
|
+
return await executeLocationIngestion(ctx, {
|
|
1387
|
+
jobId,
|
|
1388
|
+
triggeredBy: 'manual',
|
|
1389
|
+
filePattern,
|
|
1390
|
+
maxFiles,
|
|
1391
|
+
forceReprocess,
|
|
1392
|
+
});
|
|
1393
|
+
})
|
|
1394
|
+
);
|
|
1395
|
+
```
|
|
1396
|
+
|
|
1397
|
+
---
|
|
1398
|
+
|
|
1399
|
+
#### `src/workflows/webhook/job-status-check.ts`
|
|
1400
|
+
|
|
1401
|
+
**Purpose**: Query job status and progress
|
|
1402
|
+
**Trigger**: HTTP POST
|
|
1403
|
+
**Endpoint**: `POST https://{workspace}.versori.run/location-sync-job-status`
|
|
1404
|
+
**Request Body**: `{ "jobId": "SCHEDULED_LOC_20250124_183045_abc123" }`
|
|
1405
|
+
|
|
1406
|
+
```typescript
|
|
1407
|
+
import { webhook, fn } from '@versori/run';
|
|
1408
|
+
import { getJobStatus } from '../../services/location-sync.service';
|
|
1409
|
+
|
|
1410
|
+
/**
|
|
1411
|
+
* Webhook: Job Status Check
|
|
1412
|
+
*
|
|
1413
|
+
* Endpoint: POST https://{workspace}.versori.run/location-sync-job-status
|
|
1414
|
+
* Request body: { "jobId": "SCHEDULED_LOC_20250124_183045_abc123" }
|
|
1415
|
+
*
|
|
1416
|
+
* Pattern: webhook().then(fn()) - no external API needed, only KV storage
|
|
1417
|
+
* Lightweight: Only queries KV store, no Fluent API calls
|
|
1418
|
+
*
|
|
1419
|
+
* SECURITY: Authentication handled via connection parameter
|
|
1420
|
+
* No manual API key validation needed - Versori manages this via connection auth
|
|
1421
|
+
*/
|
|
1422
|
+
export const locationSyncJobStatus = webhook('location-sync-job-status', {
|
|
1423
|
+
response: { mode: 'sync' },
|
|
1424
|
+
connection: 'location-sync-job-status',
|
|
1425
|
+
}).then(
|
|
1426
|
+
fn('query-job-status', async ctx => {
|
|
1427
|
+
const { jobId } = ctx.data;
|
|
1428
|
+
const status = await getJobStatus(ctx.openKv(':project:'), jobId, ctx.log);
|
|
1429
|
+
return { success: !!status, jobId, ...status };
|
|
1430
|
+
})
|
|
1431
|
+
);
|
|
1432
|
+
```
|
|
1433
|
+
|
|
1434
|
+
---
|
|
1435
|
+
|
|
1436
|
+
### 3. Entry Point (`index.ts`)
|
|
1437
|
+
|
|
1438
|
+
**Purpose**: Register all workflows with Versori platform
|
|
1439
|
+
|
|
1440
|
+
```typescript
|
|
1441
|
+
/**
|
|
1442
|
+
* Entry Point - Registers all workflows with Versori platform
|
|
1443
|
+
*
|
|
1444
|
+
* Versori automatically discovers and registers exported workflows
|
|
1445
|
+
*
|
|
1446
|
+
* File Structure:
|
|
1447
|
+
* - src/workflows/scheduled/ → Time-based triggers (cron)
|
|
1448
|
+
* - src/workflows/webhook/ → HTTP-based triggers (webhooks)
|
|
1449
|
+
*/
|
|
1450
|
+
|
|
1451
|
+
import { MemoryInterpreter } from '@versori/run';
|
|
1452
|
+
|
|
1453
|
+
// Import scheduled workflows
|
|
1454
|
+
import { dailyLocationSync } from './src/workflows/scheduled/daily-location-sync';
|
|
1455
|
+
|
|
1456
|
+
// Import webhook workflows
|
|
1457
|
+
import { adhocLocationSync } from './src/workflows/webhook/adhoc-location-sync';
|
|
1458
|
+
import { locationSyncJobStatus } from './src/workflows/webhook/job-status-check';
|
|
1459
|
+
|
|
1460
|
+
// Register all workflows
|
|
1461
|
+
export {
|
|
1462
|
+
// Scheduled (time-based triggers)
|
|
1463
|
+
dailyLocationSync,
|
|
1464
|
+
|
|
1465
|
+
// Webhooks (HTTP-based triggers)
|
|
1466
|
+
adhocLocationSync,
|
|
1467
|
+
locationSyncJobStatus,
|
|
1468
|
+
};
|
|
1469
|
+
|
|
1470
|
+
// ✅ Memory interpreter for local development
|
|
1471
|
+
export const interpreter = new MemoryInterpreter({
|
|
1472
|
+
workflows: [dailyLocationSync, adhocLocationSync, locationSyncJobStatus],
|
|
1473
|
+
});
|
|
1474
|
+
```
|
|
1475
|
+
|
|
1476
|
+
**What Gets Exposed:**
|
|
1477
|
+
- ✅ `adhocLocationSync` → `https://{workspace}.versori.run/location-sync-adhoc`
|
|
1478
|
+
- ✅ `locationSyncJobStatus` → `https://{workspace}.versori.run/location-sync-job-status`
|
|
1479
|
+
- ❌ `dailyLocationSync` → NOT exposed (runs automatically on cron)
|
|
1480
|
+
|
|
1481
|
+
---
|
|
1482
|
+
|
|
1483
|
+
## Package Configuration
|
|
1484
|
+
|
|
1485
|
+
**File:** `package.json`
|
|
1486
|
+
|
|
1487
|
+
```json
|
|
1488
|
+
{
|
|
1489
|
+
"name": "sftp-json-location-graphql",
|
|
1490
|
+
"type": "module",
|
|
1491
|
+
"main": "index.ts",
|
|
1492
|
+
"dependencies": {
|
|
1493
|
+
"@fluentcommerce/fc-connect-sdk": "^0.1.39",
|
|
1494
|
+
"@versori/run": "latest"
|
|
1495
|
+
},
|
|
1496
|
+
"devDependencies": {
|
|
1497
|
+
"@types/node": "^20.0.0",
|
|
1498
|
+
"typescript": "^5.0.0"
|
|
1499
|
+
}
|
|
1500
|
+
}
|
|
1501
|
+
```
|
|
1502
|
+
|
|
1503
|
+
**File:** `tsconfig.json`
|
|
1504
|
+
|
|
1505
|
+
```json
|
|
1506
|
+
{
|
|
1507
|
+
"compilerOptions": {
|
|
1508
|
+
"module": "ES2022",
|
|
1509
|
+
"target": "ES2024",
|
|
1510
|
+
"moduleResolution": "node"
|
|
1511
|
+
}
|
|
1512
|
+
}
|
|
1513
|
+
```
|
|
1514
|
+
|
|
1515
|
+
---
|
|
1516
|
+
|
|
1517
|
+
## Deployment
|
|
1518
|
+
|
|
1519
|
+
```bash
|
|
1520
|
+
# 1. Install dependencies
|
|
1521
|
+
npm install
|
|
1522
|
+
|
|
1523
|
+
# 2. Deploy to Versori
|
|
1524
|
+
npm run deploy
|
|
1525
|
+
|
|
1526
|
+
# 3. Configure activation variables in Versori Dashboard
|
|
1527
|
+
# CRITICAL: Set fluentRetailerId
|
|
1528
|
+
|
|
1529
|
+
# 4. Test with sample file
|
|
1530
|
+
curl -X POST https://your-workspace.versori.run/location-ingestion-adhoc \
|
|
1531
|
+
-H "Content-Type: application/json" \
|
|
1532
|
+
-d '{"forceReprocess": false}'
|
|
1533
|
+
```
|
|
1534
|
+
|
|
1535
|
+
---
|
|
1536
|
+
|
|
1537
|
+
## Testing
|
|
1538
|
+
|
|
1539
|
+
### Test Scheduled Run
|
|
1540
|
+
Upload test JSON to SFTP and wait for scheduled execution (2 AM daily).
|
|
1541
|
+
|
|
1542
|
+
### Test Ad hoc Trigger
|
|
1543
|
+
```bash
|
|
1544
|
+
curl -X POST https://workspace.versori.run/location-ingestion-adhoc \
|
|
1545
|
+
-d '{"filePattern": "test_*.json"}'
|
|
1546
|
+
```
|
|
1547
|
+
|
|
1548
|
+
### Test Job Status
|
|
1549
|
+
```bash
|
|
1550
|
+
curl -X POST https://workspace.versori.run/location-ingestion-job-status \
|
|
1551
|
+
-d '{"jobId": "ADHOC_LOC_20251101_183045_abc123"}'
|
|
1552
|
+
```
|
|
1553
|
+
|
|
1554
|
+
---
|
|
1555
|
+
|
|
1556
|
+
## Monitoring
|
|
1557
|
+
|
|
1558
|
+
### Success Response
|
|
1559
|
+
|
|
1560
|
+
```json
|
|
1561
|
+
{
|
|
1562
|
+
"success": true,
|
|
1563
|
+
"filesProcessed": 1,
|
|
1564
|
+
"filesSkipped": 0,
|
|
1565
|
+
"filesFailed": 0,
|
|
1566
|
+
"totalRecords": 50,
|
|
1567
|
+
"mutationsExecuted": 50,
|
|
1568
|
+
"mutationsFailed": 0,
|
|
1569
|
+
"results": [
|
|
1570
|
+
{
|
|
1571
|
+
"file": "locations_2025-01-22.json",
|
|
1572
|
+
"success": true,
|
|
1573
|
+
"recordsProcessed": 50,
|
|
1574
|
+
"mutationsExecuted": 50,
|
|
1575
|
+
"mutationsFailed": 0
|
|
1576
|
+
}
|
|
1577
|
+
],
|
|
1578
|
+
"duration": 12345
|
|
1579
|
+
}
|
|
1580
|
+
```
|
|
1581
|
+
|
|
1582
|
+
### Partial Success Response
|
|
1583
|
+
|
|
1584
|
+
```json
|
|
1585
|
+
{
|
|
1586
|
+
"success": true,
|
|
1587
|
+
"filesProcessed": 1,
|
|
1588
|
+
"filesSkipped": 0,
|
|
1589
|
+
"filesFailed": 0,
|
|
1590
|
+
"totalRecords": 50,
|
|
1591
|
+
"mutationsExecuted": 45,
|
|
1592
|
+
"mutationsFailed": 5,
|
|
1593
|
+
"results": [
|
|
1594
|
+
{
|
|
1595
|
+
"file": "locations_2025-01-22.json",
|
|
1596
|
+
"success": true,
|
|
1597
|
+
"recordsProcessed": 50,
|
|
1598
|
+
"mutationsExecuted": 45,
|
|
1599
|
+
"mutationsFailed": 5,
|
|
1600
|
+
"errors": ["LOC-001: Invalid location ref", "LOC-002: Missing required field"]
|
|
1601
|
+
}
|
|
1602
|
+
],
|
|
1603
|
+
"duration": 12345
|
|
1604
|
+
}
|
|
1605
|
+
```
|
|
1606
|
+
|
|
1607
|
+
### Error Response
|
|
1608
|
+
|
|
1609
|
+
```json
|
|
1610
|
+
{
|
|
1611
|
+
"success": false,
|
|
1612
|
+
"filesProcessed": 0,
|
|
1613
|
+
"filesFailed": 1,
|
|
1614
|
+
"totalRecords": 0,
|
|
1615
|
+
"mutationsExecuted": 0,
|
|
1616
|
+
"mutationsFailed": 0,
|
|
1617
|
+
"results": [
|
|
1618
|
+
{
|
|
1619
|
+
"file": "locations_2025-01-22.json",
|
|
1620
|
+
"success": false,
|
|
1621
|
+
"error": "JSON parse error: Invalid structure"
|
|
1622
|
+
}
|
|
1623
|
+
],
|
|
1624
|
+
"duration": 876
|
|
1625
|
+
}
|
|
1626
|
+
```
|
|
1627
|
+
|
|
1628
|
+
### Monitoring Metrics
|
|
1629
|
+
|
|
1630
|
+
Monitor these metrics via Versori logs filtered by `jobId` or `stage`:
|
|
1631
|
+
|
|
1632
|
+
- **Files Processed** - Total files successfully processed
|
|
1633
|
+
- **Mutations Executed** - Total GraphQL mutations executed successfully
|
|
1634
|
+
- **Mutations Failed** - Mutations that failed (check error logs)
|
|
1635
|
+
- **Processing Duration** - Time taken for complete workflow
|
|
1636
|
+
- **Rate Limiting** - Watch for 429 errors indicating GraphQL throttling
|
|
1637
|
+
|
|
1638
|
+
Use the status webhook for dashboards and automated monitoring.
|
|
1639
|
+
|
|
1640
|
+
---
|
|
1641
|
+
|
|
1642
|
+
Before deployment, verify:
|
|
1643
|
+
|
|
1644
|
+
- [ ] ✅ `GraphQLMutationMapper` imported (NOT `UniversalMapper`)
|
|
1645
|
+
- [ ] ✅ NO `client.setRetailerId()` call (not needed for GraphQL mutations)
|
|
1646
|
+
- [ ] ✅ `fluentRetailerId` activation variable configured (only if mutation schema requires it)
|
|
1647
|
+
- [ ] ✅ Mapping config in external JSON file
|
|
1648
|
+
- [ ] ✅ Mapping config uses correct structure (`mutation` property, `arguments.input`, resolver names without `sdk.` prefix)
|
|
1649
|
+
- [ ] ✅ Uses `mapper.map()` method (returns `{ query, variables }`)
|
|
1650
|
+
- [ ] ✅ Type definitions include `LocationRecord`, `MappedLocation`
|
|
1651
|
+
- [ ] ✅ Services follow modular pattern (no logger params in business logic)
|
|
1652
|
+
- [ ] ✅ Three workflows implemented (scheduled, ad hoc, status)
|
|
1653
|
+
- [ ] ✅ Buffer imported from `node:buffer`
|
|
1654
|
+
- [ ] ✅ **Double-finally SFTP disposal** pattern implemented (inner + outer finally)
|
|
1655
|
+
- [ ] ✅ Complete service implementations (processor, sender, logger)
|
|
1656
|
+
- [ ] ✅ Error handling with per-record tracking
|
|
1657
|
+
- [ ] ✅ Processing mode documented (per-file vs chunked)
|
|
1658
|
+
|
|
1659
|
+
---
|
|
1660
|
+
|
|
1661
|
+
## Troubleshooting
|
|
1662
|
+
|
|
1663
|
+
### GraphQL Mutations Failing
|
|
1664
|
+
**Problem:** Mutations return auth errors
|
|
1665
|
+
**Solution:**
|
|
1666
|
+
1. Verify OAuth2 credentials are correct in Versori connection
|
|
1667
|
+
2. Check if mutation schema requires retailerId in input variables
|
|
1668
|
+
3. Do NOT use `client.setRetailerId()` - it's only for Job/Event API
|
|
1669
|
+
|
|
1670
|
+
### Wrong Mapper Error
|
|
1671
|
+
**Problem:** "UniversalMapper not suitable for GraphQL"
|
|
1672
|
+
**Solution:** Replace `UniversalMapper` with `GraphQLMutationMapper`
|
|
1673
|
+
|
|
1674
|
+
### Mapping Config Errors
|
|
1675
|
+
**Problem:** "Invalid mapping structure"
|
|
1676
|
+
**Solution:** Ensure config has `mutation` property (not `mutationName`) and `arguments.input` structure. Resolver names should be without `sdk.` prefix (`trim`, `toUpperCase`, `parseInt`, `parseFloat`, `toBoolean`)
|
|
1677
|
+
|
|
1678
|
+
### Wrong Method Name
|
|
1679
|
+
**Problem:** "generateMutation is not a function"
|
|
1680
|
+
**Solution:** Use `mapper.map()` method, not `generateMutation()`. Returns `{ query, variables }`
|
|
1681
|
+
|
|
1682
|
+
---
|
|
1683
|
+
|
|
1684
|
+
## Key Takeaways
|
|
1685
|
+
|
|
1686
|
+
- ✅ **Use GraphQLMutationMapper** for GraphQL mutations (NOT UniversalMapper)
|
|
1687
|
+
- ⚠️ **Set retailerId** on client before mutations (VERIFICATION REQUIRED - may need mutation input instead)
|
|
1688
|
+
- ✅ **Double-finally SFTP disposal** for resource safety (matches Event API gold standard)
|
|
1689
|
+
- ✅ **External JSON config** for production (not inline)
|
|
1690
|
+
- ✅ **Modular services** for testability and reusability
|
|
1691
|
+
- ✅ **Complete error handling** with per-record tracking
|
|
1692
|
+
- ✅ **Processing modes** documented (per-file recommended, chunked optional)
|
|
1693
|
+
- ✅ **Type-safe** with comprehensive interfaces
|
|
1694
|
+
- ✅ **Native Versori logging** (LoggingService removed - use native log)
|
|
1695
|
+
- ✅ **GraphQL alias batching** support for high-volume scenarios (mutationsPerAliasBatch parameter)
|
|
1696
|
+
|
|
1697
|
+
---
|
|
1698
|
+
|
|
1699
|
+
## References
|
|
1700
|
+
|
|
1701
|
+
- **Gold Standard Guide:** `internal/GOLD-STANDARD-APPLICATION-GUIDE.md`
|
|
1702
|
+
- **GraphQL Mutation Mapping:** `fc-connect-sdk/docs/02-CORE-GUIDES/mapping/graphql-mutation-mapping/`
|
|
1703
|
+
- **GraphQL Alias Batching:** `fc-connect-sdk/docs/02-CORE-GUIDES/mapping/graphql-alias-batching-guide.md`
|
|
1704
|
+
- **retailerId Configuration:** `fc-connect-sdk/docs/00-START-HERE/retailerid-configuration.md`
|
|
1705
|
+
- **Event API Template:** `template-ingestion-sftp-xml-product-event.md` (architectural reference)
|
|
1706
|
+
|
|
1707
|
+
---
|
|
1708
|
+
|
|
1709
|
+
**Status:** ✅ Gold Standard Compliant
|
|
1710
|
+
**Compliance:** All patterns verified against SDK and Fluent GraphQL schema
|